††thanks: These authors contribute equally to this work.††thanks: These
authors contribute equally to this work.

# Extrinsic vs Intrinsic Criticality in Systems with Many Components

Vudtiwat Ngampruetikorn<EMAIL_ADDRESS>Initiative for the Theoretical
Sciences, The Graduate Center, CUNY, New York, New York 10016, USA  Ilya
Nemenman  Department of Physics, Department of Biology, and Initiative in
Theory and Modeling of Living Systems, Emory University, Atlanta, Georgia
30322, USA  David J. Schwab  Initiative for the Theoretical Sciences, The
Graduate Center, CUNY, New York, New York 10016, USA

###### Abstract

Biological systems with many components often exhibit seemingly critical
behaviors, characterized by atypically large correlated fluctuations. Yet the
underlying causes remain unclear. Here we define and examine two types of
criticality. _Intrinsic_ criticality arises from interactions within the
system which are fine-tuned to a critical point. _Extrinsic_ criticality, in
contrast, emerges without fine tuning when observable degrees of freedom are
coupled to unobserved fluctuating variables. We unify both types of
criticality using the language of learning and information theory. We show
that critical correlations, intrinsic or extrinsic, lead to diverging mutual
information between two halves of the system, and are a feature of learning
problems, in which the unobserved fluctuations are inferred from the
observable degrees of freedom. We argue that extrinsic criticality is
equivalent to standard inference, whereas intrinsic criticality describes
_fractional learning_ , in which the amount to be learned depends on the
system size. We show further that both types of criticality are on the same
continuum, connected by a smooth crossover. In addition, we investigate the
observability of Zipf’s law, a power-law rank-frequency distribution often
used as an empirical signature of criticality. We find that Zipf’s law is a
robust feature of extrinsic criticality but can be nontrivial to observe for
some intrinsically critical systems, including critical mean-field models. We
further demonstrate that models with global dynamics, such as oscillatory
models, can produce observable Zipf’s law without relying on either external
fluctuations or fine tuning. Our findings suggest that while possible in
theory, fine tuning is not the only, nor the most likely, explanation for the
apparent ubiquity of criticality in biological systems with many components.
Our work offers an alternative interpretation in which criticality,
specifically extrinsic criticality, results from the adaptation of collective
behavior to external stimuli.

Life emerges from an intricate interplay among a large number of components,
yet how it achieves such exquisite organization remains unexplained. Several
aspects of this question fall within the domain of statistical physics, which
studies the emergence of collective behaviors from the interaction between
microscopic degrees of freedom. Perhaps the greatest success of statistical
physics is in describing spontaneous transitions between two phases of matter
such as liquid and gas. For a class of phase transitions, such as between
ferromagnetic and paramagnetic states, the critical point, which separates the
two phases, displays unique properties, absent from either phase. Many of
these properties are relevant to biological function: scale invariance allows
scaling up without the need for redesign [1, 2, 3], insensitivity to
microscopic details can form a basis for robust behaviors [4, 5], and strong
correlations between components appear useful for effective information
propagation [6, 7, 8, 9].

A tantalizing question arises whether biology operates near a critical point
[10, 11]. This idea has a long history, see, e.g., Ref. [12]. However, it is
not until recently that high-precision, simultaneous measurements of hundreds
to thousands of components in biological systems allow quantitative empirical
tests of the criticality hypothesis.

Modern quantitative biology experiments have indeed observed seemingly
critical behaviors in many systems across scales, from amino acid sequences
[13] to spatiotemporal dynamics of gene expressions [14, 4] to firing patterns
of neurons [15, 16, 17, 18, 19, 20] to velocity fluctuations in bird flocks
[21, 22, 23]. In these systems, correlations among the components and
susceptibilities to perturbations often appear to diverge with the system
size. Yet, this ubiquity is somewhat surprising, not least because equilibrium
statistical physics tells us that criticality requires a hard to achieve fine
tuning of models to a special point in their parameter space.

While biology may well be capable of fine tuning [24, 25], alternative
explanations for the observed criticality exists [26, 27, 28]. Some signatures
of criticality arise without fine tuning when observable degrees of freedom
are coupled to an unobserved fluctuating variable or variables [26, 27, 29,
30]. Provided that the number of fluctuating variables is relatively small and
their fluctuations are sufficiently large, this latent fluctuation needs not
depend on the specifics of the observable degrees of freedom such as the
system size. In this case, criticality results from an _extrinsic_ effect.
Although this mechanism seems to differ from the fine-tuning explanation, they
are not entirely unrelated; interacting systems at criticality also generate
large fluctuations. In fact, the usual definition of criticality describes not
its mechanisms, but rather the behavior of the observable degrees of freedom
such as diverging correlation length, scale invariance and nonanalytic
thermodynamic functions (see, e.g., Refs. [31, 32, 33]). Such properties can
emerge intrinsically from interactions between components when model
parameters are carefully chosen, as is often the case in statistical physics.
However, this _intrinsic_ mechanism is by no means the only one, nor is it a
defining feature of criticality.

Here we introduce a new definition of criticality that spans both intrinsic
and extrinsic mechanisms. Using the languages of learning and information
theory, we show that a unifying feature of both types of criticality is a
divergence of mutual information between two halves of the system.111Divergent
information is a direct result of long-ranged correlations at critical points,
at which the entire system becomes correlated. Indeed, it has proved a useful
characterization of criticality. See, e.g., Refs. [34, 35, 36, 37, 38, 39,
40]. The rates of divergence depend on how the fluctuations scale with the
system size and generally differ between intrinsic and extrinsic criticality.
We show further that critical systems are equivalent to the problem of
learning parameters from iid samples, with the fluctuating fields playing the
role of the parameters and system components of iid samples. This learning
problem is characterized by diverging information between the parameters and
samples. Through the learning-theoretic lens, we interpret intrinsic
criticality as a _fractional_ learning problem, in which only a fraction of
parameters is available for learning due to a sharpening of _a priori_
distribution as the system size grows. In contrast, extrinsic criticality has
a fixed _a priori_ distribution whose entropy, i.e., information available to
be learned, does not shrink with the system size.

In addition, we examine the observability of Zipf’s law—the inverse
relationship between the rank and frequency of system states—commonly used as
an empirical signature of criticality [10, 17]. While the correspondence
between Zipf’s law and critical behaviors is expected in the thermodynamic
limit [10], little is known whether it remains robust under finite sample size
and especially in finite systems, for which the definition of criticality
becomes blurred. Our analyses reveal that the extrinsic mechanism is a
likelier explanation of experimentally observed Zipfian behaviors than typical
intrinsic critical systems, such as critical mean-field models, which usually
generate fluctuations that are too small. We show further that, when endowed
with certain dynamics, intrinsically induced fluctuations can become large
enough to support empirically observable Zipf-like distributions over a range
of model parameters without the need for fine tuning.

We emphasize that the distinction between both types of criticality is not
sharp. Varying the system-size scaling of critical fluctuations leads to a
smooth crossover between intrinsic and extrinsic criticality with the latter
corresponding to the limit, in which the fluctuations are independent of the
system size. Importantly, this crossover behavior means that intrinsically
critical fluctuations that depend very weakly on the system size are
empirically indistinguishable from extrinsic critical ones.

##  I Information and learning–theoretic view of criticality

Large fluctuations are a defining feature of criticality. Criticality is often
characterized by nonanalytic behaviors of the derivatives of thermodynamic
energy or the divergence of correlations of an order parameter [31, Ch XIV].
Such definitions assume the knowledge of the probabilistic model that governs
the system, and rely on quantities that are not readily accessible in biology
experiments. As a result, they are hard to extend to living systems.
Alternatively, one can attempt to define criticality, directly using
properties of the joint probability distribution of the system’s components
which can be estimated directly from data. Zipf’s law in the rank-ordered
frequencies of the system states is one possibility [10]. This definition of
criticality also describes the samples that encode maximum information about
the unknown generative process at a fixed level of compression [41, 42, 43,
44]. More generally, energy is an extensive quantity and thus the central
limit theorem ensures that the variance of energy fluctuations for a typical,
weakly correlated statistical physics system of size $N$ scales as
$\operatorname{var}(E)\\!\sim\\!N$. A faster growth,
$\operatorname{var}(E)\\!\sim\\!N^{\alpha}$ with $\alpha\\!>\\!1$, indicates
correlated microscopic fluctuations that violate the iid assumptions of the
central limit theorem.222We assume here that the variance of the energy of
each component, $\operatorname{var}(E_{i})$, does not depend on $N$. Our
argument does not rely on this assumption. If
$\operatorname{var}(E_{i})\\!\sim\\!N^{\gamma}$, then the central limit
theorem implies $\operatorname{var}(E)\\!\sim\\!N^{1+\gamma}$. Then
criticality describes the scenarios, in which the energy fluctuations grow
faster (or decrease more slowly) than this rate. We use the presence of these
atypically large fluctuations as a _definition_ of criticality.

Figure 1: Divergent information signifies criticality. A: Intrinsic
criticality in equilibrium generally requires fine tuning. We depict the
mutual information between two equal halves of the system,
$I_{1/2}(\bm{\sigma})\\!=\\!I(\bm{\sigma}^{A};\bm{\sigma}^{B})$ with
$N_{A}\\!=\\!N_{B}\\!=\\!N/2$, for the fully connected Ising model as a
function of temperature for a range of system sizes $N$ (see legend). We see
that the information diverges with $N$ only at the critical temperature
$T\\!=\\!T_{c}$. This divergence is logarithmic, with the asymptotic
information given approximately by $\frac{1}{4}\log_{2}N$ (inset). B:
Extrinsic criticality, on the other hand, emerges without fine tuning. For a
system of noninteracting spins, coupled to a common Gaussian fluctuating field
$\phi$ (inset), the information $I_{1/2}(\bm{\sigma})$ always diverges with
$N$. In contrast to the fully connected Ising model at criticality, the
asymptotic information grows faster with $N$,
$I_{1/2}(\bm{\sigma})\approx\frac{1}{2}\log_{2}N$. C: The scaling exponent of
the critical fluctuation, $\operatorname{var}(\phi)\\!\sim\\!N^{-\gamma}$,
controls the information divergence rate. We illustrate the information
between two halves of the system $I_{1/2}(\bm{\sigma})$ (filled circles), and
between the system and the latent field $I(\bm{\sigma};\phi)$ (empty circles)
for conditionally independent identical spins, Eq. (5), under a Gaussian
fluctuating field
$\phi\\!\sim\\!\mathcal{N}(\mu\\!=\\!0,s^{2}\\!=\\!N^{-\gamma})$ for various
values of $\gamma$ (see legend). The asymptotic scaling of the information is
in good agreement with the expected logarithmic divergence
$\frac{1-\gamma}{2}\log_{2}N$ (lines), Eq. (19). We also see that
$I_{1/2}(\bm{\sigma})\\!\leq\\!I(\bm{\sigma};\phi)$, as expected from the data
processing inequality for the Markov chain $\bm{\sigma}^{A}$ – $\phi$ –
$\bm{\sigma}^{B}$.

Divergent mutual information between subsystems is another signature of
criticality. Large correlated fluctuations allow predictions of the state of
one part of the system, $\mathbf{x}^{A}\\!=\\!(x_{1},x_{2},\dots,x_{N_{A}})$,
from measurements of its other part,
$\mathbf{x}^{B}\\!=\\!(x_{N_{A}+1},x_{N_{A}+2},\dots,x_{N_{A}+N_{B}})$, with
$\mathbf{x}\\!=\\!(x_{1},x_{2},\dots,x_{N})$ and $N_{A}+N_{B}=N$. Specifically
the mutual information between these parts reads [45]

$I(\mathbf{x}^{A};\mathbf{x}^{B})=S(\mathbf{x}^{A})+S(\mathbf{x}^{B})-S(\mathbf{x}),$
(1)

where $S(\cdot)$ denotes the entropy of a random variable. At a continuous
phase transition, whether equilibrium or not, this mutual information diverges
logarithmically, $I(\mathbf{x}^{A};\mathbf{x}^{B})\propto\log_{2}N$ [34, 35,
36, 37, 38, 39, 40]. We take this divergence as another, _equivalent_
definition of criticality. We will show that this definition provides a
unifying way of treating extrinsic and intrinsic criticality (see Fig. 1), and
to isolate the effects of critical correlations from that due to geometry, we
will focus on non-spatial systems here and in the following.

Diverging information also characterizes statistical inference. Generally, the
information that iid samples encode about the parameters of a distribution
grow logarithmically with sample size $N$ in the limit $N\\!\to\\!\infty$ [35,
36]. Take, for example, a process that generates iid samples
$\mathbf{x}=(x_{1},x_{2},\dots,x_{N})$ from a probability distribution
$P(x_{i}\,|\,\phi)$ where $\phi$ is an unknown, real-valued, and possibly
multi-dimensional parameter, which one wants to estimate from $\mathbf{x}$.
Given an _a priori_ distribution $P(\phi)$, the joint marginal distribution of
the samples is

$P(\mathbf{x})=\int\\!\\!d\phi\,P(\phi)\prod\nolimits_{i=1}^{N}P(x_{i}\mid\phi).$
(2)

If we divide the samples into two macroscopic groups, i.e.,
$\mathbf{x}\\!=\\!(\mathbf{x}^{A},\mathbf{x}^{B})$, then the mutual
information between these groups reads [35] (see also Appendix A)

$I(\mathbf{x}^{A};\mathbf{x}^{B})=\frac{k}{2}\log_{2}\frac{N_{A}N_{B}}{N}+O(N^{0})\approx\frac{k}{2}\log_{2}N.$
(3)

where $k$ denotes the dimensionality of the parameter $\phi$, or, more
precisely, the dimensionality of the subset of the parameters that can be
inferred from $\mathbf{x}$.333We assume that the number of parameters $k$ is
much smaller than the sample size $N$. For $1\\!\ll\\!N\\!<\\!k$, the
information can grow _linearly_ with $N$ [46]. Importantly, the information
between the two halves exists only to the extent that both depend on $\phi$.
In fact, this information is nearly the same as the information one has about
the parameter $\phi$ having observed $\mathbf{x}$, i.e.,
$I(\mathbf{x}^{A};\mathbf{x}^{B})\\!\approx\\!I(\mathbf{x};\phi)$ [35] (see
Fig. 1C). We emphasize that this logarithmic divergence arises _without_ fine
tuning and for a broad range of assumptions about $P(\phi)$ and
$P(x_{i}\,|\,\phi)$.

Extrinsic criticality is equivalent to statistical learning. We can interpret
the information divergence in inference problems as a signature of
criticality, imposed extrinsically by the unknown, fluctuating variable
$\phi$. We consider a physical manifestation of an extrinsically critical
system [26, 27] and write down the joint probability of $N$ noninteracting,
identical binary spins in an external magnetic field $\phi$,

$P(\bm{\sigma}\mid\phi)=\prod\nolimits_{i=1}^{N}\frac{e^{\phi\sigma_{i}}}{2\cosh\phi}$
(4)

where $\bm{\sigma}\\!=\\!(\sigma_{1},\sigma_{2},\dots,\sigma_{N})$ and
$\sigma_{i}\\!\in\\!\\{\pm 1\\}$. If the field is not constant, but a random
variable that fluctuates for different realizations of the system, then
marginalizing over this field yields

$P(\bm{\sigma})=\int\\!\\!d\phi\,P(\phi)\prod\nolimits_{i=1}^{N}\frac{e^{\phi\sigma_{i}}}{2\cosh\phi},$
(5)

where $P(\phi)$ is the marginal distribution of the fluctuating field. This
equation takes the exact same form as Eq. (2), with the magnetic field playing
the role of the model parameter and the spins of the iid samples. As a result,
it follows immediately that
$I(\bm{\sigma};\phi)\\!\approx\\!I(\bm{\sigma}^{A};\bm{\sigma}^{B})\\!\approx\\!\frac{1}{2}\log_{2}N$
[35], where $\bm{\sigma}^{A}$ and $\bm{\sigma}^{B}$ denote two halves of the
system, see Fig. 1B. In fact, we can derive this result by noticing that the
_a posteriori_ distribution,
$P(\phi\,|\,\bm{\sigma})\\!\sim\\!P(\phi)\exp(\phi\sum_{i}\sigma_{i}\\!-\\!N\ln\cosh\phi)$,
sharpens as $N$ grows, with the asymptotic variance decreasing as
$\operatorname{var}(\phi\,|\,\bm{\sigma})\\!\sim\\!1/N$. In other words, as
our knowledge of the parameter improves with $N$, its _a posteriori_
differential entropy decreases as
$S(\phi\,|\,\bm{\sigma})\\!\approx\\!-\frac{1}{2}\log_{2}N$. Given that the _a
priori_ entropy $S(\phi)$ is finite and does not change with $N$, we obtain

$I(\bm{\sigma}^{A};\bm{\sigma}^{B})\approx
I(\bm{\sigma};\phi)=S(\phi)-S(\phi\mid\bm{\sigma})\approx\tfrac{1}{2}\log_{2}N,$
(6)

where the approximations hide the terms of order $O(N^{0})$.

We can glean additional insights from a more traditional statistical mechanics
argument. First we recast Eq. (5) as

$P(\bm{\sigma})=2^{-N}\int\\!\\!d\phi\,P(\phi)e^{N(\phi m-\ln\cosh\phi)},$ (7)

where $m\\!=\\!m(\bm{\sigma})\\!=\\!\sum_{i}\sigma_{i}/N$ is the
magnetization. For large $N$, we evaluate the above integral, using the saddle
point approximation and assuming that $P(\phi)$ satisfies certain technical
conditions [26],

$P(\bm{\sigma})\approx
2^{-N}\sqrt{\frac{2\pi}{N(1-m^{2})}}P(\phi^{*})e^{N(\phi^{*}m-\ln\cosh\phi^{*})},$
(8)

where $\phi^{*}\\!=\\!\phi^{*}(m)\\!=\\!\tanh^{-1}m$ denotes the saddle point.
Defining the energy function as $E(\bm{\sigma})\\!=\\!-\ln P(\bm{\sigma})$ and
recalling that the thermodynamic entropy is the logarithm of the density of
states—i.e., $\mathcal{S}(m)\\!=\\!\ln[\frac{N}{2}\binom{N}{N_{+}}]$ with
$N_{+}\\!=\\!N(1+m)/2$—we obtain to the leading order in $N$ [26]

$E(m)-\mathcal{S}(m)=\ln(1-m^{2})-\ln P(\phi^{*})+O(N^{-1}),$ (9)

which does not grow with $N$. This equivalence between the energy and entropy
to all increasing order in $N$ signifies a very strong form of criticality
[10].

To compute the mutual information, we write down the entropy of the system

$\\!\\!S(\bm{\sigma})=-\sum_{\bm{\sigma}}P(\bm{\sigma})\ln
P(\bm{\sigma})=\int_{-1}^{1}\\!\\!\\!\\!dm\,e^{\mathcal{S}(m)-E(m)}E(m).\\!$
(10)

Using Eq. (9) and noting that $\frac{d\phi^{*}}{dm}\\!=\\!\frac{1}{1-m^{2}}$,
we have

$\displaystyle S(\bm{\sigma})$
$\displaystyle\approx\int_{-1}^{1}\\!\frac{dm}{1-m^{2}}P(\phi^{*})E(m)$
$\displaystyle\approx N\langle\ln
2-\phi\tanh\phi+\ln\cosh\phi\rangle_{\phi\sim P(\phi)}+\tfrac{1}{2}\ln N,$
(11)

where we drop the terms of order $O(N^{0})$ and smaller. As a result, the
information between the two halves of the system reads

$I(\bm{\sigma}^{A};\bm{\sigma}^{B})\approx\tfrac{1}{2}\log_{2}N_{A}+\tfrac{1}{2}\log_{2}N_{B}-\tfrac{1}{2}\log_{2}N\approx\tfrac{1}{2}\log_{2}N,$
(12)

which exhibits the same logarithmic divergence as one-parameter learning, Eq.
(6).

The specifics of our calculations (conditional independence of spins, binary
spins, etc.) are not important for the main result: an extrinsically varying
parameter gives the mutual information between two subparts of the system
(surface interaction terms excluded) that grows as $\frac{1}{2}\log_{2}N$.
This is a very specific form of criticality, mathematically equivalent to a
problem of learning the said parameter from observations of the system’s
microstate. For larger dimensional parameters, the calculation generalizes:
mutual information is $\frac{1}{2}\log_{2}N$ per inferable parameter component
[35] (so long as the number of parameters is much smaller than $N$).

Intrinsic criticality is described by ‘fractional’ parameter learning. To
illustrate the physics of intrinsic criticality, we consider a minimal model
of fully connected identical Ising spins, defined by the energy function
$E(\bm{\sigma})\\!=\\!-\frac{1}{4N}(\sum_{i}\sigma_{i})^{2}$. As usual, the
joint probability distribution of the spins is $P(\bm{\sigma})\\!=\\!e^{-\beta
E(\bm{\sigma})}/Z$, where $\beta\\!=\\!1/T$ denotes the inverse temperature.
The partition function $Z$ guarantees proper normalization,

$Z=\sum_{\bm{\sigma}}e^{\frac{\beta}{4N}(\sum_{i}\sigma_{i})^{2}}=\sqrt{\frac{N}{\pi\beta}}\int\\!\\!\\!d\phi\sum_{\bm{\sigma}}e^{-\frac{N}{\beta}\phi^{2}+\phi\sum_{i}\sigma_{i}},$
(13)

where the last equality follows from the Hubbard–Stratonovich transformation.
We see that the system consists of fluctuating degrees of freedom that include
the spins $\bm{\sigma}$ as well as a scalar field $\phi$, and the joint
distribution of these variables reads

$P(\bm{\sigma},\phi)=\frac{1}{Z}\sqrt{\frac{N}{\pi\beta}}e^{-\frac{N}{\beta}\phi^{2}+\phi\sum_{i}\sigma_{i}}.$
(14)

Marginalizing out the spins yields

$P_{N}(\phi)=\frac{2^{N}}{Z}\sqrt{\frac{N}{\pi\beta}}e^{-\frac{N}{\beta}\phi^{2}+N\ln\cosh\phi},$
(15)

where the subscript $N$ emphasizes that this distribution is $N$-dependent.
Now the distribution over the spin states is

$P(\bm{\sigma})=\int\\!\\!\\!d\phi\,P_{N}(\phi)\frac{P(\bm{\sigma},\phi)}{P_{N}(\phi)}=\int\\!\\!\\!d\phi\,P_{N}(\phi)\prod_{i}\frac{e^{\phi\sigma_{i}}}{2\cosh\phi},$
(16)

which is the same form as the extrinsically critical model in Eq. (5), albeit
with an _a priori_ distribution that depends on $N$.

Intrinsically induced fluctuations behave differently at and away from the
critical point. To see this, we recall that $\ln\cosh
x\\!\approx\\!\frac{1}{2}x^{2}-\frac{1}{12}x^{4}$ for small $x$. Substituting
this approximation into the exponent of Eq. (15) gives, for small $\phi$,

$P_{N}(\phi)\sim
e^{-N(\frac{1}{\beta}-\frac{1}{2})\phi^{2}-N\frac{1}{12}\phi^{4}}.$ (17)

We see that $\operatorname{var}(\phi)\\!\sim\\!1/N$ for $\beta\\!<\\!2$, but
$\operatorname{var}(\phi)\\!\sim\\!1/\sqrt{N}$ at $\beta\\!=\\!2$. For
$\beta\\!>\\!2$, we have again $\operatorname{var}(\phi)\\!\sim\\!1/N$ with a
prefactor that depends on the curvature around the maxima of $P_{N}(\phi)$.
This change in scaling behaviors is what defines criticality: the fluctuation
becomes atypically large only at the critical temperature $\beta_{c}\\!=\\!2$,
with a variance that scales as $1/\sqrt{N}$ instead of $1/N$ for this simple
fully connected model.

Divergent mutual information emerges only at the critical point, see Fig. 1A.
The scaling behaviors of the variance of intrinsically induced fluctuations
result in the asymptotic _a priori_ differential entropy,
$S(\phi)\\!\approx\\!-\frac{1}{4}\log_{2}N$ at $\beta\\!=\\!\beta_{c}$ and
$S(\phi)\\!\approx\\!-\frac{1}{2}\log_{2}N$, otherwise. From Eq. (14), we have
$\operatorname{var}(\phi\,|\,\bm{\sigma})\\!\sim\\!1/N$, hence the _a
posteriori_ differential entropy scales as
$S(\phi\,|\,\bm{\sigma})\\!\approx\\!-\frac{1}{2}\log_{2}N$. We see that the
logarithmic divergences in _a priori_ and _a posteriori_ entropy cancel
exactly away from criticality, leading to mutual information that does not
grow with $N$. At criticality, on the other hand, we have [38]

$I(\bm{\sigma}^{A};\bm{\sigma}^{B})\approx
I(\bm{\sigma};\phi)\approx\tfrac{1}{4}\log_{2}N.$ (18)

We emphasize that the prefactor here is $1/4$ instead of $1/2$—that is, we
learn only _half_ of a parameter in this intrinsically critical setting.

Fractional learning is typical of intrinsically induced criticality. In the
above examples of intrinsic and extrinsic criticality, we see that the _a
posteriori_ differential entropy of the fluctuating field is asymptotically
identical, i.e., $S(\phi\,|\,\bm{\sigma})\\!\approx\\!-\frac{1}{2}\log_{2}N$.
The _a priori_ entropy, in contrast, differs: $S(\phi)$ does not depend on $N$
for extrinsic fluctuations, whereas
$S(\phi)\\!\approx\\!-\frac{1}{4}\log_{2}N$ for the fully connected Ising
model at criticality. As a result, the mutual information diverges at
different rates [Eqs. (6) & (18)]. More generally, intrinsically induced
fluctuations depend on the system size but need not take the same form as Eq.
(15). For a unimodal fluctuating field with
$\operatorname{var}(\phi)\\!\sim\\!N^{-\gamma}$, the differential entropy
reads $S(\phi)\\!\approx\\!-\frac{\gamma}{2}\log_{2}N$. Therefore (see
Appendix A),

$I(\bm{\sigma}^{A};\bm{\sigma}^{B})\approx
I(\bm{\sigma};\phi)\approx\frac{1-\gamma}{2}\log_{2}N.$ (19)

Away from criticality, $\operatorname{var}(\phi)\\!\sim\\!N^{-1}$, and the
mutual information is finite: we cannot learn much about the field because, in
noncritical thermodynamic systems, fluctuations are small, and we already know
almost everything _a priori_. In contrast, critical fluctuations are larger,
$\gamma\\!<\\!1$, so that observing the spins provides information about the
specific realization of $\phi$, and hence about the other spins. However, the
entropy of intrinsically induced critical fluctuations decreases with $N$
quite generally, $0\\!<\\!\gamma\\!<\\!1$, resulting effectively in only a
_fraction_ of a parameter being available for learning, thereby a decrease in
the information from its maximum possible of $\frac{1}{2}\log_{2}N$. In Fig.
1C, we see that Eq. (19) agrees well with the asymptotic behavior of mutual
information for a range of $\gamma$.

Thus, we have shown that, at least for a simple model, intrinsic criticality
can be viewed as a learning problem, where the underlying large fluctuations
in the order parameter leave sufficient freedom to learn its specific
realization from observations of the system state. The expression of the
information in terms of the difference of the _a priori_ and the _a
posteriori_ entropy, Eq. (6), shows that these results will generalize to
other critical systems: critical exponents will govern the _a priori_
differential entropy of the order parameter, while the _a posteriori_
differential entropy remains approximately $-\frac{1}{2}\log_{2}N$. Similarly,
for multi-dimensional order parameters, each dimension will contribute to the
mutual information essentially independently. For correlated fluctuating
parameters, the logarithmic divergence rate provides a measure of effective
dimensions of the parameters [47]. Finally, extrinsic and intrinsic
criticality will add up as well so that each extrinsically or intrinsically
critical field (i.e., with $N$-independent or $N$-dependent fluctuations) will
contribute $1/2$ or a smaller amount to the coefficient in front of
$\log_{2}N$ in the information.

Figure 2: Zipf’s law is an inaccurate description of critical mean-field
systems. We depict exact rank-ordered distributions for the fully connected
Ising model [Eq. (13)] at various temperatures $T$ (see legend) for the system
size $N\\!=\\!2^{6}$ and $2^{10}$ (A and B, respectively). Note that the axes
correspond to log-rank and log-frequency. The slope of the smoothed log-log
plot illustrates how close the models are to Zipf’s law (dashed). Here we
obtain the approximate slope (bottom row) by fitting a cubic polynomial to the
‘knees’ of the rank-frequency log-log plot. We see that the rank-frequency
plots deviate from the power-law behavior at all temperatures, and this
deviation does not appear to improve as the system grows. Importantly the
critical model ($T\\!=\\!T_{c}$) exhibits Zipf’s scaling (dashed) only for the
least frequent states in the tail region of the rank-ordered distributions and
only when the system is large enough (B). In fact, in a smaller system, the
rank-frequency plot can appear more Zipf-like at $T\\!<\\!T_{c}$ (A), see also
Fig. 6. Figure 3: Zipf’s law emerges robustly for adequately large extrinsic
fluctuations. We show exact rank-ordered distributions for independent spins
under a Gaussian fluctuating field of varying variance (see legend). We see
that Zipf’s law becomes more accurate as the variance of the fluctuations and
the system size increase. However, the rank-ordered distributions approach
Zipf’s scaling over the entire range only when the system is adequately large
(B). Here we obtain the approximate slope (bottom row) using the same method
as in Fig. 2.

##  II Signature of criticality in finite systems

In reality, we can only observe a finite number of components, and the
analysis of asymptotic behaviors, while instructive, becomes less precise. To
this end, we now turn to the observability of criticality in finite systems.
Criticality admits a number of potentially observable signatures. We focus on
the properties of the empirical joint distribution of the system components,
which can be constructed directly from observational data and thus is readily
usable in the context of living systems. A critical system is expected to
exhibit Zipf’s law, i.e., an inverse relationship between ranks and
frequencies of the system states [10].

Figure 4: Rank-ordered distributions exhibit a smooth crossover between
intrinsic and extrinsic criticality. Normalized exact rank-frequency plots for
a range of system sizes $N$ (see legend in F) illustrate a gradual crossover
from the fully connected Ising model at criticality (A) to spins under an
extrinsic fluctuating field (F). This crossover is induced by varying the
scaling exponent of the variance of fluctuations
$\operatorname{var}(\phi)\\!\sim\\!N^{-\gamma}$, from $\gamma\\!=\\!1/2$ for
the mean-field criticality in A [see Eq. (15)] to $\gamma\\!<\\!1/2$ for non-
mean-field intrinsic criticality in B-E (see label) and $\gamma\\!=\\!0$ for
extrinsic criticality in F. We see that the rank-frequency plots approach
Zipf’s law (dashed) as the system grows, and at a faster rate for a smaller
scaling exponent $\gamma$. For mean-field criticality, the agreement with Zipf
scaling neither improves, nor degrades, with increasing $N$ (Panel A). In B-E,
the prior over fluctuations takes the form $P(\phi)\\!\sim\\!e^{-c\phi^{4}}$
where $c\\!\propto\\!N^{2\gamma}$ [cf. Eq. (17)]. In all panels, the
fluctuation variance at $N\\!=\\!64$ (the smallest $N$ shown) is the same and
equal to that of the critical mean-field case. That is, in B-F, we have
$\operatorname{var}(\phi)\\!=\\!\lambda_{0}(64/N)^{\gamma}$ with $\lambda_{0}$
denoting the variance of critical mean-field fluctuation at $N\\!=\\!64$.

First we recap how Zipf’s behavior emerges from large fluctuations [26, 27] in
the asymptotic limit. Consider the joint distribution of $N$ conditionally
independent spins,

$P(\bm{\sigma})=\int\\!\\!\\!d\phi\,P(\phi)\prod\nolimits_{i=1}^{N}P(\sigma_{i}\mid\phi).$
(20)

We see that $\sum_{i}\ln P(\sigma_{i}\,|\,\phi)\\!\sim\\!N$, and thus
$P(\bm{\sigma}\,|\,\phi)$ becomes a sharper function of $\phi$ as $N$
increases, with a characteristic width that scales as $1/\sqrt{N}$. This
scaling sets a threshold above which fluctuations are critical—that is,
critical fluctuations are characterized by
$\operatorname{var}(\phi)\\!\sim\\!N^{-\gamma}$ with $\gamma\\!<\\!1$. As
$N\\!\to\\!\infty$, a critical prior, $P(\phi)$, appears flat with respect to
$P(\bm{\sigma}\,|\,\phi)$ which becomes infinitely sharp. Therefore, we can
make the approximation,

$P(\bm{\sigma}\mid\phi)\approx
e^{-\mathcal{S}(\phi)}\delta(\phi-\phi^{*}_{\bm{\sigma}}),$ (21)

where
$\mathcal{S}(\phi)\\!\equiv\\!\ln\sum_{\bm{\sigma}}\delta(\phi-\phi^{*}_{\bm{\sigma}})$
plays the role of the thermodynamic entropy and $\phi^{*}_{\bm{\sigma}}$ is
the maximum of $P(\bm{\sigma}\,|\,\phi)$, assuming only one exists.
Substituting the above approximation into Eq. (20) yields

$P(\bm{\sigma})\approx
P(\phi^{*}_{\bm{\sigma}})e^{-\mathcal{S}(\phi^{*}_{\bm{\sigma}})}\equiv
e^{-\mathcal{E}(\phi^{*}_{\bm{\sigma}})},$ (22)

which illustrates that the energy function depends on $\bm{\sigma}$ only
through $\phi^{*}_{\bm{\sigma}}$, i.e., $E(\bm{\sigma})\\!=\\!-\ln
P(\bm{\sigma})\\!=\\!\mathcal{E}(\phi^{*}_{\bm{\sigma}})$. Importantly, the
above equation signifies Zipf’s law via the equivalence between the extensive
parts of the entropy and energy [10],

$\lim_{N\to\infty}\left(\mathcal{E}(\phi)-\mathcal{S}(\phi)\right)/N=\lim_{N\to\infty}-N^{-1}\ln
P(\phi)=0.$ (23)

For mean-field criticality, $\ln P(\phi)\\!\sim\\!N\phi^{4}$ [Eq. (17)] and
the above cancellation holds when $\phi$ is adequately small. This condition
is guaranteed for a typical realization of $\phi$ since
$\operatorname{var}(\phi)\\!\sim\\!1/\sqrt{N}$. For extrinsic criticality,
$\ln P(\phi)\\!\sim\\!O(N^{0})$ and the entropy-energy equivalence needs not
rely on $\phi$ being small.

In Fig. 2, we depict exact (infinite samples) rank-frequency plots for the
fully connected Ising model at a range of temperatures. We see a clear
deviation from the power-law behavior at all temperatures. The rank-frequency
plot at $T_{c}$ approaches Zipf’s scaling, but only in the tail region and for
an adequately large system (Fig. 2B). In a smaller system, Zipf’s law can
appear more accurate at $T\\!<\\!T_{c}$ (Fig. 2A). This disconnect between the
Zipf behavior and criticality in mean-field models is likely to be more
visible under finite samples, which can only probe parts of the exact rank-
frequency plots (see also Fig. 6). We emphasize here that the number of
observations required to resolve the tail of this rank-frequency plot would be
experimentally impractical, $2^{N}\\!\sim\\!10^{20}$ for $N\\!=\\!2^{6}$ and
$2^{N}\\!\sim\\!10^{300}$ for $N\\!=\\!2^{10}$ (Fig. 2A&B).

On the other hand, Fig. 3 shows that the rank-ordered distributions of
identical spins under extrinsic fluctuations become more Zipf-like over the
_entire_ range of frequencies and ranks as the fluctuation variance increases
and as the system grows. However, the rank-frequency plots approach Zipf’s law
only when the system is sufficiently large (Fig. 3B).

Although mean-field criticality results in a Zipf-like rank-frequency plot
only in the hard-to-observe tail region, intrinsic criticality with a more
general critical exponent—i.e.,
$\operatorname{var}(\phi)\\!\sim\\!N^{-\gamma}$ with $\gamma\\!\neq\\!1/2$—can
generate rank-ordered distributions that are much closer to Zipf’s law. In
Fig. 4, we compare the system-size dependence of the rank-frequency plots for
the fully connected Ising model at $T_{c}$ ($\gamma\\!=\\!1/2$) to several
models with larger fluctuations, including those potentially induced by
intrinsic criticality of non-mean-field models ($0\\!<\\!\gamma\\!<\\!1/2$) as
well as the limiting case of extrinsic criticality ($\gamma\\!=\\!0$). Figure
4A shows that the critical mean-field model exhibits Zipf scaling only in the
tail region of the rank-ordered distribution (see also Fig. 2) and the
agreement with Zipf’s law does not improve, nor degrade, as the system grows.
On the other hand, we see that if the fluctuation variance decreases more
slowly with $N$, i.e., $\gamma\\!<\\!1/2$, Zipf’s law gradually becomes more
accurate as $N$ increases, see Fig. 4B-F. This behavior implies that
empirically observed Zipf behavior is likely to indicate either extrinsic
criticality or intrinsic criticality of non-mean-field type, characterized by
critical fluctuations that scale only weakly with the system size.

Figure 5: Critical mean-field fluctuations are too small to support
empirically observable Zipf behavior. Panels A and B illustrate the effects of
the width and the structure of the tails of the _a priori_ distribution
$P(\phi)$ (inset) on finite-sample rank-frequency plots, respectively. The red
dotted curves correspond to the rank-one Ising model at criticality [Eqs.
(26-27)]. In A, we consider linear scaling of critical mean-field fluctuations
such that $\operatorname{var}(\phi)\\!=\\!\delta^{2}\lambda_{c}$, where
$\lambda_{c}$ is the fluctuation variance of the rank-one Ising model at
$T_{c}$, for various scaling coefficient $\delta$ (see legend). We see that
critical mean-field fluctuations ($\delta\\!=\\!1$) do not result in Zipf
behavior. Increasing the fluctuation variance while maintaining the overall
shape of the fluctuation prior produces rank-frequency plots that
progressively appear closer to Zipf scaling (dashed). In B, we consider the
fluctuating field of the form $P(\phi)\\!\sim\\!e^{-c|\phi|^{q}}$. We vary the
probability in the tails of $P(\phi)$ with the shape parameter $q$ (see
legend), and choose the scale parameter $c$ such that the fluctuation variance
is fixed and equal to that of critical mean-field fluctuations (red dotted
lines). Decreasing $q$ increases the probability of large $\phi$, i.e., puts
more mass in the tails of $P(\phi)$, and improves the agreement between rank-
frequency plots and Zipf’s law. Overall, while instructive in understanding
critical behaviors, mean-field models are an unlikely candidate for explaining
experimentally observed Zipf behavior. Here the results are for a system of
$60$ spins, $10^{8}$ realizations per model and
$w_{i}\\!\sim\\!\mathcal{N}(\mu\\!=\\!1,s\\!=\\!0.3)$ [see Eq. (24)]. Figure
6: Empirical Zipf behavior needs not coincide with the critical point of the
system. A: We depict empirical rank-ordered distributions of the rank-one
Ising model [Eqs. (26-27)] at various temperatures (see color legend in B). We
see that the distribution is closest to Zipf’s law at an intermediate
temperature, significantly lower than the critical temperature $T_{c}$. As an
empirical, alternative definition of a critical temperature for finite
systems, we consider the maximum of the specific heat or, equivalently the
energy variance. This definition results in $T^{*}\\!<\\!T_{c}$ (see B), which
is still significantly higher than the temperature that exhibits approximate
Zipf behavior. B: The energy variance $\operatorname{var}(E)$ (solid, left
axis) and mutual information $I(\bm{\sigma};\phi)$ (dashed, right axis)
provide measures of correlations in the system. Both maximize below $T_{c}$
roughly at the same temperature ($T^{*}$ for $\operatorname{var}(E)$). C: The
prior $P(\phi)$ changes from unimodal to bimodal at $T_{c}$, which indicates
maximum correlations in the thermodynamic limit. However, in finite systems,
$P(\phi)$ cannot be accurately described by its behavior near maxima. We see
that the specific heat peaks at $T^{*}\\!<\\!T_{c}$ (see B), where $P(\phi)$
is bimodal but with significant density at $\phi\\!=\\!0$. Here the results
are for a system of $60$ spins, $10^{8}$ realizations per model and
$w_{i}\\!\sim\\!\mathcal{N}(\mu\\!=\\!1,s\\!=\\!0.3)$ [see Eq. (24)].

##  III Zipf’s law as a signature of criticality under finite samples

Experimental measurements are finite in not only the number of observable
degrees of freedom but also the number of observations. We now turn to examine
the behavior of rank-frequency plots constructed from finite samples.

In the following, we generalize our conditionally independent model, Eq. (5),
to describe nonidentical spins,

$P(\bm{\sigma}\mid\phi)=\prod_{i}\frac{e^{\sigma_{i}w_{i}\phi}}{2\cosh
w_{i}\phi}.$ (24)

Here $w_{i}\\!\sim\\!O(N^{0})$ is the coupling strength between the spin
$\sigma_{i}$ and the field $\phi$, which can differ from one spin to another.
For convenience, we also define

$\Delta\equiv 1/\sqrt{\sum\nolimits_{i}w_{i}^{2}},$ (25)

which is the characteristic width of $P(\bm{\sigma}\,|\,\phi)$ at large $N$,
i.e.,
$P(\bm{\sigma}\,|\,\phi)\\!\sim\\!e^{-(\phi-\phi^{*}_{\bm{\sigma}})^{2}/2\Delta^{2}}$
with $\phi^{*}_{\bm{\sigma}}\\!=\\!\sum_{i}w_{i}\sigma_{i}/\sum_{i}w_{i}^{2}$.
Following the argument in the preceding section, we expect Zipf behavior when
$\operatorname{var}(\phi)\\!\gg\\!\Delta^{2}\\!\sim\\!O(N^{-1})$.

Similarly, for mean-field criticality, we consider the rank-one Ising model,
which generalizes the fully connected Ising model to nonidentical spins and is
defined by the energy function,

$E(\bm{\sigma})=-\frac{1}{4N}\sum\nolimits_{ij}w_{i}w_{j}\sigma_{i}\sigma_{j}=-\frac{1}{4N}\left(\sum\nolimits_{i}w_{i}\sigma_{i}\right)^{2},$
(26)

where $w_{i}\\!\sim\\!O(N^{0})$ and $w_{i}w_{j}$ describes the pairwise
interaction between spins $i$ and $j$. This model can be recast as a
conditionally independent model, with the conditional distribution of the
spins given by Eq. (24) and an _a priori_ distribution that depends on both
$N$ and $\\{w_{i}\\}$ (see Appendix B),

$P_{N}(\phi)=\frac{2^{N}}{Z}\sqrt{\frac{N}{\pi\beta}}e^{-\frac{N}{\beta}\phi^{2}+\sum_{i}\ln\cosh
w_{i}\phi},$ (27)

where $Z\\!=\\!\sum_{\bm{\sigma}}e^{-\beta E(\bm{\sigma})}$. The thermodynamic
critical temperature, $\beta_{c}\\!=\\!2N/\sum_{i}w_{i}^{2}$, marks the point
at which this distribution changes from unimodal to bimodal.

Figure 5 illustrates that critical mean-field fluctuations are too small to
generate experimentally observable Zipf’s law. We consider a system of 60
conditionally independent spins under a number of _a priori_ distributions,
$P(\phi)$ (see inset), including that induced by a rank-one Ising model at
$T_{c}$ [Eq. (27)]. For each _a priori_ distribution, we draw $10^{8}$ iid
realizations of the system and construct an empirical rank-frequency plot. In
Fig. 5A, we see that the rank-one Ising model at $T_{c}$ does not produce
Zipf’s law. Yet, if we make the fluctuation larger while fixing the shape
(standardized moments) of the fluctuation prior, the resulting rank-frequency
plot edges closer to Zipf scaling. However, the fluctuation variance is not
the only factor that controls the behavior of the rank-frequency plot. In Fig.
5B, we see that at a fixed variance, an _a priori_ distribution with thicker
tails (larger standardized moments) produces a more Zipf-like rank-order plot.
We emphasize that the resolution of these plots, especially in the tails, is
limited by the number of samples. While we do not rule out the possibility
that mean-field criticality may exhibit Zipf behavior in the tail region (see
also Fig. 2), observing such behavior would require orders of magnitude more
samples than $10^{8}$ and would therefore be experimentally impractical.

Extrapolating the asymptotic critical temperature to finite systems is, of
course, somewhat dubious. In Fig. 6, we consider another frequently used
empirical definition of criticality which identifies the critical point with
the maximum in the specific heat or equivalently the energy variance. In the
asymptotic limit $N\\!\to\\!\infty$, this definition is identical to the
thermodynamic critical temperature $T_{c}$. For finite systems, however, the
specific heat maximum occurs at a lower temperature $T^{*}\\!<\\!T_{c}$ (Fig.
6B). This temperature also coincides roughly with another possible empirical
definition of criticality, namely the maximum of the mutual information
$I(\bm{\sigma};\phi)$ which indicates maximum correlations and learnability
(Fig. 6B). In Fig. 6A, we see that lowering the temperature of the rank-one
Ising model from $T_{c}$ to $T^{*}$ makes the system closer to, but still
visibly different from, Zipf’s law. In fact, the closest agreement to Zipf’s
law occurs at an even lower temperature. Two factors contribute to this
intriguing temperature dependence. First, for mean-field criticality, Zipf
scaling is expected only in the tail of the rank-frequency plot (see Fig. 2B),
which requires a very large number of samples to resolve. Second, the
correspondence between criticality and Zipf behavior is blurred in finite
systems with the tendency for Zipf’s law to be more accurate at subcritical
temperatures (see Fig. 2A). In sum, we demonstrate that for mean-field models,
empirically observable Zipf behavior can be completely uncoupled from the
usual notion of criticality.

Figure 6C illustrates the interaction-induced _a priori_ distribution
$P(\phi)$ at various temperatures. We see that $P(\phi)$ is flat around its
maximum at $T_{c}$. In the thermodynamic limit, this condition leads to non-
Gaussian fluctuations which break the central limit theorem and generate
critical correlations. However, in finite systems, the field $\phi$ is not
well described by fluctuations in the immediate vicinity of its most likely
values. We see that at the specific heat maximum $T^{*}$, the _a priori_
distribution has a non-negligible density at $\phi\\!=\\!0$ even though it is
bimodal with maxima at $\phi\\!\neq\\!0$. This distribution results in a
larger fluctuation than at $T_{c}$, resulting in higher energy variance as
well as a rank-ordered plot closer to Zipf’s law. As the temperature drops
below $T^{*}$, the most likely field values move further away from zero.
Larger values of $\phi$ suppress the variability of the system: each spin
$\sigma_{i}$ aligns with $\mathrm{sign}(w_{i}\phi)$ with increasing
probability, thereby the decrease in the energy variance. At high temperatures
$T\\!>\\!T_{c}$, the _a priori_ distribution becomes sharply peaked at
$\phi\\!=\\!0$, resulting in more random systems and thus a decrease in
correlations. Rank-ordered plots also reflect this competition; reduced
variability leads to a rank-ordered plot that decays faster than Zipf’s law at
low $T$, whereas increased randomness yields a plot that appears flatter than
Zipf’s law at high $T$.

Figure 7: Zipf’s law can emerge from intrinsically induced fluctuations
without fine tuning. A: Rank-ordered distributions for a dynamical spin model,
Eq. (28), display Zipf’s law for a range of effective inverse temperatures
$\beta$ (see legend). B&C: We illustrate typical empirical distributions of
the fluctuating field $\phi$ and its typical dynamics (same legend as A). The
results shown are for a system of $60$ spins, $10^{8}$ realizations per model
and $w_{i}\\!\sim\\!\mathcal{N}(\mu\\!=\\!1,s\\!=\\!0.6)$, and we set
$\alpha\\!=\\!-0.8$ [see Eq. (28)].

Intrinsic fluctuations can lead to Zipf’s law without fine tuning. So far we
see that, in the absence of external fluctuations, the empirical signatures of
large correlated fluctuations, such as Zipf-like distributions, are hard to
observe. While this statement is true for equilibrium systems, critically
large intrinsic fluctuations can emerge generically when the system is endowed
with certain dynamics. To illustrate this point, we consider a discrete-time,
dynamical generalization of the conditionally independent spin model [Eq.
(24)],

$P(\bm{\sigma}_{t+1}\mid\phi_{t})=\prod_{i}\frac{e^{\sigma_{i,t}w_{i}\phi_{t}}}{2\cosh
w_{i}\phi_{t}},$ (28)

with $\phi_{t}\\!=\\!\beta(m_{t}+\alpha m_{t-1})$ and
$m_{t}\\!=\\!\frac{1}{N}\sum_{i}w_{i}\sigma_{i,t}$. Here the index $i$ labels
each spin and $t$ the time step. The parameter $\beta$ controls the
stochasticity of the spins much like the inverse temperature and $\alpha$
couples states separated by two time steps. We illustrate the dynamics of the
fluctuating field $\phi_{t}$ for various $\beta$ and the corresponding _a
priori_ distributions in Fig. 7B&C, respectively. In Fig. 7A, we see that this
model can result in Zipf behavior over a range of model parameters,
demonstrating that fine tuning is not a requirement for empirically observable
Zipf behavior in the absence of external fields. Although the fluctuations are
generated entirely internally, we can interpret this emergence of Zipf’s law
as extrinsic criticality. To see this, we note that the model parameters
$\alpha$ and $\beta$ control the amplitude of the oscillations and thus the
fluctuation variance (Fig. 7B&C). The system size $N$ only determines the
stochasticity of the dynamics. As a result, the _a priori_ distribution
depends only weakly on $N$ and becomes completely independent of $N$ in the
asymptotic limit. This diminishing system-size dependence makes the resulting
fluctuations indistinguishable from extrinsic ones.

##  IV Discussion

Here we introduced general definitions of criticality encompassing
extrinsically and intrinsically critical systems, and we showed that
information-theoretic and learning-theoretic considerations allow us to view
all non-spatial, critical systems on a similar footing. Namely, criticality
leads to the logarithmic divergence in the information between two subsystems
or between the system’s observable degrees of freedom and its fluctuating
latent field. The coefficient in front of the divergence is semi-integer for
systems with extrinsic criticality, and other fractions for intrinsic
criticality. Both situations can be viewed as learning the parameter from
measurements of the state of system components, and the _a priori_ variance of
this parameter is independent of the system size (extrinsic criticality) or
decreases as the system size grows (intrinsic criticality).

We focused on the scenario where the most critical a system can be is when it
has $\frac{1}{2}\log_{2}N$ bits of mutual information per dimension of the
order parameter, which is equivalent to the iid learning problems. Additional
intrinsic couplings would then reduce the _a priori_ variance of the order
parameter and hence reduce the mutual information. However, some biological
systems may have the _a priori_ parameter variance that increases with $N$
[48], so that more than $\frac{1}{2}\log_{2}N$ bits are contributed to the
mutual information per latent field.

One can imagine this happening in an optimally designed sensory system, where
spins are coupled to the field in a way to reduce the redundancy of the
information they obtain about it. Investigating the properties of such systems
from information-theoretic, learning, and statistical physics angles is
clearly needed. Similarly, it is worth investigating systems in which the
mutual information between macroscopic parts scales as a sublinear power of
$N$, (rather than a logarithm), which correspond to an infinite number of
latent fields with hierarchically smaller _a priori_ variances [35].444We note
that some models, such as the random energy model, can generate extensive
information [49], but these models are generally not learnable from finite
data and hence of less experimental relevance [50]. Finally, since all of our
learning and information-theoretic arguments are asymptotic, and $O(1)$
corrections may not be negligibly small compared to $\log_{2}N$, subleading
corrections are also worth investigating.

In addition, there is a striking similarity between the critical behavior of
mutual information in classical systems and entanglement entropy, a quantum
information-theoretic measure of correlations. At quantum critical points,
long-ranged correlations lead to diverging entanglement entropy, violating the
_area law_ [51]. For infinite quantum critical spin chains, this divergence is
logarithmic in the subsystem size with a universal prefactor that is related
to the central charge of the corresponding conformal field theory [52, 53, 54,
55, 56]. It would be interesting to develop a learning-theoretic picture of
quantum criticality and explore whether and how the central charge relates to
the effective number of latent parameters.

We also investigated the observability of an empirical signature of
criticality, namely Zipf’s law. While the correspondence between Zipf behavior
and criticality is precise in the thermodynamic limit, whether it holds for a
finite system depends on how critical the system is, i.e., how large the _a
priori_ variance $\operatorname{var}(\phi)$ is, compared to the width of the
conditional distribution $P(\bm{\sigma}\,|\,\phi)$. For extrinsic criticality,
Zipf’s law emerges robustly under an adequately broad _a priori_ fluctuation
distribution. Intrinsic criticality, on the other hand, does not always induce
large enough fluctuations to support Zipf behavior. In particular, mean-field
critical fluctuations are too small to generate Zipf’s law even in the
infinite-sample limit. Indeed, under finite samples, the closest agreement to
Zipf’s law can occur at a temperature significantly lower than the
thermodynamic critical temperature as well as the specific heat maximum, an
alternative, empirical definition of a critical point. This approximate Zipf
behavior at intermediate temperature results from the competition between
order-promoting interactions and thermal noise, and is unlikely to be a
signature of equilibrium intrinsic criticality in the usual sense.

Perhaps the disconnect between intrinsic criticality and Zipf behavior in
finite systems is unsurprising, not least because of the blurred notion of
criticality away from the thermodynamic limit. While using the specific heat
maximum to indicate criticality is tempting, it requires assumptions on the
probabilistic model that describes the data. Zipf behavior offers an
alternative, model-free definition, but as we showed, it can be nontrivial to
observe in intrinsically critical systems. We emphasize that no finite-system
definition of criticality captures all of its thermodynamic signatures; for
instance, finite-sample Zipf behavior does not correspond to maximum energy
fluctuations or maximum correlations (see Fig. 6).

While our asymptotic analysis suggests that the correspondence between
intrinsic criticality and Zipf behavior is more precise for larger systems, we
focus on a relatively small system of 60 spins (Figs. 5-7) since it is more
relevant to real measurements. In particular, a well-sampled rank-ordered plot
becomes exceedingly difficult to achieve as the system size grows. For
example, a rank-ordered distribution for the critical rank-one Ising model
with 80 spins shows almost no structure even at $10^{8}$ samples (see Fig. 8).
This loss of structure due to finite samples is less severe for extrinsic
fluctuations but the agreement with Zipf’s law degrades with increasing $N$
(Fig. 9), in contrast to the infinite-sample case, in which Zipf’s law becomes
more accurate as the system grows (Fig. 4F).

We show further that some oscillatory systems can generate observable Zipf’s
law without fine tuning or external fluctuations. We argue that the mechanism
behind this behavior is mathematically equivalent to the extrinsic mechanism
since the scale of the collective dynamics—hence the variance of the
fluctuations once the time variable is integrated out—is often independent of
the system size. Indeed, collective oscillations are common both in
mathematical models (e.g., Refs. [57, 58]) and in biological systems (e.g.,
Refs. [59, 60, 61]). Our results suggest that models with a global dynamical
variable that stays within a certain range could offer another plausible
explanation for empirically observed Zipf’s law, without the need for
extrinsic fluctuations.

Finally, we discuss how subsampling may affect the observability of Zipf
behavior. Many experiments do not measure the system in its entirety and what
we can vary is the number of the observed components rather than the system
size. Perhaps the unobserved degrees of freedom could play the role of an
extrinsic source of fluctuations for the observed ones, resulting in
extrinsically induced criticality and thus making observation of Zipf’s law
more probable. However, our simple model of intrinsic criticality does not
support this thinking. Suppose we observe $K$ out of the total of $N$ spins.
Intrinsic fluctuations quite generally become smaller with $N$, i.e.,
$\operatorname{var}(\phi)\\!\sim\\!N^{-\gamma}$ with $0\\!<\\!\gamma$, whereas
the width of the conditional probability $P(\bm{\sigma}\,|\,\phi)$ decreases
with $K$, i.e., $\Delta\\!\sim\\!K^{-1/2}$ [cf. Eq. (25)]. As a result, the
relative fluctuation variance is
$\operatorname{var}(\phi)/\Delta^{2}\\!\sim\\!(K/N)\times N^{1-\gamma}$ with
$\gamma\\!<\\!1$ for critical systems. In other words, decreasing the
observable fraction makes the fluctuation appear smaller and Zipf’s law less
likely (even though the smaller number of degrees of freedom makes it easier
to obtain better-sampled rank-frequency plots, see Fig. 8). This effect is
even more acute when the system size far outnumbers the observed components,
e.g., a recording of neural spikes in the brain.

Real systems can of course be more complicated than our simple model. For
example, in spatially extended systems, the order parameter could be a field
in space and the number of inferable parameters, e.g., the Fourier components
of the order parameter, can depend on how many spins we observe. Thus, the
bits available to be learned can depend on the number of observed spins, even
though the _a priori_ variances of the parameters do not (as they are set by
the size of the entire system). Investigations of criticality and its
empirical signatures in this setting are in order.

We end by pointing out that many biological critical systems become more Zipf-
like as they grow [10], which begs the question of why this happens. As
pointed out in Ref. [26], consider a sensory system that is learning the state
of the outside world (that is, responds to its different values differently);
one would expect this system to be constructed in a way not to decrease the
variability of the world when the system size grows. Such systems would always
be critical, and specifically extrinsic critical, maybe explaining their
ubiquity.

###### Acknowledgements.

We thank William Bialek, Stephanie Palmer, and Pankaj Mehta for valuable
discussions. VN and DJS acknowledge support from the National Science
Foundation, through the Center for the Physics of Biological Function
(PHY-1734030). DJS is supported in part by the Simons Foundation and by the
Sloan Foundation. IN was supported in part by the Simons Foundation
Investigator grant and the NIH grants 1R01NS099375 and 2R01NS084844.

Figure 8: Subsampling does not make intrinsically critical systems appear more
Zipf-like. We depict the rank-ordered frequency for the rank-one Ising model
of $N$ spins, Eq. (26), at the critical temperature, for $N\\!=\\!40,80$ (a &
b), and a range of observed subsystem size $K$ (see legend). We see that
limiting observation to a fraction of the full system, i.e., $K\\!<\\!N$, does
not result in more Zipf-like behavior (dashed). It leads however to more
structured rank-ordered distributions, especially when the system is large
(b), since a system of fewer spins requires a smaller sample size to be well-
sampled. The results shown are for $10^{8}$ realizations per model and
$w_{i}\\!\sim\\!\mathcal{N}(\mu\\!=\\!1,s\\!=\\!0.3)$. Figure 9: Zipf’s law
emerges robustly from large extrinsic fluctuations. We display rank-ordered
distributions for the conditionally independent model with $N$ spins for
$N\\!=\\!40,60,80$ (left to right) under a Gaussian fluctuating field of
different widths (see legend). When the fluctuation $\phi$ is large compared
to the width of the conditional distribution $P(\bm{\sigma}\,|\,\phi)$—i.e.,
when $\operatorname{std}(\phi)\\!\gg\\!\Delta$, see Eq. (25)—the rank-ordered
plots exhibit Zipf behavior (dashed). The empirical rank-ordered distribution
displays meaningful structures only when constructed from adequate samples.
Larger systems require more samples; for $N\\!=\\!80$, the distribution is
completely flat for $\operatorname{std}(\phi)/\Delta\\!=\\!2$ even with
$10^{8}$ samples (c). The results shown are for $10^{8}$ realizations per
model and $w_{i}\\!\sim\\!\mathcal{N}(\mu\\!=\\!1,s\\!=\\!0.3)$.

##  Appendix A Mutual information in conditionally independent models

In this appendix, we consider conditionally independent spin models and derive
the leading contribution to the mutual information between the spins and the
fluctuating field and between two halves of the system in the many-spin limit.

First, we write down the probability distribution of the spins,

$P(\bm{\sigma})=\int\\!\\!\\!d\phi\,P(\phi)\prod\nolimits_{i}P(\sigma_{i}\mid\phi),$
(29)

where $P(\phi)$ denotes the distribution of the fluctuating field $\phi$. The
conditional probability of each spin reads

$P(\sigma_{i}\mid\phi)=\frac{e^{\sigma_{i}w_{i}\phi}}{2\cosh w_{i}\phi},$ (30)

where $w_{i}$ parametrizes the influence of the fluctuating field $\phi$ on
spin $i$. For convenience, we introduce

$g(\phi;\bm{\sigma})=-\tfrac{1}{N}\sum\nolimits_{i}(\sigma_{i}w_{i}\phi-\ln\cosh
w_{i}\phi),$ (31)

such that

$P(\bm{\sigma}\mid\phi)=\prod\nolimits_{i}P(\sigma_{i}\mid\phi)=e^{-Ng(\phi;\bm{\sigma})}/2^{N}.$
(32)

Therefore the full joint distribution can be written as

$P(\bm{\sigma},\phi)=P(\bm{\sigma}\mid\phi)P(\phi)=P(\phi)\times
e^{-Ng(\phi;\bm{\sigma})}/2^{N}.$ (33)

We now consider the limit $N\\!\to\\!\infty$. We assume that the weights
$\\{w_{i}\\}$ are independent of the system size $N$ (e.g., they are drawn
from a fixed distribution) such that $g(\phi;\bm{\sigma})$ is intensive. For a
_smooth_ prior—i.e., $\lim_{N\to\infty}\frac{1}{N}\ln P(\phi)\\!=\\!0$—the
joint distribution [Eq. (33)] is dominated by fluctuations around the minimum
of $g(\phi;\bm{\sigma})$,

$P(\bm{\sigma},\phi)\approx
P(\phi^{*}_{\bm{\sigma}})P(\bm{\sigma}\mid\phi^{*}_{\bm{\sigma}})\times
e^{-\frac{N}{2}g^{\prime\prime}(\phi^{*}_{\bm{\sigma}})(\phi-\phi^{*}_{\bm{\sigma}})^{2}}\\!\\!,$
(34)

where $\phi^{*}_{\bm{\sigma}}$ is the root of
$g^{\prime}(\phi;\bm{\sigma})\\!=\\!0$ and we drop the superfluous dependence
on $\bm{\sigma}$ from
$g^{\prime\prime}(\phi;\bm{\sigma})\\!=\\!\tfrac{1}{N}\sum\nolimits_{i}w_{i}^{2}\operatorname{sech}^{2}w_{i}\phi$.
We see that when conditioned on the spins, the fluctuation is Gaussian,

$P(\phi\mid\bm{\sigma})\approx\mathcal{N}\left(\phi\mid\mu\\!=\\!\phi^{*}_{\bm{\sigma}},\
s^{2}\\!=\\!\frac{1}{Ng^{\prime\prime}(\phi^{*}_{\bm{\sigma}})}\right).$ (35)

As a result, we obtain the conditional differential entropy,

$S(\phi\mid\bm{\sigma})\approx-\frac{1}{2}\ln N+\frac{1}{2}\ln 2\pi
e-\frac{1}{2}\sum_{\bm{\sigma}}P(\bm{\sigma})\ln
g^{\prime\prime}(\phi^{*}_{\bm{\sigma}}).$ (36)

We see that the logarithmic divergence is the leading contribution since the
last two terms do not grow with $N$. For extrinsic fluctuations, $P(\phi)$,
and thus $S(\phi)$, is independent of $N$; therefore, we obtain

$I(\phi;\bm{\sigma})=S(\phi)-S(\phi\mid\bm{\sigma})\approx\frac{1}{2}\ln
N+O(N^{0}).$ (37)

On the other hand, if $P(\phi)\\!\sim\\!e^{-N^{\gamma}\times
c(\phi-\phi_{0})^{2}}$ for some $\gamma\\!\in\\![0,1)$ and a constant
$c\\!>\\!0$, its entropy diverges logarithmically
$S(\phi)\\!\approx\\!-\frac{\gamma}{2}\ln N+O(N^{0})$ and the mutual
information reads

$I(\phi;\bm{\sigma})\approx\frac{1-\gamma}{2}\ln N+O(N^{0}).$ (38)

We see that the decrease in information results from the fluctuation entropy
that decreases logarithmically with $N$.

The same logarithmic divergence also emerges in the mutual information between
two macroscopic halves of the system. To see this, we note that the entropy of
the spins is given by

$S(\bm{\sigma})=S(\phi)-S(\phi\mid\bm{\sigma})+S(\bm{\sigma}\mid\phi)$ (39)

Similarly, the entropy of each half of the system reads

$S(\bm{\sigma}^{\nu})=S(\phi)-S(\phi\mid\bm{\sigma}^{\nu})+S(\bm{\sigma}^{\nu}\mid\phi)$
(40)

where $\nu\\!\in\\!\\{A,B\\}$ and
$\bm{\sigma}\\!=\\!(\bm{\sigma}^{A},\bm{\sigma}^{B})$. When the subsystems are
large and the spins in each half are randomly chosen, we have [see Eq. (36)]

$S(\phi\mid\bm{\sigma}^{\nu})\approx
S(\phi\mid\bm{\sigma})-\frac{1}{2}\ln\frac{N_{\nu}}{N}.$ (41)

We now write the mutual information between the two halves in terms of the
above entropy,

$\displaystyle I(\bm{\sigma}^{A};\bm{\sigma}^{B})$
$\displaystyle=S(\bm{\sigma}^{A})+S(\bm{\sigma}^{B})-S(\bm{\sigma})$ (42)
$\displaystyle\approx
I(\phi;\bm{\sigma})+\frac{1}{2}\ln\frac{N_{A}N_{B}}{N^{2}}.$ (43)

where we use the fact that
$I(\phi;\bm{\sigma})\\!=\\!S(\phi)\\!-\\!S(\phi\,|\,\bm{\sigma})$ and the
property of conditional independence,
$S(\bm{\sigma}\,|\,\phi)=S(\bm{\sigma}^{A}\,|\,\phi)+S(\bm{\sigma}^{B}\,|\,\phi)$.
For $N_{A}\\!=\\!N_{B}\\!=\\!N/2$, we see that
$I(\bm{\sigma}^{A};\bm{\sigma}^{B})$ is smaller than $I(\phi;\bm{\sigma})$ by
one bit.

##  Appendix B Rank-one Ising models

Here we provide an analysis of rank-one Ising models—those with pairwise
interaction matrices of rank one—defined by the energy function,

$E(\bm{\sigma})=-\frac{1}{4N}\sum\nolimits_{ij}w_{i}w_{j}\sigma_{i}\sigma_{j}=-\frac{1}{4N}\left(\sum\nolimits_{i}w_{i}\sigma_{i}\right)^{2},$
(44)

where $\bm{\sigma}\\!=\\!(\sigma_{1},\sigma_{2},\dots,\sigma_{N})$ denotes the
state of the system, $\sigma_{i}\\!\in\\!\\{\pm 1\\}$ the spin at site
$i\\!\in\\!\\{1,2,\dots,N\\}$, and the product $w_{i}w_{j}$ describes the
interaction between spins $i$ and $j$. We note that the terms with $i\\!=\\!j$
only add an irrelevant constant. This model generalizes the fully-connected
Ising model which corresponds to setting $w_{i}\\!=\\!w_{j}$ for all $i$ and
$j$. As usual, the probability distribution of the system configuration is
given by

$P(\bm{\sigma})=e^{-\beta E({\bm{\sigma}})}/Z,$ (45)

where we introduce the inverse temperature $\beta\\!=\\!1/T$ and the partition
function $Z\\!=\\!\sum_{\bm{\sigma}}e^{-\beta E(\bm{\sigma})}$.

Computing the partition function by directly summing over all possible spin
states is generally analytically intractable. Instead, we trade this summation
for an integral using the Hubbard–Stratonovich transformation,

$Z=\sum_{\bm{\sigma}}e^{\frac{\beta}{4N}\left(\sum_{i}w_{i}\sigma_{i}\right)^{2}}=\sqrt{\frac{N}{\pi\beta}}\int\\!\\!\\!d\phi\sum_{\bm{\sigma}}e^{-\frac{N}{\beta}\phi^{2}+\phi\sum_{i}w_{i}\sigma_{i}}.$
(46)

We see that the spins become noninteracting at the cost of introducing a new
fluctuating field $\phi$ which correlates with the spins via the joint
distribution,

$P(\bm{\sigma},\phi)=\frac{1}{Z}\sqrt{\frac{N}{\pi\beta}}e^{-\frac{N}{\beta}\phi^{2}+\phi\sum_{i}w_{i}\sigma_{i}}.$
(47)

Summing out each spin variable from Eq. (46) yields

$Z=2^{N}\sqrt{\frac{N}{\pi\beta}}\int\\!\\!\\!d\phi\,e^{-\frac{N}{\beta}\phi^{2}+\sum_{i}\ln\cosh
w_{i}\phi}.$ (48)

As a result, we can express various thermodynamic variables of the spins as
integrals over a continuous field which are usually more convenient than
summations over discrete spin states. In particular, the internal energy,
entropy and heat capacity—$U$, $S$ and $C$, respectively—read

$\displaystyle U$ $\displaystyle=-\frac{\partial}{\partial\beta}\ln
Z=\frac{1}{2\beta}\left(1-\frac{2N}{\beta}\langle\phi^{2}\rangle\right)$ (49)
$\displaystyle S$ $\displaystyle=\beta U+\ln Z$ (50) $\displaystyle C$
$\displaystyle=\beta^{2}\frac{\partial^{2}}{\partial\beta^{2}}\ln Z=2\beta
U-\frac{1}{2}+\frac{N^{2}}{\beta^{2}}\operatorname{var}(\phi^{2}),$ (51)

where $\langle\phi^{2}\rangle$ and $\operatorname{var}(\phi^{2})$ denote the
mean and variance of $\phi^{2}$.

In addition, we see that a rank-one Ising model is equivalent to a
conditionally independent model with a fluctuation field, induced by the
intrinsic interactions between spins. The marginal distribution of this field
is given by

$P_{N}(\phi)=\sum_{\bm{\sigma}}P(\bm{\sigma},\phi)=\frac{2^{N}}{Z}\sqrt{\frac{N}{\pi\beta}}e^{-\frac{N}{\beta}\phi^{2}+\sum_{i}\ln\cosh
w_{i}\phi},$ (52)

and thus we have

$P(\bm{\sigma}\mid\phi)=\frac{P(\bm{\sigma},\phi)}{P_{N}(\phi)}=\prod_{i}\frac{e^{\sigma_{i}w_{i}\phi}}{2\cosh
w_{i}\phi}.$ (53)

We see again that conditioning on the fluctuating field removes the
interactions between spins.

Recalling that $\ln\cosh x\approx\frac{x^{2}}{2}\\!-\\!\frac{x^{4}}{12}$ for
small $x$, we see that the fluctuation distribution $P_{N}(\phi)$ exhibits a
structural transition at

$\beta_{c}=\frac{2}{\frac{1}{N}\sum_{i}w_{i}^{2}}$ (54)

where it changes from unimodal at $\beta\\!<\\!\beta_{c}$ to bimodal at
$\beta\\!>\\!\beta_{c}$. In the limit $N\\!\to\\!\infty$, this point
corresponds to the critical temperature of an order-disorder phase transition.
In the disordered phase at high temperatures $\beta\\!<\\!\beta_{c}$, the
spins are mostly random. In the ordered phase at low temperatures
$\beta\\!>\\!\beta_{c}$, on the other hand, they mimic the pattern set by the
signs of $\\{w_{i}\\}$.

## References

  * West  _et al._ [1997] G. B. West, J. H. Brown, and B. J. Enquist, A general model for the origin of allometric scaling laws in biology, Science 276, 122 (1997).
  * Stevens [2001] C. F. Stevens, An evolutionary scaling law for the primate visual system and its basis in cortical function, Nature 411, 193 (2001).
  * Lee and Stevens [2007] S. Lee and C. F. Stevens, General design principle for scalable neural circuits in a vertebrate retina, Proceedings of the National Academy of Sciences 104, 12931 (2007).
  * Krotov  _et al._ [2014] D. Krotov, J. O. Dubuis, T. Gregor, and W. Bialek, Morphogenesis at criticality, Proceedings of the National Academy of Sciences 111, 3683 (2014).
  * Zhang  _et al._ [2022] W. Zhang, P. Scerbo, M. Delagrange, V. Candat, V. Mayr, S. Vriz, M. Distel, B. Ducos, and D. Bensimon, Fgf8 dynamics and critical slowing down may account for the temperature independence of somitogenesis, Communications Biology 5, 113 (2022).
  * Beggs [2008] J. M. Beggs, The criticality hypothesis: how local cortical networks might optimize information processing, Philosophical Transactions of the Royal Society A 366, 329 (2008).
  * Chialvo [2010] D. R. Chialvo, Emergent complex neural dynamics, Nature Physics 6, 744 (2010).
  * Toyoizumi and Abbott [2011] T. Toyoizumi and L. F. Abbott, Beyond the edge of chaos: Amplification and temporal integration by recurrent networks in the chaotic regime, Phys. Rev. E 84, 051908 (2011).
  * Mathijssen  _et al._ [2019] A. J. T. M. Mathijssen, J. Culver, M. S. Bhamla, and M. Prakash, Collective intercellular communication through ultra-fast hydrodynamic trigger waves, Nature 571, 560 (2019).
  * Mora and Bialek [2011] T. Mora and W. Bialek, Are biological systems poised at criticality?, Journal of Statistical Physics 144, 268 (2011).
  * Beggs [2022] J. M. Beggs, Addressing skepticism of the critical brain hypothesis, Frontiers in Computational Neuroscience 16, 703865 (2022).
  * Muñoz [2018] M. A. Muñoz, Colloquium: Criticality and dynamical scaling in living systems, Rev. Mod. Phys. 90, 031001 (2018).
  * Mora  _et al._ [2010] T. Mora, A. M. Walczak, W. Bialek, and C. G. Callan, Maximum entropy models for antibody diversity, Proceedings of the National Academy of Sciences 107, 5405 (2010).
  * Nykter  _et al._ [2008] M. Nykter, N. D. Price, M. Aldana, S. A. Ramsey, S. A. Kauffman, L. E. Hood, O. Yli-Harja, and I. Shmulevich, Gene expression dynamics in the macrophage exhibit criticality, Proceedings of the National Academy of Sciences 105, 1897 (2008).
  * Beggs and Plenz [2003] J. M. Beggs and D. Plenz, Neuronal avalanches in neocortical circuits, Journal of Neuroscience 23, 11167 (2003).
  * Haimovici  _et al._ [2013] A. Haimovici, E. Tagliazucchi, P. Balenzuela, and D. R. Chialvo, Brain organization into resting state networks emerges at criticality on a model of the human connectome, Phys. Rev. Lett. 110, 178101 (2013).
  * Tkačik  _et al._ [2015] G. Tkačik, T. Mora, O. Marre, D. Amodei, S. E. Palmer, M. J. Berry II, and W. Bialek, Thermodynamics and signatures of criticality in a network of neurons, Proceedings of the National Academy of Sciences 112, 11508 (2015).
  * Mora  _et al._ [2015] T. Mora, S. Deny, and O. Marre, Dynamical criticality in the collective activity of a population of retinal neurons, Phys. Rev. Lett. 114, 078105 (2015).
  * Kastner  _et al._ [2015] D. B. Kastner, S. A. Baccus, and T. O. Sharpee, Critical and maximally informative encoding between neural populations in the retina, Proceedings of the National Academy of Sciences 112, 2533 (2015).
  * Meshulam  _et al._ [2019] L. Meshulam, J. L. Gauthier, C. D. Brody, D. W. Tank, and W. Bialek, Coarse graining, fixed points, and scaling in a large population of neurons, Phys. Rev. Lett. 123, 178103 (2019).
  * Cavagna  _et al._ [2010] A. Cavagna, A. Cimarelli, I. Giardina, G. Parisi, R. Santagati, F. Stefanini, and M. Viale, Scale-free correlations in starling flocks, Proceedings of the National Academy of Sciences 107, 11865 (2010).
  * Bialek  _et al._ [2012] W. Bialek, A. Cavagna, I. Giardina, T. Mora, E. Silvestri, M. Viale, and A. M. Walczak, Statistical mechanics for natural flocks of birds, Proceedings of the National Academy of Sciences 109, 4786 (2012).
  * Bialek  _et al._ [2014] W. Bialek, A. Cavagna, I. Giardina, T. Mora, O. Pohl, E. Silvestri, M. Viale, and A. M. Walczak, Social interactions dominate speed control in poising natural flocks near criticality, Proceedings of the National Academy of Sciences 111, 7212 (2014).
  * Attanasi  _et al._ [2014] A. Attanasi, A. Cavagna, L. Del Castello, I. Giardina, S. Melillo, L. Parisi, O. Pohl, B. Rossaro, E. Shen, E. Silvestri, and M. Viale, Finite-size scaling as a way to probe near-criticality in natural swarms, Phys. Rev. Lett. 113, 238102 (2014).
  * Ma  _et al._ [2019] Z. Ma, G. G. Turrigiano, R. Wessel, and K. B. Hengen, Cortical circuit dynamics are homeostatically tuned to criticality in vivo, Neuron 104, 655 (2019).
  * Schwab  _et al._ [2014] D. J. Schwab, I. Nemenman, and P. Mehta, Zipf’s law and criticality in multivariate data without fine-tuning, Phys. Rev. Lett. 113, 068102 (2014).
  * Aitchison  _et al._ [2016] L. Aitchison, N. Corradi, and P. E. Latham, Zipf’s law arises naturally when there are underlying, unobserved variables, PLoS Computational Biology 12, e1005110 (2016).
  * Touboul and Destexhe [2017] J. Touboul and A. Destexhe, Power-law statistics and universal scaling in the absence of criticality, Phys. Rev. E 95, 012413 (2017).
  * Morrell  _et al._ [2021] M. C. Morrell, A. J. Sederberg, and I. Nemenman, Latent dynamical variables produce signatures of spatiotemporal criticality in large biological systems, Phys. Rev. Lett. 126, 118302 (2021).
  * Morrell  _et al._ [2023] M. Morrell, I. Nemenman, and A. J. Sederberg, Neural criticality from effective latent variable, eLife 12, RP89337 (2023).
  * Landau and Lifshitz [1980] L. D. Landau and E. M. Lifshitz,  _Statistical Physics_ , 3rd ed., Course of Theoretical Physics, Vol. 5 (Butterworth-Heinemann, Oxford, UK, 1980).
  * Goldenfeld [1992] N. Goldenfeld,  _Lectures On Phase Transitions And The Renormalization Group_ (CRC Press, 1992).
  * Cardy [1996] J. Cardy,  _Scaling and Renormalization in Statistical Physics_ , Cambridge Lecture Notes in Physics (Cambridge University Press, Cambridge, UK, 1996).
  * Grassberger [1986] P. Grassberger, Toward a quantitative theory of self-generated complexity, International Journal of Theoretical Physics 25, 907 (1986).
  * Bialek  _et al._ [2001a] W. Bialek, I. Nemenman, and N. Tishby, Predictability, complexity, and learning, Neural Computation 13, 2409 (2001a).
  * Bialek  _et al._ [2001b] W. Bialek, I. Nemenman, and N. Tishby, Complexity through nonextensivity, Physica A: Statistical Mechanics and its Applications 302, 89 (2001b), Proc. Int. Workshop on Frontiers in the Physics of Complex Systems.
  * Tchernookov and Nemenman [2013] M. Tchernookov and I. Nemenman, Predictive information in a nonequilibrium critical model, Journal of Statistical Physics 153, 442 (2013).
  * Wilms  _et al._ [2012] J. Wilms, J. Vidal, F. Verstraete, and S. Dusuel, Finite-temperature mutual information in a simple phase transition, Journal of Statistical Mechanics: Theory and Experiment 2012, P01023 (2012).
  * Lau and Grassberger [2013] H. W. Lau and P. Grassberger, Information theoretic aspects of the two-dimensional ising model, Phys. Rev. E 87, 022128 (2013).
  * Cohen  _et al._ [2015] O. Cohen, V. Rittenberg, and T. Sadhu, Shared information in classical mean-field models, Journal of Physics A: Mathematical and Theoretical 48, 055002 (2015).
  * Marsili  _et al._ [2013] M. Marsili, I. Mastromatteo, and Y. Roudi, On sampling and modeling complex systems, Journal of Statistical Mechanics: Theory and Experiment 2013, P09003 (2013).
  * Cubero  _et al._ [2019] R. J. Cubero, J. Jo, M. Marsili, Y. Roudi, and J. Song, Statistical criticality arises in most informative representations, Journal of Statistical Mechanics: Theory and Experiment 2019, 063402 (2019).
  * Duranthon  _et al._ [2021] O. Duranthon, M. Marsili, and R. Xie, Maximal relevance and optimal learning machines, Journal of Statistical Mechanics: Theory and Experiment 2021, 033409 (2021).
  * Marsili and Roudi [2022] M. Marsili and Y. Roudi, Quantifying relevance in learning and inference, Physics Reports 963, 1 (2022).
  * Cover and Thomas [2006] T. M. Cover and J. A. Thomas,  _Elements of Information Theory_ , 2nd ed. (Wiley-Interscience, 2006).
  * Ngampruetikorn and Schwab [2022] V. Ngampruetikorn and D. J. Schwab, Information bottleneck theory of high-dimensional regression: relevancy, efficiency and optimality, in  _Advances in Neural Information Processing Systems_, Vol. 35, edited by S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Curran Associates, Inc., 2022) pp. 9784–9796.
  * Bialek [2022] W. Bialek, On the dimensionality of behavior, Proceedings of the National Academy of Sciences 119, e2021860119 (2022).
  * Humplik and Tkačik [2017] J. Humplik and G. Tkačik, Probabilistic models for neural populations that naturally capture global coupling and criticality, PLOS Computational Biology 13, 1 (2017).
  * Ngampruetikorn and Schwab [2023] V. Ngampruetikorn and D. J. Schwab, Random-energy secret sharing via extreme synergy (2023), in preparation.
  * Bialek  _et al._ [2020] W. Bialek, S. E. Palmer, and D. J. Schwab, What makes it possible to learn probability distributions in the natural world? (2020), arXiv:2008.12279 [cond-mat.stat-mech] .
  * Eisert  _et al._ [2010] J. Eisert, M. Cramer, and M. B. Plenio, Colloquium: Area laws for the entanglement entropy, Rev. Mod. Phys. 82, 277 (2010).
  * Holzhey  _et al._ [1994] C. Holzhey, F. Larsen, and F. Wilczek, Geometric and renormalized entropy in conformal field theory, Nuclear Physics B 424, 443 (1994).
  * Vidal  _et al._ [2003] G. Vidal, J. I. Latorre, E. Rico, and A. Kitaev, Entanglement in quantum critical phenomena, Phys. Rev. Lett. 90, 227902 (2003).
  * Calabrese and Cardy [2004] P. Calabrese and J. Cardy, Entanglement entropy and quantum field theory, Journal of Statistical Mechanics: Theory and Experiment 2004, P06002 (2004).
  * Refael and Moore [2004] G. Refael and J. E. Moore, Entanglement entropy of random quantum critical points in one dimension, Phys. Rev. Lett. 93, 260602 (2004).
  * Ryu and Takayanagi [2006] S. Ryu and T. Takayanagi, Holographic derivation of entanglement entropy from the anti–de sitter space/conformal field theory correspondence, Phys. Rev. Lett. 96, 181602 (2006).
  * Mirollo and Strogatz [1990] R. E. Mirollo and S. H. Strogatz, Synchronization of pulse-coupled biological oscillators, SIAM Journal on Applied Mathematics 50, 1645 (1990).
  * Acebrón  _et al._ [2005] J. A. Acebrón, L. L. Bonilla, C. J. Pérez Vicente, F. Ritort, and R. Spigler, The kuramoto model: A simple paradigm for synchronization phenomena, Rev. Mod. Phys. 77, 137 (2005).
  * BuzsÃ¡ki and Draguhn [2004] G. BuzsÃ¡ki and A. Draguhn, Neuronal oscillations in cortical networks, Science 304, 1926 (2004).
  * Gregor  _et al._ [2010] T. Gregor, K. Fujimoto, N. Masaki, and S. Sawai, The onset of collective behavior in social amoebae, Science 328, 1021 (2010).
  * Di Talia and Vergassola [2022] S. Di Talia and M. Vergassola, Waves in embryonic development, Annual Review of Biophysics 51, 327 (2022).