SIAdditional References

# Phase Transitions in the Output Distribution
of Large Language Models

Julian Arnold
Department of Physics
University of Basel
Klingelbergstrasse 82, 4056 Basel, Switzerland
<EMAIL_ADDRESS>&Flemming Holtorf
CSAIL
Massachusetts Institute of Technology
Cambridge, Massachusetts 02139, USA
<EMAIL_ADDRESS>&&Frank Schäfer
CSAIL
Massachusetts Institute of Technology
Cambridge, Massachusetts 02139, USA
<EMAIL_ADDRESS>&Niels Lörch
Department of Physics
University of Basel
Klingelbergstrasse 82, 4056 Basel, Switzerland
<EMAIL_ADDRESS>

###### Abstract

In a physical system, changing parameters such as temperature can induce a
phase transition: an abrupt change from one state of matter to another.
Analogous phenomena have recently been observed in large language models.
Typically, the task of identifying phase transitions requires human analysis
and some prior understanding of the system to narrow down which low-
dimensional properties to monitor and analyze. Statistical methods for the
automated detection of phase transitions from data have recently been proposed
within the physics community. These methods are largely system agnostic and,
as shown here, can be adapted to study the behavior of large language models.
In particular, we quantify distributional changes in the generated output via
statistical distances, which can be efficiently estimated with access to the
probability distribution over next-tokens. This versatile approach is capable
of discovering new phases of behavior and unexplored transitions – an ability
that is particularly exciting in light of the rapid development of language
models and their emergent capabilities.

##  1 Introduction

Colloquially, the term phase transition refers to a change among the basic
phases of matter. For example, in response to changes in external conditions
such as temperature or pressure, water can transition to a solid, liquid, or
gaseous state. More broadly, in physics a phase transition refers to an abrupt
change in the macroscopic behavior of a large-scale system of interacting
constituents [1, 2]. Notable examples include transitions in the magnetic
properties of materials [3], transitions from a normal conducting state to a
superconductor [4], transitions in the entanglement properties of quantum
circuit [5], or the collective motion of active matter such as a flock of
birds [6].

In the context of artificial intelligence, “phase transition”-like phenomena
have also been observed in the learning behavior of neural networks (NNs) [1,
7, 8, 9, 10, 11, 12, 13, 14, 15, 16]. For example, during training, AlphaZero
[17] underwent periods of rapid knowledge acquisition in which increasingly
sophisticated chess openings were favored by the engine [9]. Large language
models (LLMs) have been observed to make sudden improvements in their
inductive abilities during training which is related to the formation of
special circuitry (so-called induction heads) [18]. Similar abrupt
improvements in specific capabilities, often referred to as breakthroughs,
have been observed for a variety of different models and tasks [19, 20, 21,
22, 23, 24, 12, 25, 26, 10, 27]. Moreover, phenomena such as double descent
[28, 29] or grokking [30, 31, 32, 33, 34, 35] are also reminiscent of phase
transitions in physics.

The detection of phase transitions111In the following, we adopt a more general
definition of a phase transition as a sudden shift in the qualitative behavior
of a system as a function of a control parameter [1, 10, 36]. in deep learning
systems may improve our understanding and eventually enable better model
training. For example, an in-depth analysis of the grokking transition [30,
37] led to a way for accelerating generalization [32]. Moreover, it has been
shown that models are highly sensitive to perturbations, such as data
corruptions, at critical points [38, 36]. Being able to predict the behavior
of models is also crucial for ensuring safe model deployment [19] as well as
for projecting the performance of future model versions and optimally
allocating resources for their training [39].

The characterization of phase transitions in physics is difficult because the
state of the systems to be studied typically lives in a very high-dimensional
space and is probabilistic in nature, meaning that for given values of the
tuning parameters we can find the system in various states. Physicists solve
this problem by finding a suitable set of a few low-dimensional quantities,
called order parameters [2], which capture the essence of each phase of the
system. For example, even though water is a highly complex system, we can
detect the liquid-gas transition by looking at the density which shows a
sudden jump at the boiling point and, in this case, serves as an order
parameter. However, finding such a suitable set of order parameters is
“considered an art” [2], as it requires a great deal of human intuition as
well as prior understanding.

Faced with the task of characterizing phase transitions in learning systems
based on large NNs, similar issues are encountered. NNs contain an enormous
amount of trainable parameters and their state space, as characterized by
their neural activations, is huge. This problem is exacerbated in generative
models such as LLMs where also the output space is large, i.e., the high
dimensionality cannot be foregone by treating the inside of the NN as a black
box and focusing solely on its output characteristics. Understanding LLMs from
first principles has been notoriously hard [40]. Theories capturing their
microscopic and macroscopic behavior, for instance based on mechanistic
interpretability [41, 18, 42, 34, 43] or neural scaling laws [44, 45, 39, 46,
47, 48, 49, 27], are still nascent. In particular, the definition of
appropriate low-dimensional quantities that facilitate the detection of
transitions has been done manually, for example through the extraction of
appropriate circuitry [50, 51, 43]. Due to this human-in-the-loop, transitions
can be easily missed [43] or spuriously induced [52].

In physics, these problems have been tackled using statistical methods for the
detection of phase transitions from data, which requires minimal prior system
knowledge and human input [53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65,
66, 67, 68, 69, 70, 71, 72, 73, 74, 75]. Inspired by this body of work, we
here adapt such an approach for the automated detection of phase transitions
in LLMs. The method is based on measuring changes in the distribution of the
text output of LLMs via generic statistical distances belonging to the family
of $f$-divergences, making it a versatile all-purpose tool for objectively and
automatically mapping out phase diagrams of generative models. Such an
approach has the potential to characterize unexplored phase transitions and
potentially discover new phases of behaviors. This is crucial in light of the
rapid development of LLMs [76, 77, 78] and their emergent capabilities [19,
20, 21, 22, 23, 24, 79, 18, 80, 21, 25, 26, 10].

As a demonstration, we characterize transitions occurring as a function of
three different control parameters in Pythia [81], Mistral (7B) [82], and
Llama3 (8B) [83] language models: an integer occurring in the input prompt,
the temperature hyperparameter for text generation, and the model’s training
epoch.

#### Specific Findings.

We find that

  * •

the instruction-tuned Llama and Mistral models seem to have the capability to
order integers whereas all considered base models do not.

  * •

changes in integer tokenization can be visible in the text output as sharp
transitions.

  * •

three distinct phases of behavior as a function of an LLM’s temperature can be
mapped out: a deterministic “frozen” phase near zero temperature, an
intermediate “coherent” phase, and a “disordered” phase at high temperatures.

  * •

an LLM’s “heat capacity” with respect to the temperature can be negative,
i.e., the LLM’s mean energy can decrease as its temperature is increased.

  * •

rapid changes in the distribution of weights during training can coincide with
transitions in the text output that are present across many prompts.

  * •

different prompts result in different transition times during training,
suggesting that distinct type of behavior can be learned rapidly at distinct
times in training.

##  2 Methodology

###  2.1 Quantifying Dissimilarity between Distributions

In this work, we view phase transitions as rapid changes in the probability
distribution $P(\cdot|T)$ governing the state of the system $\bm{x}\sim
P(\cdot|T)$ as the control parameter $T$ is varied.222This definition
encompasses phase transitions in physics, i.e., abrupt changes in the
distribution governing large-scale systems of interacting constituents. That
is, values of the parameter at which the distribution changes strongly are
considered critical points where phase transitions occur. While it is possible
to generalize our approach to distributions conditioned on multiple control
parameters (see [73, 75]), for simplicity we consider the one-dimensional
scenario in the following.

We quantify the rate of change using $f$-divergences [84], as they have
particularly nice properties, such as satisfying the data processing
inequality. Given a convex function $f:\mathbb{R}_{\geq
0}\rightarrow\mathbb{R}$ with $f(1)=0$, the corresponding $f$-divergence is a
statistical distance defined as

$D_{f}[p,q]=\sum_{\bm{x}}q(\bm{x})f\left(\frac{p(\bm{x})}{q(\bm{x})}\right)\geq
0.$ (1)

Prominent examples of $f$-divergences include the Kullback-Leibler (KL)
divergence, the Jensen-Shannon (JS) divergence, which corresponds to a
symmetrized and smoothened version of the KL divergence, as well as the total
variation (TV) distance. Ideally, we would also like the statistical distance
we choose to be symmetric $D[p,q]=D[q,p]$. This condition is only satisfied by
the TV distance and the JS divergence among the examples above.

Hence, in this work, we will focus on the TV distance

$D_{\rm TV}[p,q]=\frac{1}{2}\sum_{\bm{x}}|p(\bm{x})-q(\bm{x})|$ (2)

corresponding to $f(x)=\frac{1}{2}|1-x|$, as well as the JS divergence

$D_{\rm JS}[p,q]=\frac{1}{2}D_{\rm
KL}\left[p,\frac{p+q}{2}\right]+\frac{1}{2}D_{\rm
KL}\left[q,\frac{p+q}{2}\right]$ (3)

corresponding to
$f(x)=\frac{1}{2}\left[x\log(\frac{2x}{1+x})+\log(\frac{2}{1+x})\right]$,
where $D_{\rm KL}$ is the KL divergence.

The TV distance and the JS divergence have also had tremendous success in
detecting phase transitions in physical systems without prior system knowledge
under the name of “learning-by-confusion” [55, 85, 86, 87, 88, 89, 90, 91, 92,
93, 94, 72, 95, 96, 97, 98, 99, 100, 73, 75, 74, 101].333Note that both the TV
distance and the JS divergence form lower bounds to the KL divergence and
other $f$-divergence, such as the $\chi^{2}$ divergence: $D_{\rm JS}[p,q]\leq
D_{\rm TV}[p,q]\leq\sqrt{D_{\rm KL}[p,q]}\leq\sqrt{D_{\chi^{2}}[p,q]}$ [102].
In this sense, detecting a large dissimilarity in terms of the TV distance or
the JS divergence also signals a large dissimilarity in other measures.

###  2.2 Detecting Phase Transitions

Having defined appropriate notions of distance between probability
distributions, we now describe their use to detect phase transitions: Consider
a sampled set $\mathcal{T}$ of control parameter values $T$, forming a uniform
one-dimensional grid. For each $T^{*}$ lying halfway in between grid points,
we assess whether it is a critical point by computing a dissimilarity score
$D(T^{*})=D\left[P_{\mathrm{left}}(\cdot|T^{*}),P_{\mathrm{right}}(\cdot|T^{*})\right]$
between the distributions underlying the segments
$\sigma_{\mathrm{left}}(T^{*})$ to the left and
$\sigma_{\mathrm{right}}(T^{*})$ to the right of $T^{*}$. Denoting the
cardinality as $|\cdot|$, we can write these probabilities as
$P_{i}(\cdot|T^{*})=\frac{1}{|\sigma_{i}(T^{*})|}\sum_{T\in\sigma_{i}(T^{*})}P(\cdot|T)$
for $i\in\\{\mathrm{left},\mathrm{right}\\}$. Critical points where phase
transitions occur can then be identified as local maxima in $D$.

For the sake of simplicity, we proceed with segments of equal length for the
rest of this article, and define the length $L=|\sigma_{\rm
left}|=|\sigma_{\rm right}|$ as the number of parameter values $T$ to the left
or right of $T^{*}$ that characterize the segment. We are free to adjust it
according to the problem, as $L$ sets a natural length scale on which changes
in the distributions are assessed. Examples will be discussed in Sec. 3. In
particular for $L=1$ and neighboring parameter points separated by $\delta T$,
$D_{f}(T^{*})=\tfrac{1}{2}{f^{\prime\prime}(1)}\mathcal{F}(T^{*})\delta
T^{2}+\mathcal{O}(\delta T^{3})$ where $\mathcal{F}$ is the Fisher information
[103]. That is, local changes in a distribution as measured by any
$f$-divergence reduce to the Fisher information in the limit $\delta T\to 0$.
Having the Fisher information as a limiting case is a desirable property: It
is a well-known, generic statistical measure for quantifying how sensitive
probability distributions are to changes in their parameters and its behavior
is well-understood when used to detect phase transitions in physical systems
[104, 105, 106, 75].

###  2.3  Application to Language Models and Numerical Implementation

In the case of language models, $\bm{x}$ is the sampled text and $T$ is any
variable that influences the sampling probability. Because of the
autoregressive structure of language models, we can efficiently sample text
$\bm{x}$ for a given prompt and evaluate its probability $P(\bm{x}|T)$. Thus,
we can obtain an unbiased estimate $\hat{D}(T^{*})$ of $D(T^{*})$ by replacing
expected values with sample means where samples correspond to text generated
with language models conditioned on different parameter settings $T$, see
Appendix A for details on implementation.

For numerical stability and efficient sampling, we express our dissimilarity
measures as parameterized by a function $g$ acting on the probability
$P(\sigma_{i}|\bm{x})=\frac{P_{i}(\bm{x})}{P_{\mathrm{left}}(\bm{x})+P_{\mathrm{right}}(\bm{x})}$
for $\bm{x}$ to stem from segment $\sigma_{i}$. Specifically, we consider

$D_{g}=\frac{1}{2L}\sum_{i\in\\{{\rm left},{\rm
right}\\}}\sum_{T\in\sigma_{i}}\mathbb{E}_{\bm{x}\sim
P(\cdot|T)}\biggl{[}g\left[P(\sigma_{i}|\bm{x})\right]\biggr{]}.$ (4)

These $g$-dissimilarities and the $f$-divergences [Eq. (1)] defined above
correspond to each other in the following sense: any $g$-dissimilarity $D_{g}$
can be rewritten in the form of an $f$-divergence $D_{f}[P_{\rm left},P_{\rm
right}]$ with

$f(x)=\frac{x}{2}\cdot g\left(\frac{x}{1+x}\right)+\frac{1}{2}\cdot
g\left(\frac{1}{1+x}\right),$ (5)

see Appendix B for the derivation and further discussion. In particular, for
the choice $g(x)=\log(x)+\log(2)$, $D_{g}$ corresponds to the JS divergence
[Eq. (3)]. For $g(x)=1-2\min\\{x,1-x\\}$, $D_{g}$ corresponds to the TV
distance [Eq. (2)].

A natural choice for $g$ is any linear function in $x$. In particular, setting
$g(x)=2x-1$ results in a dissimilarity measure that quantitifies the ability
of an optimal classifier to tell whether a sample $\bm{x}$ has been drawn in
the left or right sector. This measure is 0 if the two distributions are
completely indistinguishable and 1 if the two distributions are perfectly
distinguishable. Moreover, $g(x)=2x-1$ has the property of being bounded
between 1 and -1, where the edge values are attained for the certain
predictions 0 and 1, and the value 0 corresponds to uncertain predictions at
0.5. This results in a low variance and favorable convergence properties for
$\hat{D}_{g(x)=2x-1}$, which we will refer to as linear dissimilarity in what
follows. This quantity is a valid $f$-divergence and reduces to the Fisher
information in lowest non-vanishing order444In fact, any $g$-dissimilarity
with $g(1/2)=0$ and a twice-differentiable $g$-function can be shown to be
proportional to the Fisher information in lowest order., as shown in Appendix
B.

###  2.4 Utilized Large Language Models

In this work, we study transitions in models of the Pythia, Mistral, and Llama
family.

#### Pythia

Pythia [81] is suite of 16 LLMs released in 2023 that were trained on public
data in the same reproducible manner ranging from 70 million (M) to 12 billion
(B) parameters in size. We consider every second model, i.e. the models with
70M, 410M, 1.4B, and 6.9B parameters.

#### Mistral

From the Mistral family, we consider the base model Mistral-7B-v0.1 with 7.3B
parameters and the corresponding fine-tuned Mistral-7B-Instruct model [82]
released in 2023.

#### Llama

Llama 3 [83] from Meta AI was released in 2024. We consider both the Llama-3
8B parameter base model and NVIDIA’s chat-tuned Llama3-ChatQA-1.5-8B [107].
For the chat model we use accordingly formatted inputs.

##  3 Results

In the following, we will explore all three fundamental ways in which a
parameter $T$ may influence the output distribution of a language model: $(i)$
As a variable within the input prompt, we scan through integers injected to
the prompt in Sec. 3.1. $(ii)$ As a hyperparameter controlling how a trained
language model is applied, we vary the temperature in Sec. 3.2. $(iii)$ As a
training hyperparameter of the language model, we vary the number of training
epochs in Sec. 3.3.

###  3.1 Transitions as a Function of a Variable in the Prompt

Figure 1: Mistral model applied to the integer ordering prompt. (a) Different
$g$-dissimilarities with $L=3$. (b) Linear dissimilarity for different
$L$-values. [Number of text outputs generated per parameter value $T$: 10280.
Number of generated output tokens: 10. Error bars indicate standard error of
the mean over 4 batches, each with batch size 2056.]

As an introduction, we start with the simplest case: The parameter $T$ to be
varied is a particular part of the prompt, and all parameters of the language
model itself are fixed. As a first such prompt, consider “$T$ is larger than
42. True or False?” with an integer $T$ as the control parameter. An LLM that
understands the order of integers should output very different answers for
$T<42$ versus $T>42$, i.e., its distribution over outputs should change
drastically around $T=42$. Thus, in such a case we expect the dissimilarities
to show a clear peak around $T=42$.

Figure 1(a) shows dissimilarities based on various $g$-functions for the
Mistral-7B-Instruct model [82]. All dissimilarities show a clear peak around
$T=42$, whereas they are relatively flat otherwise. This is a clear example of
an abrupt transition between two distinct phases of behaviors of an LLM as a
function of a tunable parameter. As compared to the linear dissimilarity, the
logarithm-based JS divergence is arguably a bit sharper in that it decays more
rapidly to baseline 0. The TV distance’s peak is the broadest due to the
$\min$ function appearing in its $g$-function. In the following, we will focus
on the linear dissimilarity as a compromise between sensitivity and numerical
stability.

The transition is also clearly visible using different $L$ settings, see Fig.
1(b). Smaller $L$ values are closer to the Fisher information limit, while
larger values generally lead to higher distinguishability of distributions and
therefore larger peaks at transition points. As we will see in more detail in
Sec. 3.3, they can also be less susceptible to outliers due to the averaging
over several parameter points.

Figure 2: Benchmarking of various models using the linear dissimilarity with
$L=3$. (a) Test of ability to compare integers in value. (b) Bare integers as
prompt reveals transition in tokenizer encoding. [Same numerical settings as
in Fig 1.]

Interestingly, when performing the same analysis on base models such as the
Llama3-8B and Mistral base models, as well as Pythia models [81] of various
sizes, the resulting linear dissimilarity is flat, signaling the absence of
any transition [see Fig. 2(a)]. In contrast to Mistral-7B-Instruct and
NVIDIA’s chat-tuned Llama3-8B, these models do not show a clear peak around
$T=42$.

A transition of a different origin can be observed in Fig. 2(b), where the
LLMs are probed using the prompt “$T$” with $T$ again being an integer.
Interestingly, all Pythia models show a peak between $T=2020$ and $T=2021$.
This behavioral transition may originate from a transition in the tokenizers
of these models, which encode numbers in a range below $T=2021$ with a single
token and numbers in a range at and above $T=2021$ with two. This explanation
is corroborated by the absence of the transition around $T=2021$ for the Llama
and Mistral models, whose tokenizers translate a number into tokens following
rules that are independent of the number’s frequency.

The Mistral models and the base Llama3-8B model show a smaller peak around
$T=2023/2024$. Both models have only encountered training data from before and
around that time given their release date in 2023/2024, which may explain the
peak. This transition is absent in the Pythia models.

###  3.2 Transitions as a Function of the Model’s Temperature

Next, we consider transitions as a function of the temperature hyperparameter
$T$ controlling how the logits $\bm{z}$ are converted to probabilities

$p_{i}=\frac{e^{z_{i}/T}}{\sum_{j}e^{z_{j}/T}}$ (6)

for next-token prediction where the sum runs over all possible tokens. Per
construction, at $T=1$ language models predict probabilities $p_{i}$ to
approximate the distribution to be learned. In the limit $T\rightarrow 0$, the
model deterministically picks the most likely next token in each step. For
$T\rightarrow\infty$ the model samples the next token uniformly.

This scenario somewhat resembles a system of a one-dimensional lattice of
spins that are coupled via long-range interactions, i.e., the one-dimensional
Ising model [108, 109], which has an order-disorder phase transition. In our
case, the tokens take the role of the spins, and the coupling is mediated via
the transformer’s attention mechanism.

In Fig. 3, the dissimilarity shows two distinct peaks corresponding to two
transition points: one at a very low temperature $T_{1}^{*}\approx 0.02$ and
one at an intermediate temperature $T_{2}^{*}\approx 0.5$. Intuitively, these
two points mark transitions between three distinct phases of behavior:
“frozen” at low temperatures $T<T_{1}^{*}$, “unfrozen and sensible” at
intermediate temperatures $T_{1}^{*}<T<T_{2}^{*}$, and “random” at high
temperatures $T_{2}^{*}<T$. The transition at low temperatures has recently
been investigated in Ref. [110] for GPT-2 using physics-inspired quantities.
Moreover, they speculated on the existence of a phase transition at higher
temperatures.

Figure 3: Temperature transitions of Pythia 70M model in response to the
prompt “There’s measuring the drapes, and then there’s measuring the drapes on
a house you haven’t bought, a” – an excerpt from OpenWebText [111]. Linear
dissimilarity measure ($L=5$) is shown in black. Heat capacity is shown in
red. Dashed lines indicate local maxima, i.e., predicted critical points.
Shaded regions indicate the error bands. (a) Temperature range $[10^{-4},2]$.
(b) Zoomed-in range $[10^{-4},0.2]$ near $T=0$. [Number of text outputs
generated per parameter value $T$: 20480. Number of generated output tokens:
10. Error bars indicate standard error of the mean over 4 batches, each with
batch size 5120.]

We perform an analysis independent of the dissimilarity-based indicators by
taking inspiration from statistical mechanics, where the state of thermal
systems is governed by the Boltzmann distribution (see Appendix C for
details). We view the LLM as such a thermal system at varying temperature
where the negative logarithmic probability at $T=1$, $-\log P(\bm{x}|T=1)$,
takes on the role of the energy $E$ of a given text output $\bm{x}$. In
physical systems governed by Boltzmann distributions, thermal phase
transitions can be detected as peaks in the heat capacity
$C(T)=\partial\mathbb{E}_{\bm{x}\sim
P(\cdot|T)}\left[E(\bm{x})\right]/\partial T$, i.e., by looking at the
temperature derivative of the mean total energy [112].

Figure 3 shows that the locations of peaks (i.e., dips) in these quantities
are close to the critical points highlighted by our method. Note that in the
LLM case, the text outputs are not truly sampled from a Boltzmann distribution
governed by the total energy. Instead, each individual token is drawn from a
Boltzmann distribution for its individual energy conditioned on the previous
tokens only. This procedure corresponds to a greedy sampling strategy. The
resulting sampling mismatch can lead to the counterintuitive phenomenon of the
mean energy of the system increasing with decreasing temperature corresponding
to a negative “heat capacity”, cf. Fig. 3(b).

The intermediate temperature transition at $T_{2}^{*}$ may be reminiscent of
the Schottky anomaly [112] occurring in systems with a finite number of energy
levels. As such, this phenomenon is perhaps better described as a crossover
rather than a phase transition in the Ehrenfest sense. In particular, we also
observed such a transition for a basic language model that samples words
according to their overall frequency without taking into account any word-to-
word interaction.

In Fig. 3 we have investigated the output distributions corresponding to a
specific prompt. While we find that the temperature behavior is strongly
dependent on the prompt, there seems to be a trend: many distinct prompts lead
to a transition at $T\approx 1$ (i.e., on the order of the natural temperature
scale), at $T\ll 1$, or both.

###  3.3 Transitions as a Function of the Training Epoch

Finally, we search for transitions as a function of the training epoch, i.e.,
we compare the output distributions of models at different stages during
training and see whether there are certain epochs at which these statistics
change drastically. Such temporal analyses are rare given that they require
access to models at checkpoints during training [113, 114, 36]. Here, we
analyze the Pythia suite of models for which such checkpoints are publicly
available.

Ref. [115] analyzed the weight distribution of the Pythia models, and similar
weight-based analyses of other NNs during training have also been performed in
previous works [7, 38, 36]. In order to study the previously observed
transitions [115], we analyze changes in the weight distributions in the same
manner as for the output distributions (see Sec. 2), i.e., to characterize
phase transitions using dissimilarities. The lists of model weights are
converted to distributions via histogram binning (10000 bins for the range -3
to 3).

The results for $L=6$ are shown in Fig. 4(a) as colored lines, each
corresponding to the distribution of the weights of a particular QKV layer.
Different layers show transitions at roughly 20K (layer 5), 40K (layers 3),
50K (layer 4), and 80K (layer 4) epochs. We also observe a large peak around
epoch 0, i.e., at the start of the training, highlighting that the LLM learns
most rapidly at the beginning stages. In the long run, the dissimilarity
curves approach 0, signaling that overall the weight distributions become less
and less distinguishable.

Figure 4: Linear dissimilarity by epoch, with checkpoints taken every 1000
epochs. (a) Computed at $L=6$ for both weights and responses to 20 random
prompts from OpenWebText (gray) and 7 short prompts (black) shown in panel
(b). (b) Computed at $L=1$ for several prompts. For reference, the mean linear
dissimilarity over short prompts and OpenWebText prompts with $L=1$ is also
shown. [Number of text outputs generated per parameter value $T$ and prompt:
1536. Number of generated output tokens: 10. Error bars indicate the standard
error of the mean over all corresponding prompts. Error bars for the
individual prompts in panel (b) are almost negligible and thus omitted to
avoid visual clutter.]

Complementarily, in the same plot, we show dissimilarities derived from the
LLM output distributions. The grey line corresponds to an average of the
dissimilarities obtained by using entries from OpenWebText [111] (which serves
as a proxy for the Pythia training dataset) as prompts. The black line
corresponds to the average of results obtained from a selection of single-
token prompts [see also panel (b)]. Both dissimilarity curves show a peak
around epoch 0 as well as a peak around 80K epochs that is potentially related
to the rapid change of layer 4 around the same time.

Figure 4(b) shows dissimilarities as a function of the training epoch for
models queried with short, generic prompts (“ ”, “0”, “I”, “?”, “1”, “You”,
and “!”) at $L=1$. These short prompts were selected to be as general as
possible and the associated output distributions seem more sensitive as
compared to the long examples from OpenWebText: their mean dissimilarity shows
clear peaks near epochs 20K, 40K, and 80K. These correspond to outliers where
the output distribution changes severely only at a single point and returns
back (close) to its original behavior immediately after. As such, these peaks
do not mark transitions between two macroscopic phases of behavior. We further
verified this outlier behavior by inspecting the dissimilarity between the
points directly to the left and right of the potential outlier. It remains an
open question if these outliers are linked to the transitions observed in the
layer weights shown in panel (a).

The larger $L$ value used in panel (a) averages out the signal stemming from
these outliers. Such a reduced susceptibility to outliers can be an advantage
of using $L\gg 1$ when searching for macroscopic transitions in particular.

Some peak locations in the dissimilarity curves are prompt-dependent,
indicating that learning progresses differently for different types of
behavior. Here we have used rather generic prompts, resulting in an analysis
of the LLM’s general behavior during training. However, in principle,
conditioning on the prompt allows one to analyze whether and when specific
knowledge emerges [113, 114]. As an outlook, one can imagine automatically
monitoring changes across a multitude of prompts on different topics and
testing different abilities at scale, without the need to design individual
metrics for each prompt.

##  4 Related Works

Before concluding, let us discuss how our method relates to other approaches
for studying transitions in LLMs.

Generic performance-based analysis. Many previous works found transitions in
LLM behavior by locating sharp changes in generic performance measures, such
as sudden drops in training loss [18, 36]. While this may capture transitions
in the overall behavior, such an approach cannot resolve transitions in
specific LLM behavior. In particular, it may miss algorithmic transitions
where the same performance is reached but by different means [43].

Prompt-specific success metrics. Other works have found transitions by looking
at success metrics tailored toward specific prompts [20, 21, 22, 23, 24, 25,
113]. Recalling the example studied in Sec. 3.1, this would correspond to
assigning a score of 1 if the LLM provided the correct answer to the question
$x<42$ and 0 otherwise. Similarly, one could compute such a score in a
temporal analysis (Sec. 3.3) or for detecting transitions as a function of
another hyperparameter (Sec. 3.2). A downside of this approach is that it is
restricted to prompts that allow for a clear score to be assigned. In
particular, choosing an appropriate scoring function may require lots of human
engineering. Moreover, discontinuous metrics can artificially induce
transitions where the underlying behavior varies smoothly [52]. Similarly,
they may miss transitions where the same performance is reached but by
different means [43].

Measures based on model internals. The aforementioned approaches are based on
the model output. Many works have also detected transitions based on changes
in the internal structure of models, such as its trainable parameters [115,
36] (similar to the weight-based analysis we have performed in Sec. 3.3).
However, access to model internals may not always be available. Moreover, the
design of measures that capture specific transitions in behavior requires lots
of human input [50, 51, 43], e.g., using insights from the field of
mechanistic interpretability.

##  5 Conclusion and Outlook

We have proposed a method for automating the detection of phase transitions in
LLMs, and demonstrated that it successfully reveals a variety of transitions.
Leveraging access to the LLMs’ next-token probability distributions, the
proposed dissimilarity measures can efficiently quantify distribution shifts
without fine-tuning or adaption to the specific scenario at hand. Because the
method is solely based on analyzing a model’s output distribution and access
to the model weights is not required, it enables black-box interpretability
studies.

The proposed method is not only applicable to language models, but can be
straightforwardly adapted to any generative model with an explicit, tractable
density [116, 73]. If one can draw samples from the output distribution but
does not have explicit access to the underlying probabilities, then the
dissimilarity measures can still be approximated using NN-based classifiers
[117, 75] tailored toward the particular data type, such as natural language.

#### Limitations.

Future large-scale investigations are needed to fully understand how the
uncovered transitions depend on variables such as the specific prompt, the
number of generated output tokens, or the selected model. In particular, due
to computational resource constraints, the size of the studied language models
has been limited.

#### Broader Impact.

Our method has the potential to enhance the development of future AI systems
due to an improved understanding of their behavior. The dual-use nature of
such systems carries inherent risks, which requires one to proceed with
caution and implement mechanisms to ensure they are used safely and ethically.

## Acknowledgments and Disclosure of Funding

We thank Christoph Bruder for stimulating discussions and helpful suggestions
on the manuscript. J.A. and N.L. acknowledge financial support from the Swiss
National Science Foundation individual grant (grant no. 200020 200481). This
material is based upon work supported by the National Science Foundation under
grant no. OAC-1835443, grant no. OAC-2103804, and grant no. DMS-2325184. The
authors acknowledge the MIT SuperCloud and Lincoln Laboratory Supercomputing
Center for providing HPC resources that have contributed to the research
results reported within this paper.

## References

  * [1] Lorenza Saitta, Attilio Giordana, and Antoine Cornuejols.  Phase transitions in machine learning.  Cambridge University Press, 2011.
  * [2] James P Sethna.  Statistical mechanics: entropy, order parameters, and complexity, volume 14.  Oxford University Press, USA, 2021.
  * [3] Lars Onsager.  Crystal Statistics. i. A Two-Dimensional Model with an Order-Disorder Transition.  Phys. Rev., 65:117–149, Feb 1944.
  * [4] Louis Taillefer.  Scattering and pairing in cuprate superconductors.  Annu. Rev. Condens. Matter Phys., 1(1):51–70, 2010.
  * [5] Ali Lavasani, Yahya Alavirad, and Maissam Barkeshli.  Measurement-induced topological entanglement transitions in symmetric random quantum circuits.  Nature Physics, 17(3):342–347, 2021.
  * [6] Tamás Vicsek, András Czirók, Eshel Ben-Jacob, Inon Cohen, and Ofer Shochet.  Novel type of phase transition in a system of self-driven particles.  Phys. Rev. Lett., 75:1226–1229, Aug 1995.
  * [7] Ravid Shwartz-Ziv and Naftali Tishby.  Opening the black box of deep neural networks via information.  arXiv:1703.00810, 2017.
  * [8] Guy Gur-Ari, Daniel A Roberts, and Ethan Dyer.  Gradient descent happens in a tiny subspace.  arXiv:1812.04754, 2018.
  * [9] Thomas McGrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearce, Martin Wattenberg, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik.  Acquisition of chess knowledge in alphazero.  Proceedings of the National Academy of Sciences, 119(47):e2206625119, 2022.
  * [10] Alexander Pan, Kush Bhatia, and Jacob Steinhardt.  The effects of reward misspecification: Mapping and mitigating misaligned models.  arXiv:2201.03544, 2022.
  * [11] Liu Ziyin and Masahito Ueda.  Exact phase transitions in deep learning.  arXiv:2205.12510, 2022.
  * [12] James B Simon, Maksis Knutins, Liu Ziyin, Daniel Geisz, Abraham J Fetterman, and Joshua Albrecht.  On the stepwise nature of self-supervised learning.  In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 31852–31876. PMLR, 23–29 Jul 2023.
  * [13] Hugo Cui, Freya Behrens, Florent Krzakala, and Lenka Zdeborová.  A phase transition between positional and semantic learning in a solvable model of dot-product attention.  arXiv:2402.03902, 2024.
  * [14] Allan Raventós, Mansheej Paul, Feng Chen, and Surya Ganguli.  Pretraining task diversity and the emergence of non-bayesian in-context learning for regression.  Advances in Neural Information Processing Systems, 36, 2024.
  * [15] Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli.  Exponential expressivity in deep neural networks through transient chaos.  Advances in neural information processing systems, 29, 2016.
  * [16] Keiichi Tamai, Tsuyoshi Okubo, Truong Vinh Truong Duy, Naotake Natori, and Synge Todo.  Absorbing phase transitions in artificial deep neural networks.  arXiv:2307.02284, 2023.
  * [17] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al.  A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.  Science, 362(6419):1140–1144, 2018.
  * [18] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al.  In-context learning and induction heads.  arXiv:2209.11895, 2022.
  * [19] Deep Ganguli, Danny Hernandez, Liane Lovitt, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova Dassarma, Dawn Drain, Nelson Elhage, et al.  Predictability and surprise in large generative models.  In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1747–1764, 2022.
  * [20] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al.  Program synthesis with large language models.  arXiv:2108.07732, 2021.
  * [21] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [22] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.  Measuring massive multitask language understanding.  arXiv:2009.03300, 2020.
  * [23] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748–8763. PMLR, 18–24 Jul 2021.
  * [24] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.  arXiv:2206.04615, 2022.
  * [25] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al.  Emergent abilities of large language models.  arXiv:2206.07682, 2022.
  * [26] Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al.  Scaling language models: Methods, analysis & insights from training gopher.  arXiv:2112.11446, 2021.
  * [27] Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger.  Broken neural scaling laws.  arXiv:2210.14891, 2022.
  * [28] Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal.  Reconciling modern machine-learning practice and the classical bias–variance trade-off.  Proceedings of the National Academy of Sciences, 116(32):15849–15854, 2019.
  * [29] Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever.  Deep double descent: Where bigger models and more data hurt.  Journal of Statistical Mechanics: Theory and Experiment, 2021(12):124003, 2021.
  * [30] Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra.  Grokking: Generalization beyond overfitting on small algorithmic datasets.  arXiv:2201.02177, 2022.
  * [31] Ziming Liu, Ouail Kitouni, Niklas S Nolte, Eric Michaud, Max Tegmark, and Mike Williams.  Towards understanding grokking: An effective theory of representation learning.  In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 34651–34663. Curran Associates, Inc., 2022.
  * [32] Ziming Liu, Eric J Michaud, and Max Tegmark.  Omnigrok: Grokking beyond algorithmic data.  In The Eleventh International Conference on Learning Representations, 2022.
  * [33] Noam Levi, Alon Beck, and Yohai Bar-Sinai.  Grokking in linear estimators–a solvable model that groks without understanding.  arXiv:2310.16441, 2023.
  * [34] Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt.  Progress measures for grokking via mechanistic interpretability.  arXiv:2301.05217, 2023.
  * [35] Noa Rubin, Inbar Seroussi, and Zohar Ringel.  Droplets of good representations: Grokking as a first order phase transition in two layer networks.  arXiv:2310.03789, 2023.
  * [36] Angelica Chen, Ravid Schwartz-Ziv, Kyunghyun Cho, Matthew L Leavitt, and Naomi Saphra.  Sudden drops in the loss: Syntax acquisition, phase transitions, and simplicity bias in mlms.  arXiv:2309.07311, 2023.
  * [37] Vimal Thilak, Etai Littwin, Shuangfei Zhai, Omid Saremi, Roni Paiss, and Joshua Susskind.  The slingshot mechanism: An empirical study of adaptive optimizers and the grokking phenomenon.  arXiv:2206.04817, 2022.
  * [38] Alessandro Achille, Matteo Rovere, and Stefano Soatto.  Critical learning periods in deep neural networks.  arXiv:1711.08856, 2017.
  * [39] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.  Scaling laws for neural language models.  arXiv:2001.08361, 2020.
  * [40] Afra Alishahi, Grzegorz Chrupała, and Tal Linzen.  Analyzing and interpreting neural networks for nlp: A report on the first blackboxnlp workshop.  Natural Language Engineering, 25(4):543–557, 2019.
  * [41] Chris Olah.  Mechanistic interpretability, variables, and the importance of interpretable bases.  https://www.transformer-circuits.pub/2022/mech-interp-essay.  Online; accessed 15 April 2024.
  * [42] Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt.  Interpretability in the wild: a circuit for indirect object identification in gpt-2 small.  arXiv:2211.00593, 2022.
  * [43] Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas.  The clock and the pizza: Two stories in mechanistic explanation of neural networks.  In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 27223–27250. Curran Associates, Inc., 2023.
  * [44] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou.  Deep learning scaling is predictable, empirically.  arXiv:1712.00409, 2017.
  * [45] Jonathan S Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit.  A constructive prediction of the generalization error across scales.  arXiv:1909.12673, 2019.
  * [46] Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al.  Scaling laws for autoregressive generative modeling.  arXiv:2010.14701, 2020.
  * [47] Mitchell A Gordon, Kevin Duh, and Jared Kaplan.  Data and parameter scaling laws for neural machine translation.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5915–5922, 2021.
  * [48] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer.  Scaling vision transformers.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12104–12113, 2022.
  * [49] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al.  Training compute-optimal large language models.  arXiv:2203.15556, 2022.
  * [50] Tilman Räuker, Anson Ho, Stephen Casper, and Dylan Hadfield-Menell.  Toward transparent ai: A survey on interpreting the inner structures of deep neural networks.  In 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pages 464–483. IEEE, 2023.
  * [51] Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso.  Towards automated circuit discovery for mechanistic interpretability.  Advances in Neural Information Processing Systems, 36:16318–16352, 2023.
  * [52] Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo.  Are emergent abilities of large language models a mirage?  In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 55565–55581. Curran Associates, Inc., 2023.
  * [53] Lei Wang.  Discovering phase transitions with unsupervised learning.  Phys. Rev. B, 94:195105, Nov 2016.
  * [54] Juan Carrasquilla and Roger G Melko.  Machine learning phases of matter.  Nat. Phys., 13(5):431–434, 2017.
  * [55] Evert PL Van Nieuwenburg, Ye-Hua Liu, and Sebastian D Huber.  Learning phase transitions by confusion.  Nat. Phys., 13(5):435–439, 2017.
  * [56] Sebastian J. Wetzel.  Unsupervised learning of phase transitions: From principal component analysis to variational autoencoders.  Phys. Rev. E, 96:022140, Aug 2017.
  * [57] Sebastian J. Wetzel and Manuel Scherzer.  Machine learning of explicit order parameters: From the Ising model to SU(2) lattice gauge theory.  Phys. Rev. B, 96:184410, Nov 2017.
  * [58] Yi Zhang and Eun-Ah Kim.  Quantum Loop Topography for Machine Learning.  Phys. Rev. Lett., 118:216401, May 2017.
  * [59] Peter Broecker, Juan Carrasquilla, Roger G Melko, and Simon Trebst.  Machine learning quantum phases of matter beyond the fermion sign problem.  Sci. Rep., 7(1):1–10, 2017.
  * [60] Kelvin Ch’ng, Juan Carrasquilla, Roger G. Melko, and Ehsan Khatami.  Machine Learning Phases of Strongly Correlated Fermions.  Phys. Rev. X, 7:031038, Aug 2017.
  * [61] Wenjian Hu, Rajiv R. P. Singh, and Richard T. Scalettar.  Discovering phases, phase transitions, and crossovers through unsupervised machine learning: A critical examination.  Phys. Rev. E, 95:062122, Jun 2017.
  * [62] Jordan Venderley, Vedika Khemani, and Eun-Ah Kim.  Machine learning out-of-equilibrium phases of matter.  Phys. Rev. Lett., 120:257204, Jun 2018.
  * [63] Giuseppe Carleo, Ignacio Cirac, Kyle Cranmer, Laurent Daudet, Maria Schuld, Naftali Tishby, Leslie Vogt-Maranto, and Lenka Zdeborová.  Machine learning and the physical sciences.  Rev. Mod. Phys., 91:045002, Dec 2019.
  * [64] Frank Schäfer and Niels Lörch.  Vector field divergence of predictive model output as indication of phase transitions.  Phys. Rev. E, 99:062107, Jun 2019.
  * [65] Benno S Rem, Niklas Käming, Matthias Tarnowski, Luca Asteria, Nick Fläschner, Christoph Becker, Klaus Sengstock, and Christof Weitenberg.  Identifying quantum phase transitions using artificial neural networks on experimental data.  Nat. Phys., 15(9):917–920, 2019.
  * [66] Eliska Greplova, Agnes Valenti, Gregor Boschung, Frank Schäfer, Niels Lörch, and Sebastian D Huber.  Unsupervised identification of topological phase transitions using predictive models.  New J. Phys., 22(4):045003, apr 2020.
  * [67] Juan Carrasquilla.  Machine learning for quantum matter.  Adv. Phys.: X, 5(1):1797528, 2020.
  * [68] Korbinian Kottmann, Patrick Huembeli, Maciej Lewenstein, and Antonio Acín.  Unsupervised phase discovery with deep anomaly detection.  Phys. Rev. Lett., 125:170603, Oct 2020.
  * [69] Julian Arnold, Frank Schäfer, Martin Žonda, and Axel U. J. Lode.  Interpretable and unsupervised phase classification.  Phys. Rev. Res., 3:033052, Jul 2021.
  * [70] Niklas Käming, Anna Dawid, Korbinian Kottmann, Maciej Lewenstein, Klaus Sengstock, Alexandre Dauphin, and Christof Weitenberg.  Unsupervised machine learning of topological phase transitions from experimental data.  Mach. Learn.: Sci. Technol., 2021.
  * [71] Anna Dawid, Julian Arnold, Borja Requena, Alexander Gresch, Marcin Płodzień, Kaelan Donatella, Kim A Nicoli, Paolo Stornati, Rouven Koch, Miriam Büttner, et al.  Modern applications of machine learning in quantum sciences.  arXiv:2204.04198, 2022.
  * [72] Julian Arnold and Frank Schäfer.  Replacing neural networks by optimal analytical predictors for the detection of phase transitions.  Phys. Rev. X, 12:031044, Sep 2022.
  * [73] Julian Arnold, Frank Schäfer, Alan Edelman, and Christoph Bruder.  Mapping out phase diagrams with generative classifiers.  arXiv:2306.14894, 2023.
  * [74] Julian Arnold, Frank Schäfer, and Niels Lörch.  Fast detection of phase transitions with multi-task learning-by-confusion.  arXiv:2311.09128, 2023.
  * [75] Julian Arnold, Niels Lörch, Flemming Holtorf, and Frank Schäfer.  Machine learning phase transitions: Connections to the fisher information.  arXiv:2311.10710, 2023.
  * [76] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  Gpt-4 technical report.  arXiv:2303.08774, 2023.
  * [77] Anthropic.  Model Card and Evaluations for Claude Models.  https://www-cdn.anthropic.com/bd2a28d2535bfb0494cc8e2a3bf135d2e7523226/Model-Card-Claude-2.pdf.  Online; accessed 15 April 2024.
  * [78] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al.  Gemini: a family of highly capable multimodal models.  arXiv:2312.11805, 2023.
  * [79] Eric Michaud, Ziming Liu, Uzay Girit, and Max Tegmark.  The quantization model of neural scaling.  In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 28699–28722. Curran Associates, Inc., 2023.
  * [80] Sanjeev Arora and Anirudh Goyal.  A theory for emergence of complex skills in language models.  arXiv:2307.15936, 2023.
  * [81] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al.  Pythia: A suite for analyzing large language models across training and scaling.  In International Conference on Machine Learning, pages 2397–2430. PMLR, 2023.
  * [82] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al.  Mistral 7b.  arXiv:2310.06825, 2023.
  * [83] AI@Meta.  Llama 3 Model Card.  https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md.  Online; accessed 21 May 2024.
  * [84] Friedrich Liese and Igor Vajda.  On divergences and informations in statistics and information theory.  IEEE Trans. Inf. Theory, 52(10):4394–4412, 2006.
  * [85] Ye-Hua Liu and Evert P. L. van Nieuwenburg.  Discriminative Cooperative Networks for Detecting Phase Transitions.  Phys. Rev. Lett., 120:176401, Apr 2018.
  * [86] Matthew J. S. Beach, Anna Golubeva, and Roger G. Melko.  Machine learning vortices at the Kosterlitz-Thouless transition.  Phys. Rev. B, 97:045207, Jan 2018.
  * [87] Philippe Suchsland and Stefan Wessel.  Parameter diagnostics of phases and phase transition learning by neural networks.  Phys. Rev. B, 97:174435, May 2018.
  * [88] Song Sub Lee and Beom Jun Kim.  Confusion scheme in machine learning detects double phase transitions and quasi-long-range order.  Phys. Rev. E, 99:043308, Apr 2019.
  * [89] Qi Ni, Ming Tang, Ying Liu, and Ying-Cheng Lai.  Machine learning dynamical phase transitions in complex networks.  Phys. Rev. E, 100:052312, Nov 2019.
  * [90] Qi Ni, Jie Kang, Ming Tang, Ying Liu, and Yong Zou.  Learning epidemic threshold in complex networks by convolutional neural network.  Chaos: An Interdisciplinary Journal of Nonlinear Science, 29(11), 2019.
  * [91] W. Guo, B. Ai, and Liang He.  Reveal flocking of birds flying in fog by machine learning.  arXiv:2005.10505, 2020.
  * [92] Y. A. Kharkov, V. E. Sotskov, A. A. Karazeev, E. O. Kiktenko, and A. K. Fedorov.  Revealing quantum chaos with machine learning.  Phys. Rev. B, 101:064406, Feb 2020.
  * [93] A. Bohrdt, S. Kim, A. Lukin, M. Rispoli, R. Schittko, M. Knap, M. Greiner, and J. Léonard.  Analyzing Nonequilibrium Quantum States through Snapshots with Artificial Neural Networks.  Phys. Rev. Lett., 127:150504, Oct 2021.
  * [94] I Corte, S Acevedo, M Arlego, and Carlos Alberto Lamas.  Exploring neural network training strategies to determine phase transitions in frustrated magnetic models.  Computational Materials Science, 198:110702, 2021.
  * [95] Monika Richter-Laskowska, Marcin Kurpas, and Maciej M. Maśka.  Learning by confusion approach to identification of discontinuous phase transitions.  Phys. Rev. E, 108:024113, Aug 2023.
  * [96] M A Gavreev, A S Mastiukova, E O Kiktenko, and A K Fedorov.  Learning entanglement breakdown as a phase transition by confusion.  New J. Phys., 24(7):073045, aug 2022.
  * [97] Daria Zvyagintseva, Helgi Sigurdsson, Valerii K Kozin, Ivan Iorsh, Ivan A Shelykh, Vladimir Ulyantsev, and Oleksandr Kyriienko.  Machine learning of phase transitions in nonlinear polariton lattices.  Commun. Phys., 5(1):8, 2022.
  * [98] Wanzhou Zhang, Huijiong Yang, and Nan Wu.  Neural network topological snake models for locating general phase diagrams.  arXiv:2205.09699, 2022.
  * [99] Henning Schlömer and Annabelle Bohrdt.  Fluctuation based interpretable analysis scheme for quantum many-body snapshots.  arXiv:2304.06029, 2023.
  * [100] W. Guo and Liang He.  Learning phase transitions from regression uncertainty: a new regression-based machine learning approach for automated detection of phases of matter.  New J. Phys., 25(8):083037, aug 2023.
  * [101] Max Cohen, Max Casebolt, Yutan Zhang, Kaden R. A. Hazzard, and Richard Scalettar.  Classical analog of quantum models in synthetic dimensions.  Phys. Rev. A, 109:013303, Jan 2024.
  * [102] Steven T Flammia and Ryan O’Donnell.  Quantum chi-squared tomography and mutual information testing.  arXiv:2305.18519, 2023.
  * [103] Shun-ichi Amari and Andrzej Cichocki.  Information geometry of divergence functions.  Bulletin of the polish academy of sciences. Technical sciences, 58(1):183–195, 2010.
  * [104] Wen-Long You, Ying-Wai Li, and Shi-Jian Gu.  Fidelity, dynamic structure factor, and susceptibility in critical phenomena.  Phys. Rev. E, 76:022101, Aug 2007.
  * [105] Shi-Jian Gu.  Fidelity approach to quantum phase transitions.  Int. J. Mod. Phys. B, 24(23):4371–4458, 2010.
  * [106] Mikhail Prokopenko, Joseph T. Lizier, Oliver Obst, and X. Rosalind Wang.  Relating fisher information to order parameters.  Phys. Rev. E, 84:041116, Oct 2011.
  * [107] Zihan Liu, Wei Ping, Rajarshi Roy, Peng Xu, Chankyu Lee, Mohammad Shoeybi, and Bryan Catanzaro.  Chatqa: Surpassing gpt-4 on conversational qa and rag.  arXiv preprint arXiv:2401.10225, 2024.
  * [108] Freeman J. Dyson.  Existence of a phase transition in a one-dimensional Ising ferromagnet.  Commun. Math. Phys., 12:91–107, 1969.
  * [109] JG Martínez-Herrera, Omar Abel Rodríguez-López, and MA Solís.  Critical temperature of one-dimensional ising model with long-range interaction revisited.  Physica A: Statistical Mechanics and its Applications, 596:127136, 2022.
  * [110] Sebastián Bahamondes.  Study of the possibility of phase transitions in LLMs.  https://community.wolfram.com/groups/-/m/t/2958851?p_p_auth=PI4XRS4b.  Online; accessed 10 April 2024.
  * [111] Aaron Gokaslan and Vanya Cohen.  OpenWebText Corpus.  https://Skylion007.github.io/OpenWebTextCorpus, 2019.
  * [112] Stephen J. Blundell and Katherine M. Blundell.  Concepts in Thermal Physics.  Oxford University Press, 10 2009.
  * [113] Leo Z Liu, Yizhong Wang, Jungo Kasai, Hannaneh Hajishirzi, and Noah A Smith.  Probing across time: What does roberta know and when?  arXiv:2104.07885, 2021.
  * [114] Wes Gurnee and Max Tegmark.  Language models represent space and time.  arXiv:2310.02207, 2023.
  * [115] Beren Millidge.  Basic facts about language models during training.  https://www.alignmentforum.org/posts/2JJtxitp6nqu6ffak/basic-facts-about-language-models-during-training-1.  Online; accessed 12 April 2024.
  * [116] Ian Goodfellow.  NIPS 2016 Tutorial: Generative Adversarial Networks.  arXiv:1701.00160, 2016.  pp. 9 – 17.
  * [117] Aditya Menon and Cheng Soon Ong.  Linking losses for density ratio and class-probability estimation.  In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 304–313, New York, New York, USA, 20–22 Jun 2016. PMLR.

##  Appendix A Implementation details

#### Estimating dissimilarity measures.

Our method for detecting phase transitions is based on estimating $D_{g}$
across the entire parameter range. Starting with a fixed set of points on a
uniform grid $\mathcal{T}$, let us denote the set of in-between points at
least $L$ points away from the border of the range as $\bar{\mathcal{T}}$
(note that $|\bar{\mathcal{T}}|=|\mathcal{T}|-2L$). For each trial point
$T^{*}\in\bar{\mathcal{T}}$, we obtain an unbiased estimate
$\hat{D}_{g}=(\hat{J}_{\rm left}+\hat{J}_{\rm right})/2$, where

$\hat{J}_{i}=\frac{1}{|\sigma_{i}|}\sum_{T\in\sigma_{i}}\frac{1}{\mathcal{D}(T)}\sum_{\bm{x}\in\mathcal{D}(T)}g\left[P(\sigma_{i}|\bm{x})\right].$
(A1)

Recall that $\sigma_{\rm left}$ and $\sigma_{\rm right}$ denote the $L$
closest points to the left or right of the trial point. Here, $\mathcal{D}(T)$
denotes a set of output texts $\bm{x}$ generated via the LLM at point
$T\in\mathcal{T}$. In this work, we choose the number of generated text
samples to be the same for all $T\in\mathcal{T}$, i.e.,
$|\mathcal{D}(T)|=|\mathcal{D}|$.

We perform the computation of
$\hat{D}_{g}(T^{*})\;\forall\,T^{*}\in\bar{\mathcal{T}}$ in two stages. In a
first stage, we go through each grid point $T\in\mathcal{T}$ and generate text
outputs that are $N_{\rm tokens}$ in length via the LLM. The associated
computation time scales as $|\mathcal{T}|\cdot N_{\rm tokens}\cdot t_{\rm
LLM,eval}(|\mathcal{D}|)$, where $t_{\rm
LLM,eval}(|\mathcal{D}|)=\mathcal{O}(|\mathcal{D}|)$ corresponds to the time
it takes the LLM to generate $|\mathcal{D}|$ different outputs (single token
in length). In a second stage, for a given trial point
$T^{*}\in\bar{\mathcal{T}}$, we evaluate the probability of each text output
generated in its vicinity $\\{\bm{x}\in\mathcal{D}(T)|T\in\sigma_{\rm
left}(T^{*})\cup\sigma_{\rm right}(T^{*})\\}$ to come from the left or right
segment. That is, we compute $P(\bm{\sigma}_{\rm left}|\bm{x})$ and
$P(\bm{\sigma}_{\rm right}|\bm{x})$, i.e., a term in the sum of Eq. (A1). The
computation time associated with the second stage scales as
$|\bar{\mathcal{T}}|\cdot N_{\rm tokens}\cdot t_{\rm
LLM,eval}(|\mathcal{D}|)\cdot 2L$. Note that in practice, one can
embarrasingly parallelize over the different grid and trial points. Moreover,
one generates and evaluates text outputs batchwise.

#### Compute Resources.

For our computations, we used an NVIDIA RTX 3090 GPU as well as NVIDIA Tesla
V100 (32 GB) and NVIDIA A100 (40GB) GPUs.

#### Code availability.

A Python implementation of our method is available at
github.com/llmtransitions/llmtransitions.

#### Utilized assets.

Our code is implemented in Python and internally uses NumPy harris:2020,
PyTorch pytorch:2019, and transformers wolf:2020. For the presented examples,
we make use of the following additional packages: pandas mckinney:2010, SciPy
scipy:2020, Matplotlib hunter:2007, and seaborn waskom:2021.

The Pythia models, the Mistral models Mistral-7B-Instruct model and
Mistral-7B-Base model, are available under the Apache-2.0 license on Hugging
Face. The Meta-Llama-3-8B model and NVIDIA’s  Llama3-ChatQA-1.5-8B are
available on Hugging Face under the Meta Llama 3 community license.

##  Appendix B Theoretical background on $g$-dissimilarities

#### Correspondence Between $f$-divergences and $g$-dissimilarities.

Let us establish a correspondence between $f$-divergences [Eq. (1)] and
$g$-dissimilarities [Eq. (4)]. We can write any $g$-dissimilarity as

$\displaystyle D_{g}$ $\displaystyle=\frac{1}{2}\left(\mathbb{E}_{\bm{x}\sim
P_{\rm left}}\biggl{[}g\left[P(\sigma_{\rm
left}|\bm{x})\right]\biggr{]}+\mathbb{E}_{\bm{x}\sim P_{\rm
right}}\biggl{[}g\left[P(\sigma_{\rm right}|\bm{x})\right]\biggr{]}\right)$
$\displaystyle=\mathbb{E}_{\bm{x}\sim P_{\rm
right}}\biggl{[}\frac{1}{2}\frac{P_{\rm left}(\bm{x})}{P_{\rm
right}(\bm{x})}g\left[P(\sigma_{\rm
left}|\bm{x})\right]\biggr{]}+\mathbb{E}_{\bm{x}\sim P_{\rm
right}}\biggl{[}\frac{1}{2}g\left[P(\sigma_{\rm
right}|\bm{x})\right]\biggr{]}$ $\displaystyle=\mathbb{E}_{\bm{x}\sim P_{\rm
right}}\biggl{[}\frac{1}{2}\frac{P_{\rm left}(\bm{x})}{P_{\rm
right}(\bm{x})}g\left[P(\sigma_{\rm
left}|\bm{x})\right]+\frac{1}{2}g\left[P(\sigma_{\rm
right}|\bm{x})\right]\biggr{]}$ $\displaystyle=\mathbb{E}_{\bm{x}\sim P_{\rm
right}}\biggl{[}\frac{1}{2}\frac{P_{\rm left}(\bm{x})}{P_{\rm
right}(\bm{x})}g\left[\frac{P_{\rm
left}(\bm{x})}{P_{\mathrm{left}}(\bm{x})+P_{\mathrm{right}}(\bm{x})}\right]+\frac{1}{2}g\left[\frac{P_{\rm
right}(\bm{x})}{P_{\mathrm{left}}(\bm{x})+P_{\mathrm{right}}(\bm{x})}\right]\biggr{]}$
$\displaystyle=\mathbb{E}_{\bm{x}\sim P_{\rm
right}}\biggl{[}\frac{1}{2}\frac{P_{\rm left}(\bm{x})}{P_{\rm
right}(\bm{x})}g\left[\frac{\frac{P_{\rm
left}(\bm{x})}{P_{\mathrm{right}}(\bm{x})}}{\frac{P_{\mathrm{left}}(\bm{x})}{P_{\mathrm{right}}(\bm{x})}+1}\right]+\frac{1}{2}g\left[\frac{1}{\frac{P_{\mathrm{left}}(\bm{x})}{P_{\mathrm{right}}(\bm{x})}+1}\right]\biggr{]}.$

Thus, any $g$-dissimilarity $D_{g}$ can be rewritten in the form of an
$f$-divergence $D_{f}[P_{\rm left},P_{\rm right}]$ with

$f(x)=\frac{x}{2}\cdot g\left(\frac{x}{1+x}\right)+\frac{1}{2}\cdot
g\left(\frac{1}{1+x}\right).$ (B1)

Note, however, that not any choice of $g$-function will lead to a proper
$f$-divergence in the sense that the resulting $f$-function may not be convex
and $f(1)$ may not be zero (recall the definition of an $f$-divergence in Sec.
2.1).

#### JS divergence.

Using the correspondence above, we have that $D_{g(x)=\log(x)+\log(2)}$ is
equivalent to an $f$-divergence $D_{f}[P_{\rm left},P_{\rm right}]$ with

$f(x)=\frac{x}{2}\cdot\log\left(\frac{2x}{1+x}\right)+\frac{1}{2}\cdot\log\left(\frac{2}{1+x}\right)$
(B2)

which corresponds to the JS divergence, see Eq. (3).

#### TV distance.

Let us further prove that the $D_{g(x)=1-2\min\\{x,1-x\\}}$ corresponds to the
TV distance $D_{\rm TV}[P_{\rm left},P_{\rm right}]$. We have

$\displaystyle D_{g(x)=1-2\min\\{x,1-x\\}}=1$
$\displaystyle-\mathbb{E}_{\bm{x}\sim P_{\rm
left}}\biggl{[}\min\\{P(\sigma_{\rm left}|\bm{x}),P(\sigma_{\rm
right}|\bm{x})\\}\biggr{]}$ $\displaystyle-\mathbb{E}_{\bm{x}\sim P_{\rm
right}}\biggl{[}\min\\{P(\sigma_{\rm left}|\bm{x}),P(\sigma_{\rm
right}|\bm{x})\\}\biggr{]}$ (B3)

Using the identity

$\min\\{P(\sigma_{\rm left}|\bm{x}),P(\sigma_{\rm
right}|\bm{x})\\}=\frac{1}{2}\left(1-\left|P(\sigma_{\rm
left}|\bm{x})-P(\sigma_{\rm right}|\bm{x})\right|\right),$

Eq. (B) can be rewritten as

$\displaystyle D_{g(x)=1-2\min\\{x,1-x\\}}$
$\displaystyle=\frac{1}{2}\mathbb{E}_{\bm{x}\sim P_{\rm
left}}\biggl{[}\left|P(\sigma_{\rm left}|\bm{x})-P(\sigma_{\rm
right}|\bm{x})\right|\biggr{]}$
$\displaystyle\quad+\frac{1}{2}\mathbb{E}_{\bm{x}\sim P_{\rm
right}}\biggl{[}\left|P(\sigma_{\rm left}|\bm{x})-P(\sigma_{\rm
right}|\bm{x})\right|\biggr{]}$
$\displaystyle=\frac{1}{2}\mathbb{E}_{\bm{x}\sim P_{\rm
right}}\biggl{[}\left(1+\frac{P_{\rm left}(\bm{x})}{P_{\rm
right}(\bm{x})}\right)\left|P(\sigma_{\rm left}|\bm{x})-P(\sigma_{\rm
right}|\bm{x})\right|\biggr{]}$
$\displaystyle=\frac{1}{2}\mathbb{E}_{\bm{x}\sim P_{\rm
right}}\biggl{[}P(\sigma_{\rm right}|\bm{x})\left(1+\frac{P_{\rm
left}(\bm{x})}{P_{\rm right}(\bm{x})}\right)\left|1-\frac{P(\sigma_{\rm
left}|\bm{x})}{P(\sigma_{\rm right}|\bm{x})}\right|\biggr{]}.$

Noting that $\frac{P(\sigma_{\rm left}|\bm{x})}{P(\sigma_{\rm
right}|\bm{x})}=\frac{P_{\rm left}(\bm{x})}{P_{\rm right}(\bm{x})}$ and
$P(\sigma_{\rm left}|\bm{x})+P(\sigma_{\rm right}|\bm{x})=1$, we finally
obtain

$D_{g(x)=1-2\min\\{x,1-x\\}}=\mathbb{E}_{\bm{x}\sim P_{\rm
right}}\biggl{[}\frac{1}{2}\left|1-\frac{P_{\rm left}(\bm{x})}{P_{\rm
right}(\bm{x})}\right|\biggr{]}$

which corresponds to an $f$-divergence $D_{f}[P_{\rm left},P_{\rm right}]$
with $f(x)=\frac{1}{2}|1-x|$, i.e., the TV distance.

#### Freedom in Choice of $g$-function.

Note that the choice of $g$-function leading to a particular $g$-dissimilarity
is not unique. In particular, we have that $D_{\tilde{g}}=D_{g}$ for any
$\tilde{g}(x)=g(x)+c(\frac{1}{x}-2)$ where $c\in\mathbb{R}$ is some constant:

$\displaystyle D_{\tilde{g}}$
$\displaystyle=D_{g}+\frac{c}{2}\left(\mathbb{E}_{\bm{x}\sim P_{\rm
left}}\biggl{[}\frac{1-P(\sigma_{\rm left}|\bm{x})}{P(\sigma_{\rm
left}|\bm{x})}-1\biggr{]}+\mathbb{E}_{\bm{x}\sim P_{\rm
right}}\biggl{[}\frac{1-P(\sigma_{\rm right}|\bm{x})}{P(\sigma_{\rm
right}|\bm{x})}-1\biggr{]}\right)$
$\displaystyle=D_{g}+\frac{c}{2}\left(\mathbb{E}_{\bm{x}\sim P_{\rm
left}}\biggl{[}\frac{P(\sigma_{\rm right}|\bm{x})}{P(\sigma_{\rm
left}|\bm{x})}-1\biggr{]}+\mathbb{E}_{\bm{x}\sim P_{\rm
right}}\biggl{[}\frac{P(\sigma_{\rm left}|\bm{x})}{P(\sigma_{\rm
right}|\bm{x})}-1\biggr{]}\right)$
$\displaystyle=D_{g}+\frac{c}{2}\left(\mathbb{E}_{\bm{x}\sim P_{\rm
left}}[1]+\mathbb{E}_{\bm{x}\sim P_{\rm right}}[1]-2\right)=D_{g}.$

#### Relation to the Fisher information.

In the following, we prove that any $g$-dissimilarity with $g(\frac{1}{2})=0$
and a twice-differentiable $g$-function reduces to the Fisher information in
lowest order.

For this, consider the case with $L=1$ where we compare the distributions at
two points in parameter space that are separated by $\delta T$. The
corresponding $g$-dissimilarity is equivalent to an $f$-divergence
$D_{f}[P(\bm{x}|T),P(\bm{x}|T+\delta T)]$. Note that
$D_{f}[P(\bm{x}|T),P(\bm{x}|T)]=0$ if $f(1)=0$ and $\left.\partial
D_{f}[P(\bm{x}|T),P(\bm{x}|B)]/\partial
B\right|_{B=T}=f^{\prime}(1)\partial\mathbb{E}_{\bm{x}\sim
P(\cdot|T)}[1]/\partial T=f^{\prime}(1)\partial 1/\partial T=0$. The second-
order derivative corresponds to

$\displaystyle\left.\frac{\partial^{2}D_{f}[P(\bm{x}|T),P(\bm{x}|B)]}{\partial
B^{2}}\right|_{B=T}$ $\displaystyle=f^{\prime}(1)\mathbb{E}_{\bm{x}\sim
P(\cdot|T)}\biggl{[}\frac{1}{P(\bm{x}|T)}\frac{\partial^{2}P(\bm{x}|T)}{\partial
T^{2}}\biggr{]}$ $\displaystyle\quad+f^{\prime\prime}(1)\mathbb{E}_{\bm{x}\sim
P(\cdot|T)}\biggl{[}\left(\frac{\partial\log P(\bm{x}|T)}{\partial
T}\right)^{2}\biggr{]}$ $\displaystyle=f^{\prime\prime}(1)\mathcal{F}(T)$

assuming $f^{\prime}(1)=0$, where $\mathcal{F}$ is the Fisher information.
Thus, for $L=1$ and parameter values separated by $\delta T$, we can express
any $g$-dissimilarity with $g(\frac{1}{2})=0$ as
$D_{g}=\frac{g^{\prime\prime}(\frac{1}{2})}{32}\mathcal{F}(T)\delta
T^{2}+\mathcal{O}(\delta T^{3})$. The fact that $f^{\prime}(1)$ must be zero
translates into the condition that $g^{\prime}(\frac{1}{2})=0$. This is not a
fundamental restriction since we have some freedom in the choice of $g$
function. That is, we can replace $g\mapsto\tilde{g}$, where
$\tilde{g}(x)=g(x)+c(\frac{1}{x}-2)$ with
$c=\frac{1}{6}g^{\prime}(\frac{1}{2})$, retaining $D_{g}=D_{\tilde{g}}$ and
ensuring that $\tilde{g}^{\prime}(\frac{1}{2})=0$.

##  Appendix C Details on Energy-Based Analysis of Temperature Transition

Let $\bm{x}=(x_{1},\dots,x_{N})$ be a sequence of $N$ tokens generated for a
fixed prompt from an autoregressive LLM such as the ones considered in this
article. The distribution of $\bm{x}$ is given by

$P(\bm{x}|T)=Q_{T}(x_{N}|x_{1},\dots,x_{N-1})Q_{T}(x_{N-1}|x_{1},\dots,x_{N-2})\cdots
Q_{T}(x_{2}|x_{1})Q_{T}(x_{1}),$

denoting the fact that the tokens are sampled sequentially. In each step, a
token is sampled from a Boltzmann distribution $Q_{T}$,

$Q_{T}(x_{i}|x_{1},\dots,x_{i-1})=e^{-E(x_{i}|x_{1},\dots,x_{i-1})/T}/Z_{i}(T).$

Here, $Z_{i}(T)=\sum_{x_{i}}e^{-E(x_{i}|x_{1},\dots,x_{i-1})/T}$ is a
normalization factor with the sum running over all possible $i$th tokens. The
conditional energies, $E(x_{i}|x_{1},\dots,x_{i-1})$ are typically referred to
as logits and learned from data. Note that while the distribution over
individual tokens can be expressed as a Boltzmann distribution at varying
temperature, the overall distribution $P(\bm{x}|T)$ cannot.555In order for a
quantity to be a valid energy of a system, it cannot itself depend on
temperature, i.e., change with temperature.

Nevertheless, we can define an energy scale for the entire system by viewing
the overall probability distribution at $T=1$ as a Boltzmann

$P(\bm{x}|T=1)=e^{-E(\bm{x})}/Z,$ (C1)

where $Z$ is a normalization constant independent of $\bm{x}$ (also referred
to as partition function). Recall that any valid probability distribution can
be written in the form of Eq. (C1) with a suitably chosen energy function.
Taking the logarithm of Eq. (C2) and reordering, we have

$E(\bm{x})=-\log P(\bm{x}|T=1)-\log Z.$ (C2)

Using Eq. (C2), we can compute the total energy up to the constant $-\log Z$
which serves as our reference point for the energy scale. In the main text, we
use $-\log P(\bm{x}|T=1)$ as the total energy to compute and compare energy
statistics at various temperatures. In particular, the heat capacity remains
the same under the mapping $-\log P(\bm{x}|T=1)-\log Z\mapsto-\log
P(\bm{x}|T=1)$.

unsrt refs_appendix