††thanks: corresponding author<EMAIL_ADDRESS>co-
corresponding author<EMAIL_ADDRESS>corresponding
author<EMAIL_ADDRESS>
# A model for spontaneous SMCs unidirectional motion

Andrea Bonato  Department of Physics, University of Strathclyde, Glasgow, G4
0NG, UK  Jae-Won Jang  Department of Physics and Astronomy, and Institute of
Applied Physics, Seoul National University, Seoul, 08826, South Korea  Kyoung-
Wook Moon  Department of Physics and Astronomy, and Institute of Applied
Physics, Seoul National University, Seoul, 08826, South Korea  Davide
Michieletto  School of Physics and Astronomy, University of Edinburgh, Peter
Guthrie Tait Road, Edinburgh, EH9 3FD, UK  MRC Human Genetics Unit, Institute
of Genetics and Cancer, University of Edinburgh, Edinburgh EH4 2XU, UK  Je-
Kyung Ryu  Department of Physics and Astronomy, and Institute of Applied
Physics, Seoul National University, Seoul, 08826, South Korea

# Spontaneous Unidirectional Loop Extrusion Emerges from Structural Symmetry
Breaking of SMC-Protein Motion

Andrea Bonato  Department of Physics, University of Strathclyde, Glasgow, G4
0NG, UK  Jae-Won Jang  Department of Physics and Astronomy, and Institute of
Applied Physics, Seoul National University, Seoul, 08826, South Korea  Kyoung-
Wook Moon  Department of Physics and Astronomy, and Institute of Applied
Physics, Seoul National University, Seoul, 08826, South Korea  Davide
Michieletto  School of Physics and Astronomy, University of Edinburgh, Peter
Guthrie Tait Road, Edinburgh, EH9 3FD, UK  MRC Human Genetics Unit, Institute
of Genetics and Cancer, University of Edinburgh, Edinburgh EH4 2XU, UK  Je-
Kyung Ryu  Department of Physics and Astronomy, and Institute of Applied
Physics, Seoul National University, Seoul, 08826, South Korea

# Spontaneous Unidirectional Loop Extrusion Emerges from Symmetry Breaking of
SMC Extension

Andrea Bonato  Department of Physics, University of Strathclyde, Glasgow, G4
0NG, UK  Jae-Won Jang  Department of Physics and Astronomy, and Institute of
Applied Physics, Seoul National University, Seoul, 08826, South Korea  Kyoung-
Wook Moon  Department of Physics and Astronomy, and Institute of Applied
Physics, Seoul National University, Seoul, 08826, South Korea  Davide
Michieletto  School of Physics and Astronomy, University of Edinburgh, Peter
Guthrie Tait Road, Edinburgh, EH9 3FD, UK  MRC Human Genetics Unit, Institute
of Genetics and Cancer, University of Edinburgh, Edinburgh EH4 2XU, UK  Je-
Kyung Ryu  Department of Physics and Astronomy, and Institute of Applied
Physics, Seoul National University, Seoul, 08826, South Korea

###### Abstract

DNA loop extrusion is arguably one of the most important players in genome
organization. The precise mechanism by which loop extruding factors (LEFs)
work is still unresolved and much debated. One of the major open questions in
this field is how do LEFs establish and maintain unidirectional motion along
DNA. In this paper, we use High-Speed AFM data to show that condensin hinge
domain displays a structural, geometric constraint on the angle within which
it can extend with respect to the DNA-bound domains. Using computer
simulations, we then show that such a geometrical constraint results in a
local symmetry breaking and is enough to rectify the extrusion process,
yielding unidirectional loop extrusion along DNA. Our work highlights an
overlooked geometric aspect of the loop extrusion process that may have a
universal impact on SMC function across organisms.

DNA loop extrusion by structural maintenance of chromosome (SMC) complexes has
emerged as a universal organizing principle for chromosomes Nasmyth (2011);
Alipour and Marko (2012); Sanborn  _et al._ (2015); Fudenberg  _et al._
(2016); Hirano (2016); Davidson and Peters (2021); Xiang and Koshland (2021).
For instance, it is now well established that in eukaryotes, cohesin complexes
are involved in shaping “topologically associating domains” (TADs) during
interphase Vian  _et al._ (2018); Rao  _et al._ (2017), while condensin
complexes direct the establishment of the cylindrical structure of mitotic
chromosomes Gibcus  _et al._ (2018).

SMC complexes have a ring-like structure, composed by a SMC dimer and an
intrinsically disordered kleisin subunit. The SMC dimer is formed through the
so-called “hinge” while the kleisin subunit is bound to the ATPase domain of
each SMC (Fig. 1a). In the case of yeast condensin, there are putative
additional DNA binding sites in the hinge, dimerized heads, Ycg1/Brn1 and
Ycs4/Brn1 for DNA anchoring Shaltiel  _et al._ (2022). Despite the wealth of
structural data, there is still contrasting evidence regarding the precise
topology and mechanics of loop extrusion process.

Among the most debated features of loop extrusion are the motoring action of
the hinge and the origin of the unidirectional motion Higashi  _et al._
(2021); Nomidis  _et al._ (2022); Takaki  _et al._ (2021); Barth  _et al._
(2023). Recent structural studies have suggested that SMC uses conformational
changes between a hinge-released state – where the hinge is extended away from
the heads – and a hinge-engaged state – where the hinge is in proximity of the
heads, in order to drive the motion in a “scrunching”, and ATP-dependent,
fashion Ryu  _et al._ (2020); Bauer  _et al._ (2021); Xiang and Koshland
(2021) (Fig. 1a, b). The scrunching model predicts that following dimerisation
of the SMC heads (due to ATP-binding), the coiled-coil arms fold to bring the
hinge closer to the heads. Following ATP-hydrolysis, the heads are released
and the hinge extends again Ryu  _et al._ (2020); Eeftens  _et al._ (2016);
Takaki  _et al._ (2021). During this step, the positively charged extended
hinge may search for a 3D proximal (but not necessarily 1D contiguous) DNA
segment to grab and to subsequently bring close to the heads in the following
ATP-binding step Bonato and Michieletto (2021); Takaki  _et al._ (2021).
Whilst this model can elegantly explain the bypassing of other SMCs Kim  _et
al._ (2020); Brandão  _et al._ (2021) and large roadblocks Brandão  _et al._
(2019); Pradhan  _et al._ (2022), it cannot explain unidirectionality. Indeed,
during the search step, there is no guarantee that the hinge will grab onto a
DNA segment ahead of the one that was reeled in the extruded loop in the
previous ATP cycle. Thus, even within the scrunching model explaining the
unidirectional motion of LEFs remains an outstanding problem in the field of
SMC-driven DNA organisation.

Figure 1: Limited solid angle of hinge movement from the position of the SMC
heads. a-b. Angle distributions of hinge-releasing a. and hinge-engaging
movements b. respect to the line connecting two heads at the middle of two
heads observed by HS AFM (N = 727 and 698, respectively). c. Distributions of
the angle between hinge and the central position of the globular domain with
respect to tangential DNA line at the central positions of the globular domain
analysed by dry-AFM images (N = 79). d. Sketch of the truncated cone hinge-
reachable region determined by the AFM data.

To gain a better understanding of the origin of the unidirectional motion, we
use High-Speed AFM (HS AFM) on yeast condensin Ryu  _et al._ (2020). We
observed that the hinge domain typically extends in an orthogonal direction
from the DNA-bound globular head domains, and we then hypothesised that this
local (angular) symmetry breaking due to the structure of condensin could
generate a geometric bias in the grabbing of new DNA segments. Thus, we
computationally implemented a loop extrusion process with such geometric
constrain, by modelling a 3D search constrained to lie within a solid angle,
and we discovered that it spontaneously displayed rectified, unidirectional
extrusion. Our findings suggest that a geometric symmetry-breaking mechanism,
without the need to impose explicit topological constraints between the SMC
complex and extruded DNA segment, could underlie the emergence of rectified,
unidirectional loop extrusion by SMC proteins.

## AFM reveals a preferred angle for hinge extension. –

We started from the hypothesis that the structure of SMC itself may present
some geometric restrictions on its allowed conformations. Specifically, we
argued that the hinge, connected to dynamic coiled-coil arms Eeftens  _et al._
(2016), largely determines the search of the DNA segment to be captured and
reeled in the extruded loop. For this reason, we wanted to measure the
possible geometric conformations assumed by a LEF, in order to quantify any
restriction on the motion of the hinge. To do this, we performed and analysed
High-Speed AFM (HS AFM) images of yeast condensin complex to obtain the
typical position of the hinge with respect to the globular head domains Ryu
_et al._ (2020) (Fig. 1). From the HS AFM movies we could identify two
distinct globular domains: the hinge and the heads, linked together by semi-
flexible arms (Fig. 1a). Furthermore, we were able to distinguish hinge-
released and hinge-engaged states by measuring the distance of the hinge from
the heads (see Fig. 1a-b and SI Fig. S4). Based on these, we measured the
angles of hinge-releasing and hinge-engaging states, with respect to the line
connecting two heads and from the middle point of two heads (Fig. 1a-b). In
addition, to compare these angles with the anchoring angle of condensin with
respect to the DNA, we analyzed dry-AFM images and measured the angle of hinge
extension with respect to the DNA-bound head domains (Fig. 1c).

We discovered that the hinge is most often extended orthogonally to the line
joining the SMC heads (Fig. 1a-b). Even in absence of DNA, we measured that
the angle distribution at which the hinge is extended and retracted is
normally distributed for both hinge releasing ($83\pm 26^{\circ}$) and
engaging ($86\pm 30^{\circ}$) steps. In addition, dry-AFM images of condensins
that bound to DNA through head domains also showed vertical angle of the
hinge-head line respect to DNA tangential direction ($88\pm 23^{\circ}$) (Fig.
1c).

It is important to notice that our measurements are done on condensin complex
absorbed on mica, i.e., in 2D. Thus, we argue that the angle distribution of
the hinge extension would define a solid angle when the complex is allowed to
move in 3D. Indeed, our results suggest that the hinge releases and engages
orthogonally to both heads and DNA, forming a solid angle $\Omega$ defined by
the width of the Gaussian distribution, $\gamma\simeq 60^{\circ}$ (see Fig.
1d).

Based on the AFM results, we defined a hinge-reachable region for the
scrunching model as a truncated cone with the estimated solid angle $\Omega$
(Fig. 1d). The maximum hinge extension is determined from the distribution of
hinge-head distances in the hinge-released state ($\simeq 40$ nm) while the
minimum hinge extension is obtained as the hinge-head distance in the hinge-
engaged state Ryu  _et al._ (2020) ($\simeq 10$ nm, see SI Fig. S4).

Figure 2:  A model for loop extrusion with geometric constraints on hinge
extension. a. Schematics of a loop extrusion model where the hinge (red) is
restricted to search for a DNA segment within a truncated cone. The segment is
then bound to the SMC heads and reeled in the extruded loop, while the hinge
returns to the search position. Throughout this process, one DNA segment is
trapped at the anchor bound to the kleisin (purple). b Implementation of the
model on a coarse-grained bead-spring polymer, where the heads and anchor are
denoted with red and purple beads, respectively. The hinge is not explicitly
modelled with a bead, but is accounted for by the geometrically restricted
search region (yellow shaded area). The simulated loop extrusion displays both
(c) forward and (d) backward steps.

## A LEF model with geometrically biased 3D search. –

In light of our HS AFM measurements, we propose a new geometrically-
constrained scrunching model as follows: first, Ycg1/Brn1 anchors DNA with a
“safety belt” mechanism Kschonsak  _et al._ (2017) then the heads/Ycs4 connect
the anchors to other part of DNA to extrude a loop (Fig. 2a). The motor action
of the hinge is limited by the angle distribution we found in Fig. 1, and can
grab DNA segments through the hinge, by extending the SMC arms at a fixed
angle $\alpha\simeq 90^{\circ}$ and with a certain width $\gamma\simeq
60^{\circ}$ from the orientation of the bound DNA. After that the hinge grabs
onto a new DNA segment, the ATP-binding-induced conformational change brings
the grabbed DNA close to the heads/Ycs4 subunits. Finally, after ATP-
hydrolysis, the heads/Ycs4 bind to the new DNA segment – thereby extending the
extruded loop – and the hinge is then free to target a new DNA segment for the
next round of DNA-loop extrusion (Fig. 2a).

To simulate this model, we implemented a coarse-grained loop extrusion process
with a geometric constraint on the region that can be reached by the hinge.
Specifically, we account for the connectivity of the anchor (YcgI) to the
heads (Smc2 and Smc4) via the kleisin subunit as beads connected by a harmonic
bond; additionally, we impose that the search of the segment to reel in the
extruded loop is to be performed within a solid angle around an axis
orthogonal to the tangent of the SMC-bound DNA (Fig. 2b). When a segment of
the coarse-grained polymer falls within a truncated cone formed by the solid
angle $\Omega$ and within a certain Euclidean distance (10 and 40 nm) the
position of the harmonic bond between anchor and heads is updated to grab onto
the anchor and such new segment (see Fig. 2b). In turn, the harmonic spring
connecting anchor and heads is temporarily extended and brings the new segment
close to the anchor. Finally, the new segment is identified with the new
position of the heads, the anchor remains at its original position, and the
hinge is then returned free to search for a new segment to grab onto (again
within the solid angle $\Omega$). This process defines a full ATP-cycle and
involves a 3D search of proximal DNA segments with a geometric constraint but
no strict topological requirements.

Figure 3: Unidirectional motion emerges from symmetry breaking. a. Individual
traces of simulated extruded loops as a function of time and with different
hinge-search angles. b. Root mean squared extruded length as a function of
time and for different hinge-search angles. The case where the search is
allowed to occur on the full sphere $\gamma=360^{\circ}$ yields a random walk
scaling as $t^{1/2}$. c. Probability of step size as a function of the search
angle. The narrower the angle, the more rectified the extrusion, i.e. the
larger the forward/reverse ratio. d. Plot of the success probability and
average step sizes as a function of the hinge search angle. These two
quantities display a trade-off which yield an optimum of the extrusion
velocity at $\gamma=180^{\circ}$ search angles, as shown in e.

The important difference of our work from previous models of loop extrusion
Sanborn  _et al._ (2015); Fudenberg  _et al._ (2016); Alipour and Marko
(2012); Banigan and Mirny (2019); Orlandini  _et al._ (2019); Bonato and
Michieletto (2021); Takaki  _et al._ (2021) is that we do not impose the
directionality of the extrusion a priori. The hinge can grab any segment
ahead, or behind, the current 1D position of the heads. In fact, in our
simulations we can observe backward extrusion steps, where the newly grabbed
segment is inside the extruded loop, as shown in Fig. 2c, d. This move yields
a reduction in the total length of the extruded loop and is also observed in
experiments Ryu  _et al._ (2022). In addition, our model accounts for an
additional spring to mimic the presence of the disordered kleisins attaching
the anchor YcgI to the SMC heads (SI). This additional spring constrains the
relative rotation of the DNA segments bound to the heads and the anchor. This
rotational constraints cannot otherwise be imposed if two beads are connected
by a single spring 111An alternative way to implement this constraints could
be to impose a dihedral or angular potential between the DNA segments bound to
heads and anchor. We expect to obtain similar results with this alternative
implementation.. The rotational constraint within the DNA-bound SMC complex is
evident from structural cryo-EM data Shaltiel  _et al._ (2022), where the
bound DNA segments sit tightly within DNA-binding pockets near the heads, and
that their juxtaposition within the SMC structure assume a well-defined angle
Shaltiel  _et al._ (2022); this implies that SMC-bound DNA segments are likely
to be restricted in their relative rotation.

Using this model, we performed a simulated loop extrusion on a bead-spring
polymer with $N=400$ beads of size $\sigma=10$ nm. We then tracked the
position of the anchor and heads, and defined an oriented extruded loop length
as $l=n_{a}-n_{h}$, where $n_{a}$ and $n_{h}$ are the positions of the anchor
and the heads; we discovered that, strikingly, the LEFs display growing loops
with a clear sign of unidirectional motion. Since we do not hard-code
directionality within the model, the LEFs in different replicas will start
extruding in different directions. The interesting finding is that they
display a tendency to maintain a rectified, unidirectional motion, once that
the “left-right” symmetry has been broken (Fig. 3a).

To understand how this spontaneous rectification is due to the broken symmetry
in the search step, we performed simulations with wider search angles up to
restoring the full spherically symmetric search $\gamma=360^{\circ}$. In the
symmetric case, we do not observe unidirectional motion, instead we find that
extruded loops shrink back, with a behaviour similar to a random walk (see
Fig. 3a). To further characterise this process we took the root mean squared
extruded length $\langle
l\rangle=\langle\left[n_{a}-n_{h}\right]^{2}\rangle^{1/2}$ and indeed found
that the spherically symmetric case displayed a scaling $\langle l\rangle\sim
t^{1/2}$, in line with a simple random walk (see Fig. 3b). Interestingly, we
also noted that the cases with $60^{\circ}<\gamma<360^{\circ}$ displayed
faster linear growth than the case with $\gamma=60^{\circ}$. Despite this, the
distribution of step sizes clearly indicate that the case $\gamma=60^{\circ}$
is the one that benefits from the greater rectification, i.e. the ratio
forward/reverse steps is the largest. In other words, wider angles increase
the probability of shorter and backward steps (Fig. 3c). In turn, this implies
that the average step size – defined as $s=\Sigma_{i}[sign(i)S_{i}]/N$, where
$S_{i}$ is the $i$-th step size – is typically smaller for wider angles. The
largest probability of large steps $\sim 50$ nm, which is in line with
experiments Ryu  _et al._ (2022), was seen for $\gamma=60^{\circ}$. Thus, to
understand why wider search angles yield faster extrusion in our simulations,
we compute the success probability of making a step. Indeed, in our algorithm
we impose that the LEF does not make a step if, in a given simulation time,
there are no DNA beads that satisfy the search criterion. This readily implies
that narrower search angles yield lower success rates (Fig. 3d). The opposite
trends of success rate (increasing with $\gamma$) and average step size
(decreasing with $\gamma$) yield a trade-off (Fig. 3d) that naturally leads to
an optimum in velocity around $\gamma\simeq 180^{\circ}$. However, we argue
that condensin may employ a narrow angle to optimize the search process within
a crowded DNA environment, within which the success probability would be
generally larger.

## Conclusions. –

Motivated by AFM data, in this Letter we have provided experimental evidence
that the structure of yeast condensin favours certain geometric conformations
where the hinge is extended perpendicularly to the local direction of the
heads-bound DNA segment. We have also quantified the width of the search
angle, $\gamma=60^{\circ}$ and computationally demonstrated that by imposing
this geometric constraint, loop extrusion can be spontaneously rectified.
Interestingly, we find that the narrower the search angle, the larger the
typical the step size and the more unidirectional the extrusion, but also the
more likely to fail to find a DNA segment to grab in a given time. This trade-
off yields an optimum extrusion speed at a predicted angle of
$\gamma=180^{\circ}$.

We argue that the emergence of spontaneously rectified, unidirectional
extrusion is due to a combination of local deformations of the underlying DNA
and the geometric constraint on the hinge-mediated 3D search. Our findings
ought to be relevant to other SMC protein complexes, as long as their
structure impose a geometric constraint on the conformational space of the
DNA:protein complex. Additionally, they can reconcile a range of recent
findings, e.g. bypassing of roadblocks, Z-loops, and also the pinching of a
negatively supercoiled loop during the hinge-grabbing step Martínez‐García
_et al._ (2023), whilst also explaining the unidirectional extrusion. Our
hypothesis could be tested by mutating the SMC coiled-coil arms to be more
flexible or rigid, thereby directly increasing or reducing the search angle
$\gamma$. Overall, we argue that our findings contribute to highlight a
largely overlooked aspect of loop extrusion that ought to be relevant for the
function of generic SMC complexes.

## Acknowledgements. –

DM acknowledges the Royal Society and the European Research Council (grant
agreement No 947918, TAP) for funding. The authors also acknowledge the
contribution of the COST Action Eutopia, CA17139. J.-K.R. acknowledges the
Institute of Applied Physics of Seoul National University, Creative-Pioneering
Researchers Program through Seoul National University (Project Number
3348-20230013), the Brain Korea 21 Four Project grant funded by the Korean
Ministry of Education, Samsung Electronics Co., Ltd. (Project Number
A0426-20220109), and the National Research Foundation of Korea (Project Number
0409-20230237, 0409-20230171, 0409-20230219).

## References

  * Nasmyth (2011) K. Nasmyth, Nature Cell Biology 13, 1170 (2011).
  * Alipour and Marko (2012) E. Alipour and J. F. Marko, Nucleic Acids Res. 40, 11202 (2012).
  * Sanborn  _et al._ (2015) A. L. Sanborn, S. S. P. Rao, S.-C. Huang, N. C. Durand, M. H. Huntley, A. I. Jewett, I. D. Bochkov, D. Chinnappan, A. Cutkosky, J. Li, K. P. Geeting, A. Gnirke, A. Melnikov, D. McKenna, E. K. Stamenova, E. S. Lander, and E. L. Aiden, Proc. Natl. Acad. Sci. USA 112, 201518552 (2015).
  * Fudenberg  _et al._ (2016) G. Fudenberg, M. Imakaev, C. Lu, A. Goloborodko, N. Abdennur, and L. A. Mirny, Cell Reports 15, 2038 (2016).
  * Hirano (2016) T. Hirano, Cell 164, 847 (2016).
  * Davidson and Peters (2021) I. F. Davidson and J. M. Peters, Nature Reviews Molecular Cell Biology 22, 445 (2021).
  * Xiang and Koshland (2021) S. Xiang and D. Koshland, eLife 10, 1 (2021).
  * Vian  _et al._ (2018) L. Vian, A. Pȩkowska, S. S. Rao, K. R. Kieffer-Kwon, S. Jung, L. Baranello, S. C. Huang, L. El Khattabi, M. Dose, N. Pruett, A. L. Sanborn, A. Canela, Y. Maman, A. Oksanen, W. Resch, X. Li, B. Lee, A. L. Kovalchuk, Z. Tang, S. Nelson, M. Di Pierro, R. R. Cheng, I. Machol, B. G. St Hilaire, N. C. Durand, M. S. Shamim, E. K. Stamenova, J. N. Onuchic, Y. Ruan, A. Nussenzweig, D. Levens, E. L. Aiden, and R. Casellas, Cell 173, 1165 (2018).
  * Rao  _et al._ (2017) S. S. Rao, S.-C. Huang, B. G. S. Hilaire, J. M. Engreitz, E. M. Perez, K.-R. Kieffer-Kwon, A. L. Sanborn, S. E. Johnstone, G. D. Bascom, I. D. Bochkov, X. Huang, M. S. Shamim, J. Shin, D. Turner, Z. Ye, A. D. Omer, J. T. Robinson, T. Schlick, B. E. Bernstein, R. Casellas, E. S. Lander, and E. L. Aiden, Cell 171, 305 (2017).
  * Gibcus  _et al._ (2018) J. H. Gibcus, K. Samejima, A. Goloborodko, I. Samejima, N. Naumova, J. Nuebler, M. T. Kanemaki, L. Xie, J. R. Paulson, W. C. Earnshaw, L. A. Mirny, and J. Dekker, Science 359, eaao6135 (2018).
  * Shaltiel  _et al._ (2022) I. A. Shaltiel, S. Datta, L. Lecomte, M. Hassler, M. Kschonsak, S. Bravo, C. Stober, J. Ormanns, S. Eustermann, and C. H. Haering, Science 376, 1087 (2022).
  * Higashi  _et al._ (2021) T. L. Higashi, G. Pobegalov, M. Tang, M. I. Molodtsov, and F. Uhlmann, eLife 10, 1 (2021).
  * Nomidis  _et al._ (2022) S. K. Nomidis, E. Carlon, S. Gruber, and J. F. Marko, Nucleic Acids Research 50, 4974 (2022).
  * Takaki  _et al._ (2021) R. Takaki, A. Dey, G. Shi, and D. Thirumalai, Nature Communications 12 (2021).
  * Barth  _et al._ (2023) R. Barth, B. Pradhan, E. Kim, I. F. Davidson, J. van der Torre, J. M. Peters, and C. Dekker, Scientific Reports 13, 1 (2023).
  * Ryu  _et al._ (2020) J. K. Ryu, A. J. Katan, E. O. van der Sluis, T. Wisse, R. de Groot, C. H. Haering, and C. Dekker, Nature Structural and Molecular Biology 27, 1134 (2020).
  * Bauer  _et al._ (2021) B. W. Bauer, I. F. Davidson, D. Canena, G. Wutz, W. Tang, G. Litos, S. Horn, P. Hinterdorfer, and J. M. Peters, Cell 184, 5448 (2021).
  * Eeftens  _et al._ (2016) J. M. Eeftens, A. J. Katan, M. Kschonsak, M. Hassler, L. de Wilde, E. M. Dief, C. H. Haering, and C. Dekker, Cell Reports 14, 1813 (2016).
  * Bonato and Michieletto (2021) A. Bonato and D. Michieletto, Biophysical Journal 120, 5544 (2021).
  * Kim  _et al._ (2020) E. Kim, J. Kerssemakers, I. A. Shaltiel, C. H. Haering, and C. Dekker, Nature 579, 438 (2020).
  * Brandão  _et al._ (2021) H. B. Brandão, Z. Ren, X. Karaboja, L. A. Mirny, and X. Wang, Nature Structural and Molecular Biology 28, 642 (2021).
  * Brandão  _et al._ (2019) H. B. Brandão, P. Paul, A. A. van den Berg, D. Z. Rudner, X. Wang, and L. A. Mirny, Proceedings of the National Academy of Sciences of the United States of America 116, 20489 (2019).
  * Pradhan  _et al._ (2022) B. Pradhan, R. Barth, E. Kim, I. F. Davidson, B. Bauer, T. van Laar, W. Yang, J. K. Ryu, J. van der Torre, J. M. Peters, and C. Dekker, Cell Reports 41, 1 (2022).
  * Kschonsak  _et al._ (2017) M. Kschonsak, F. Merkel, S. Bisht, J. Metz, V. Rybin, M. Hassler, and C. H. Haering, Cell 171, 588 (2017).
  * Banigan and Mirny (2019) E. J. Banigan and L. A. Mirny, Physical Review X 9, 031007 (2019).
  * Orlandini  _et al._ (2019) E. Orlandini, D. Marenduzzo, and D. Michieletto, Proceedings of the National Academy of Sciences 116, 8149 (2019).
  * Ryu  _et al._ (2022) J.-K. Ryu, S.-H. Rah, R. Janissen, J. W. J. Kerssemakers, A. Bonato, D. Michieletto, and C. Dekker, Nucleic acids research 50, 820 (2022).
  * Note (1) An alternative way to implement this constraints could be to impose a dihedral or angular potential between the DNA segments bound to heads and anchor. We expect to obtain similar results with this alternative implementation.
  * Martínez‐García  _et al._ (2023) B. Martínez‐García, S. Dyson, J. Segura, A. Ayats, E. E. Cutts, P. Gutierrez‐Escribano, L. Aragón, and J. Roca, The EMBO Journal 42, 1 (2023).


# Understanding Transformer Memorization Recall Through Idioms

Adi Havivτ Ido Cohenτ Jacob Gidronτ Roei Schusterν Yoav Goldbergαβ Mor Gevaα

τTel Aviv University νVector Institute for AI βBar-Ilan University αAllen
Institute for AI

###### Abstract

To produce accurate predictions, language models (LMs) must balance between
generalization and memorization. Yet, little is known about the mechanism by
which transformer LMs employ their memorization capacity. When does a model
decide to output a memorized phrase, and how is this phrase then retrieved
from memory? In this work, we offer the first methodological framework for
probing and characterizing recall of memorized sequences in transformer LMs.
First, we lay out criteria for detecting model inputs that trigger memory
recall, and propose _idioms_ as inputs that fulfill these criteria. Next, we
construct a dataset of English idioms and use it to compare model behavior on
memorized vs. non-memorized inputs. Specifically, we analyze the internal
prediction construction process by interpreting the model’s hidden
representations as a gradual refinement of the output probability
distribution. We find that across different model sizes and architectures,
memorized predictions are a two-step process: early layers promote the
predicted token to the top of the output distribution, and upper layers
increase model confidence. This suggests that memorized information is stored
and retrieved in the early layers of the network. Last, we demonstrate the
utility of our methodology beyond idioms in memorized factual statements.
Overall, our work makes a first step towards understanding memory recall, and
provides a methodological basis for future studies of transformer
memorization.111Our code and data are available at
https://github.com/adihaviv/idiomem/.

##  1 Introduction

Transformer language models (LMs) memorize instances from their training data
(Carlini et al., 2021; Zhang et al., 2021b), and evidence is building that
such memorization is an important precondition for their predictive abilities
(Lee et al., 2022; Feldman, 2020; Feldman and Zhang, 2020). Still, it is
unknown when models decide to output memorized sequences, and how these
sequences are being retrieved internally from memory. Current methods for
analyzing memorization (Feldman and Zhang, 2020; Zhang et al., 2021b; Carlini
et al., 2022) use definitions that are based on models performance, which
changes between models and often also between training runs. Moreover, these
methods study memorization behavior in terms of the model’s “black-box”
behavior rather than deriving a behavioral profile of memory recall itself.

Our first contributions are to provide a definition and construct a dataset
that allows probing memorization recall in LMs. We define a set of criteria
for identifying memorized sequences that does not depend on model
behavior:222Literature often purports to “define memorization”, resulting in a
multitude of technical definitions with subtle differences, although we would
expect this concept to be consistent and intuitive. Thus, instead of
explicitly defining “memorization”, we will define _sufficient criteria_ for
detecting memorized instances. sequences that have a single plausible
completion that is independent of context and can be inferred only given the
entire sequence. We show that many idioms (e.g. “in one ear and out the
other”) fulfill these conditions, allowing us to probe and analyze
memorization behavior. Furthermore, we construct a dataset of such English
idioms, dubbed IdioMem, and release it publicly for the research community.

Next, to analyze memory recall behavior, we compare the construction process
of predictions that involve memory recall with those that do not. To this end,
we adopt a view of the transformer inference pass as a gradual refinement of
the output probability distribution (Geva et al., 2021; Elhage et al., 2021).
Concretely, the token representation at any layer is interpreted as a “hidden”
probability distribution over the output vocabulary (Geva et al., 2022). This
interpretation allows to track the prediction across layers in the evolving
distribution.

We find a clear difference in model behavior between memorized and non-
memorized predictions. This difference persists across different transformer
architectures and sizes: retrieval from memory happens in two distinct phases,
corresponding to distinct roles of the transformer parameters and layers: (1)
_candidate promotion_ of memorized predictions’ rank in the hidden
distribution in the first layers, and (2) _confidence boosting_ where, in the
last few layers, the prediction’s probability grows substantially faster than
before. This is unlike non-memorized predictions, where the two phases are
less pronounced and often indistinct. Fig. 1 illustrates this. We further
confirm these phases of memorized predictions through intervention in the
network’s FFN sublayers, which have been shown to play an important role in
the prediction construction process (Geva et al., 2022; Mickus et al., 2022).
Concretely, zeroing-out hidden FFN neurons in early layers deteriorates
memory-recall, while intervention in upper layers does not affect it.

Last, we show our findings extend to types of memory recall beyond idioms by
applying our method to factual statements from the LAMA-UHN dataset (Poerner
et al., 2020) (e.g. “The native language of Jean Marais is French”). For
factual statements that were completed correctly by the LM, we observe the
same two phases as in memorized idioms, further indicating their connection to
memory recall.

To summarize, we construct a novel dataset of idioms, usable for probing LM
memorization irrespective of the model architecture or training
parameterization. We then design a probing methodology that extracts
carefully-devised features of the internal inference procedure in
transformers. By applying our methodology and using our new dataset, we
discover a profile that characterizes memory recall, across transformer LMs
and types of memorized instances. Our released dataset, probing framework, and
findings, open the door for future work on transformer memorization, to
ultimately demystify the internals of neural memory in LMs.

##  2 Criteria for Detecting Memory Recall

To study memory recall, we require a set of inputs that trigger this process.
Prior work on memorization focused on detecting instances whose inclusion in
the training data has a specific influence on model behavior, such as
increased accuracy on those instances (Feldman and Zhang, 2020; Magar and
Schwartz, 2022; Carlini et al., 2022, 2021, 2019). As a result, memorized
instances differ across models and training parameterization. Our goal is
instead to find a stable dataset of sequences that correctly predicting their
completion indicates memorization recall. This will greatly reduce the
overhead of studying memorization and facilitate useful comparisons across
models and studies.

To build such a dataset, we start by defining a general set of criteria that
are predicates on sequence features, entirely independent of the LM being
probed. Given a textual sequence of $n$ words, we call the first $n-1$ words
the prompt and the $n$th word the target. We focus on the task of predicting
the target given the prompt, i.e., predicting the last word in a sequence
given its prefix.333In cases where tokenization divides the target to sub-
tokens, our task becomes predicting the target’s first token. Such predictions
can be based on either generalization or memorization, and we are interested
in isolating memorized cases in order to study model behavior on them.
Particularly, we are looking for _sequences for which success in this task
necessarily implies memorization recall_.

We argue that the following criteria are _sufficient_ for detecting such
memorized sequences:

  1. 1.

Single target, independent of context: We require that the target is the only
correct continuation, regardless of the textual context where the prompt is
placed.444We assume that contexts are naturally-occurring and not adversarial.

  2. 2.

Irreducible prompt: The target is the single correct completion only if the
entire prompt is given exactly. Changing or removing parts from the prompt
would make the correct target non-unique.

######  Claim 2.1.

Assume a sequence fulfills the above criteria. Then, if a LM correctly
predicts the target, it is highly likely that this prediction involves memory
recall.

###### Justification.

First, observe that most natural-language prompts have many possible
continuations. For example consider the sentence “to get there fast, you can
take this ____”. Likely continuations include “route”, “highway”, “road”,
“train”, “plane”, “advice”, inter alia. Note that there are several divergent
interpretations or contexts for the prompt, and for each, language offers many
different ways to express similar meaning.

A prediction that is a product of generalization — i.e., it is derived from
context and knowledge of language — _always has plausible alternatives_ ,
depending on the context and stylistic choice of words. Hence, the
relationship between the entire prompt and the target, where the target is the
single correct continuation, is something that needs to be memorized rather
than derived via generalization. A LM that predicts _the_ single correct
continuation either memorized this relationship, or used “cues” from the
prompt that happen to provide indication towards the correct continuation. To
illustrate the latter, consider the sequence “it’s raining cats and ____”
which has a single correct continuation, “dogs”, but a LM might predict it
without observing this sequence during training, due to the semantic proximity
of “cats” and “dogs”. Our second criterion excludes such cases by requiring
that the correct continuation is only likely given the entire sequence.

Therefore, a LM that correctly completes a sequence that fulfills both
criteria, necessarily recalled it from memory. ∎

In the next section, we argue that idioms are a special case of such
sequences, and are thus useful for studying memorization (§3).

##  3 The Utility of Idioms for Studying Memorization

An idiom is a group of words with a meaning that is not deducible from the
meanings of its individual words. For example, consider the phrase “play it by
ear” — there is a disconnect between its nonsensical literal meaning (to play
something by a human-body organ called ‘ear’) and its intended idiomatic
meaning (to improvise).

A key observation is that _idioms often satisfy our criteria (§ 2), and
therefore they can probe memorization_. First, by definition, idioms are
expected to be non-compositional Dankers et al. (2022a). They are special
“hard-coded” phrases that carry a specific meaning. As a result, their prompts
each have a single correct continuation, regardless of their context
(criterion 1). For example, consider the prompt “crying over spilt ____” — a
generalizing prediction would predict that this slot may be filled by any
spillable item, like wine, water or juice, while a memorized prediction will
retrieve only milk in this context. Second, many idioms are “irreducible”, for
example the sub-sequences “crying over” or “over spilt” by themselves have but
scant connection to the word “milk”.

Still, we note that not all idioms fulfill the criteria. For example, even
when the idiom is far from literal, its constituents sometimes strongly
indicate the correct continuation, such as with the case of “it’s raining cats
and ____” (as explained in §2). To construct a dataset of memorization-probing
sequences, we will carefully curate a set of English idioms and filter out
ones that do not fulfill our criteria.

###  3.1 The IdioMem Dataset

We begin with existing datasets of English idioms, namely MAGPIE (Haagsma et
al., 2020), EPIE (Saxena and Paul, 2020), and the English subset of LIdioms
(Moussallem et al., 2018). We enrich this collection with idioms scraped from
the website “Education First” (EF).555https://www.ef.com/wwen/english-
resources/english-idioms/ We then split each idiom into a prompt containing
all but the last word, and a target, which is the last word.

Next, we filter out idioms that do not comply with our criteria (§2) or whose
target can be predicted from their prompt based on spurious correlations
rather than memorization. To this end, we use three simple rules:

  * •

Short idioms. We observe that prompts of idioms with just a few words often
have multiple plausible continuations, that are not necessarily the idiom’s
target, violating our first criterion. For example, the prompt “break a ____”
has many possible continuations (e.g. “window”, “promise”, and “heart”) in
addition to its idiomatic continuation “leg”. To exclude such cases, we filter
out idioms with $<4$ words.

  * •

Idioms whose target is commonly predicted from the prompt’s subsequence. We
filter such cases to ensure the prompt fulfills our second criterion (prompt
irreducibility).

To detect these cases, we use an ensemble of pretrained LMs:
GPT2${}_{\textsc{M}}$, RoBERTa-base (Liu et al., 2019), T5-base (Kale and
Rastogi, 2020) and ELECTRA-base-generator (Clark et al., 2020), and check for
each model if there is an n-gram ($1\leq n\leq 4$) in the prompt from which
the model predicts the target. We filter out idioms for which a majority
($\geq 3$) of models predicted the target (for some n-gram).

  * •

Idioms whose targets are semantically similar to tokens in the prompt. To
further ensure prompt irreducibility, we embed the prompt’s tokens and the
target token using GloVe word embeddings (Pennington et al., 2014). We measure
the cosine distance between the target token to each token in the prompt
separately, and take the maximum of all the tokens. We filter out idioms where
this number is higher than 0.75 (this number was tuned manually using a small
validation set of idioms).

Overall, 55.7% of the idioms were filtered out, including 48.5% by length,
6.1% by the predictable-target test and an additional 1.6% by the prompt-
target similarity, resulting in an 814 idioms dataset, named IdioMem. Further
statistics are provided in Tab. 1, and example idioms in Tab. 2.

We now turn to describe our probing method, using idioms from IdioMem or any
sequences fulfilling our criteria (§2).

Source | # of Idioms |  Idiom Length (words)
---|---|---
MAGPIE | 590 | $4.5\pm 0.9$
LIdioms | 149 | $5.1\pm 1.2$
EF | 97 | $5.6\pm 1.9$
EPIE | 76 | $4.4\pm 0.7$
Total (unique) | 814 | $4.7\pm 1.8$
Table 1: Statistics per data source in IdioMem.

##  4 Probing Methodology

#### Background and Notation

Assuming a transformer LM with $L$ layers, a hidden dimension $d$ and an
input/output-embedding matrix $E\in\mathbb{R}^{\lvert V\rvert\times d}$ over a
vocabulary $V$. Denote by $\mathbf{s}=\langle s_{1},...,s_{t}\rangle$ the
input sequence to the LM, and let $\mathbf{h}_{i}^{\ell}$ be the output for
token $i$ at layer $\ell$, for all $\ell\in 1,...,L$ and $i\in 1,...,t$. The
model’s prediction for a token $s_{i}$ is obtained by projecting its last
hidden representation $\mathbf{h}_{i}^{L}$ to the embedding matrix, i.e.
$softmax(E\mathbf{h}_{i}^{L})$.

Following Geva et al. (2021, 2022), we interpret the prediction for a token
$s_{i}$ by viewing its corresponding sequence of hidden representations
$\mathbf{h}_{i}^{1},...,\mathbf{h}_{i}^{L}$ as an evolving distribution over
the vocabulary. Concretely, we read the “hidden” distribution at layer $\ell$
by applying the same projection to the hidden representation at that layer:
$\mathbf{p}_{i}^{\ell}=softmax(E\mathbf{h}_{i}^{\ell})$. Using this
interpretation, we track the probability and rank of the predicted token in
the output distribution across layers. A token’s rank is its position in the
output distribution when sorted by probability from highest to lowest (e.g.
the rank of the final predicted token is zero).

#### Probing Procedure

Our key method to understand how transformer LMs retrieve information from
memory is _comparing features of memory recall_ to inference that does not
necessarily include memory recall. Given a set of sequences that fulfill the
criteria in §2, we split them into a “memorized” set whose targets’ first
token is predicted correctly by the model being analyzed given (and are
therefore memorized), and a “non-memorized” set whose target is predicted
incorrectly. We additionally include a second set of “non-memorized”
instances: natural-language sequences randomly sampled from a large corpus (we
assume most naturally-occurring sequences are not memorized).

To probe a LM, we run it on the 3 sets, and for each set and each layer, we
(a) extract the rank and probability of the final predicted token in the
hidden distribution for each prompt, and (b) compute the mean rank and
probability over all prompts.

Prompt | Target | Pred. | Sim. | IdioMem
---|---|---|---|---
“make a mountain out of a” | molehill |  |  | ✓
“think outside the” | box |  |  | ✓
“there’s no such thing as a free” | lunch |  |  | ✓
“go back to the drawing” | board | ✓ |  |
“boys will be” | boys |  | ✓ |
“take it or leave” | it | ✓ | ✓ |
Table 2: Example English idioms included and excluded from IdioMem by our
filters of predictable target (Pred.) and prompt-target similarity (Sim.).

##  5 Probing Memorization Using Idioms

###  5.1 Experimental Setup

| GPT2${}_{\textsc{M}}$ | GPT2${}_{\textsc{L}}$ | BERT${}_{\textsc{B}}$ | BERT${}_{\textsc{L}}$
---|---|---|---|---
Memorized idioms (mem-idiom) | 364 44.7% | 392 48.2% | 230 28.3% | 305 37.5%
Non-memorized idioms (non-mem-idiom) | 450 55.3% | 422 51.8% | 584 71.7% | 509 62.5%
Table 3: Number of memorized idioms vs. non-memorized idioms from the IdioMem
dataset for each model. An instance is considered a memorized example if the
model correctly predicts the target.

#### Datasets

For each LM under analysis (see below), we split IdioMem into two disjoint
subsets of memorized and non-memorized idioms, denoted as mem-idiom and non-
mem-idiom, respectively, according to whether or not the LM succeeds in
completing them. We produce an additional set of non-memorized instances,
wiki, by sampling prompts from the WikiText-103 dataset (Merity et al., 2017)
of the same length distribution as in IdioMem (see Tab. 1).

#### Models

We use multiple transformer LMs that are different in size, architecture, and
optimization objective. we use GPT2 (medium and large) (Radford et al., 2019),
an autoregressive transformer decoder, and BERT (base and large) (Devlin et
al., 2019), a transformer encoder trained with a masked language modeling
(MLM) objective. To evaluate BERT on a specific idiom, we feed the idiom’s
prompt concatenated with the special mask token and a period (e.g. “think
outside the [MASK].”). Further details on each model are presented in Tab. 4.
The number of memorized and non-memorized idioms from IdioMem for each model
are provided in Tab. 3.

Figure 1: The predicted token’s probability and rank across layers, for
memorized idioms (mem-idiom), non-memorized idioms (non-mem-idiom) and short
sequences from Wikipedia (wiki). Memory recall exhibits two characteristic
phases of candidate promotion and confidence boosting.

###  5.2 Memorized Predictions are a Two-Step Process

Fig. 1 shows the probability and rank of the output token across layers, for
memorized and non-memorized idioms and short sequences from Wikipedia, by
GPT2${}_{\textsc{M}}$, GPT2${}_{\textsc{L}}$, BERT${}_{\textsc{B}}$, and
BERT${}_{\textsc{L}}$. Naturally, the prediction’s rank decreases across
layers as the prediction probability increases. However, for memorized
predictions these trends occur as two distinct and sharp inference phases. In
lower layers, the prediction’s rank decreases from a high rank to near zero,
while its probability is also close to zero. For example, in
GPT2${}_{\textsc{L}}$ the rank decreases until layer 20 while the probability
remains below 0.1. We refer to this phase as _candidate promotion_ , as the
predicted token is being promoted to be a top candidate in the output
distribution.

Compared to non-memorized predictions, the initial rank of memorized
predictions is generally higher, especially in GPT2 (6000 vs. 3000 in
GPT2${}_{\textsc{M}}$, and 3000 vs. 1500 in GPT2${}_{\textsc{L}}$). This is
not due to the predicted tokens’ unigram frequencies, as those are not
correlated with the prediction rank at the first layer (see §B). Instead, a
possible explanation is that non-memorized predictions are often promoted in
early layers that detect local “shallow” patterns such as common bigrams Geva
et al. (2021). Prediction for memorized idioms is never local, because it
requires the entire sequence.

In the middle layers, once the predicted token reaches the top of the hidden
distribution, its probability increases until the last layer. We refer to this
phase as _confidence boosting_ , as the distribution shifts towards the
predicted token. For memorized idioms, this increase is abrupt and dramatic,
with a final probability of $>0.6$ across all models. In comparison,
predictions on short sequences from Wikipedia and non-memorized idioms have
substantially lower probability of $\sim 0.2$. This can be explained by the
fact that memorized idioms have a single correct target, rather than many
possible continuations as in the instances from Wikipedia. In addition, low-
probability predictions for non-memorized idioms are expected as the model did
not memorize the idioms and does not know their continuation.

We further verify that our extracted hidden-distribution features distinguish
between memorized and non-memorized predictions by training linear classifiers
over combinations of these features (details in §C). We observe that, indeed,
our features enable separating between memorized and non-memorized predictions
at a high accuracy (77%-85% across models). Moreover, classifiers that use
hidden distribution features are more accurate than those relying only on the
model’s output. Overall, these findings provide a profile of memorized
predictions, suggesting that the memorized information is retrieved in early
layers at inference.

Figure 2:  Intervention in non-dominant (left) and dominant (right) FFN sub-
updates in GPT2${}_{\textsc{M}}$. Each cell shows the percentage of memorized
idioms for which the prediction was changed by zeroing-out the FFN sub-updates
between the start and end layers.

###  5.3 Testing the Roles of Different Layers Through Intervention

Our analysis in the previous section interprets hidden representations as
distributions over the output vocabulary. We now conduct intervention
experiments to verify that this interpretation is meaningful for studying
memory recall, and to test layers’ roles in that process. Concretely, we zero
out FFN sub-updates to the hidden representation (§4) and measure changes in
memorized predictions.

#### A Short Primer on Transformer FFN Sublayers.

FFN sublayers are the final computation in transformer layers, which output
the hidden distribution at the center of our analysis. In general, they have a
key role in capturing knowledge in transformer LMs (Dai et al., 2022; Meng et
al., 2022). We follow (Geva et al., 2022) and view the computation of each FFN
sublayer as a weighted collection of $d_{m}$ sub-updates to the output
distribution, each promoting a concept in the vocabulary space, e.g. “past-
tense verbs” or “female athletes”. To understand this, consider the
computation of the FFN at layer $\ell$, given by
$\texttt{FFN}^{\ell}(\mathbf{h}^{\ell}_{i})=f(W_{K}^{\ell}\mathbf{h}_{i}^{\ell})W_{V}^{\ell}$,
where $W_{K}^{\ell},W_{V}^{\ell}\in\mathbb{R}^{d_{m}\times d}$ are parameter
matrices, $d_{m}$ is the intermediate hidden dimension, and $f$ is a point-
wise non-linearity activation function. This computation can be decomposed as:
$\texttt{FFN}^{\ell}(\mathbf{h}^{\ell}_{i})=\sum_{j=1}^{d_{m}}f(\mathbf{h}_{i}^{\ell}\cdot\mathbf{k}_{j}^{\ell})\mathbf{v}_{j}^{\ell}=\sum_{j=1}^{d_{m}}m_{j}^{\ell}\mathbf{v}_{j}^{\ell}$,
where $\mathbf{k}_{j}^{\ell}$ and $\mathbf{v}_{j}^{\ell}$ are the $j$-th row
in $W_{K}^{\ell}$ and the $j$-th column in $W_{V}^{\ell}$, respectively. Geva
et al. argue that each weight $m_{j}^{\ell}$ is the score assigned by the
model for some textual pattern, and each vector $\mathbf{v}_{j}$ promotes a
concept that follows that pattern.

#### Experiment

First, we sample 100 random instances from IdioMem that the model memorized.
Then, for each range of up to 3 consecutive layers, we perform two
complementary experiments, where we run GPT2${}_{\textsc{M}}$’s inference on
the 100 instances while intervening in the chosen layers to cancel the
contribution of FFN sub-updates to the prediction. Specifically, for each
layer $\ell$ in the layer range, we perform the following two interventions:
first, we zero out (i.e. artificially set to 0 during inference) the weights
of the 10 most dominant sub-updates, which are known to be particularly
salient for predictions Geva et al. (2022) (there are $d_{m}$ sub-updates per
layer). Concretely, we sort the sub-updates by their contribution to the FFN
output $|m_{i}^{\ell}|||\mathbf{v}_{i}^{\ell}||\;\forall i\in[1,...,d_{m}]$,
and set $m_{i}^{\ell}=0$ for the 10 sub-updates with the highest contribution.
Next, we zero out non-dominant sub-updates, i.e. all the sub-updates _except_
for the 10 most dominant ones. For each intervention, we measure how often it
changes the predicted token. Further measurements of the change in rank and
probability of the target token are reported in §D.

#### Results

Fig. 2 shows, for each layer range, the percentage of memorized idioms where
the predicted token has changed. Focusing on zeroing out non-dominant FFN sub-
updates (Fig. 2, left), we observe a _two-phase pattern of decreasing “layer
importance” which corresponds to the two-phase pattern of decreasing rank and
increasing probability during inference (§ 5.2)_: layers’ effect on memory
recall drops precipitously in the first 10 layers, corresponding to the
candidate promotion phase. Then, from around layer 10 onwards, the drop in
effect is much less steep, corresponding to the confidence boosting phase.
Intervention in upper layers rarely changes the predicted token, and its
effect is limited to reducing the model’s confidence (§D). We further
visualize this two-phase behavior in Fig. 4 in §D.

These findings suggest that the candidate promotion phase, while having a
smaller effect on the prediction’s assigned probability compared to the later
confidence-boosting stage, in fact has a crucial role in memory recall.

We also observe that interventions in the first layer are by far the most
destructive, with 100% change in prediction for non-dominant updates (Fig. 2,
left). Unlike for other layers, this is also observable when zeroing out the
dominant sub-updates (Fig. 2, right) which constitute only 0.1% of layer sub-
updates. This suggests the first layer is especially critical for memorized
predictions.

##  6 Memorization of Factual Statements

We now examine if our findings generalize beyond idioms to other types of
memory recall, focusing on completion of statements expressing facts.

#### Data

Datasets for evaluating memorization of factual knowledge typically contain
simple queries such as “The continent of Kuwait is”, where predicting the next
token requires knowledge of the triplet $\langle s,r,t\rangle$ where $s$ is a
source entity (e.g. Kuwait), $t$ a target entity (e.g. Asia), and $r$ is the
relation between them (e.g. is the continent of). However, unlike idioms, such
queries are not suitable for probing memorization since they often fail to
satisfy the criteria in §2. Concretely, queries often include “clues” that
could make the prediction easy to guess and based on generalization (Poerner
et al., 2020) (e.g. predicting a Spanish-speaking country for the query
“Federico López was born in”), and the same fact can be expressed in multiple
different ways (e.g. “Kuwait is a country in Western Asia” also encodes the
above fact).

To test memorization of facts, we therefore collect factual statements where
such cases are excluded. We use LAMA-UHN (Poerner et al., 2020), a subset of
the LAMA dataset (Petroni et al., 2019), where “easy-to-guess” queries are
filtered out. LAMA comprises of queries structured as “fill-in-the-blank”
cloze statements (e.g. “Gordon Scholes is a member of the ____political
party.”). To accommodate autoregressive LMs, we consider only queries where
the blank appears at the end. In addition, we keep only queries with a single
correct completion (based on LAMA). This turns our definition of memorized and
non-memorized sets (§4) equivalent to separating based on the evaluation
metric of LAMA: an instance is considered memorized if the model predicts the
single correct completion, and non-memorized otherwise. Overall, the resulting
collection consists of 17,855 factual statements with 22 unique relations.

#### Memorized Facts Exhibit a Similar Prediction Process to Idioms

We repeat our analysis (§5.1), using the collected factual statements and
GPT2${}_{\textsc{M}}$. Splitting the statements to those memorized and non-
memorized by the model results in 786 mem-facts vs. 17,069 mem-facts
statements.

Fig. 3 shows the rank and probability of the predicted token across layers.
Like idioms, memorized facts exhibit a clear two-phase prediction process,
where the prediction probability rapidly increases once the candidate reaches
a low rank (at layer 16). This is in contrast to non-memorized facts and short
sequences from Wikipedia, where the rank (probability) gradually decreases
(increases) across layers without a distinct two phases.

Figure 3: The predicted token’s probability and rank across layers of
GPT2${}_{\textsc{M}}$, for memorized (mem-fact) and non-memorized (non-mem-
fact) facts and short sequences from Wikipedia (wiki).

#### Differences from Memorized Idioms Stem from Ill-Defined Targets

There is one major difference compared to memorized idioms (Fig. 3 vs. Fig. 1
upper left), which is a substantial drop in probability ($0.62\rightarrow
0.21$) in the last two layers. We hypothesize that this is because the input
query has multiple plausible completions that were not specified as “correct”
targets in LAMA. We verify this by manually analyzing predictions, and find
that for 82 out of 100 queries there is more than one correct continuation in
top 5 predicted tokens. We posit that the above deficiency of LAMA is inherent
because, in violation of our criterion (§2), factual statements can usually be
expressed in many ways so their prompt has no single correct target.

##  7 Related Work

#### Memorization as Training-Data Influence.

Memorization in LMs has attracted immense attention due to their rich
sensitive training data (Carlini et al., 2019; Song and Shmatikov, 2019;
Carlini et al., 2021; Zhang et al., 2021b; Carlini et al., 2022; Tirumala et
al., 2022; Tänzer et al., 2022). Recent work suggests that memorization is
_necessary_ for performant ML due to the “long tail” of infrequently-observed
patterns (Zhang et al., 2021a; Feldman, 2020; Brown et al., 2021). This line
of work has two key limitations: (a) only black-box behavior is measured
rather than looking at models’ internal prediction process, and (b) it detects
memorized instances by measuring the effect of their inclusion in the training
set on inference behavior. This results in a set of memorized examples that is
specific to the model, training data, and even training pass, making it
difficult to build on these results in future research.

#### Transformers and Idioms.

Nedumpozhimana and Kelleher (2021) showed that idioms are identified using
textual cues within the expression; Dankers et al. (2022b) showed that idioms
tend to be internally processed as single units of meaning. It has also been
known (Fakharian and Cook, 2021; Salton et al., 2016) that LM contextual
embeddings encode information about whether or not an expression is idiomatic
(vs. literal). (Shwartz and Dagan, 2019; Chakrabarty et al., 2022) also
studied representations and interpretation of non-compositional sequences,
such as idioms. No prior work used idioms to probe LM memorization.

#### Memorization of Factual Knowledge.

An extensive line of work (Petroni et al., 2019; Jiang et al., 2020; Poerner
et al., 2019; Lewis et al., 2020; Elazar et al., 2021) studied LMs’ capacity
to acquire relational knowledge during training. Less attention has been given
to understanding the inner-workings of inference on factual statements,
although Wallat et al. (2020) showed that some factual knowledge is retrieved
from bottom and intermediate layers. Since factual statements do not fulfill
our criteria, it is difficult to convincingly argue that correct predictions
indicate memory recall, making it impossible to use them to isolate the effect
of memorization.

##  8 Conclusion

We introduce a methodological framework for detecting and analyzing memorized
predictions in transformer LMs. This includes a set of criteria on textual
sequences for probing memorized predictions, the IdioMem dataset of idioms
fulfilling these criteria, and an interpretation method of prediction
internals. We characterize a behavioral profile that is unique to predictions
involving memory recall and is observable across different LMs and data types.
By providing these fundamental tools and initiating a thread of research on
the phenomena we observe, we hope to empower future work towards demystifying
transformer memorization.

## Limitations

Our criteria for detecting memorized instances are sufficient but not
necessary, which raises the question of what other sequences that trigger
memory recall satisfy them.

Additionally, while correct prediction for sequences that fulfill our criteria
implies memory recall, incorrect prediction does not necessarily imply that no
memory was used. This means that our set non-mem-idiom might include some
portion of memorized sequences. This does not qualitatively affect our results
as long as mem-idiom still contains _more_ memorized predictions than non-mem-
idiom, as is evidenced by the stark differences we observe between LMs’
internal behaviors on these sets.

Our work focuses on showing the utility of idioms for probing memorization,
and opening up a new thread of research in this vein. We therefore leave
further investigation of the above gaps for future work.

## Acknowledgements

We thank Gabriel Stanovsky for helpful feedback. This project has received
funding and resources from CIFAR through the Canada CIFAR AI Chair program,
the Province of Ontario, companies sponsoring the Vector Institute, as well as
the Computer Science Scholarship granted by the Séphora Berrebi Foundation and
the European Research Council (ERC) under the European Union’s Horizon 2020
research and innovation programme, grant agreement No. 802774 (iEXTRACT).

## References

  * Brown et al. (2021) Gavin Brown, Mark Bun, Vitaly Feldman, Adam Smith, and Kunal Talwar. 2021.  When is memorization of irrelevant training data necessary for high-accuracy learning?  In _Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing_ , pages 123–132.
  * Carlini et al. (2022) Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2022.  Quantifying memorization across neural language models.  _arXiv preprint arXiv:2202.07646_.
  * Carlini et al. (2019) Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. 2019\.  The secret sharer: Evaluating and testing unintended memorization in neural networks.  In _28th USENIX Security Symposium (USENIX Security 19)_ , pages 267–284, Santa Clara, CA. USENIX Association.
  * Carlini et al. (2021) Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021.  Extracting training data from large language models.  In _30th USENIX Security Symposium (USENIX Security 21)_ , pages 2633–2650. USENIX Association.
  * Chakrabarty et al. (2022) Tuhin Chakrabarty, Yejin Choi, and Vered Shwartz. 2022.  It’s not rocket science: Interpreting figurative language in narratives.  _Transactions of the Association for Computational Linguistics_ , 10:589–606.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020.  Electra: Pre-training text encoders as discriminators rather than generators.  In _International Conference on Learning Representations_.
  * Dai et al. (2022) Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022.  Knowledge neurons in pretrained transformers.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8493–8502, Dublin, Ireland. Association for Computational Linguistics.
  * Dankers et al. (2022a) Verna Dankers, Christopher Lucas, and Ivan Titov. 2022a.  Can transformer be too compositional? analysing idiom processing in neural machine translation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3608–3626, Dublin, Ireland. Association for Computational Linguistics.
  * Dankers et al. (2022b) Verna Dankers, Christopher G Lucas, and Ivan Titov. 2022b.  Can transformer be too compositional? analysing idiom processing in neural machine translation.  _arXiv preprint arXiv:2205.15301_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Elazar et al. (2021) Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. 2021.  Measuring and improving consistency in pretrained language models.  _Transactions of the Association for Computational Linguistics_ , 9:1012–1031.
  * Elhage et al. (2021) Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. 2021.  A mathematical framework for transformer circuits.  _Transformer Circuits Thread_.  Https://transformer-circuits.pub/2021/framework/index.html.
  * Fakharian and Cook (2021) Samin Fakharian and Paul Cook. 2021.  Contextualized embeddings encode monolingual and cross-lingual knowledge of idiomaticity.  In _Proceedings of the 17th Workshop on Multiword Expressions (MWE 2021)_ , pages 23–32.
  * Feldman (2020) Vitaly Feldman. 2020.  Does learning require memorization? a short tale about a long tail.  In _Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing_ , STOC 2020, page 954–959, New York, NY, USA. Association for Computing Machinery.
  * Feldman and Zhang (2020) Vitaly Feldman and Chiyuan Zhang. 2020.  What neural networks memorize and why: Discovering the long tail via influence estimation.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 2881–2891. Curran Associates, Inc.
  * Geva et al. (2022) Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. 2022.  Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space.  _arXiv preprint arXiv:2203.14680_.
  * Geva et al. (2021) Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021.  Transformer feed-forward layers are key-value memories.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 5484–5495, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Haagsma et al. (2020) Hessel Haagsma, Johan Bos, and Malvina Nissim. 2020.  MAGPIE: A large corpus of potentially idiomatic expressions.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 279–287, Marseille, France. European Language Resources Association.
  * Jiang et al. (2020) Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. 2020.  How can we know what language models know?  _Transactions of the Association for Computational Linguistics_ , 8:423–438.
  * Kale and Rastogi (2020) Mihir Kale and Abhinav Rastogi. 2020.  Text-to-text pre-training for data-to-text tasks.  In _Proceedings of the 13th International Conference on Natural Language Generation_ , pages 97–102, Dublin, Ireland. Association for Computational Linguistics.
  * Lee et al. (2022) Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2022.  Deduplicating training data makes language models better.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8424–8445, Dublin, Ireland. Association for Computational Linguistics.
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  _Advances in Neural Information Processing Systems_ , 33:9459–9474.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _ArXiv_ , abs/1907.11692.
  * Magar and Schwartz (2022) Inbal Magar and Roy Schwartz. 2022.  Data contamination: From memorization to exploitation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 157–165, Dublin, Ireland. Association for Computational Linguistics.
  * Meng et al. (2022) Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022.  Locating and editing factual knowledge in gpt.  _arXiv preprint arXiv:2202.05262_.
  * Merity et al. (2017) Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017.  Pointer sentinel mixture models.  In _International Conference on Learning Representations_.
  * Mickus et al. (2022) Timothee Mickus, Denis Paperno, and Mathieu Constant. 2022.  How to dissect a Muppet: The structure of transformer embedding spaces.  _Transactions of the Association for Computational Linguistics_ , 10:981–996.
  * Moussallem et al. (2018) Diego Moussallem, Mohamed Ahmed Sherif, Diego Esteves, Marcos Zampieri, and Axel-Cyrille Ngonga Ngomo. 2018.  LIdioms: A multilingual linked idioms data set.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan. European Language Resources Association (ELRA).
  * Nedumpozhimana and Kelleher (2021) Vasudevan Nedumpozhimana and John Kelleher. 2021.  Finding bert’s idiomatic key.  In _Proceedings of the 17th Workshop on Multiword Expressions (MWE 2021)_ , pages 57–62.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  GloVe: Global vectors for word representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019.  Language models as knowledge bases? In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2463–2473, Hong Kong, China. Association for Computational Linguistics.
  * Poerner et al. (2019) Nina Poerner, Ulli Waltinger, and Hinrich Schütze. 2019.  Bert is not a knowledge base (yet): Factual knowledge vs. name-based reasoning in unsupervised qa.  _arXiv preprint arXiv:1911.03681_ , 3.
  * Poerner et al. (2020) Nina Poerner, Ulli Waltinger, and Hinrich Schütze. 2020.  E-BERT: Efficient-yet-effective entity embeddings for BERT.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 803–818, Online. Association for Computational Linguistics.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Salton et al. (2016) Giancarlo Salton, Robert Ross, and John Kelleher. 2016.  Idiom token classification using sentential distributed semantics.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 194–204.
  * Saxena and Paul (2020) Prateek Saxena and Soma Paul. 2020.  Epie dataset: A corpus for possible idiomatic expressions.  In _Text, Speech, and Dialogue_ , pages 87–94, Cham. Springer International Publishing.
  * Shwartz and Dagan (2019) Vered Shwartz and Ido Dagan. 2019.  Still a pain in the neck: Evaluating text representations on lexical composition.  _Transactions of the Association for Computational Linguistics_ , 7:403–419.
  * Song and Shmatikov (2019) Congzheng Song and Vitaly Shmatikov. 2019.  Auditing data provenance in text-generation models.  In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, KDD ’19, page 196–206, New York, NY, USA. Association for Computing Machinery.
  * Tänzer et al. (2022) Michael Tänzer, Sebastian Ruder, and Marek Rei. 2022.  Memorisation versus generalisation in pre-trained language models.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 7564–7578, Dublin, Ireland. Association for Computational Linguistics.
  * Tirumala et al. (2022) Kushal Tirumala, Aram H Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. 2022\.  Memorization without overfitting: Analyzing the training dynamics of large language models.  _arXiv preprint arXiv:2205.10770_.
  * Wallat et al. (2020) Jonas Wallat, Jaspreet Singh, and Avishek Anand. 2020.  BERTnesia: Investigating the capture and forgetting of knowledge in BERT.  In _Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_ , pages 174–183, Online. Association for Computational Linguistics.
  * Zhang et al. (2021a) Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. 2021a.  Understanding deep learning (still) requires rethinking generalization.  _Communications of the ACM_ , 64(3):107–115.
  * Zhang et al. (2021b) Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. 2021b.  Counterfactual memorization in neural language models.  _arXiv preprint arXiv:2112.12938_.

| GPT2${}_{\textsc{M}}$ | GPT2${}_{\textsc{L}}$ | BERT${}_{\textsc{B}}$ | BERT${}_{\textsc{L}}$
---|---|---|---|---
Layers | 24 | 36 | 12 | 24
Model hidden dimensions ($d$) | 1024 | 1280 | 768 | 1024
Feed-forward dimensions ($d_{m}$) | 4096 | 5120 | 3072 | 4096
Attention heads | 12 | 20 | 12 | 16
Parameters | 345M | 774M | 110M | 340M
Vocabulary size (# of tokens) | 50,256 | 50,256 | 30,522 | 30,522
Table 4: The models hyperparameters.

##  Appendix A Experimental Setting Details

Tab. 4 shows the evaluated models hyperparameters.

| correlation | p-value
---|---|---
GPT2${}_{\textsc{M}}$ | -0.22 | $2.9e^{-21}$
GPT2${}_{\textsc{L}}$ | -0.18 | $5.4e^{-14}$
BERT${}_{\textsc{B}}$ | -0.19 | $1.8e^{-15}$
BERT${}_{\textsc{L}}$ | 0.15 | $3.5e^{-10}$
Table 5: Correlation between the predicted token’s rank at the first layer and
its general frequency in Wikipedia.

##  Appendix B Predicted Token Frequency

In §5.2, we observe that during candidate promotion phase, the predicted token
has a higher initial rank for memorized idioms than for non-memorized idioms
and short sequences from Wikipedia. Arguably, this can be a result of a
generally lower frequency of the predicted token for memorized idioms. To rule
out this hypothesis, we check whether the rank of the predicted token at the
first layer is correlated with its frequency. First, for each model, we
calculate the average token frequency over the training set of WikiText-103
(103M tokens). Then, we compute the Pearson correlation between the rank at
the first layer and the frequency of the predicted token. Results are shown in
Tab. 5. For all models, there is only low correlation between the initial rank
of the predicted token and its frequency.

| GPT2${}_{\textsc{M}}$ | GPT2${}_{\textsc{L}}$ | BERT${}_{\textsc{B}}$ | BERT${}_{\textsc{L}}$
---|---|---|---|---
probability | $\bm{84.6\pm 2.8}$ | $\bm{84.0\pm 2.3}$ | $\bm{84.2\pm 1.8}$ | $\bm{77.6\pm 3.3}$
ranks | $63.2\pm 1.6$ | $59.5\pm 2.7$ | $73.3\pm 3.3$ | $64.5\pm 3.6$
probability + ranks | $83.6\pm 2.7$ | $82.9\pm 2.9$ | $82.7\pm 2.2$ | $76.9\pm 2.9$
ranks layer 1-12 + probability | $\bm{84.3\pm 2.8}$ | $83.4\pm 2.7$ | $82.7\pm 2.2$ | $\bm{77.2\pm 4.2}$
probability last layer | $79.5\pm 3.5$ | $83.3\pm 2.3$ | $77.7\pm 2.3$ | $74.2\pm 4.8$
final hidden state | $72\pm 2.1$ | $72.5\pm 3.7$ | $72.7\pm 2.9$ | $66.0\pm 3.0$
token ids | $58\pm 0.2$ | $59.4\pm 0.2$ | $58.8\pm 0.2$ | $71.6\pm 0.2$
random | $49.5\pm 3.8$ | $49.7\pm 4.7$ | $50.4\pm 4.9$ | $50.1\pm 3.5$
Table 6: Cross-validation accuracy of a logistic-regression classifier trained
to distinguish between memorized and non-memorized idioms.

##  Appendix C Distinguishing Memorization Using Hidden-Distribution Features

§5.2 shows differences in our extracted hidden-distribution features, namely
the rank and probability across layers, between memorized idioms and non-
memorized sequences. To verify that these features are distinguishing between
memorized and non-memorized predictions, we build a classifier that receives
them as input, as follows.

#### Experiment

To answer the above, we represent every instance in IdioMem, for every LM we
experiment with, as a sequence of probabilities and ranks assigned to the
predicted token at each layer. This results in a feature vector for each
instance in IdioMem for each of our LMs (GPT2${}_{\textsc{M}}$,
GPT2${}_{\textsc{L}}$, BERT${}_{\textsc{B}}$, and BERT${}_{\textsc{L}}$). We
then append a class label for each LM’s instances corresponding to whether it
memorized them. Then, for each LM’s dataset, we perform 10-fold cross
validation with an 80%-20% train-test split, to evaluate the accuracy of a
logistic-regression classifier using the LogisticRegresion classifier of
scikit-learn,666https://scikit-learn.org/ specifying L1 penalty and a bilinear
solver as constructor parameters. After each split and before evaluating the
classifier, we balance the test set by replacing the larger class with a
random subsample the size of the smaller class. To isolate the distinguishing
utility of ranks from that of probabilities, we repeat this process while only
taking either of them as features at each time. We also repeat this process
while separately considering just the last-layer probability as a single
feature, the last hidden state vector, and finally, the ranks in layers of
layers 1-12 (omitting ranks in layers 13-16 where ranks are usually 0)
appended to all layer probabilities.

#### Results

Results are given in Tab. 6. We observe that, across all models, most
classifiers perform well over the 50%-accuracy baseline for their class-
balanced test sets. The output probability alone is often highly
distinguishing with around 78% accuracy, but the vector of 16 hidden-
distribution probabilities seems to contain additional distinguishing
information, as using it alongside ranks results in higher accuracy, typically
around 84%. Using the ranks in addition to probabilities usually _decreases_
accuracy, but omitting the ranks in layers 13-16 (which we know are mostly 0
as this is the confidence-boosting phase) attenuates this effect. We
conjecture that ranks have little meaningful information, especially in the
confidence boosting phase.

Figure 4: We visualize the effect of intervening in each 3-consecutive-layer
range according to the procedure in §5.3.

##  Appendix D Intervention Experiments: Additional Analysis

###  D.1 Additional Visualization

In §5.3, we performed an intervention-based experiment to test the effect of
zeroing out FFN sub-updates in layer computation. This produced a heat-map of
values corresponding to intervention’s effect for each layer range. Here, we
plot the intervention’s effect across all 3-layer layer ranges. Note that
since there are 24 layers, there are a total of 22 ranges of 3 consecutive
layers. Fig. 4 visualizes the effect of intervening in each such range.

As discussed in §5.3, we observe a steep drop in effect in the first 10
layers, followed by a more leveled slope of decrease in upper layers.

###  D.2 Analyzing Changes in Rank and Probability of the Target Token

In addition to measuring how often an intervention changes memorized
predictions (§5.3), we further measure the average change in the rank and
probability of the target token. Note that the original target rank for
memorized predictions is always zero, as the target token is the top candidate
in the original output distribution.

#### Change in target rank

Fig. 5 show the change in the target token’s rank for all intervention
experiments. Overall, we observe similar trends as in §5.3. First, zeroing-out
either dominant or non-dominant FFN sub-updates in upper layers (layers 11-24)
does not affect memory recall, as the target token is still ranked as the top
candidate in the output distribution. Moreover, zeroing out in early layers
(1-10) damages memory recall as the target rank increases by $>20$ positions.
Specifically, zeroing-out non-dominant FFN sub-updates in layers 2-4 increases
the target rank by 60, and disabling either dominant or non-dominant sub-
updates in the first layer completely eliminates memory recall as the rank
increases to $>6000$.

#### Change in target probability

Fig. 6 shows the change in the target token’s probability for all intervention
experiments. Unlike the prediction rank, which is mostly influenced by the
lower layers during memory recall, the prediction probability is highly
influenced by the intermediate and upper layers, where confidence boosting
happens. Disabling FFN sub-updates in only three of these layers reduces the
prediction probability by up to 33%. Considering the lower layers (1-9), zero-
outs lead to a large probability decrease (up to 50%). This is expected since
these interventions often change the prediction, i.e. they eliminate the
target from the top of the output distribution.

Figure 5: Change in the rank of the target token following intervention
zeroing out non-dominant (left) and dominant (right) FFN sub-updates in
GPT2${}_{\textsc{M}}$. Each cell shows the average change in rank of the
target token after zeroing-out the sub-updates in the layers between the start
and end layers. For readability, we provide plots with the first layer (top)
and without (bottom). Figure 6: Change in the probability of the target token
following intervention zeroing out non-dominant (left) and dominant (right)
FFN sub-updates in GPT2${}_{\textsc{M}}$. Each cell shows the average change
in probability of the target token after zeroing-out the sub-updates in the
layers between the start and end layers. For readability, we provide plots
with the first layer (top) and without (bottom).