# Developing a Chemical and Structural Understanding of the Surface Oxide in a
Niobium Superconducting Qubit

Akshay A. Murthy  Superconducting Quantum Materials and Systems Division,
Fermi National Accelerator Laboratory (FNAL), Batavia, IL 60510, USA
<EMAIL_ADDRESS>Paul Masih Das  Department of Materials Science and
Engineering, Northwestern University, Evanston, IL, 60208 USA  Stephanie M.
Ribet  Department of Materials Science and Engineering, Northwestern
University, Evanston, IL, 60208 USA  Cameron Kopas  Rigetti Computing,
Berkeley, CA 94710, USA  Jaeyel Lee  Superconducting Quantum Materials and
Systems Division, Fermi National Accelerator Laboratory (FNAL), Batavia, IL
60510, USA  Matthew J. Reagor  Rigetti Computing, Berkeley, CA 94710, USA  Lin
Zhou  Ames Laboratory, U.S. Department of Energy, Ames, IA 50011, United
States  Matthew J. Kramer  Ames Laboratory, U.S. Department of Energy, Ames,
IA 50011, United States  Mark C. Hersam  Department of Materials Science and
Engineering, Northwestern University, Evanston, IL, 60208 USA  Mattia Checchin
Superconducting Quantum Materials and Systems Division, Fermi National
Accelerator Laboratory (FNAL), Batavia, IL 60510, USA  Anna Grasselino
Superconducting Quantum Materials and Systems Division, Fermi National
Accelerator Laboratory (FNAL), Batavia, IL 60510, USA  Roberto dos Reis
Department of Materials Science and Engineering, Northwestern University,
Evanston, IL, 60208 USA  Vinayak P. Dravid  Department of Materials Science
and Engineering, Northwestern University, Evanston, IL, 60208 USA
<EMAIL_ADDRESS>Alexander Romanenko  Superconducting Quantum
Materials and Systems Division, Fermi National Accelerator Laboratory (FNAL),
Batavia, IL 60510, USA<EMAIL_ADDRESS>Akshay A. Murthy  ∗,† Paul Masih Das
‡ Stephanie M. Ribet  ‡,¶ Cameron Kopas  § Jaeyel Lee  †
Matthew J. Reagor  § Lin Zhou  ∥ Matthew J. Kramer  ∥ Mark C. Hersam  ‡,⟂,#
Mattia Checchin  † Anna Grassellino  † Roberto dos Reis  ‡,@ Vinayak P. Dravid
∗,‡,@,¶

( †Superconducting Quantum Materials and Systems Division, Fermi National
Accelerator Laboratory (FNAL), Batavia, IL 60510, USA
‡epartment of Materials Science and Engineering, Northwestern University,
Evanston, IL 60208, USA
¶International Institute of Nanotechnology, Northwestern University, Evanston,
IL 60208, USA
§Rigetti Computing, Berkeley, CA 94710, USA
∥Ames Laboratory, U.S. Department of Energy, Ames, IA 50011, United States
⟂Department of Chemistry, Northwestern University, Evanston, IL 60208
#Department of Electrical and Computer Engineering, Northwestern University,
Evanston, IL 60208
@The NUANCE Center, Northwestern University, Evanston, IL 60208, USA

∗Corresponding Authors:
Akshay Murthy<EMAIL_ADDRESS>
Vinayak P. Dravid<EMAIL_ADDRESS>
Alexander Romanenko<EMAIL_ADDRESS>)

###### Abstract

Superconducting thin films of niobium have been extensively employed in
transmon qubit architectures. Although these architectures have demonstrated
remarkable improvements in recent years, further improvements in performance
through materials engineering will aid in large-scale deployment. Here, we use
information retrieved from secondary ion mass spectrometry and electron
microscopy to conduct a detailed assessment of the surface oxide that forms in
ambient conditions for transmon test qubit devices patterned from a niobium
film. We observe that this oxide exhibits a varying stoichiometry with NbO and
NbO2 found closer to the niobium film and Nb2O5 found closer to the surface.
In terms of structural analysis, we find that the Nb2O5 region is
semicrystalline in nature and exhibits randomly oriented grains on the order
of 1-2 nm corresponding to monoclinic N-Nb2O5 that are dispersed throughout an
amorphous matrix. Using fluctuation electron microscopy, we are able to map
the relative crystallinity in the Nb2O5 region with nanometer spatial
resolution. Through this correlative method, we observe that amorphous regions
are more likely to contain oxygen vacancies and exhibit weaker bonds between
the niobium and oxygen atoms. Based on these findings, we expect that oxygen
vacancies likely serve as a decoherence mechanism in quantum systems.

International Institute of Nanotechnology, Northwestern University, Evanston,
IL, 60208 USA Department of Chemistry, Northwestern University, Evanston, IL
60208 Department of Electrical and Computer Engineering, Northwestern
University, Evanston, IL 60208 The NUANCE Center, Northwestern University,
Evanston, IL, 60208 USA The NUANCE Center, Northwestern University, Evanston,
IL, 60208 USA International Institute of Nanotechnology, Northwestern
University, Evanston, IL, 60208 USA

KEYWORDS: superconducting qubits, 4D-STEM, electron diffraction, Nb thin
films, interfaces, hydrides, decoherence mechanisms

##  1 Introduction

Over the last two decades, many significant advances have been made towards
constructing large-scale quantum computers. In particular, superconducting
quantum information technology has emerged as a leading architecture to
interrogate complex problems commonly deemed intractable with the most
efficient classical computing platforms 1, 2, 3, 4. Nonetheless, extending
this technology to large-scale devices requires continued progress to improve
reliability and performance. Such improvements require higher quality
materials and specifically, an increased understanding and control over
imperfections including interfaces and surfaces 5.

As an example, in the case of niobium (Nb)-superconducting transmon qubits,
the amorphous surface oxides that form upon ambient exposure serve as major
sources of microwave dissipation. At milliKelvin (mK) operating temperatures,
they also display loss tangent values that are three orders of magnitude
larger compared to the Nb thin films and Si substrates 6, 7, 8. This loss can
be largely attributed to two-state defects in the amorphous surface oxide,
i.e. two-level system (TLS) defects 6. These states emerge as a result of
deviations from long-range order. The system is able to transition between the
two states through low energy excitations at the operating temperatures for
superconducting transmon qubit devices 9, 10. While the density of these TLS
can be calculated through spectroscopic methods 11 and several TLS noise
mechanisms have been proposed (tunneling of atoms,12 tunneling of electrons,13
paramagnetic spins14, quasiparticle traps 15), an understanding of the atomic
defects or nanoscale deviations from which this noise fundamentally arises
remains unclear.

A combination of various spectroscopy and microscopy techniques have led
researchers to understand that deviations from crystalline order on the
nanoscale introduce quantum decoherence 10. As such, scanning transmission
electron microscopy (STEM) is an indispensable tool for identification of
materials that host TLS defects in a broader effort to eliminate these
sources. Namely, the ability to procure a variety of analytical and
spectroscopic signals from nanometric volumes allows for detection of
chemical, structural, and electromagnetic fluctuations on these relevant
length scales 16, 17. We employ a combination of electron microscopy and time-
of-flight secondary ion mass spectrometry (TOF-SIMS) to identify nanoscale
defects and chemical inhomogeneities within the surface oxide associated with
a transmon test qubit device. The use of cutting edge techniques to map the
relative crystallinity allows us to link the presence of sub-stoichiometric
oxygen concentrations to local disorder, which suggests that oxygen vacancies
may serve as a decoherence mechanism in niobium-based quantum systems.

##  2 Results and Discussion

Nb transmon test qubits were fabricated following the procedures detailed by
Nersisyan et al. 18 This involved preparation of a Si (001) wafer (float-zone
$>$10,000 Ohm-cm) with an RCA surface treatment 19, 18, followed by deposition
of Nb films via high-power impulse magnetron sputtering (HiPIMS) with a base
pressure less than 1E-8 Torr at room temperature. TOF-SIMS measurements were
performed using a dual beam system (IONTOF 5) to analyze the concentration and
depth distribution of oxygen in the Nb film. Secondary ion measurements were
performed using a liquid bismuth ion beam (Bi+). A cesium ion gun with an
energy of 500 eV was used for sputtering the surface for depth profile
measurements to detect anions. TEM samples were prepared from the Nb contact
pad using a 30 kV focused Ga+ ion beam. We first provide an assessment of the
chemical composition of the Nb surface oxide before discussing nanoscale
inhomogeneities in this region using advanced electron microscopy techniques.

###  2.1 Surface Oxide Chemical Composition

Figure 1: TOF-SIMS depth profiles taken from Nb thin films. (a) Normalized
oxygen signal is plotted as a function of depth taken from a Nb pad in a
transmon test qubit. (b) Different NbOx signals are plotted as a function of
depth for the top 10 nm of a Nb pad in a transmon test qubit. We observe that
the Nb2O5 signal peaks closer to the surface whereas the NbO2 and NbO signals
persist further into the film. (c) Nb2O5 signal is plotted as a function of
depth taken from a Nb pad in a transmon test qubit and an unprocessed Nb thin
film deposited with HiPIMS. We find that the lithography and etching processes
performed following film deposition lead to inward diffusion of oxygen.

TOF-SIMS depth profiles were taken from transmon test qubit devices and the
averaged profiles are provided in Figure 1. TOF-SIMS is a valuable method for
examining impurities such as oxygen in these films as it combines sensitivity
to light elements and high mass resolution with $<$100 nm spatial resolution.
From these measurements (Figure 1a), we observe an appreciable content of
oxygen in the immediate vicinity of the surface, which corresponds to the
surface oxide as discussed previously 5. We also observe that this oxygen
signal decays to a baseline level of roughly 0.5 O-/Nb- counts, which we
expect results from the incorporation of oxygen interstitial atoms in the Nb
film 20.

Further, TOF-SIMS enables the identification of different oxide species based
on their mass to charge (m/z) ratios. While direct quantification of these
oxides can be challenging due to the possibility for molecular fragmentation
prior to detection by the analyzer, monitoring these signals provides
qualitative information regarding their spatial location within the oxide
layer 21. In Figure 1b, we observe that the signal associated with Nb2O5
exhibits a peak closer to the surface whereas the NbO2 and NbO signals persist
further into the film. This is in agreement with the literature indicating a
change in valence state for the Nb throughout the oxide 22, 21. As the
presence of Nb2O5 has been specifically linked to decoherence in Nb
superconducting qubit systems 6, we focus our analysis to this region
throughout the text.

To get a better understanding for how this oxide evolves with qubit device
processing, we compare the Nb2O5 signal taken from Nb thin films prior to the
lithography and etching processes required to fabricate transmon qubits as
well as that taken from similar processed test qubit devices. The averaged
signals for each are plotted in Figure 1c. We observe that device processing
steps following film deposition lead to oxygen diffusion into the Nb film.
Such an effect has been previously observed in Nb superconducting
radiofrequency (SRF) cavities 23. In that situation, the authors observed that
this oxygen diffusion led to a significant degradation in performance and was
likely caused by an increased oxygen vacancy concentration. As the two
research areas are intrinsically linked, observations in the SRF cavity arena
can help inform methods to improve superconducting qubit performance metrics
as well.

###  2.2 Structural and Chemical Analysis

To probe the surface oxide further, we performed electron microscopy and
diffraction analysis on a cross-section taken from the contact pad of the
transmon test qubit (Figure 2a). An annular dark field (ADF) image of this TEM
lamella using collection angles varying between 10 mrad to 70 mrad is provided
in Figure 2b with the NbOx region labeled. From this image, we observe that
the Nb film exhibits columnar grains that are roughly 50 nm by 170 nm and are
characteristic of the HiPIMS process. A diffraction pattern taken when the Nb
film was tilted along the [111] zone axis is provided as an inset.

Figure 2: Electron microscopy of transmon test qubit (a) Schematic of transmon
qubit architecture. Black box represents the Nb contact pad region
investigated in this study. (b) Annular dark field image taken from a
representative, cross-sectional TEM sample. The surface oxide is indicated. A
diffraction pattern taken from the Nb metal film is provided as an inset.

To clearly identify the structural and chemical features associated with the
surface oxide, a dark field STEM image of this area and the underlying Nb thin
film is provided in Figure 3a. This image was constructed by applying a
virtual detector with an inner collection angle of 10 mrad and an outer
collection angle of 15 mrad, which was designed specifically to match the
initial diffraction ring observed in the NbOx diffraction pattern (Figure S2).
Representative diffraction patterns captured from the indicated regions in the
dark field image in Figure 3a are provided in Figure 3d-h. We observe a
combination of intense diffraction spots as well as broad diffuse rings. Based
on comparison with a simulated diffraction pattern, we conclude that these
spots likely arise from N-Nb2O5, which exhibits a monoclinic structure (Figure
S3) 24. These spots are present for spots 1-4 in Figure 3a, which suggests
that this area corresponds to Nb2O5. Conversely, the broad diffuse rings are
indicative of a lack of long-range order. The appearance of both of these
features suggests that Nb2O5 is semi-crystalline in nature and once again we
focus our analysis to this region.

To gain a qualitative understanding of the size of the crystallites in the
Nb2O5, we implement virtual apertures, or binary masks, to each captured
diffraction pattern in order to selectively filter signal associated with a
few of the most prominent diffraction spots. This use of virtual apertures
produces the dark-field image in Figure S3. Based on this image, it appears
that the N-Nb2O5 crystallites are roughly around 1-3 nm in dimension and
distributed throughout the entire oxide layer.

Figure 3: Nb/O interface (a) DF image of Nb oxide/Nb constructed using a
virtual detector that matched the diffraction ring of Nb oxide seen in Figure
S2b. (b-f) Diffraction patterns taken from regions 1, 2, 3, 4, and 5,
respectively. The distinct diffraction spots indicated by the arrows are
associated with N-Nb2O5 as indicated by the simulation provided in Figure S3.
As these spots fade in intensity from region 1 to region 5, it is apparent
that the Nb2O5 content decreases in the direction away from the surface. (g)
Comparison between experimental RDF as a function of position and simulated
RDF for crystalline Nb2O5. This simulated profile is also decomposed into a
pair distribution function to understand which pairs of atoms give rise to the
features observed.

To better understand the nature of the amorphous regions within the Nb2O5, we
need to employ specialized methods as conventional diffraction and analysis
techniques are insufficient in providing nanoscale structural information from
these regions. Instead fluctuation electron microscopy (FEM) allows us to
better understand the medium range ordering present in the sample. This
analysis was conducted following the approaches detailed previously. 25, 26,
27, 28. First, this process involves using a 1 nm probe and capturing
diffraction patterns as a function of position. Following elliptical
correction, the radial intensity was then calculated as a function of
scattering angle. By subtracting the contribution associated with the thermal
motion of atoms as well as the scattering resulting from individual atoms, we
were able to calculate the scattering factor as a function of position.
Finally upon performing a Fourier transform, the radial distribution function
(RDF) as a function of real-space distance is accessible. An RDF profile
represents the probability of finding an atom $r$ distance away from a
reference atom. As such, atomic bond distances will produce peaks in the
profile.

Figure 4: Fluctuation Electron Microscopy of Surface Oxide (a) DF image of Nb
oxide/Nb constructed using a virtual detector. (b) Line profile of EDS
intensity as a function of position. (c-h) 2D maps and averaged 1D profiles of
various peak parameters for Nb-O bond through the Nb2O5 region. Peak amplitude
values are provided in c-d. Peak FWHM values are provided in e-f. Peak center
values are provided in g-h. Regions with a relatively large peak FWHM values
are indicative of relatively disordered regions. A few of these regions are
circled in (e). Based on the associated peak center and peak amplitude maps,
it appears that disordered regions in the Nb2O5 are more likely to contain
reduced oxygen concentrations and larger bond distances between Nb and O.

The RDF calculated for positions 1, 3, and 5 in the dark field image are seen
in Figure 3c. From these profiles, we observe that the most prominent feature
is centered about 1.8 Å. Based on the calculated radial distribution function
profile for crystalline Nb2O5 (Figure S1) 29, this primary feature is expected
to arise from Nb-O bonding. As the experimental RDF is obtained from a
semicrystalline region, disorder arising from local deviations in the bond
length and bond angles introduce the experimental peak broadening observed
compared to the simulated profile. Nonetheless, we can monitor how this
feature evolves throughout the oxide to obtain information associated with the
nature of the Nb-O bonds in this region.

To verify the assignment of this feature in the RDF profile to the Nb-O bond,
we map and provide the averaged line profiles of various parameters associated
with this peak in Figure 4 for the Nb2O5 region. Specifically, the parameters
of the Nb-O peak provide information about the distribution of Nb-O bonds as a
function of position. For instance, in addition to the peak amplitude
representing the local density of Nb-O bonds, the peak full width at half
maximum (FWHM) provides insight into the degree of local order or disorder,
and the peak center provides information regarding the local bond distances.
The Nb-O feature was fitted using a Gaussian distribution (details are
provided in SI). From Figure 4d, we observe that the amplitude of this feature
steadily decreases beginning at the surface in the direction towards the Nb
film. This is in agreement with the EDS line profile provided in Figure 4b as
well as the TOF-SIMS data discussed previously. In conjunction with the
simulated RDF profiles, this further corroborates that this primary feature
arises from Nb-O bonding.

Having established that this initial peak represents the Nb-O bond, we
calculate and plot additional peak parameters which provide information on the
relative crystallinity and bond distances. With respect to the peak FWHM, we
observe that this parameter varies throughout the Nb2O5 region. In Figure 3e,
regions exhibiting large FWHM values are circled. These areas correspond to
regions that are highly disordered and as discussed previously, more prone to
hosting TLS. Upon evaluation of the peak amplitudes present in these
disordered regions (circled in Figure 3c), we find that local disorder tends
to be linked to Nb-O peak amplitude suppression, which potentially arises
through an increase in oxygen vacancy concentration as suggested by the TOF-
SIMS measurements. We find that the cross-correlation R2 value between these
two parameters in the Nb2O5 region is 0.29. Similarly, we observe that these
disordered regions tend to exhibit an increased peak center as well (Figure
3g). This is indicative of weaker bonds between Nb and O and in this case, we
find that the R2 value between these two parameters within the Nb2O5 region is
roughly 0.22.

Since TLS likely arise from disordered regions in Nb2O5, the correlation in
the peak parameters suggests that in the case of this system, TLS may arise
due to the presence of oxygen vacancies. Further, the weak Nb-O bonds in this
region may assist in permitting atomic tunneling at low temperatures 12. As
theoretical work 30 and experimental findings 31 have suggested that oxygen
vacancies introduce local paramagnetic moments, such a tunneling process at
the qubit operating temperatures is expected to introduce electromagnetic
noise and potentially lead to quantum decoherence.

##  3 Concluding Remarks

In this study, we applied TOF-SIMS and STEM imaging and diffraction methods to
identify structural inhomogeneities and defects in the surface oxide
associated with a transmon test qubit. Using these methods, we observe that
this oxide exhibits a varying stoichiometry with NbO and NbO2 found closer to
the niobium film and Nb2O5 found closer to the surface. Further, we find that
the Nb2O5 region is semicrystalline in nature and exhibits randomly oriented
grains on the order of 1-2 nm corresponding to N-Nb2O5 that are dispersed
through an amorphous matrix. By subsequently mapping the relative
crystallinity in the Nb2O5 region, we observe that amorphous regions are more
likely to contain oxygen vacancies and exhibit weaker bonds between the
niobium and oxygen atoms. Together, this work suggests that oxygen vacancies
in the surface oxide likely serve as a decoherence mechanism in quantum
systems. As a result, identifying methodologies for passivation of oxygen
vacancies or preventing the formation of this lossy surface oxide are critical
to improving coherence times in superconducting qubits.

##  4 Methods

### TEM Sample Preparation

TEM samples were prepared from the Nb electrode using a 30 kV focused Ga+ ion
beam. In order to protect the surface oxide during the ion milling process,
the sample was first coated with 50 nm of Ni. The samples were finely polished
to a thickness of roughly 50 nm using 5 kV and 2 kV Ga+ ions in an effort to
remove surface damage and amorphization in the regions of interest.

### STEM Data Collection

S/TEM data was collected on JEOL 300F Grand ARM S/TEM using an accelerating
voltage of 300kV for the low magnification analysis of the Nb film. In this
case, the camera length is set to 80 cm and the condenser aperture is selected
to provide a convergence semi-angle of 10 mrad. Four-dimensional STEM (4DSTEM)
data sets 16 were acquired in a 360 x 62 mesh (with pixel size of 3.33 nm)
across the Nb films using a Gatan® OneView camera and synchronized using
STEMx. In the case of the Nb surface oxide, an accelerating voltage of 80kV
was selected to reduce beam damage in the amorphous compounds and data
acquired using a Gatan® Stela hybrid-pixelated camera. STEM-EDS measurements
were collected at 200kV using a JEOL Silicon Drift Detector (SDD) with a solid
angle of 1.7 sr.

### Fluctuation Electron Microscopy

A condenser aperture was selected to provide a convergence semi-angle of 1.5
mrad for fluctuation electron microscopy measurements. This yields a probe
size that is roughly 1 nm and allows for characterizing amorphous regions that
are on the order of 5 nm in dimension. The resultant image was subsequently
passed through a Gaussian filter with a filter size of 2 pixels to eliminate
scanning distortions. Analysis of scanning diffraction-based fluctuation
electron microscopy measurements was performed using the py4DSTEM package 28
per the methods outlined by Souza Junior et al. 32. Upon obtaining diffraction
patterns from the regions of interest, the diffraction patterns were fit
radially using least squares fitting of the first diffraction ring. The radial
intensity was then calculated as a function of scattering angle. By
subtracting the contribution associated with the thermal motion of atoms as
well as the scattering resulting from individual atoms, we were able to
calculate the scattering factor as a function of position as discussed. This
contribution was approximated by fitting these functions to the tail of the
radial intensity and this process yields the reduced structure factor. Finally
upon performing a Fourier transform, the radial distribution function (RDF) as
a function of real-space distance can be obtained. The initial peak in the RDF
was fit with a Gaussian lineshape. The parameters of this Gaussian function
are reported.

This material is based upon work supported by the U.S. Department of Energy,
Office of Science, National Quantum Information Science Research Centers,
Superconducting Quantum Materials and Systems Center (SQMS) under the contract
No. DE-AC02-07CH11359. Fermilab is operated by the Fermi Research Alliance,
LLC under contract No. DE-AC02-07CH11359 with the United States Department of
Energy. This work made use of the EPIC facility of Northwestern University’s
NUANCE Center, which received support from the Soft and Hybrid Nanotechnology
Experimental (SHyNE) Resource (NSF ECCS-1542205); the MRSEC program (NSF
DMR-1720139) at the Materials Research Center; the International Institute for
Nanotechnology (IIN); the Keck Foundation; and the State of Illinois, through
the IIN. The authors thank members of the Superconducting Quantum Materials
and Systems (SQMS) Center for valuable discussion. S. M. R. gratefully
acknowledges support from IIN and 3M. The authors thank Dr. Anahita Pakzad
from Ametek/Gatan, Inc, Pleasanton, CA, for the valuable feedback on the usage
of the Stela pixelated detector.

##  5 Author Contributions

The manuscript was written through contributions of all authors. All authors
have given approval to the final version of the manuscript.

##  6 Competing Financial Interests:

The authors declare no competing financial interests.

EDS map, evolution of diffraction patterns in Nb oxide along with simulated
diffraction patterns associated with Nb2O5 are provided in supporting
information.

## References

  * Kjaergaard et al. 2020 Kjaergaard, M.; Schwartz, M. E.; Braumüller, J.; Krantz, P.; Wang, J. I.-J.; Gustavsson, S.; Oliver, W. D. Superconducting Qubits: Current State of Play. _Annual Review of Condensed Matter Physics_ 2020, _11_ , 369–395
  * Wendin 2017 Wendin, G. Quantum information processing with superconducting circuits: a review. _Reports on Progress in Physics_ 2017, _80_ , 106001
  * de Leon et al. 2021 de Leon, N. P.; Itoh, K. M.; Kim, D.; Mehta, K. K.; Northup, T. E.; Paik, H.; Palmer, B. S.; Samarth, N.; Sangtawesin, S.; Steuerman, D. W. Materials challenges and opportunities for quantum computing hardware. _Science_ 2021, _372_
  * Murray 2021 Murray, C. E. Material matters in superconducting qubits. 2021
  * Murthy et al. 2022 Murthy, A. A.; Lee, J.; Kopas, C.; Reagor, M. J.; McFadden, A. P.; Pappas, D. P.; Checchin, M.; Grassellino, A.; Romanenko, A. TOF-SIMS analysis of decoherence sources in superconducting qubits. _Applied Physics Letters_ 2022, _120_ , 044002
  * Romanenko et al. 2020 Romanenko, A.; Pilipenko, R.; Zorzetti, S.; Frolov, D.; Awida, M.; Belomestnykh, S.; Posen, S.; Grassellino, A. Three-Dimensional Superconducting Resonators at T $<$ 20 mK with Photon Lifetimes up to $\tau=2$ s. _Phys. Rev. Applied_ 2020, _13_ , 034032
  * McRae et al. 2020 McRae, C. R. H.; Wang, H.; Gao, J.; Vissers, M. R.; Brecht, T.; Dunsworth, A.; Pappas, D. P.; Mutus, J. Materials loss measurements using superconducting microwave resonators. _Review of Scientific Instruments_ 2020, _91_ , 091101
  * Verjauw et al. 2021 Verjauw, J. et al. Investigation of Microwave Loss Induced by Oxide Regrowth in High-Q Niobium Resonators. _Phys. Rev. Applied_ 2021, _16_ , 014018
  * Premkumar et al. 2020 Premkumar, A. et al. Microscopic Relaxation Channels in Materials for Superconducting Qubits. _arXiv_ 2020,
  * Müller et al. 2019 Müller, C.; Cole, J. H.; Lisenfeld, J. Towards understanding two-level-systems in amorphous solids: insights from quantum circuits. _Reports on Progress in Physics_ 2019, _82_ , 124501
  * Lisenfeld et al. 2019 Lisenfeld, J.; Bilmes, A.; Megrant, A.; Barends, R.; Kelly, J.; Klimov, P.; Weiss, G.; Martinis, J. M.; Ustinov, A. V. Electric field spectroscopy of material defects in transmon qubits. _npj Quantum Information_ 2019, _5_ , 105
  * Martinis et al. 2005 Martinis, J. M.; Cooper, K. B.; McDermott, R.; Steffen, M.; Ansmann, M.; Osborn, K. D.; Cicak, K.; Oh, S.; Pappas, D. P.; Simmonds, R. W.; Yu, C. C. Decoherence in Josephson Qubits from Dielectric Loss. _Phys. Rev. Lett._ 2005, _95_ , 210503
  * Agarwal et al. 2013 Agarwal, K.; Martin, I.; Lukin, M. D.; Demler, E. Polaronic model of two-level systems in amorphous solids. _Phys. Rev. B_ 2013, _87_ , 144201
  * Paladino et al. 2014 Paladino, E.; Galperin, Y. M.; Falci, G.; Altshuler, B. L. $\frac{1}{f}$ noise: Implications for solid-state quantum information. _Rev. Mod. Phys._ 2014, _86_ , 361–418
  * de Graaf et al. 2020 de Graaf, S. E.; Faoro, L.; Ioffe, L. B.; Mahashabde, S.; Burnett, J. J.; Lindström, T.; Kubatkin, S. E.; Danilov, A. V.; Tzalenchuk, A. Y. Two-level systems in superconducting quantum devices due to trapped quasiparticles. _Science Advances_ 2020, _6_ , eabc5055
  * Ophus 2019 Ophus, C. Four-Dimensional Scanning Transmission Electron Microscopy (4D-STEM): From Scanning Nanodiffraction to Ptychography and Beyond. _Microscopy and Microanalysis_ 2019, _25_ , 563–582
  * Ribet et al. 2021 Ribet, S. M.; Murthy, A. A.; Roth, E. W.; dos Reis, R.; Dravid, V. P. Making the most of your electrons: Challenges and opportunities in characterizing hybrid interfaces with STEM. _Materials Today_ 2021,
  * Nersisyan et al. 2019 Nersisyan, A.; Poletto, S.; Alidoust, N.; Manenti, R.; Renzas, R.; Bui, C.-V.; Vu, K.; Whyland, T.; Mohan, Y.; Sete, E. A.; Stanwyck, S.; Bestwick, A.; Reagor, M. Manufacturing low dissipation superconducting quantum processors. 2019 IEEE International Electron Devices Meeting (IEDM). 2019; pp 31.1.1–31.1.4
  * Vissers et al. 2010 Vissers, M. R.; Gao, J.; Wisbey, D. S.; Hite, D. A.; Tsuei, C. C.; Corcoles, A. D.; Steffen, M.; Pappas, D. P. Low loss superconducting titanium nitride coplanar waveguide resonators. _Applied Physics Letters_ 2010, _97_ , 232509
  * Koch et al. 1974 Koch, C. C.; Scarbrough, J. O.; Kroeger, D. M. Effects of interstitial oxygen on the superconductivity of niobium. _Phys. Rev. B_ 1974, _9_ , 888–897
  * Bose et al. 2020 Bose, A.; Mondal, P.; Bhalerao, G.; Kokil, S.; Raghavendra, S.; Joshi, S.; Srivastava, A.; Tewari, R. Evolution of surface oxides and impurities in high vacuum heat treated Nb: A TEM and TOF-SIMS in-situ study, mechanism and repercussions on SRF cavity applications. _Applied Surface Science_ 2020, _510_ , 145464
  * Lindau and Spicer 1974 Lindau, I.; Spicer, W. E. Oxidation of Nb as studied by the uv‐photoemission technique. _Journal of Applied Physics_ 1974, _45_ , 3720–3725
  * Bafia et al. 2021 Bafia, D.; Grassellino, A.; Romanenko, A. Probing the Role of Low Temperature Vacuum Baking on Photon Lifetimes in Superconducting Niobium 3-D Resonators. 2021; https://arxiv.org/abs/2108.13352
  * Nico et al. 2016 Nico, C.; Monteiro, T.; Graça, M. Niobium oxides and niobates physical properties: Review and prospects. _Progress in Materials Science_ 2016, _80_ , 1–37
  * Kennedy et al. 2020 Kennedy, E.; Reynolds, N.; Rangel DaCosta, L.; Hellman, F.; Ophus, C.; Scott, M. C. Tilted fluctuation electron microscopy. _Applied Physics Letters_ 2020, _117_ , 091903
  * Harrelson et al. 2021 Harrelson, T. F.; Sheridan, E.; Kennedy, E.; Vinson, J.; N’Diaye, A. T.; Altoé, M. V. P.; Schwartzberg, A.; Siddiqi, I.; Ogletree, D. F.; Scott, M. C.; Griffin, S. M. Elucidating the local atomic and electronic structure of amorphous oxidized superconducting niobium films. _Applied Physics Letters_ 2021, _119_ , 244004
  * Mu et al. 2016 Mu, X.; Wang, D.; Feng, T.; Kübel, C. Radial distribution function imaging by STEM diffraction: Phase mapping and analysis of heterogeneous nanostructured glasses. _Ultramicroscopy_ 2016, _168_ , 1–6
  * Savitzky et al. 2021 Savitzky, B. H.; Zeltmann, S. E.; Hughes, L. A.; Brown, H. G.; Zhao, S.; Pelz, P. M.; Pekin, T. C.; Barnard, E. S.; Donohue, J.; Rangel DaCosta, L.; et al., py4DSTEM: A Software Package for Four-Dimensional Scanning Transmission Electron Microscopy Data Analysis. _Microscopy and Microanalysis_ 2021, _27_ , 712–743
  * Farrow et al. 2007 Farrow, C. L.; Juhas, P.; Liu, J. W.; Bryndin, D.; Božin, E. S.; Bloch, J.; Proffen, T.; Billinge, S. J. L. PDFfit2 and PDFgui: computer programs for studying nanostructure in crystals. _Journal of Physics: Condensed Matter_ 2007, _19_ , 335219
  * Sheridan et al. 2021 Sheridan, E.; Harrelson, T. F.; Sivonxay, E.; Persson, K. A.; Altoé, M. V. P.; Siddiqi, I.; Ogletree, D. F.; Santiago, D. I.; Griffin, S. M. Microscopic Theory of Magnetic Disorder-Induced Decoherence in Superconducting Nb Films. 2021
  * Proslier et al. 2011 Proslier, T.; Kharitonov, M.; Pellin, M.; Zasadzinski, J.; Ciovati, Evidence of Surface Paramagnetism in Niobium and Consequences for the Superconducting Cavity Surface Impedance. _IEEE Transactions on Applied Superconductivity_ 2011, _21_ , 2619–2622
  * Souza Junior et al. 2021 Souza Junior, J. B.; Schleder, G. R.; Bettini, J.; Nogueira, I. C.; Fazzio, A.; Leite, E. R. Pair Distribution Function Obtained from Electron Diffraction: An Advanced Real-Space Structural Characterization Tool. _Matter_ 2021, _4_ , 441–460

###  S0.1 Supplementary Information

##  S1 Developing a Chemical and Structural Understanding of the Surface Oxide
in a Niobium Superconducting Qubit

and Alexander Romanenko∗,†

KEYWORDS: superconducting qubits, 4D-STEM, electron diffraction, Nb thin
films, interfaces, hydrides, decoherence mechanisms

Table of contents:
S1: Energy dispersive spectroscopy map of the Nb surface oxide
S2: Diffraction patterns taken from Nb oxide
S3: Diffraction patterns taken from Nb/Si interface

Figure S1: Energy dispersive spectroscopy map of the Nb surface oxide. Figure
S2: (a) DF image of Nb oxide/Nb constructed using a virtual detector that
matched the diffraction ring of Nb oxide. The dotted arrows indicate regions
over which changes in the diffraction pattern are provided in (c-f). (b)
Diffraction pattern taken from NbOx with a virtual annular detector with an
inner collection angle of 10 mrad and an outer collection angle of 15 mrad to
preferentially produce a NbOx dark field image. (c-f) Evolution of the
electron diffraction pattern in the direction of the arrows indicated in (a)
for the labeled regions 1, 2, 3, 4, and 5, respectively. In all cases,
prominent diffraction spots tend to decay in intensity when moving away from
the surface. Figure S3: (a) Representative diffraction pattern of Nb2O5 taken
from region 3 in Figure 3. (b) Simulated diffraction pattern for N-Nb2O5.
Based on the consistency in the location of the diffraction rings, we
hypothesize that the crystalline features that form exhibit this phase. (c)
Precipitate map produced by placing the virtual apertures seen in (a). We
observe that the precipitates in this oxide are between 1-3 nm in lateral
dimension.


# Detection and Measurement of Syntactic Templates
in Generated Text

Chantal Shaib1 Yanai Elazar2,3 Junyi Jessy Li4 Byron C. Wallace1
1Northeastern University, 2Allen Institute for AI,
3University of Washington, 4The University of Texas at Austin
{shaib.c<EMAIL_ADDRESS>

###### Abstract

Recent work on evaluating the diversity of text generated by LLMs has focused
on word-level features. Here we offer an analysis of syntactic features to
characterize general repetition in models, beyond frequent $n$-grams.
Specifically, we define syntactic templates and show that models tend to
produce templated text in downstream tasks at a higher rate than what is found
in human-reference texts. We find that most (76%) templates in model-generated
text can be found in pre-training data (compared to only 35% of human-authored
text), and are not overwritten during fine-tuning processes such as RLHF. This
connection to the pre-training data allows us to analyze syntactic templates
in models where we do not have the pre-training data. We also find that
templates as features are able to differentiate between models, tasks, and
domains, and are useful for qualitatively evaluating common model
constructions. Finally, we demonstrate the use of templates as a useful tool
for analyzing style memorization of training data in LLMs.

##  1 Introduction

An open question about LLMs is what patterns such models learn from pre-
training data (Goldberg, 2019; Petroni et al., 2019; Bender et al., 2021; Chen
et al., 2024), and whether these patterns generalize across downstream tasks
and datasets (Hupkes et al., 2023). While prior work has focused on the
quality of generation (Zhang et al., 2019; Dou et al., 2022; Kryściński et
al., 2020), and more recently on text generation novelty (McCoy et al., 2023),
there has been limited work on characterizing the sorts of patterns that are
learned by LLMs.

Consider, for instance, the text from OLMo-Instruct in Figure 1, which is
sampled from a corpus of generated movie review summaries. This was produced
by prompting the model to summarize a collection of human-written movie
reviews: “ _The Last Black Man in San Francisco is a poignant, beautifully
shot film […] creates a unique and intense viewing experience […]_ ”. In this
example, we find a total of 35 repeated sequences of part-of-speech (POS) tags
of lengths $n=5$ to $8$. While the generated text was not seen in OLMo’s
training data, we find that 33 out of the 35 (95%) sequences appear in the
pre-training data. As such, while the generated text itself is novel, it
relies on common syntactic sequences learned from the training data.

Figure 1: Sample movie meta-reviews generated by OLMo-7B (top) and Mistral-7B
(bottom) by prompting the Rotten Tomatoes dataset. Templates appear at varying
rates, and differ across models. We extract templates from the entire corpus
of generated text for each model, and match the text to the part-of-speech
templates (highlighted), following by the frequency of each template.

In this work we quantify and measure LLMs’ usages of repetitive sequences in
text generation. We introduce and focus on _syntactic templates_ , namely POS
sequences, a syntactic abstraction over texts that showcase diverse sets of
patterns. We begin by establishing the connection between frequent templates
in the training data and models’ reliance on such templates during generation.
Using this finding, we draw a direct connection between training data template
statistics and the tendency of models trained on this data to generate these
templates. We then evaluate several closed-source models and the syntactic
templates they are prone to generating, which teaches us about properties of
their training data.

We start by introducing syntactic templates, and defining methods for
detecting and measuring such templates in generated texts (§3). Given this
definition, we seek to answer the following questions:

RQ1 How templated are outputs generated by instruction-tuned LLMs?

RQ2 Can we locate model-generated templates in (pre-)training data?

RQ3 Can syntactic templates be used for detection of data memorization?

In all, we evaluate eight models on three different tasks (§5). We show how
training data templates are memorized and subsequently generated by models
trained on them (§6). We then show how such insights allow one to draw
conclusions about the training data used by closed models in a downstream
summarization task (§7). Finally, we show that our metrics can also be used as
a softer version of memorization. For instance, while Carlini et al. (2022)
estimates that 1% of texts to be memorized, we find between 0.8 - 3.1% more
soft-memorized texts over verbatim memorization, often by replacing numbers
and generating synonyms (§8).

##  2 Related Work

#### Diversity in Text Generation

Past evaluations of diversity in LLM outputs have primarily focused on token-
level diversity (Montahaei et al., 2019; Bache et al., 2013). Diverse sampling
strategies have been introduced to address the lower token diversity observed
in neural text generation (Holtzman et al., 2020; Roberts et al., 2020),
however it is unclear whether such sampling strategies also increase the
diversity of the syntactic structure in LLMs. Beyond lexical diversity,
Padmakumar and He (2023) extend definitions for measuring content diversity,
which has broad applications in downstream tasks such as summarization and
creative story generation. Recent work has quantified the drop in generated-
text diversity specifically relative to the RLHF training process, however
this again focused primarily on token level diversity (Kirk et al., 2024). Our
work aligns more closely with the first body of work; we measure the syntactic
structure of text rather than its semantic content. Most similar to our work
is Bär et al. (2012), which broadly evaluates text repetition metrics at the
stylistic, content, and lexical level.

#### Structural Analysis of Text

DiMarco and Hirst (1993) provide a computational approach comprising lexical
and syntactic components to describe stylistic elements in model-generated
text. The discussion around style in writing has been adopted broadly for a
variety of downstream tasks such as author or model attribution (Wu et al.,
2023; Lample et al., 2018; Rosenfeld and Lazebnik, 2024). While our main goal
is to provide measurements and characterizations of repetitive syntactic
features in text, definitions of stylistic elements are closely related and
help contextualize our findings. One can use our definitions of templates to
ask broader questions about the prevailing syntactic style in a given corpus.
Indeed, recent works adopt various measures of linguistic analysis to address
differences in writing style in both human-written and model-generated texts
(Krishna et al., 2020; Soler-Company and Wanner, 2017).

#### AI Text Detection

In identifying $n$-gram features that appear in high frequencies in model-
generated text, a natural question arises as to whether such features can be
used to reliably _detect_ model-generated text. Prior work has established
that this is difficult, and that text-level features at the corpus level
correlate with text being model-generated (Liang et al., 2024a, b). In this
work, we make no claims for the use of templates in AI-text detection. Our aim
is to _characterize_ patterns rather than _detect_ generated outputs, and to
provide a basis for future work on model linguistic diversity.

##  3 Detecting Syntactic Templates

Our goal is to propose an abstract representation of texts, that is able to
capture more subtle repetitions than mere text memorization. Repeated runs of
literal tokens may not be sufficient for describing such redundancy nor why a
summary produced by, e.g. ChatGPT, might seem familiar.

Focusing on syntactic patterns rather than tokens allows us to capture such
repetitions. For example, a pattern consisting of the part-of-speech sequence
DT JJ NN IN DT JJ NN will match to phrases in movie reviews (“a romantic
comedy about a corporate executive”) and in news summarization (“a humorous
insight into the perceived class”), despite these sentences having only one
token overlap.

###  3.1 Defining Templates

Given a sequence of tokens $T=(t_{1},t_{2},\ldots,t_{n})$, and a function $f$
that computes an abstraction over $T$ (e.g., part-of-speech tags), we define a
_template_ as a sub-sequence of abstractions over the tokens $f(T)$ that
repeats at least $\tau$ times in $T$. Figure 1 shows examples of templates and
their counts across the Rotten Tomatoes dataset (Leone, 2020).

###  3.2 Extracting Syntactic Templates from Text

We operationalize the definition in 3.1 as parts-of-speech (POS). For sub-
sequences of POS we consider templates of length $n\in\\{4,5,6,7,8\\}$.
Templates are characterized by their high frequency across the texts in a
given corpus (e.g., one comprising texts generated by a particular LLM).
Practically, we choose $\tau$ relative to the sample size. For Rotten Tomatoes
we retain the top 100 most common template where the least frequent template
appears 4 times in the dataset.

To extract templates we use diversity (Shaib et al.,
2024),111https://pypi.org/project/diversity/ a library providing tools to
evaluate token and POS diversity in a dataset. We use this tool to first tag
all tokens in a corpus with their corresponding POS tags, then search for the
top 100 most frequent $n$-grams across these tags. diversity uses the NLTK POS
tagger,222https://www.nltk.org/api/nltk.tag.html which relies on the Penn
Treebank set of 36 tags (Taylor et al., 2003). After tagging, we use the tool
to return frequent $n$-grams, and the corresponding matched text.

The pipeline for identifying templates can be further extended to other
tagging libraries and types. For example, we also explore constituency parses
as an alternative to POS tags. Extracting templates and matching over tokens
is non-trivial for constituency parses. We provide examples of the patterns
identified by this abstraction in Appendix A and leave further analysis to
future work.

###  3.3 Metrics for Measuring Templates

Our goal for extracting templates is to assess and characterize different
levels of repetition in LLMs. We use templates to measure three metrics (1)
the diversity of the POS tags that are generated, (2) the fraction of texts
generated with a template, and (3) the number of templates that appear within
each text. We now elaborate on each one.

#### CR-POS.

At the most granular level, we are interested in quantifying the $n$-gram
diversity of the POS tag sequences present in the text. Lossless text
compression algorithms—such as gZip—are optimized to detect repeated
characters in sequences, and rely on this to compress documents without any
loss of information. Prior work shows that using gZip to calculate a
compression ratio (CR) can provide an efficient measure for capturing lexical
diversity, specifically $n$-gram repetition (Shaib et al., 2024).

We calculate CR over a set of POS-tagged text, with higher values indicating
that text is highly compressable (and therefore shows lower diversity). To
calculate the CR, we concatenate all POS-tagged text into a sequence, and
measure the ratio between the original document size and the compressed
document size:

$\text{CR}(f(T\oplus))=\frac{|f(T\oplus)|}{\text{compressed }|f(T\oplus)|}$
(1)

Where $T\oplus$ is the concatenated sequence of text, and $f(T\oplus)$ is POS-
tagged text. Higher compression ratios imply more redundancy in the text, and
therefore lower diversity of the sequence.

#### Template Rate and Count

We measure the fraction of texts that contain at least 1 template to quantify
how frequently templates appear across an entire corpus of text.

#### Templates-per-Token

In practice, text can contain many templates. Measures of diversity are
confounded by text length (Salkar et al., 2022), which also applies to
template counts; if a model tends to produce longer texts, there is a higher
chance that any given output will contain a template. To compare between text
sources, we can length normalize:

$\text{TPT}(T\oplus)=\frac{\frac{1}{T}\sum_{i=1}^{T}\text{\\# templates in
$t_{i}$}}{\frac{1}{T}\sum_{i=1}^{T}\text{\\# words in $t_{i}$}}$ (2)

Where $T$ is a concatenated sequence of tokens forming a corpus for a
particular text source, and $t$ the individual text.

##  4 Experimental Setup

We first evaluate the incidence of templated text in two open-ended generation
tasks using OLMo-7B Instruct (Groeneveld et al., 2024), a completely open
source model that released model training checkpoints and its training data.
This allows us to evaluate templates in its training datasets: Dolma (Soldaini
et al., 2024), Tulu-V2 (Ivison et al., 2023), and Ultra-feedback (Cui et al.,
2023).

We then evaluate templates across closed source models (which do not release
training data), specifically: Mistral, Llama (-2, -3), Alpaca, and GPT-4o.

#### Decoding

While greedy decoding is a common decoding strategy for many popular
downstream generation tasks, one can explicitly control token diversity at
inference time via choice of decoding hyperparameters such as temperature. We
evaluate generation under various decoding strategies and model sizes. We
refer to Wiher et al. (2022) for an in-depth discussion on the impact of
sampling on generated text, and here focus specifically on varying
hyperparameters and resultant impact of the appearance and frequency of
templates. For the former, we use greedy decoding, and separately vary
temperature and top-$p$ for decoding with sampling. Top-$p$ (nucleus) sample
restricts the subset of tokens such that the combined probability reaches a
threshold $p$ (Holtzman et al., 2020).

###  4.1 Fine-tuned (Instruction) Models

We experiment with a total of 8 instruction-tuned models. We use Mistral
(Instruct, 7B; Jiang et al. 2023), Alpaca (7B, 13B; Taori et al. 2023; Wang et
al. 2022; Touvron et al. 2023a). In addition, 5 models are further trained on
human preferences: OLMo (Instruct, 7B; Groeneveld et al. 2024), Llama-2 (Chat-
HF, 7B-70B; Touvron et al. 2023b), and Llama-3 (Instruct, 70B AI@Meta 2024).

###  4.2 Tasks and Datasets

#### Open-Ended Generation

To evaluate intrinsic template behaviour we evaluate over an open-ended
generation task. In the first setting, we sample generations from the model
given only a special token denoting beginning of sequence ([BOS]). In the
second, we randomly sample 100 tokens from Dolma and use these tokens to
prompt further open-ended generation from the model.

#### Synthetic Data Generation

LLMs are increasingly used to create synthetic training datasets (e.g., Wang
et al. 2022). We evaluate templates in Cosmopedia, a synthetic dataset
generated by prompting Mixtral-8x7B-Instruct with instructions to produce text
relating to textbooks, blogposts, stories, posts and WikiHow articles (Ben
Allal et al., 2024). We prompt OLMo-7B with the Cosmopedia instructions and
evaluate the resulting generations.

| Open Generation | Rotten Tomatoes
---|---|---
| Decoding
---
Strategy
| CR:
---
POS
|  $\geq 1$ Templates
---
% ($n=6$)
| CR:
---
POS
|  $\geq 1$ Templates
---
% ($n=6$)
Greedy | 702.8 | 100.0 (0.065) | 6.45 | 97.0 (0.041)
Default | 5.81 | 75.5 (0.009) | 6.33 | 96.6 (0.041)
temp 0.8 | 6.74 | 71.8 (0.007) | 6.26 | 96.6 (0.043)
temp 0.85 | 6.48 | 74.0 (0.007) | 6.22 | 96.6 (0.041)
temp 0.9 | 6.22 | 73.4 (0.009) | 6.17 | 97.4 (0.039)
temp 0.95 | 5.98 | 75.4 (0.010) | 6.14 | 97.2 (0.040)
top_p 0.8 | 7.03 | 76.5 (0.007) | 6.31 | 97.8 (0.041)
top_p 0.85 | 6.71 | 71.0 (0.007) | 6.27 | 96.6 (0.041)
top_p 0.9 | 6.50 | 75.3 (0.008) | 6.22 | 96.2 (0.039)
top_p 0.95 | 6.17 | 77.2 (0.009) | 6.31 | 95.8 (0.041)

Table 1: Compression ratio with POS (CR-POS), average text length and
percentage of generated outputs with at least 1 template of size $n=6$, when
varying temperature and top_p for OLMo-7B decoding.

Dataset |  | CR:
---
POS
|  $\geq 1$ Templates
---
% ($n=6$)
Dolma-100 | 5.65 | 82.6 (0.012)
Cosmopedia | 5.76 | 99.1 (0.014)

Table 2: CR-POS, template-per-token, and template counts for templates of size
$n=6$ reported for OLMo-7B text generated with Cosmopedia Instructions, and
100 sampled tokens from the Dolma dataset, with greedy decoding.

#### Summarization

is a common benchmark for long text generation. We evaluate models on a
handful of summarization datasets, including single- and multi-document tasks.
Such datasets allow us to study templates in longer sequences that would not
be evident in tasks where only a few tokens are generated. For general
English-language tasks, we generate summaries and reviews over news (CNN/Daily
Mail; Nallapati et al. 2016), movies (Rotten Tomatoes; Leone 2020), and books
(BooookScore; Chang et al. 2023).

We also look at templates in the biomedical domain as an example of a domain-
specific task. Cochrane is a dataset of systematic reviews summarizing the
evidence over medical interventions (Wallace et al., 2020). We prompt models
to generate systematic reviews. Importantly, these datasets include human-
written reference summaries, which serve as a baseline to compare our task-
specific template analysis.

##  5 Templates in Model-Generated Text

We first evaluate OLMo-7B Instruct on 4 tasks: open-ended generation,
synthetic data generation, and summarization, using both greedy and varying
temperature and top-$p$ sampling strategies (RQ 1).

Table 1 shows the effect of varying sampling hyperparameters temperature and
top-$p$ on the overall diversity of the generated text with OLMo-7B with open-
generation and summarization. Varying sampling strategies in the open-
generation task results in a higher variance of template rates (74.4% $\pm$
2.1) compared to templates rates in the summarization task (96.8% $\pm$ 0.6).
These results indicate that templatic text in summarization appears in spite
of sampling strategies intended to increase (lexical) diversity.

Overall, the rate of templates is much higher in the Rotten Tomatoes dataset
than for open-generation, indicating downstream tasks such as summarization,
which often entail prompting with instructions, may yield more repetitive
structures.333We show the incidence of templates given different instructions
in Appendix D and find that, while the rates somewhat vary, there is still a
high incidence of templates across all prompts.

Table 2 shows the rate of templates on two additional tasks: Synthetic data
generation and data generation with Dolma Cosmopedia results in a higher
incidence of templates (99.1%) and templates per token (0.014), compared to
Dolma (82.6%, 0.012).

##  6 Searching For Templates in Pre-training Data

One hypothesis for the emergence of templates in generated text is that these
templates are over-represented in the training data (RQ 2). Here we
interrogate this empirically.

###  6.1 Emergence of Templates in Training

We first aim to understand when during training models start to produce
templates. We measure this by looking at the perplexity of a set of identified
templates across OLMo model checkpoints. Higher perplexity values indicate the
templates are assigned low likelihood at that checkpoint. For each model
checkpoint, we average the perplexities of templates of length $n=6$ and
compare to the perplexities of randomly sampled 6-grams. We calculate the
average perplexity for the dataset using:

$\frac{1}{|D|}\frac{1}{|N|}\sum_{j=1}^{|D|}\sum_{k=1}^{|N|}2^{H(p_{k})}$ (3)

Where $N$ is the total number of templates in the document, and $D$ the total
number of documents in the dataset We repeat this process for randomly sampled
6-grams (distinct from the templates) to match the number of templates.

Figure 2: Perplexity of matched template at different model checkpoints.
Templates initially have higher perplexity than other tokens, but quickly drop
after initial training steps.

Figure 3: Coverage of templates found in pre-training data, fine-tuning data,
and both datasets combined. Templates are found at a much higher rate in the
training data than random n-gram sequences.

Figure 4: Ranking of OLMo templates and non-templated sequences in frequency
of pre-training data. Templates appear at higher ranks (and are therefore more
frequent) than non-templated sequences.

#### Results

Figure 2 shows the average perplexities across model checkpoints. We find that
templates are learned quickly—by the first model checkpoint (which was trained
on 4B tokens). Average perplexity drops to around 500 for non-template tokens,
compared to 200 for templates.

These findings are surprising, and indicate that templates are learned early
in pre-training, rather than during the fine-tuning process. The average
perplexities remain lower for template tokens for the remainder of the
training process.

###  6.2 Templates in Pre-training Data

The above finding indicates that templates are seen fairly early on in pre-
training. We next measure the incidence and types of templates in the
(pre-)training data, and whether they correspond to the templates that models
produce.

To search for template coverage by OLMo, we start by selecting a random subset
of the Dolma dataset, containing 10 billion tokens. We then annotate all of
the sequences with a POS tagger using the Dolma toolkit (Soldaini et al.,
2024). Finally, we find the 50K most common POS-grams in the data for sequence
length of six using the WIMBD toolkit (Elazar et al., 2023), which is
optimized for search and count at large scales.

#### Results

Figure 4 shows the coverage of templates produced by OLMo in the pre-training
data, the fine-tuning data, and their concatenation. We find that 75% of
templates produced by OLMo are found in the pre-training data, indicating that
templates are not a novel construction learned during fine-tuning. Rather,
they are learned directly from pre-training data. In comparison, only 34% of
randomly sampled non-templated sequences are found in the pre-training data.
Further, Figure 4 shows that the templates OLMo generates consistently rank
higher in frequency in the pre-training dataset, compared to randomly sampled
non-templates. The difference in ranks between the templates and non-templates
is statistically significant; the median rank in templates and non-templates
are 337.5 and 9651.0 (Mann–Whitney $U=6043$, $p<0.05\text{ two-tailed}$).

|  | Rotten Tomatoes
---
Cochrane | CNN/DM
Model |  | CR:
---
POS
|  $\geq 1$ Templates
---
% ($n=6$)
| CR:
---
POS
|  $\geq 1$ Templates
---
% ($n=6$)
| CR:
---
POS
|  $\geq 1$ Templates
---
% ($n=6$)
Reference | 5.31 | 46.4 (0.040) | 5.63 | 83.3 (0.049) | 5.33 | 36.0 (0.013)
Input Documents | 5.82 | 29.3 (0.001) | 5.96 | 98.5 (0.021) | 5.54 | 98.4 (0.020)
OLMo-7B | 6.45 | 97.0 (0.041) | 6.53 | 74.0 (0.030) | 5.83 | 91.2 (0.025)
Mistral-7B | 6.29 | 99.6 (0.043) | 6.10 | 99.5 (0.043) | 5.70 | 89.9 (0.029)
Llama-2-7B | 6.87 | 93.0 (0.047) | 6.43 | 88.4 (0.042) | 5.71 | 90.4 (0.028)
Llama-2-13B | 6.70 | 99.0 (0.060) | 6.65 | 95.1 (0.052) | 5.91 | 97.4 (0.042)
Llama-2-70B | 6.36 | 99.3 (0.123) | 6.51 | 99.7 (0.042) | 5.69 | 87.4 (0.027)
Llama-3-70B | 6.39 | 99.2 (0.151) | 6.50 | 99.5 (0.030) | 5.66 | 83.2 (0.024)
Alpaca-7B | 6.65 | 92.4 (0.070) | 7.82 | 75.9 (0.051) | 6.65 | 90.0 (0.027)
Alpaca-13B | 6.28 | 89.2 (0.053) | 6.26 | 67.0 (0.043) | 5.59 | 85.4 (0.028)
GPT-4o | 6.11 | 98.2 (0.041) | 6.12 | 95.7 (0.011) | 5.71 | 91.0 (0.026)

Table 3: Compression ratio with POS (CR-POS) reported for each model-generated
output over a random sample (n=500) of the Rotten Tomatoes, Cochrane, and
CNN/DM datasets using greedy decoding, and the prompt ‘‘Write a short
summary". For Cochrane, we use the prompt ‘‘Write a meta-analysis" to match
the task. Larger values in CR-POS indicate less diversity in the sequences. We
report the percentage of generated outputs with at least 1 template of size
$n=6$, and the rate of templates-per-token in parentheses (avg. num. templates
per summary normalized by avg. length). Models producing higher templates-per-
token than the human-written references are marked in bold.

Figure 5: Incidence of generated text with at least 1 template of sizes
$n=4,5,6,7,8$ over the (a) Rotten Tomatoes and (b) Cochrane datasets. Longer
templates appear less frequently but at higher rates in model-generated text
than in human-written references (dashed lines).

##  7 Templates in Closed-Source Models

With OLMo, we find that most templates are in the pre-training data. We
evaluate the incidence of templates in other closed-source models, which we
define as models that do not release their training data. Addressing RQ1, we
characterize the rates of templatic texts in these models, and posit that
templates may be indicators of the pre-training data sources models are
trained on.

#### Summarization

We report the rate of templates in Table 3, and measure the percentage of
outputs that contain at least one template of length $n$. We find that, on
average in the Rotten Tomatoes dataset, 95% of outputs contain templates of
length $n=6$ across different model types and sizes. This is in contrast to
human-written reference and input documents, which contain templates of the
same size on average in 38% of cases. We find a similar trend for templates of
length $n=[4,8]$ (Figure 5).

While the average number of templates is higher in model-generated output,
this could be attributed to models simply producing lengthier texts than the
human written references. To quantify this, we also compute the template-per-
token as a length normalized value capturing the average templates per
summary. Even controlling for length, most models produce more templates per
token than human authors, as shown in Table 3.

The CNN/DM datasets show similar trends, but with lower rates of templates
compared to the Rotten Tomatoes dataset. In contrast, the percentage of
templates is high for model-generated and human-written references in the
Cochrane dataset. This owes to the nature of meta-analysis texts, which are
formulaic (Higgins and Green, 2010).444Appendix Table C provides examples of
the human-written references

Figure 5 illustrates the rate of templates for each model as the template
length grows from length $n=4$ to $8$ for Rotten Tomatoes and Cochrane. For
Rotten Tomatoes (and CNN; Appendix B), all models produce templates at higher
rates than human-written summaries across all template lengths. With Cochrane,
template lengths $\geq 6$ show the majority of models produce higher rates of
templates than human authored references.

The BooookScore dataset provides text generated by models using long-document
summarization strategies. We report results over the hierarchical strategy,
where the final summary is merged together from smaller summarized chunks.
Similar to the other summarization tasks, we observe high rates of templates
across all the models available in this dataset (Table 4).

| BooookScore, Hierarchical
---|---
Model |  | CR:
---
POS
|  $\geq 1$ Templates
---
% ($n=6$)
Claude-2048 | 5.63 | 95.0 (0.010)
Claude-88000 | 5.60 | 94.0 (0.004)
ChatGPT-2048 | 6.17 | 100.0 (0.017)
GPT4-2048 | 6.04 | 100.0 (0.013)
GPT4-4096 | 6.01 | 99.0 (0.013)
Mixtral-2048 | 6.01 | 100.0 (0.017)

Table 4: Compression ratio with POS (CR-POS) reported for the BooookScore
dataset. We report the percentage of generated outputs with at least 1
template of size $n=6$, and the rate of templates-per-token in parentheses.

#### Model Size

Table 3 reports differences in the rate of templates between different sizes
of Llama-2 and Alpaca. In the case of Alpaca, the larger model yields outputs
with less repetition and fewer templates. With the Llama-2 and Llama-3 models,
we observe a surprising trend as model size increases: CR-POS and average text
length decrease, however the rate of summaries that contain one or more
templates stays the same (and increases slightly in some cases). These results
indicate that larger models do not necessarily produce less templated outputs.
The templates-per-token value further supports this, showing an increase in
template rate (per token) for larger models.

Figure 6: Percent memorized POS sequences, stratified by frequency in training
data.

Figure 7: Sampled sentences from OLMo-7B prompted with a prefix of training
data. The model substitutes synonyms or numbers.

##  8 Style Memorization

Past work has shown that models sometimes memorize portions of pre-training
data. Carlini et al. (2022) show a lower bound of 1% verbatim memorized data.
We next show how our syntactic template analysis can be used to evaluate how
much models memorize from pre-training data, beyond strict token sequence
matches (RQ 3).

#### Definition: Exact-Text Memorization

We borrow the definition for extractable memorization from Carlini et al.
(2022). A string $s$ is memorized by a model $\mathcal{M}$ if, when prompted
with context $p$, $\mathcal{M}(p)$ produces a string $g$ that is an exact text
match to the source string $s$ under greedy decoding.

#### Definition: Style (POS) Memorization

For style memorization, we follow the same definition for Exact-Text, but
modified to operate over POS (rather than token) sequences to capture
instances of syntactic “style”. Specifically, given a POS tagger $f$, sequence
$f(s)$ is memorized by $\mathcal{m}$ if, when prompted with context $p$,
$\mathcal{m}(p)$ produces a sequence $g$ such that $f(g)$ is an exact match to
the source string $f(s)$.

###  8.1 Experimental Setup

We follow a similar experimental setup as Carlini et al. (2022), focusing on
creating sampled datasets that contain $n$-grams repeated in the pre-training
dataset. We use WIMBD to build our subset over the Dolma dataset and return
the top 50k most common 100-grams in the Dolma dataset from § 6.2. For each
100-gram, we tokenize the sequence with NLTK and truncate to 50 tokens
(Carlini et al., 2022). Following the setup for extracting memorized
sequences, we prompt OLMo-7B with 50 tokens of the training data sequence and
generate a maximum of 1000 tokens using greedy decoding. We apply NLTK’s POS
model to tag the original string $s$ and the model-generated string $g$.

###  8.2 Results

We randomly sample 10k documents and look at the fraction of memorized outputs
based on exact-match and the POS sequence (“style”) memorization. We average
the fraction memorized over 1,000 seeds for sampling the datasets.

On average, the POS memorization definition finds 6.4% ($\pm$ 0.7) memorized,
whereas exact text match only reports 5.3% ($\pm$ 0.6) memorized of the
training dataset. Figure 6 shows the percent templates memorized stratified by
frequency of the 100-gram in the training dataset. We divide the sampled data
point into 10 buckets each containing 4,138 samples with counts that fall in
each bucket. In all buckets, POS memorization captures a higher rate of
memorized sequences.

The implications of our looser definition of memorization allows us to capture
instances of memorization where exact tokens may be substituted during
generation, but where an output span is nonetheless structurally the same as a
source string. In Figure 7, we provide examples of substitutions that occur
that are not captured by exact-memorization definitions, yet demonstrate that
the particular style of that training point has been memorized. We find that
these cases often include synonym swaps, or different numbers being generated.

##  9 Conclusions

In this work, we introduce syntactic templates as a framework for analyzing
subtle repetitive characteristics in model-generated text. We show that this
analysis can also extend to human-written references and downstream tasks, and
find that the pre-training data contains many of these identified templates.
We show that this definition of repetition is useful for detecting subtle
types of data “memorization”. Our hope is that this work inspires additional
research into characterizing where (in data) observed stylistic patterns in
LLM outputs originate.

## Limitations

There are a few limitations to this work that we address here.

First, this type of analysis requires an entire corpus that is representative
of a text source. For paid models, this can be costly to obtain. For large
datasets, this can be resource intensive. These considerations provide a
potential barrier based on available resources.

Second, we use third party tools to tag our text abstractions; however these
tools are not deterministic and can contain errors in the tags they assign to
sequences, particularly if a sequence contains text from another language. We
assume that the majority of the text we analyze is in English, and that any
errors are superseded by the frequency of common templates.

Finally, this work only examines English texts, in part due to availability of
datasets at the scale necessary to evaluate models.

## Acknowledgements

This work was supported in part by the National Science Foundation (NSF)
grants IIS 2211954, IIS 2145479, IIS 2107524.

## References

  * AI@Meta (2024) AI@Meta. 2024.  Llama 3 model card.
  * Bache et al. (2013) Kevin Bache, David Newman, and Padhraic Smyth. 2013.  Text-based measures of document diversity.  _Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining_.
  * Bär et al. (2012) Daniel Bär, Torsten Zesch, and Iryna Gurevych. 2012.  Text reuse detection using a composition of text similarity measures.  In _Proceedings of COLING 2012_ , pages 167–184.
  * Bauer et al. (2023) John Bauer, Chloé Kiddon, Eric Yeh, Alex Shan, and Christopher D. Manning. 2023.  Semgrex and ssurgeon, searching and manipulating dependency graphs.  In _Proceedings of the 21st International Workshop on Treebanks and Linguistic Theories (TLT, GURT/SyntaxFest 2023)_ , pages 67–73, Washington, D.C. Association for Computational Linguistics.
  * Ben Allal et al. (2024) Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra. 2024.  Cosmopedia.
  * Bender et al. (2021) Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021.  On the dangers of stochastic parrots: Can language models be too big?  In _Proceedings of the 2021 ACM conference on fairness, accountability, and transparency_ , pages 610–623.
  * Carlini et al. (2022) Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2022.  Quantifying memorization across neural language models.  _arXiv preprint arXiv:2202.07646_.
  * Chang et al. (2023) Yapei Chang, Kyle Lo, Tanya Goyal, and Mohit Iyyer. 2023.  Booookscore: A systematic exploration of book-length summarization in the era of llms.  _ArXiv_ , abs/2310.00785.
  * Chen et al. (2024) Mayee Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, and Christopher Ré. 2024.  Skill-it! a data-driven skills framework for understanding and training language models.  _Advances in Neural Information Processing Systems_ , 36.
  * Cui et al. (2023) Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. 2023.  Ultrafeedback: Boosting language models with high-quality feedback.
  * DiMarco and Hirst (1993) Chrysanne DiMarco and Graeme Hirst. 1993.  A computational theory of goal-directed style in syntax.  _Computational Linguistics_ , 19(3):451–500.
  * Dou et al. (2022) Yao Dou, Maxwell Forbes, Rik Koncel-Kedziorski, Noah A Smith, and Yejin Choi. 2022.  Is gpt-3 text indistinguishable from human text? scarecrow: A framework for scrutinizing machine text.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 7250–7274.
  * Elazar et al. (2023) Yanai Elazar, Akshita Bhagia, Ian Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, et al. 2023.  What’s in my big data?  _arXiv preprint arXiv:2310.20707_.
  * Goldberg (2019) Yoav Goldberg. 2019.  Assessing bert’s syntactic abilities.  _arXiv preprint arXiv:1901.05287_.
  * Groeneveld et al. (2024) Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, A. Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Daniel Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Hanna Hajishirzi. 2024.  Olmo: Accelerating the science of language models.  _ArXiv_ , abs/2402.00838.
  * Higgins and Green (2010) Julian P T Higgins and Sally Green. 2010.  Cochrane handbook for systematic reviews of interventions.  _International Coaching Psychology Review_.
  * Holtzman et al. (2020) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020.  The curious case of neural text degeneration.  In _International Conference on Learning Representations_.
  * Hupkes et al. (2023) Dieuwke Hupkes, Mario Giulianelli, Verna Dankers, Mikel Artetxe, Yanai Elazar, Tiago Pimentel, Christos Christodoulopoulos, Karim Lasri, Naomi Saphra, Arabella Sinclair, Dennis Ulmer, Florian Schottmann, Khuyagbaatar Batsuren, Kaiser Sun, Koustuv Sinha, Leila Khalatbari, Maria Ryskina, Rita Frieske, Ryan Cotterell, and Zhijing Jin. 2023.  A taxonomy and review of generalization research in nlp.  _Nature Machine Intelligence_ , 5(10):1161–1174.
  * Ivison et al. (2023) Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew E. Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hanna Hajishirzi. 2023.  Camels in a changing climate: Enhancing lm adaptation with tulu 2.  _ArXiv_ , abs/2311.10702.
  * Jiang et al. (2023) Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L’elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023.  Mistral 7b.  _ArXiv_ , abs/2310.06825.
  * Kirk et al. (2024) Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. 2024.  Understanding the effects of rlhf on llm generalisation and diversity.
  * Krishna et al. (2020) Kalpesh Krishna, John Wieting, and Mohit Iyyer. 2020.  Reformulating unsupervised style transfer as paraphrase generation.  _arXiv preprint arXiv:2010.05700_.
  * Kryściński et al. (2020) Wojciech Kryściński, Bryan McCann, Caiming Xiong, and Richard Socher. 2020.  Evaluating the factual consistency of abstractive text summarization.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9332–9346.
  * Lample et al. (2018) Guillaume Lample, Sandeep Subramanian, Eric Michael Smith, Ludovic Denoyer, Marc’Aurelio Ranzato, and Y-Lan Boureau. 2018.  Multiple-attribute text rewriting.  In _International Conference on Learning Representations_.
  * Leone (2020) Stefano Leone. 2020.  Rotten tomatoes movies and critic reviews dataset.
  * Liang et al. (2024a) Weixin Liang, Zachary Izzo, Yaohui Zhang, Haley Lepp, Hancheng Cao, Xuandong Zhao, Lingjiao Chen, Haotian Ye, Sheng Liu, Zhi Huang, et al. 2024a.  Monitoring ai-modified content at scale: A case study on the impact of chatgpt on ai conference peer reviews.  _arXiv preprint arXiv:2403.07183_.
  * Liang et al. (2024b) Weixin Liang, Yaohui Zhang, Zhengxuan Wu, Haley Lepp, Wenlong Ji, Xuandong Zhao, Hancheng Cao, Sheng Liu, Siyu He, Zhi Huang, et al. 2024b.  Mapping the increasing use of llms in scientific papers.  _arXiv preprint arXiv:2404.01268_.
  * McCoy et al. (2023) R. Thomas McCoy, Paul Smolensky, Tal Linzen, Jianfeng Gao, and Asli Celikyilmaz. 2023.  How Much Do Language Models Copy From Their Training Data? Evaluating Linguistic Novelty in Text Generation Using RAVEN.  _Transactions of the Association for Computational Linguistics_ , 11:652–670.
  * Montahaei et al. (2019) Ehsan Montahaei, Danial Alihosseini, and Mahdieh Soleymani Baghshah. 2019.  Jointly measuring diversity and quality in text generation models.  _ArXiv_ , abs/1904.03971.
  * Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016.  Abstractive text summarization using sequence-to-sequence rnns and beyond.  _arXiv preprint arXiv:1602.06023_.
  * Padmakumar and He (2023) Vishakh Padmakumar and He He. 2023.  Does writing with language models reduce content diversity?  _arXiv preprint arXiv:2309.05196_.
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019.  Language models as knowledge bases?  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2463–2473.
  * Roberts et al. (2020) Nicholas Roberts, Davis Liang, Graham Neubig, and Zachary C Lipton. 2020.  Decoding and diversity in machine translation.  _arXiv preprint arXiv:2011.13477_.
  * Rosenfeld and Lazebnik (2024) Ariel Rosenfeld and Teddy Lazebnik. 2024.  Whose llm is it anyway? linguistic comparison and llm attribution for gpt-3.5, gpt-4 and bard.  _arXiv preprint arXiv:2402.14533_.
  * Salkar et al. (2022) Nikita Salkar, Thomas Trikalinos, Byron Wallace, and Ani Nenkova. 2022.  Self-repetition in abstractive neural summarizers.  In _Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 341–350, Online only. Association for Computational Linguistics.
  * Shaib et al. (2024) Chantal Shaib, Joe Barrow, Jiuding Sun, Alexa F Siu, Byron C Wallace, and Ani Nenkova. 2024.  Standardizing the measurement of text diversity: A tool and a comparative analysis of scores.  _arXiv preprint arXiv:2403.00553_.
  * Soldaini et al. (2024) Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. 2024.  Dolma: An open corpus of three trillion tokens for language model pretraining research.  _arXiv preprint arXiv:2402.00159_.
  * Soler-Company and Wanner (2017) Juan Soler-Company and Leo Wanner. 2017.  On the relevance of syntactic and discourse features for author profiling and identification.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_ , pages 681–687, Valencia, Spain. Association for Computational Linguistics.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca.
  * Taylor et al. (2003) Ann Taylor, Mitchell Marcus, and Beatrice Santorini. 2003.  The penn treebank: an overview.  _Treebanks: Building and using parsed corpora_ , pages 5–22.
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a.  Llama: Open and efficient foundation language models.  _ArXiv_ , abs/2302.13971.
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023b.  Llama 2: Open foundation and fine-tuned chat models.  _ArXiv_ , abs/2307.09288.
  * Wallace et al. (2020) Byron C. Wallace, Sayantani Saha, Frank Soboczenski, and Iain James Marshall. 2020.  Generating (factual?) narrative summaries of rcts: Experiments with neural multi-document summarization.  _AMIA … Annual Symposium proceedings. AMIA Symposium_ , 2021:605–614.
  * Wang et al. (2022) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022.  Self-instruct: Aligning language models with self-generated instructions.  In _Annual Meeting of the Association for Computational Linguistics_.
  * Wiher et al. (2022) Gian Wiher, Clara Meister, and Ryan Cotterell. 2022.  On decoding strategies for neural text generators.  _Transactions of the Association for Computational Linguistics_ , 10:997–1012.
  * Wu et al. (2023) Junchao Wu, Shu Yang, Runzhe Zhan, Yulin Yuan, Derek F Wong, and Lidia S Chao. 2023.  A survey on llm-gernerated text detection: Necessity, methods, and future directions.  _arXiv preprint arXiv:2310.14724_.
  * Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations_.

##  Appendix A Constituency Trees

For constituency parsing, we use the Stanza library (Bauer et al., 2023), and
linearize sequences using a breadth-first search approach. Table 5 shows some
examples of linearized constituency trees and their matching text.

Template (Constituents) | Frequency | Matched Text
---|---|---
(VP (VB ) (NP (NP (DT ) (JJ ) (NN )) (PP (IN ) (NP) | 458 | Trace the intellectual history of ancient
Examine the early history of automobiles
guarantee a seamless ascent into another
reach a broad audience of buyers
(PP (IN ) (NP (NP (DT ) (JJ ) (NN )) (PP (IN ) (NP) | 948 | as a key component of your
by a high abundance of free
across a global range of cultures
for the comprehensive study of the
(DT ) (JJ ) (NN )) (PP (IN ) (NP) | 1680 | a strong grasp of various
a solid understanding of
a radical change in

Table 5: Example templates using constituency trees over the Cosmopedia
dataset.

##  Appendix B CNN/DM Trends

Figure 8 demonstrates the same trend of high templatic text in the model
generated text compared to human authored references.

Figure 8: Incidence of generated text with at least 1 template of sizes
$n=4,5,6,7,8$ over the CNN/DM dataset.

Prompts, CNN/DM
---
1 Please write a summary.
2 Please write a summary of the article.
3 Summarize.
4 Write a short summary.
5 Write a short summary and be creative.
6 Write a meta-analysis.
7 Write an aggregate summary based on the above facts.

Table 6: Prompts used for the CNN/DM summarization task.

Prompts, Cochrane
---
1 Please write a summary.
2 Please write a summary of the evidence.
3 Summarize.
4 Write a short summary.
5 Write a short summary and be creative.
6 Write a meta-analysis.
7 Write an aggregate analysis based on the above evidence.

Table 7: Prompts used for the Cochrane meta-analysis task.

##  Appendix C Cochrane References

Cochrane systematic reviews follow guidelines for how they should be written.
Figure 9 shows an example of human-authored Cochrane text.

Figure 9: Sample of Cochrane (Human-Authored) references.

Prompts, Rotten Tomatoes
---
1 Please write a summary.
2 Please write a summary of the movie reviews.
3 Summarize.
4 Write a short summary.
5 Write a short summary and be creative.
6 Write a meta-analysis.
7 Write an aggregate movie review based on the above reviews.

Table 8: Prompts used for the Rotten Tomatoes summarization task.

##  Appendix D Choice of Prompt

The choice of prompt can impact the incidence of templates. Here, we provide
an analysis of 7 different prompts for the downstream summarization task. We
find that shorter and more generic prompts result in lower performance. It is
possible that with the shorter and more generic instructions instructions,
there may be a higher overlap in template types across models.

Model |  $\geq 1$ Templates % ($n=6$) |
---|---|---
Prompt | 1 | 2 | 3 | 4 | 5 | 6 | 7 | $\Delta$
OLMo-7B | 98.1 | 94.4 | 92.5 | 97.0 | 96.2 | 97.2 | 97.2 | 5.6
Mistral-7B | 98.5 | 98.5 | 97 | 99.6 | 99.5 | 100 | 100 | 3.0
Llama-2-7B | 91.5 | 94.5 | 88.9 | 93.0 | 94.8 | 95.4 | 94.9 | 6.5
Llama-2-13B | 99.1 | 99.8 | 94.9 | 99.0 | 98.0 | 98.0 | 98.6 | 4.9
Llama-2-70B | 96.2 | 96.8 | 97.2 | 99.3 | 98.8 | 99.6 | 99.2 | 3.4
Llama-3-70B | 95.7 | 97.3 | 93.7 | 99.2 | 97.8 | 99.8 | 99.4 | 6.1
Alpaca-7B | 92.3 | 87.5 | 86.8 | 92.4 | 92.7 | 84.1 | 91.6 | 8.6
Alpaca-13B | 92.5 | 91.0 | 90.3 | 89.2 | 91.6 | 94.5 | 94.4 | 5.3
GPT-4o | 98.2 | 98.0 | 95.2 | 97.0 | 99.6 | 100.0 | 99.8 | 4.8

Table 9: Percentage of generated summaries with at least 1 template of length
$n=6$ under different instruction prompts. Some instruction prompts result is
more templates outputs for certain models.

The choice of prompt can affect the rate of templates. Appendix Table 9 shows
the rate of templates with different prompts while the choice of prompt
impacts the rate of templates, the incidence remains on average higher than
90%.

##  Appendix E Template Measures, Text Length

For completeness, we provide the full tables for all datasets that include the
average text length. In the case of Booookscore, we also provide results over
the incremental dataset.

|  | Rotten Tomatoes
---
Cochrane | CNN/DM
Model |  | CR:
---
POS
| Avg. Text
---
Length
|  $\geq 1$ Templates
---
% ($n=6$)
| CR:
---
POS
| Avg. Text
---
Length
|  $\geq 1$ Templates
---
% ($n=6$)
| CR:
---
POS
| Avg. Text
---
Length
|  $\geq 1$ Templates
---
% ($n=6$)
Reference | 5.31 | 25.1 | 46.4 (0.040) | 5.63 | 73.8 | 83.3 (0.049) | 5.33 | 57.7 | 36.0 (0.013)
Input Documents | 5.82 | 668.2 | 29.3 (0.001) | 5.96 | 1555.3 | 98.5 (0.021) | 5.54 | 514.7 | 98.4 (0.020)
OLMo-7B | 6.45 | 194.4 | 97.0 (0.041) | 6.53 | 158.1 | 74.0 (0.030) | 5.83 | 177.1 | 91.2 (0.025)
Mistral-7B | 6.29 | 185.6 | 99.6 (0.043) | 6.10 | 177.2 | 99.5 (0.043) | 5.70 | 153.0 | 89.9 (0.029)
Llama-2-7B | 6.87 | 126.5 | 93.0 (0.047) | 6.43 | 151.0 | 88.4 (0.042) | 5.71 | 153.9 | 90.4 (0.028)
Llama-2-13B | 6.70 | 117.0 | 99.0 (0.060) | 6.65 | 157.7 | 95.1 (0.052) | 5.91 | 143.5 | 97.4 (0.042)
Llama-2-70B | 6.36 | 114.2 | 99.3 (0.123) | 6.51 | 324.7 | 99.7 (0.042) | 5.69 | 138.3 | 87.4 (0.027)
Llama-3-70B | 6.39 | 106.1 | 99.2 (0.151) | 6.50 | 387.5 | 99.5 (0.030) | 5.66 | 132.7 | 83.2 (0.024)
Alpaca-7B | 6.65 | 99.4 | 92.4 (0.070) | 7.82 | 98.1 | 75.9 (0.051) | 6.65 | 145.2 | 90.0 (0.027)
Alpaca-13B | 6.28 | 93.0 | 89.2 (0.053) | 6.26 | 69.7 | 67.0 (0.043) | 5.59 | 138.1 | 85.4 (0.028)
GPT-4o | 6.11 | 203.5 | 98.2 (0.041) | 6.12 | 560.7 | 95.7 (0.011) | 5.71 | 167.6 | 91.0 (0.026)

Table 10: Compression ratio with POS (CR-POS) reported for each model-
generated output over a random sample (n=500) of the Rotten Tomatoes,
Cochrane, and CNN/DM datasets using greedy decoding, and the prompt ‘‘Write a
short summary". For Cochrane, we use the prompt ‘‘Write a meta-analysis" to
match the task. Larger values in CR-POS indicate less diversity in the
sequences. We report the percentage of generated outputs with at least 1
template of size $n=6$, and the rate of templates-per-token in parentheses
(avg. num. templates per summary normalized by avg. length). Models producing
higher templates-per-token than the human-written references are marked in
bold.

| Open Generation | Rotten Tomatoes
---|---|---
Decoding Strategy |  | CR:
---
POS
| Avg. Text
---
Length
|  $\geq 1$ Templates
---
% ($n=6$)
| CR:
---
POS
| Avg. Text
---
Length
|  $\geq 1$ Templates
---
% ($n=6$)
Greedy | 702.8 | 48.3 | 100.0 (0.065) | 6.45 | 194.4 | 97.0 (0.041)
Default Sampling | 5.81 | 479.2 | 75.5 (0.009) | 6.33 | 194.2 | 96.6 (0.041)
temp 0.8 | 6.74 | 517.9 | 71.8 (0.007) | 6.26 | 191.1 | 96.6 (0.043)
temp 0.85 | 6.48 | 506.8 | 74.0 (0.007) | 6.22 | 188.5 | 96.6 (0.041)
temp 0.9 | 6.22 | 497.3 | 73.4 (0.009) | 6.17 | 190.3 | 97.4 (0.039)
temp 0.95 | 5.98 | 500.2 | 75.4 (0.01) | 6.14 | 186.6 | 97.2 (0.040)
top_p 0.8 | 7.03 | 541.0 | 76.5 (0.007) | 6.31 | 190.2 | 97.8 (0.041)
top_p 0.85 | 6.71 | 526.3 | 71.0 (0.007) | 6.27 | 190.3 | 96.6 (0.041)
top_p 0.9 | 6.50 | 495.1 | 75.3 (0.008) | 6.22 | 190.9 | 96.2 (0.039)
top_p 0.95 | 6.17 | 513.0 | 77.2 (0.009) | 6.31 | 192.1 | 95.8 (0.041)

Table 11: Compression ratio with POS (CR-POS), average text length and
percentage of generated outputs with at least 1 template of size $n=6$, when
varying temperature and top_p for OLMo-7B decoding.

| BooookScore, Hierarchical | BooookScore, Incremental
---|---|---
Model |  | CR:
---
POS
| Avg. Text
---
Length
|  $\geq 1$ Templates
---
% ($n=6$)
| CR:
---
POS
| Avg. Text
---
Length
|  $\geq 1$ Templates
---
% ($n=6$)
Claude-2048 | 5.63 | 461.9 | 95.0 (0.010) | 5.71 | 575.4 | 86.0 (0.007)
Claude-88000 | 5.60 | 487.9 | 94.0 (0.004) | 5.60 | 439.5 | 96.0 (0.010)
ChatGPT-2048 | 6.17 | 613.8 | 100.0 (0.017) | 5.76 | 439.2 | 95.0 (0.011)
GPT4-2048 | 6.04 | 706.5 | 100.0 (0.013) | 5.95 | 721.0 | 99.0 (0.009)
GPT4-4096 | 6.01 | 855.4 | 99.0 (0.013) | 6.05 | 1,013.2 | 100.0 (0.010)
Mixtral-2048 | 6.01 | 619.2 | 100.0 (0.017 ) | 5.59 | 496.1 | 99.0 (0.012)

Table 12: Compression ratio with POS (CR-POS) reported for the BooookScore
dataset. We report the percentage of generated outputs with at least 1
template of size $n=6$, and the rate of templates-per-token in parentheses.

Dataset |  | CR:
---
POS
| Avg. Text
---
Length
|  $\geq 1$ Templates
---
% ($n=6$)
Dolma-100 | 5.65 | 483.2 | 82.6 (0.012)
Cosmopedia | 5.76 | 768.0 | 99.1 (0.014)

Table 13: CR-POS, template-per-token, and template counts for templates of
size $n=6$ reported for OLMo-7B text generated with Cosmopedia Instructions,
and 100 sampled tokens from the Dolma dataset, with greedy decoding.