#  Towards More Realistic Extraction Attacks:
An Adversarial Perspective

Yash More∗
McGill University, Mila
<EMAIL_ADDRESS>
Prakhar Ganesh∗
McGill University, Mila
<EMAIL_ADDRESS>
Golnoosh Farnadi
McGill University, Mila
<EMAIL_ADDRESS>

###### Abstract

Language models are prone to memorizing large parts of their training data,
making them vulnerable to extraction attacks. Existing research on these
attacks remains limited in scope, often studying isolated trends rather than
the real-world interactions with these models. In this paper, we revisit
extraction attacks from an adversarial perspective, exploiting the brittleness
of language models. We find significant churn in extraction attack trends,
i.e., even minor, unintuitive changes to the prompt, or targeting smaller
models and older checkpoints, can exacerbate the risks of extraction by up to
$2-4\times$. Moreover, relying solely on the widely accepted verbatim match
underestimates the extent of extracted information, and we provide various
alternatives to more accurately capture the true risks of extraction. We
conclude our discussion with data deduplication, a commonly suggested
mitigation strategy, and find that while it addresses some memorization
concerns, it remains vulnerable to the same escalation of extraction risks
against a real-world adversary. Our findings highlight the necessity of
acknowledging an adversary’s true capabilities to avoid underestimating
extraction risks.

11footnotetext: Equal contribution

##  1 Introduction

Large language models (LLMs) have grown considerably in size and function [33,
50], becoming integral to a wide range of tasks like knowledge retrieval,
question answering, code generation, machine translation, etc. To complement
this growing scale, LLMs require large amounts of training data [37, 44, 20,
39], introducing challenges in data collection. Given its sheer volume,
meticulous curation of the training data becomes impractical, instead relying
on data scraping, which raises significant concerns about data memorization
and potential information leakage in LLMs.

Extraction attacks offer an empirical method to measure memorization in LLMs
[17, 15, 35, 30], thereby quantifying the information leakage risks in
practice. One commonly studied form of extraction attack is discoverable
memorization [15, 26], where the model is prompted with a portion of a
sentence from the training data to extract the rest. Discoverable memorization
enables adversaries to perform targeted attacks, highlighting its significance
in the literature.

Recent studies have examined extraction attacks and their trends across
various parameters, such as prompt length, model size, generation
hyperparameters, and learning dynamics [15, 49, 9]. However, these trends are
often investigated in isolation. In practice, an adversary can prompt the
model multiple times, use models of different sizes (e.g., Llama3 8B and 70B
[33]; Gemini Pro, Ultra, Flash, and Nano [45]; Falcon2 1.3B, 7.5B, and 40B
[4]), and prompt models over time (e.g., ChatGPT updates [36]; GPT 3, 3.5, and
4 [2]; Llama, Llama2, and Llama3 [33]), combining knowledge from various
sources. In this paper, we consider this realistic scenario and explore the
actual risks posed by extraction attacks in the current ecosystem of LLMs.
More specifically, we investigate:

  1. 1.

Can adversaries exploit repeated prompting? We find that extraction attacks
are highly sensitive to the prompt design, extracting over $20\%$ more data
with even minor, unintuitive changes to the prompt (§6.1). Thus, an adversary,
given the opportunity to prompt the model multiple times, can extract more
data than previously observed.

  2. 2.

Does access to multiple checkpoints increase extraction risk? An adversary
with access to multiple model checkpoints or sizes gains a broader black-box
access to the underlying dataset. We find such an adversary can increase the
extraction rates up to $2\times$, significantly heightening the risk of
information leakage (§6.2).

  3. 3.

Is verbatim match a reliable measure of information leakage? We show the
limitations of using a verbatim match to assess the success of extraction
attacks and the resulting underestimation of extraction risk (§4). As
alternatives, we propose the use of approximate matching to estimate the
actual risk of information leakage better and observe a more than $2\times$
increase in extraction rates even under strict thresholds for approximation
(§7.2).

  4. 4.

How effective is data deduplication in reducing the extraction risks? We find
that data deduplication does reduce the extraction risks, in line with the
existing literature [15]. However, adversaries with access to repeated
prompting and multiple checkpoints continue to extract more information, with
the risks of extraction surpassing even the rates without deduplication
reported in the literature (§7.3). Thus, our concerns of a powerful adversary
persist despite deduplication.

With this work, we bring to light the existence of a stronger adversary in the
current landscape of LLMs, urging the community to focus on developing
solutions for these realistic extraction attacks. The rest of the paper is
organized as follows. §2 reviews related work. §3 introduces the real-world
adversary and §4 proposes approximate matching to replace the widely used
verbatim matching. §5 defines the experiment setup. §6 evaluates the
extraction risk under our real-world adversary and §7 combines it with
approximate matching to quantify the overall extraction risk. §8 concludes the
discussion, with a focus on limitations and future work.

##  2 Background and Related work

Extraction Attacks in LLMs. Unintended memorization in LLMs can make it prone
to information leakage [46, 16, 31, 13], particularly through extraction
attacks [11, 17, 15, 35]. These attacks allow adversaries to extract training
data from the model, raising concerns of leaking sensitive information about
individuals or corporations [11]. Extraction attacks have gained significant
attention in recent years, studied under two primary frameworks: Discoverable
Memorization [15, 26, 30, 9, 46, 21], where the adversary attempts to extract
targeted information, and Extractable Memorization [35, 25, 38], where the
adversary attempts to extract any information from data.

We add to the growing body of research on targeted extraction attacks by
highlighting several limitations of the current definition of discoverable
memorization. We emphasize the presence of a stronger real-world adversary
capable of combining information from various attacks, thereby defining a
composite form of discoverable memorization (§3). Furthermore, we show that
existing evaluation metrics like verbatim match [17] can severely
underestimate the risks of memorization, and suggest alternatives to capture
the true information leakage (§4). A concurrent work by Schwarzschild et al.
[42] also redefines discoverable memorization, using prompt optimization and
adversarial compression ratio (ACR) to quantify memorization as information
compression. In contrast, rather than relying on optimization, we provide a
more scalable and efficient method of evaluating extraction risks by
emphasizing the brittleness of LLM generation. We also highlight the hidden
churn [34] present in extraction attack trends throughout the existing
literature.

Prompt Sensitivity in LLMs. LLMs are shown to be sensitive to changes in their
prompts, leading to fluctuations in their performance [43, 28]. This
sensitivity persists across varying model sizes and through fine-tuning and
other downstream modifications [41, 51]. The sensitivity of prompts can also
be misused, and adversarial modifications to prompts can trigger the model to
act in unintended ways [40, 29, 22, 29]. While several overarching trends
studying the impact of prompt design on extraction attacks are present in the
literature [15, 26, 38, 46], these trends are often evaluated in isolation.
Motivated by the composability of privacy leakage [32], we argue that an
adversary capable of repeated prompting can combine these trends. We show that
such an adversary can extract more information about the training data than
previously reported in the literature (§6.1).

Training Dynamics of LLMs. Several recent works have studied the training
dynamics of LLMs and the evolution of their capabilities over time [46, 27,
48]. In the context of memorization, recent work by Biderman et al. [9]
explored the impact of model size and intermediate checkpoints during training
on the dynamics of memorization, revealing a considerable variance in
memorized data over time and size. The practice of releasing models in various
sizes and regularly updating them over time can thus increase the
vulnerability of extraction attacks on the underlying dataset (§6.2).

Predictive Churn. The instability of model predictions under updates has
gained significant attention in recent years, studied under the umbrella of
predictive churn [34, 19, 8, 7, 24, 3, 47]. Defined on exactly two models,
predictive churn quantifies the inconsistency in predictions between a system
pre-update vs post-update, by measuring the fraction of individual test
examples whose predictions diverge [34]. While the term churn is traditionally
used in the literature to describe such regressive trends in model
predictions, we extend its use in our work by highlighting similar regressive
trends and instability of extraction attacks under changing prompts and
models. In simpler terms, churn occurs when information is extractable with
weaker setups like shorter prompts, smaller models, or earlier checkpoints,
but not with a stronger setup.

##  3 Re-evaluating adversarial strengths

The adversary is central to our work. We begin our discussion by defining its
capabilities and limitations, arguing that current work has underestimated the
strength of real-world adversaries. To ensure broad applicability across
different models and settings, we assume the target model to be a black box
that only provides generation output based on the input sentence.
Consequently, the adversary cannot access model weights, gradients, control
generation hyperparameters, or even view output probabilities. Despite these
constraints, we will demonstrate that an adversary in the current LLM
ecosystem possesses far greater power than what has been recognized in
existing literature.

###  3.1 Adversary capabilities

Composability (or self-composability) of privacy leakage is well known in the
literature [32]. When an adversary gains access to multiple outputs from
algorithms on the same underlying dataset—whether through multiple queries
from the same algorithm or queries across multiple algorithms—the risk of
information leakage grows. Consequently, an adversary with multiple points of
access is significantly more powerful than one with only a single point of
access. In the current landscape of LLMs, such access is not only unsurprising
but also easily obtainable (as illustrated in Figure 1). Specifically, we
consider two types of multi-access scenarios:

Figure 1: Composability in the current LLM ecosystem. In the real world, an
adversary has multi-faceted access to a dataset through (a) repeated
prompting, and (b) several model checkpoints.

Repeated Prompting. LLMs are highly sensitive to their input, including its
structure, content, or even the presence of noisy text within the prompt [43,
28, 41, 51]. While existing studies have focused on improving the prompts for
stronger attacks, the nuance of prompt sensitivity in LLMs often defies
intuitive expectations. For instance, while longer prompts are known to
increase the success of extraction attacks [15], our work demonstrates that
shorter prompts can at times exploit vulnerabilities that longer prompts
overlook (§6.1), thus indicating churn in these trends.

Given the widespread use of LLMs through both chat interfaces and through
rapid API calls, restricting repeated prompting and prompt engineering is
impractical. This accessibility introduces vulnerabilities where adversaries
can exploit input sensitivity and prompt the model multiple times to increase
their likelihood of extracting sensitive information from the model.

Multiple Checkpoints. LLMs are typically deployed in various sizes, catering
to different needs for accuracy and efficiency among users. However, due to
the stochastic nature of their training and the impact of scaling, different
model sizes might memorize unique portions of the underlying dataset [9].
Consequently, an adversary with access to multiple model sizes can effectively
aggregate extracted information rather than one limited to a single model
(§6.2).

Similarly, deployed LLMs undergo regular updates driven by new data, better
learning techniques, evolving security measures, and novel functionalities.
The stochastic training process means that data resilient to attacks at a
certain time step may become vulnerable in subsequent model updates [9]. Such
fluctuations can enable adversaries to exploit multiple checkpoints over time,
potentially extracting more information than from a static model (§6.2).

###  3.2 Combining extraction attacks

We argued for the heightened risk posed by multifaceted access to LLMs, either
through repeated prompting or multiple model checkpoints. Before diving into
our empirical study, we first quantify the risks associated with this stronger
adversary. We argue that when an adversary gains such extensive access, any
successful extraction of information—even if achieved once—renders that
specific information vulnerable to the adversary.

Formally, adapting the definition of discoverable memorization from Nasr et
al. [35], we propose:

######  Definition 3.1 (Composite Discoverable Memorization).

For a set of $k$ models $Gen_{1},Gen_{2},...,$ $Gen_{k}$, a set of $r$ prompt
modifiers $F_{1},F_{2},...,F_{r}$, and an example
$[{\mathbf{p}}\parallel{\mathbf{x}}]$ from the training set ${X}$, we say
${\mathbf{x}}$ is composite discoverably memorized if $\exists\;i\in[1\dots
k]\;\textrm{and}\;j\in[1\dots
r]\;\textrm{s.t.}\;Gen_{i}(F_{j}({\mathbf{p}}))={\mathbf{x}}$.

##  4 Beyond verbatim match

In the previous section, we argued that real-world adversaries possess greater
capabilities than previously acknowledged. However, even adversaries using
verbatim matches to quantify the success of an attack can underestimate the
information extracted. In this section, we explore the limitations of existing
metrics and propose alternative methods to better quantify potential
information leakage.

(a)

(b)

$0$$100$$200$$300$$400$$10^{1}$$10^{2}$$10^{3}$$10^{4}$$10^{5}$ (c) Verbatim
Match Length # of Sequences

Figure 2:  We visualize the limitations of current evaluation techniques. (a) We note with an example that by restricting ourselves to verbatim match, we miss out on potential information leakage despite the content being the same. (b) Similarly, we show that restricting the generations to 50 tokens can miss additional information leaked under continued generations. (c) This is further shown in the distribution of verbatim match lengths where we observe overlap significantly beyond $50$ tokens. Metric |  Advantages |  Disadvantages
---|---|---
Verbatim match: Exact overlap between generated and original text. |  Strong evidence of memorization. No hyperparameter. |  Highly brittle to noisy generation. Limited in scope.
Levenshtein Ratio: Minimum single-character edits (insertions, deletions or substitutions) between two sequences. |  Can recognize fragments of extracted information. Robust to noise. Continuous metric. |  Ambiguity at smaller thresholds due to coincidental similarity. Lacks interpretability.
Longest Common Substring (LCS): Longest common substring present in both sequences. |  Compromise between edit distance metrics and verbatim match. Interpretable. |  Does not account for the presence of multiple substrings.
$n$-gram similarity: Ratio of $n$-grams common between two sequences. |  Granular. Can also detect multiple fragments of information. |  Highly sensitive to the hyperparameter $n$.
Table 1: Overview of various similarity metrics in the context of extraction
attacks.

###  4.1 Challenges

Extraction attacks, specifically discoverable memorization, are often
evaluated in the literature using a verbatim match [17, 15, 9, 35, 46, 6, 10,
21, 18], i.e., the generated text must match the original test perfectly (see
Definition 3.1). However, this rigid metric does not take into account the
brittleness of LLM generations. For instance, consider the example of
generated text in Figure 2(a). Due to a single noisy token at the start, the
generated text is marked an unsuccessful attack. Yet, this example does indeed
extract the relevant data and leak sensitive information underscoring the
limitations of verbatim matching.

Another arbitrary convention in the literature is binarizing the success of an
attack based on extracting exactly $50$ tokens [15]. While this setup can be
useful for studying memorization trends, it oversimplifies real-world risks.
For instance, consider the distribution of verbatim match lengths for
generated text in Figure 2(c). Extractions of length close to $50$ tokens,
such as $49$, $48$, are marked unsuccessful, while extractions exceeding $50$
tokens (some even extending to more than $500$ tokens) are considered as
dangerous as $50$ tokens. These interpretations underestimate the real risks
of extraction (see Figure 2(a)). To navigate these challenges, we propose
using a continuous measure of risk that does not rely on verbatim match. We
now turn to approximate matching.

###  4.2 Approximate matching

Given our concerns about verbatim matching, we now turn to approximate
matching to quantify extraction risks. This approach is not entirely new and
has been previously used to quantify extraction risks for images [14].
Moreover, we also found several recent works, concurrent to our own, that
utilize approximate matching to quantify extraction risks for LLMs [38, 26,
30, 23]. We formalize this thread of research in the context of our stronger
adversary and define a new measure for composite discoverable memorization,
that provides a more precise metric to evaluate the risks of extraction:

######  Definition 4.1 (($s,\delta$)-Composite Discoverable Memorization).

For a set of $k$ models $Gen_{1},Gen_{2},...,$ $Gen_{k}$, a set of $r$ prompt
modifiers $F_{1},F_{2},...,F_{r}$, and an example
$[{\mathbf{p}}\parallel{\mathbf{x}}]$ from the training set ${X}$, we say
${\mathbf{x}}$ is ($s,\delta$)-composite discoverably memorized if
$\exists\;i\in[1\dots k]\;\textrm{and}\;j\in[1\dots r]\;\textrm{s.t.}\;$
$s(Gen_{i}(F_{j}({\mathbf{p}})),{\mathbf{x}})\geq\delta$.

Here, $s$ is a similarity metric, and $\delta$ is a threshold that controls
the degree of approximate matching. Furthermore, we do not restrict model
generations or true completion to $50$ tokens (more details in §5). By using a
similarity metric and allowing longer text generations, we move away from the
challenges of existing evaluation methods as described above. Although the
similarity metric is eventually binarized using $\delta$, this occurs only at
the final step, ensuring no information is lost. Additionally, the parameter
$\delta$ can be adjusted to study the trends of extraction under changing
flexibility (§7.2).

Similarity Metric. Our definition of approximate matching relies on a
similarity metric, $s$. We borrow several similarity metrics from the
extensive literature on text similarity. Table 1 provides a brief overview of
these metrics, highlighting their advantages and disadvantages. For a thorough
exploration of their definitions and the rationale for selection, we defer to
§A.

##  5 Experimental setup

In this section, we outline our experiment setup, to set the stage for our
empirical study to follow.

###  5.1 Models and Dataset

We use the Pythia suite [10] for all our experiments. Pythia suite contains
decoder-only language models with the same architecture as EleutherAI’s GPT-
Neo [12], albeit different training, across various model sizes and with open
source access to the complete training data and the intermediate checkpoints.
Pythia models were trained using GPT-NeoX library [5] on the Pile dataset [20]
and have not undergone any form of instruction-tuning. The standard version of
Pythia was trained over a single epoch of the Pile dataset, i.e., $\approx
143k$ steps with a batch size of $1024$, while the deduplicated version of
Pythia was trained over $\approx 1.5$ epochs of the deduplicated Pile dataset,
maintaining the same number of training steps as the standard version.

Pythia suite of models was developed with an emphasis on facilitating open-
source investigation into the training dynamics of LLMs. As such, they offer
access to (a) models of various sizes (we use model sizes: 1b, 1.4b, 2.8b,
6.9b, and 12b), (b) intermediate model checkpoints during training (a total of
$154$ checkpoints, with $144$ of them equally spaced, i.e., at every $1k$
training steps), and (c) the complete training data order, which is the same
for all model sizes. This level of accessibility and control over the training
setup allows us to simulate the real-world availability of models across
various sizes and with updating checkpoints over time.

Figure 3: Choosing prompts (pre $100k$ steps) and checkpoints (post $100k$
steps) for evaluation.

###  5.2 Evaluation methodology

We now describe our approach to the design and evaluation of extraction
attacks. Similar to Carlini et al. [15], we sample a representative portion of
the dataset for analyzing the performance of our extraction attacks. More
specifically, we uniformly sample $100,000$ sequences from the first $100k$
steps (batches) of the training data. This sampling strategy is crucial
because we choose model checkpoints for evaluation starting at step $100k$,
ensuring that every sentence evaluated for memorization has been seen by each
checkpoint under consideration, as illustrated in Figure 3.

Each sequence sampled is exactly $2049$ tokens. For our analysis, we employ a
consistent method of partitioning each sequence into a prompt and completion
at the midpoint, i.e., $1024$ tokens. Formally, for a sentence $s_{1:2049}$,
prompt length $l_{p}$, and completion length $l_{x}$, the example
$[{\mathbf{p}}\parallel{\mathbf{x}}]$ is defined as
${\mathbf{p}}=s_{1024-l_{p}:1024}$ and ${\mathbf{x}}=s_{1024:1024+l_{x}}$.
This partitioning allows us to systematically vary the prompt length and other
aspects of the prompt design while comparing the same completion.

Based on the discussion in §4, we increase the completion length from
$l_{x}=50$ in the previous literature [15, 17] to $l_{x}=500$. We argue this
gives us a more robust platform to study the concerns of risk underestimation
due to limited generation. However, this also scales down our extraction
rates; now, a success score of $1$ applies only to extractions of $500$
tokens, compared to the previous $50$ tokens in the literature. For readers
interested in direct comparisons with previous studies, we refer them to §C,
where we repeat our empirical study with completion length $l_{x}=50$ and
highlight similar trends as we see in our main paper.

Unless otherwise specified, we use a prompt length of $l_{p}=50$, Pythia-6.9b
model, and the 140k training step checkpoint, evaluating the extraction
attacks using verbatim match.

##  6 Churn in extraction trends

Churn [34], as defined in the literature, refers to regressive variance for
individuals despite an overall improvement. For instance, in our setup, while
using a longer prompt, targeting a larger model, or a model checkpoint that
has undergone longer training, is often recommended in the literature for
successful extraction attacks [15, 9], in this section, we demonstrate that
these trends exhibit churn, i.e., certain information extracted with shorter
prompts, smaller models, or earlier checkpoints is not extracted with the
stronger setups suggested in the literature. This churn can be exploited by an
adversary with multifaceted access to the data to execute a composite
extraction attack.

###  6.1 Prompt sensitivity

We start by examining prompt sensitivity, focusing on how trends in prompt
design can lead to churn.

$50$$200$$350$$500$$0.4$%$0.6$%$0.8$%Composite Ext. Rate Prompt Length
Extraction Rate (a) Extraction rate over Prompt Length

1b1.4b2.8b6.9b12b$0.1$%$0.2$%$0.3$%$0.4$%Composite Ext. Rate Model Size p (b)
Extraction rate over Model Size p

100k110k120k130k140k$0.2$%$0.3$%$0.4$%Composite Ext. Rate Training Step (c)
Extraction rate over Training Steps

100300500$0$%$0.2$%$0.4$%$0.6$% Prompt Length Extraction Rate (d) Extraction
rate over Masking and Removal

BaselineMask ($5\%$)Mask ($10\%$)Remove ($5\%$)Remove ($10\%$)Composite
Prefix$0$%$0.1$%$0.2$%$0.3$%Prefix g Extraction Rate (e) Extraction rate over
Prefixes g No PrefixNumbersAlphanumericComposite

Figure 4: Extraction rates under prompt sensitivity and across multiple
models. (a) Increasing prompt length results in better extraction rates, with
the composite extraction rate better than even at prompt length $500$. (b, c)
We see similar trends for increasing model size and training steps,
respectively. Specifically, we see the largest impact of the composite
extraction rate across training steps, with the extraction rate almost doubled
compared to any single checkpoint. (d) Adding noise to the prompt severely
hurts the extraction rate and does not provide any discernable churn,
highlighting the importance of prompt structure. (e) Adding a random prefix,
however, does not degrade the extraction rate as rapidly and can contribute
minor improvements to the composite extraction rate.

Prompt Length. Prompt length is a commonly studied parameter in extraction
attacks, and it has been shown that longer prompts lead to better extraction.
This is intuitive, as conditioning the model with more text from training
would increase the likelihood of obtaining the expected completion. We will
now show that the composite extraction rate (Definition 3.1) across varying
prompt lengths exceeds the extraction rate at even the largest prompt length.
As illustrated in Figure 4(a), the extraction rate increases with longer
prompts. However, the composite extraction rate is noticeably higher than any
single prompt length, including the longest at 500 tokens. This suggests that
certain information is extractable with shorter prompts but remains elusive
with the longest prompt. Consequently, an adversary can exploit this churn in
prompt length to extract more information.

Prompt Structure. Next, we explore the structure of prompts to identify what
makes a prompt potent and where churn can emerge. We first introduce noise
into the prompts by masking and removing random tokens; results are collected
in Figure 4(d). Despite introducing only a small amount of noise, we observed
a significant drop in extraction rates and no noticeable churn. This indicates
that the contiguous prompt from the training data is crucial for extracting
information, and any disruption inside this prompt can significantly hurt its
capabilities.

Given that noise within the prompts was ineffective, we next added noise as a
prefix in the form of random numeric and alphanumeric strings, results shown
in Figure 4(e). Although the performance with a noisy prefix still does not
surpass that of the clean prompt, the degradation in performance is less
severe compared to noise within the prompt. More importantly, we observed a
higher composite extraction rate. This suggests that adding a noisy prefix can
help extract unique information that was previously inaccessible. This further
highlights how an adversary can exploit repeated prompting to extract more
information, even with seemingly unintuitive changes like adding noise.

###  6.2 Multiple checkpoints

Model Size. The scale of a model has long been known to influence learning
trends, and our results of extraction trends in Figure 4(b) reflect this
phenomenon. As expected, we find that larger models tend to memorize more
information, making them more vulnerable to extraction attacks. However, our
results also indicate that the composite extraction rate is higher than the
extraction rate of any single model size, highlighting the churn present in
these trends. Biderman et al. [9] conducted a similar empirical study on the
overlap between memorized data across model sizes and found comparable
results. Their study showed that up to $10\%$ of the data memorized by smaller
models is not memorized by larger models. Combining our insights with existing
literature, it’s clear that releasing models in different sizes creates an
increased risk of information leakage.

Model Updates. We now analyze model updates over time by using intermediate
checkpoints released by Pythia, where we observe the most significant churn in
our study. Figure 4(c) illustrates the extraction rate across checkpoints over
time and the composite extraction rate. Unsurprisingly, attacking models at
later stages of training is more successful, as previously seen in the
literature. But remarkably, this churn is significantly powerful and by
exploiting composability across intermediate checkpoints, an adversary can
nearly double their extraction rate. This underscores the impact of
stochasticity in model training on extraction attacks and reveals that regular
model updates, typically considered beneficial in the current LLM ecosystem,
can create a considerably powerful adversary.

##  7 Towards realistic extraction attacks

With a better understanding of the underlying trends of extraction attacks
across various setups, we now evaluate a more realistic measure of information
leakage in LLMs under such attacks.

(a) Composite Extraction Rates Across Multiple Axes

0.23%0.34%0.44%0.38%0.52%0.67%0.43%0.59%0.75%Prompt LengthModel
Size1.4b+2.8b+6.9b100+200+300

0.25%0.48%0.55%0.42%0.67%0.77%0.48%0.74%0.84%Training StepsPrompt
Length100+200+300100k+120k+140k

0.33%0.45%0.50%0.44%0.59%0.66%0.56%0.77%0.84%Model SizeTraining
Steps100k+120k+140k1.4b+2.8b+6.9b

$0.1$$0.4$$0.7$$1$$2$%$4$% Similarity Threshold ($\delta$) Extraction Rate (b)
Comparing various similarity metrics

VerbatimLevenshteinLCS$n$-gram PilePile-Deduped$0$%$1$%$2$% Dataset Extraction
Rate (c) Towards realistic g extraction attacks g BaselineBest Single SetupAll
Setups CompositeLeven. ($\delta=0.95$)LCS ($\delta=0.5$)$n$-gram
($\delta=0.5$)

Figure 5: Towards more realistic extraction rates by combining various churn
trends and incorporating approximate matching. (a) Combining two axes at a
time, we see a monotonically increasing trend in extraction rates as we gain
more points of access to the underlying dataset, highlighting the growing
power of the adversary. (b) Various similarity metrics have distinct trends as
we decrease the threshold value and allow for looser approximations, thus
highlighting the importance of choosing the correct approximation metric based
on the context. (c) Combining multiple axes of attack and various approximate
matching metrics, highlighting the dramatic increase (up to $4\times$) in
extraction rate for both the standard setup and the deduplicated dataset.

###  7.1 Combining multiple axes of churn

In the previous section, we saw how churn can impact individual axes of
variability, such as prompt sensitivity, model size, or intermediate
checkpoints. However, a real-world adversary could have access to all these
factors simultaneously. We show that an adversary who can combine multiple
axes can significantly increase their extraction rates. We start by analyzing
two axes at a time, as shown in Figure 5(a). For all pairs of variability, the
overall composite extraction rate (bottom right) is 3-4 times higher than the
base setup (top left) and nearly double the composite extraction rates along
one axis (top right and bottom left). Furthermore, when all three axes are
combined, as depicted in Figure 5(c), the extraction rates grow even higher,
albeit with diminishing gains. Thus, we show that a real-world adversary is
capable of extracting far more information about the training data than has
been previously seen in the literature.

###  7.2 Approximate matching

Till now, our focus has been on how adversaries with multiple points of access
can extract more information. However, as discussed in §4, even the evaluation
of extraction information is not adequate in existing literature, and can
significantly underestimate the true risk of extraction. To address this gap,
we introduced $s,\delta$-composite discoverable memorization (Definition 4.1)
based on approximate matching. We will now analyze various similarity metrics
$s$ and examine how their extraction rates vary with different values of
$\delta$, reported in Figure 5(b).

Our results reveal intriguing trends. First, we analyze evaluations based on
the Levenshtein ratio metric and observe that setting a threshold of
$\delta=0.95$ results in a doubling of the extraction attack rate compared to
a verbatim match. This threshold signifies a minimum $95\%$ overlap between
generated and original text. Even under such a strict threshold, the doubling
of the extraction rate underscores the significant underestimation of
extraction risks when relying solely on verbatim matches. As $\delta$
decreases, however, the extraction rate increases exponentially, as the
Levenshtein ratio becomes less reliable under looser constraints.

Transitioning to other similarity metrics — longest common substring (LCS) and
$n$-gram matching — we find that even lower values of similarity($\delta$) can
contribute meaningfully to extraction attacks. For instance, $\delta=0.5$ for
LCS implies a contiguous sequence of $250$ tokens in the generated text
matching verbatim with the original text. Similarly, $\delta=0.5$ in $n$-gram
matching indicates at least $50\%$ overlap in $50$-grams ($n=50$) between both
sequences. In both cases, we observe patterns of rising extraction rates
similar to what we saw earlier with Levenshtein distance, reinforcing our
hypothesis that verbatim matching inadequately quantifies the information
leakage from the model.

###  7.3 Data deduplication

A commonly recommended solution to extraction attacks and memorization is data
deduplication, involving the removal of duplicate data entries within a
dataset [15]. While costly, data deduplication represents a critical aspect of
data curation and has been shown to mitigate extraction risks. To understand
the role of data deduplication in our discussion, we repeat our experiments
using models from Pythia trained on the deduplicated Pile dataset. The results
are collected in Figure 5(c).

In line with existing literature, we find that data deduplication effectively
reduces memorization, significantly lowering the extraction rate.
Interestingly, however, we observe persistent trends: the presence of a
stronger adversary due to multi-faceted dataset access, and challenges with
verbatim matching. We see that the increase in extract rate under our setup is
more pronounced in the deduplicated dataset compared to the original. We even
see the final extraction rate of a real-world adversary employing approximate
matching on the deduplicated dataset exceeds that achieved by the standard
setup in the existing literature on the original dataset.

Thus, while beneficial, data deduplication does not alter our fundamental
conclusions; real-world adversaries with multifaceted access to the underlying
training data can still extract substantial information even post-
deduplication. Future work is needed on incorporating and advancing more
concrete frameworks like differential privacy, to better understand such
adversaries, particularly from the perspective of privacy protection under
multi-access systems.

##  8 Discussion and future work

By highlighting the reality of an adversary performing extraction attacks in
the current LLM landscape, our work reveals a severe underestimation of
information leakage risks in the existing literature. It examines various
facets of access available to adversaries and the necessity of adopting
approximate matching techniques. By integrating these factors, the paper shows
an increased extraction rate of up to $2-4\times$ higher than even those
achieved by adversaries using the most vulnerable isolated setups with
verbatim matching. Thus, the composability of information leakage needs to be
explicitly considered when discussing data extraction risks, and we emphasize
the importance of considering an adversarial perspective in extraction
attacks.

Building on our study of real-world extraction risks, future research should
focus on translating these risks into concrete adversarial scenarios or
applications, such as identifying copyright claims, assessing the extent of
personally identifiable information (PII) released, etc. Similarly, as most of
our analysis focuses on the risks posed by extraction attacks under the lens
of discoverable memorization, future research should explore how our findings
translate to other forms of privacy attacks. Finally, our study addresses the
threats posed by powerful real-world adversaries but does not propose specific
defence methods. Further exploration is needed to navigate the current LLM
ecosystem and mitigate the risks posed by these strong adversaries. Our work
provides a foundation for future exploration and defence against more
realistic extraction attacks, contributing to a more secure and robust
management of the risks associated with extraction attacks in LLMs.

## Acknowledgments and Disclosure of Funding

Funding support for project activities has been partially provided by Canada
CIFAR AI Chair, and Google award. We also express our gratitude to Compute
Canada for their support in providing facilities for our evaluations.

## References

  * [1]
  * Achiam et al. [2023] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023\.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ (2023).
  * Adam et al. [2023] George Adam, Benjamin Haibe-Kains, and Anna Goldenberg. 2023.  Maintaining Stability and Plasticity for Predictive Churn Reduction.  _arXiv preprint arXiv:2305.04135_ (2023).
  * Almazrouei et al. [2023] Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. 2023\.  The falcon series of open language models.  _arXiv preprint arXiv:2311.16867_ (2023).
  * Andonian et al. [2023] Alex Andonian, Quentin Anthony, Stella Biderman, Sid Black, Preetham Gali, Leo Gao, Eric Hallahan, Josh Levy-Kramer, Connor Leahy, Lucas Nestler, Kip Parker, Michael Pieler, Jason Phang, Shivanshu Purohit, Hailey Schoelkopf, Dashiell Stander, Tri Songz, Curt Tigges, Benjamin Thérien, Phil Wang, and Samuel Weinbach. 2023.  _GPT-NeoX: Large Scale Autoregressive Language Modeling in PyTorch_.  https://doi.org/10.5281/zenodo.5879544
  * Anil et al. [2023] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023\.  Palm 2 technical report.  _arXiv preprint arXiv:2305.10403_ (2023).
  * Anil et al. [2018] Rohan Anil, Gabriel Pereyra, Alexandre Passos, Robert Ormandi, George E Dahl, and Geoffrey E Hinton. 2018.  Large scale distributed neural network training through online distillation. In _International Conference on Learning Representations_.
  * Bahri and Jiang [2021] Dara Bahri and Heinrich Jiang. 2021.  Locally Adaptive Label Smoothing Improves Predictive Churn. In _Proceedings of the 38th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 139)_ , Marina Meila and Tong Zhang (Eds.). PMLR, 532–542.
  * Biderman et al. [2023a] Stella Biderman, USVSN PRASHANTH, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raff. 2023a.  Emergent and predictable memorization in large language models.  _Advances in Neural Information Processing Systems_ 36 (2023).
  * Biderman et al. [2023b] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023b.  Pythia: A suite for analyzing large language models across training and scaling. In _International Conference on Machine Learning_. PMLR, 2397–2430.
  * Birch et al. [2023] Lewis Birch, William Hackett, Stefan Trawicki, Neeraj Suri, and Peter Garraghan. 2023.  Model Leeching: An Extraction Attack Targeting LLMs.  arXiv:2309.10544 [cs.LG]
  * Black et al. [2022] Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. 2022\.  GPT-NeoX-20B: An Open-Source Autoregressive Language Model.  _Challenges & Perspectives in Creating Large Language Models_ (2022), 95.
  * Carlini et al. [2022] Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. 2022.  Membership inference attacks from first principles. In _IEEE Symposium on Security and Privacy_. IEEE.
  * Carlini et al. [2023a] N Carlini, J Hayes, M Na sr, M Jagielski, V Sehwag, F Tramèr, B Balle, D Ippolito, and E Wallace. 2023a.  Extracting Training Data from Diffusion Models. In _32nd USENIX Security Symposium (USENIX Security 23)_. USENIX Association.
  * Carlini et al. [2023b] Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2023b.  Quantifying Memorization Across Neural Language Models. In _The Eleventh International Conference on Learning Representations_.  https://openreview.net/forum?id=TatRHT_1cK
  * Carlini et al. [2019] Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. 2019.  The secret sharer: evaluating and testing unintended memorization in neural networks. In _Proceedings of the 28th USENIX Conference on Security Symposium_ (Santa Clara, CA, USA) _(SEC’19)_. USENIX Association, USA, 267–284.
  * Carlini et al. [2021] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021\.  Extracting training data from large language models. In _30th USENIX Security Symposium (USENIX Security 21)_. 2633–2650.
  * Chowdhery et al. [2023] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023\.  Palm: Scaling language modeling with pathways.  _Journal of Machine Learning Research_ 24, 240 (2023), 1–113.
  * Cotter et al. [2019] Andrew Cotter, Heinrich Jiang, Maya Gupta, Serena Wang, Taman Narayan, Seungil You, and Karthik Sridharan. 2019.  Optimization with non-differentiable constraints with applications to fairness, recall, churn, and other goals.  _Journal of Machine Learning Research_ 20, 172 (2019), 1–59.
  * Gao et al. [2020] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020\.  The pile: An 800gb dataset of diverse text for language modeling.  _arXiv preprint arXiv:2101.00027_ (2020).
  * Huang et al. [2022] Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang. 2022.  Are Large Pre-Trained Language Models Leaking Your Personal Information?. In _Findings of the Association for Computational Linguistics: EMNLP 2022_. 2038–2047.
  * Hubinger et al. [2024] Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tamera Lanham, Daniel M. Ziegler, Tim Maxwell, Newton Cheng, Adam Jermyn, Amanda Askell, Ansh Radhakrishnan, Cem Anil, David Duvenaud, Deep Ganguli, Fazl Barez, Jack Clark, Kamal Ndousse, Kshitij Sachan, Michael Sellitto, Mrinank Sharma, Nova DasSarma, Roger Grosse, Shauna Kravec, Yuntao Bai, Zachary Witten, Marina Favaro, Jan Brauner, Holden Karnofsky, Paul Christiano, Samuel R. Bowman, Logan Graham, Jared Kaplan, Sören Mindermann, Ryan Greenblatt, Buck Shlegeris, Nicholas Schiefer, and Ethan Perez. 2024.  Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training.  arXiv:2401.05566 [cs.CR]
  * Ippolito et al. [2022] Daphne Ippolito, Florian Tramèr, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher A Choquette-Choo, and Nicholas Carlini. 2022.  Preventing verbatim memorization in language models gives a false sense of privacy.  _arXiv preprint arXiv:2210.17546_ (2022).
  * Jiang et al. [2021] Heinrich Jiang, Harikrishna Narasimhan, Dara Bahri, Andrew Cotter, and Afshin Rostamizadeh. 2021.  Churn Reduction via Distillation. In _International Conference on Learning Representations_.
  * Kandpal et al. [2022] Nikhil Kandpal, Eric Wallace, and Colin Raffel. 2022.  Deduplicating training data mitigates privacy risks in language models. In _International Conference on Machine Learning_. PMLR, 10697–10707.
  * Kassem et al. [2024] Aly M Kassem, Omar Mahmoud, Niloofar Mireshghallah, Hyunwoo Kim, Yulia Tsvetkov, Yejin Choi, Sherif Saad, and Santu Rana. 2024.  Alpaca against Vicuna: Using LLMs to Uncover Memorization of LLMs.  _arXiv preprint arXiv:2403.04801_ (2024).
  * Liu et al. [2021] Leo Z. Liu, Yizhong Wang, Jungo Kasai, Hannaneh Hajishirzi, and Noah A. Smith. 2021.  Probing Across Time: What Does RoBERTa Know and When?  _CoRR_ abs/2104.07885 (2021).  arXiv:2104.07885 https://arxiv.org/abs/2104.07885
  * Liu et al. [2023b] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023b.  Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing.  _ACM Comput. Surv._ 55, 9, Article 195 (jan 2023), 35 pages.  https://doi.org/10.1145/3560815
  * Liu et al. [2024] Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, Kailong Wang, and Yang Liu. 2024.  Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study.  arXiv:2305.13860 [cs.SE]
  * Liu et al. [2023a] Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo, Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. 2023a.  Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models’ Alignment. In _Socially Responsible Language Modelling Research_.
  * Mattern et al. [2023] Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schoelkopf, Mrinmaya Sachan, and Taylor Berg-Kirkpatrick. 2023.  Membership Inference Attacks against Language Models via Neighbourhood Comparison. In _Findings of the Association for Computational Linguistics: ACL 2023_. 11330–11343.
  * McSherry [2009] Frank D McSherry. 2009.  Privacy integrated queries: an extensible platform for privacy-preserving data analysis. In _Proceedings of the 2009 ACM SIGMOD International Conference on Management of data_. 19–30.
  * Meta AI [2024] Meta AI Meta AI. 2024.  Introducing Meta Llama 3: The most capable openly available LLM to date.  https://ai.meta.com/blog/meta-llama-3/
  * Milani Fard et al. [2016] Mahdi Milani Fard, Quentin Cormier, Kevin Canini, and Maya Gupta. 2016.  Launch and iterate: Reducing prediction churn.  _Advances in Neural Information Processing Systems_ 29 (2016).
  * Nasr et al. [2023] Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A. Feder Cooper, Daphne Ippolito, Christopher A. Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. 2023.  Scalable Extraction of Training Data from (Production) Language Models.  arXiv:2311.17035 [cs.LG]
  * OpenAI [2024] OpenAI. 2024.  ChatGPT Documentation: Models.  https://platform.openai.com/docs/models/overview
  * Penedo et al. [2024] Guilherme Penedo, Hynek Kydlíček, Leandro von Werra, and Thomas Wolf. 2024.  _FineWeb_.  https://doi.org/10.57967/hf/2092
  * Qi et al. [2024] Zhenting Qi, Hanlin Zhang, Eric Xing, Sham Kakade, and Himabindu Lakkaraju. 2024.  Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systems.  _arXiv preprint arXiv:2402.17840_ (2024).
  * Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of machine learning research_ 21, 140 (2020), 1–67.
  * Rossi et al. [2024] Sippo Rossi, Alisia Marianne Michel, Raghava Rao Mukkamala, and Jason Bennett Thatcher. 2024.  An Early Categorization of Prompt Injection Attacks on Large Language Models.  arXiv:2402.00898 [cs.CR]
  * Salinas and Morstatter [2024] Abel Salinas and Fred Morstatter. 2024.  The Butterfly Effect of Altering Prompts: How Small Changes and Jailbreaks Affect Large Language Model Performance.  arXiv:arXiv:2401.03729
  * Schwarzschild et al. [2024] Avi Schwarzschild, Zhili Feng, Pratyush Maini, Zachary C. Lipton, and J. Zico Kolter. 2024.  Rethinking LLM Memorization through the Lens of Adversarial Compression.  arXiv:2404.15146 [cs.LG]
  * Sclar et al. [2024] Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024.  Quantifying Language Models’ Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting. In _The Twelfth International Conference on Learning Representations_.  https://openreview.net/forum?id=RIu5lyNXjT
  * Soboleva et al. [2023] Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. 2023.  SlimPajama: A 627B token cleaned and deduplicated version of RedPajama.  https://huggingface.co/datasets/cerebras/SlimPajama-627B
  * Team et al. [2023] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023\.  Gemini: a family of highly capable multimodal models.  _arXiv preprint arXiv:2312.11805_ (2023).
  * Tirumala et al. [2022] Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. 2022.  Memorization without overfitting: Analyzing the training dynamics of large language models.  _Advances in Neural Information Processing Systems_ 35 (2022), 38274–38290.
  * Watson-Daniels et al. [2024] Jamelle Watson-Daniels, Flavio du Pin Calmon, Alexander D’Amour, Carol Long, David C. Parkes, and Berk Ustun. 2024.  Predictive Churn with the Set of Good Models.  _arxiv_ (2024).
  * Xia et al. [2023] Mengzhou Xia, Mikel Artetxe, Chunting Zhou, Xi Victoria Lin, Ramakanth Pasunuru, Danqi Chen, Luke Zettlemoyer, and Ves Stoyanov. 2023.  Training Trajectories of Language Models Across Scales. In _61st Annual Meeting of the Association for Computational Linguistics, ACL 2023_. Association for Computational Linguistics (ACL), 13711–13738.
  * Yu et al. [2023] Weichen Yu, Tianyu Pang, Qian Liu, Chao Du, Bingyi Kang, Yan Huang, Min Lin, and Shuicheng Yan. 2023.  Bag of tricks for training data extraction from language models. In _International Conference on Machine Learning_. PMLR, 40306–40320.
  * Zhao et al. [2023] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023\.  A survey of large language models.  _arXiv preprint arXiv:2303.18223_ (2023).
  * Zhu et al. [2023] Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Yue Zhang, Neil Zhenqiang Gong, and Xing Xie. 2023.  PromptBench: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts.  arXiv:2306.04528 [cs.CL]

##  Appendix A Similarity metrics

In this section, we define the approximate matching metrics used in our paper,
as well as our rationale for not using some other common approximate matching
methods that are present in the literature.

Verbatim Match: The verbatim match score between two strings $a,b$ is $1$ if
both strings match exactly, and $0$ otherwise. Formally,

$\operatorname{verbatimmatch}(a,b)=\begin{cases}1&\text{ if }a=b,\\\ 0&\text{
otherwise }\end{cases}$

Levenshtein Ratio: The levenshtein ratio between two strings $a,b$ of length
$|a|,|b|$ respectively is given as,

$\operatorname{levenratio}(a,b)=1-\dfrac{\operatorname{levendistance}(a,b)}{|a|+|b|}$

where,

$\operatorname{levendistance}(a,b)=\begin{cases}|a|&\text{ if }|b|=0,\\\
|b|&\text{ if }|a|=0,\\\
\operatorname{levendistance}(\operatorname{tail}(a),\operatorname{tail}(b))&\text{
if }\operatorname{head}(a)=\operatorname{head}(b),\\\
1+\min\left\\{\begin{array}[]{l}\operatorname{levendistance}(\operatorname{tail}(a),b)\\\
\operatorname{levendistance}(a,\operatorname{tail}(b))\\\
\operatorname{levendistance}(\operatorname{tail}(a),\operatorname{tail}(b))\end{array}\right.&\text{
otherwise }\end{cases}$

Here the tail of string $x$ is a string of all but the first character of $x$,
i.e. $\operatorname{tail}\left(x_{0}x_{1}\ldots x_{n}\right)=x_{1}x_{2}\ldots
x_{n}$, and the head of string $x$ is the first character of $x$, i.e.
$\operatorname{head}\left(x_{0}x_{1}\ldots x_{n}\right)=x_{0}$.

Longest Common Substring (LCS): The longest common substring (LCS) between two
strings $a,b$ is the longest contiguous subsequence which is common between
both strings. We define the LCS score by normalizing it with the length of the
longest string (i.e., $\operatorname{max}(|a|,|b|)$). Formally,

$\operatorname{lcsratio}(a,b)=\dfrac{\operatorname{lcs}(a,b)}{\operatorname{max}(|a|,|b|)}$

where,

$\operatorname{lcs}(a,b)=max_{i\in|a|;j\in|b|}\operatorname{L}(a,b,i,j)$
$\operatorname{L}(a,b,i,j)=\begin{cases}0&\text{ if }i=0,\\\ 0&\text{ if
}j=0,\\\ \operatorname{L}(a,b,i-1,j-1)+1&\text{ if }a[i]=b[j],\\\ 0&\text{
otherwise }\end{cases}$

Note that this is different from another commonly used metric called longest
common subsequence, which does not restrict the matching subsequence to be
contiguous, and is hence a more lenient definition of a match compared to the
longest common substring used here.

$n$-gram Similarity: The n-gram similarity is a measure of the number of
common $n$-grams between two strings $a,b$ and is defined as the Jaccard
similarity between the set of $n$-grams for both strings,

$\operatorname{n-gramsim}(a,b)=\dfrac{|\operatorname{n-grams}(a)\cap\operatorname{n-grams}(b)|}{|\operatorname{n-grams}(a)\cup\operatorname{n-grams}(b)|}$

where $\operatorname{n-grams}(x)$ is simply the set of all $n$-grams present
in some string $x$. We use $n=50$ for all our experiments in this paper, i.e.,
we are measuring $\operatorname{50-gramsim}(a,b)$.

There are many other text similarity metrics present in the literature.
However, we argue that many of them do not align with the requirements of
measuring memorization. Simply the presence of certain words (as recorded by
ROGUE and BLEU scores) or similar meanings of the sentence (as recorded by
cosine similarity, BERT score, etc.) should not by itself be considered a form
of memorization. Thus, we chose the string overlap metrics defined above
instead of other alternatives in string matching literature. Nevertheless, our
framework can be easily extended to other approximate matching metrics.

##  Appendix B Additional results

We provide additional results for trends under Levenshtein distance
$\delta=0.95$ (Figure 6) and for the models trained on the deduplicated
dataset (Figure 7).

$50$$200$$350$$500$$0.5$%$1$%Composite Ext. Rate Prompt Length Extraction Rate

1b1.4b2.8b6.9b12b$0.2$%$0.4$%$0.6$%Composite Ext. Rate Model Size

100k110k120k130k140k$0.4$%$0.6$%$0.8$%Composite Ext. Rate Training Step

100300500$0$%$0.5$%$1$% Prompt Length Extraction Rate

BaselineMask ($5\%$)Mask ($10\%$)Remove ($5\%$)Remove ($10\%$)Composite
Prefix$0$%$0.2$%$0.4$%Prefix Extraction Rate No
PrefixNumbersAlphanumericComposite

Figure 6: Extraction rates under prompt sensitivity and across models, under
Levenshtein $\delta=0.95$.

$50$$200$$350$$500$$0.1$%$0.2$%$0.3$%Composite Ext. Rate Prompt Length
Extraction Rate

1b1.4b2.8b6.9b12b$0.1$%$0.2$%Composite Ext. Rate Model Size

100k110k120k130k140k$0.1$%$0.2$%Composite Ext. Rate Training Step

Figure 7: Extraction rates under prompt sensitivity and across models, for the
deduplicated setup.

##  Appendix C Direct comparison with Carlini et al. [15]

We compare our results directly with Carlini et al. [15] by using their setup,
i.e., completion length $l_{x}=50$. The results are collected in Figure 8 and
show similar trends as seen in the main paper.

$50$$200$$350$$500$$1$%$2$%$3$%$4$%Composite Ext. Rate Prompt Length
Extraction Rate

1b1.4b2.8b6.9b12b$0.6$%$0.8$%$1$%$1.2$%$1.4$%Composite Ext. Rate Model Size

100k110k120k130k140k$1.2$%$1.4$%$1.6$%$1.8$%$2$%Composite Ext. Rate Training
Step

Figure 8: Extraction rates under prompt sensitivity and across models, for
completion length 50.