# Many-Shot Regurgitation (MSR) Prompting

Shashank Sonkar
Rice University
Houston, TX
<EMAIL_ADDRESS>
&Richard G. Baraniuk
Rice University
Houston, TX
<EMAIL_ADDRESS>

###### Abstract

We introduce Many-Shot Regurgitation (MSR) prompting, a new black-box
membership inference attack framework for examining verbatim content
reproduction in large language models (LLMs). MSR prompting involves dividing
the input text into multiple segments and creating a single prompt that
includes a series of faux conversation rounds between a user and a language
model to elicit verbatim regurgitation. We apply MSR prompting to diverse text
sources, including Wikipedia articles and open educational resources (OER)
textbooks, which provide high-quality, factual content and are continuously
updated over time. For each source, we curate two dataset types: one that LLMs
were likely exposed to during training ($D_{\rm pre}$) and another consisting
of documents published after the models’ training cutoff dates ($D_{\rm
post}$). To quantify the occurrence of verbatim matches, we employ the Longest
Common Substring algorithm and count the frequency of matches at different
length thresholds. We then use statistical measures such as Cliff’s delta,
Kolmogorov-Smirnov (KS) distance, and Kruskal-Wallis H test to determine
whether the distribution of verbatim matches differs significantly between
$D_{\rm pre}$ and $D_{\rm post}$. Our findings reveal a striking difference in
the distribution of verbatim matches between $D_{\rm pre}$ and $D_{\rm post}$,
with the frequency of verbatim reproduction being significantly higher when
LLMs (e.g. GPT models and LLaMAs) are prompted with text from datasets they
were likely trained on. For instance, when using GPT-3.5 on Wikipedia
articles, we observe a substantial effect size (Cliff’s delta $=-0.984$) and a
large KS distance ($0.875$) between the distributions of $D_{\rm pre}$ and
$D_{\rm post}$. Our results provide compelling evidence that LLMs are more
prone to reproducing verbatim content when the input text is likely sourced
from their training data.

##  1 Introduction

Large language models (LLMs) have revolutionized the field of natural language
processing (NLP) with their remarkable ability to generate human-like text
across a wide range of domains. These models, such as GPT [1], Gemini [2], and
LLaMA [3, 4], are trained on vast amounts of diverse text data, enabling them
to produce coherent and contextually relevant responses to given prompts.
Important emerging questions relevant to artificial intelligence (AI)
alignment [5, 6] revolve around the how and where a model’s training data was
obtained. In this paper, we study the following specific question that is
relevant to studying training data provenance: to what extent do these models
reproduce verbatim content from their training data? We develop the Many-Shot
Regurgitation (MSR) prompting technique, a new approach designed to
efficiently investigate this phenomenon. Code is available on
github111https://github.com/luffycodes/Many-Shot-Regurgitation-MIA.

MSR represents a new framework for membership inference attack (MIA) against
language models. MIAs aim to determine whether a given input was part of a
model’s training data and have been studied extensively in both computer
vision [7, 8] and natural language processing domains [9, 10, 11]. However,
most prior MIA research on language models has focused on the fine-tuning
stage [12, 11, 10]and assumes access to model internals such as logits, token
probabilities, or loss values [13, 14, 15]. In contrast, MSR operates in a
black-box setting, requiring only the ability to prompt the model and observe
its outputs. This makes MSR applicable to studying real-world deployed models
like GPT-3.5 [16] and GPT-4 [1, 17], where access to model internals is no
longer available due to API changes.

The MSR prompting technique, inspired by the many-shot jailbreaking approach
[18, 19], involves dividing the input text into multiple segments and creating
a single prompt that includes a series of faux conversation rounds between a
user and a language model designed to elicit verbatim regurgitation. The input
text is segmented into multiple parts, denoted as $T_{1}$, $T_{2}$, …,
$T_{n}$, where $n$ is the total number of segments. A single prompt is then
constructed, creating a simulated conversation sequence between a user and a
language model. Throughout these rounds of conversation, the user supplies the
odd-numbered segments ($T_{1}$, $T_{3}$, …, $T_{n-1}$), while the even-
numbered segments ($T_{2}$, $T_{4}$, …, $T_{n-2}$) are presented as if they
were generated by the language model. In the final round, the user presents
the penultimate segment ($T_{n-1}$), prompting the language model to generate
the concluding segment ($T_{n}^{\prime}$). It is worth highlighting that the
segments presented as responses from the language model are not genuine
outputs but are integrated into the prompt to mimic a conversation. The sole
genuine output from the LLM is the final segment ($T_{n}^{\prime}$).

While quantifying the occurrence of verbatim matches between the generated
text ($T_{n}^{\prime}$) and the original segment ($T_{n}$) is important, it
doesn’t provide a complete understanding of verbatim reproduction. Thus, we
also explore whether the distribution of these verbatim matches is
statistically different from matches in documents that the LLMs are unlikely
to have been trained on. To achieve this, we curate two types of dataset
sources: $D_{\rm pre}$, which includes documents that the LLMs were likely
exposed to during training, and $D_{\rm post}$, which consists of documents
that were published after the LLMs’ training cutoff dates. Specifically, we
utilize Wikipedia articles and Open Educational Resource (OER) textbooks as
they provide high-quality, factual content and are continuously updated over
time.

Through experiments employing the MSR prompting technique on both $D_{\rm
pre}$ and $D_{\rm post}$ datasets, we aim to compare the distributions of
verbatim matches. We employ statistical tests such as Cliff’s Delta [20],
Kolmogorov-Smirnov (KS) Distance [21], and the Kruskal-Wallis H Test [22] to
evaluate the statistical significance of the differences in verbatim
reproduction between the two dataset types. Our experiments are performed
using three state-of-the-art LLMs: GPT-3.5 [16], GPT-4 [1], and LLaMA-3 [4].

Our experiments reveal that verbatim reproduction occurs in text generated by
different LLMs, with the extent of reproduction varying across dataset sources
and language models. We find that the frequency of verbatim matches is
generally higher for $D_{\rm pre}$ compared to $D_{\rm post}$, suggesting that
the availability of the dataset influences the likelihood of verbatim
reproduction. For example, using GPT-3.5 on Wikipedia articles, we observe a
Cliff’s delta of $-0.984$ and a Kolmogorov-Smirnov distance of $0.875$,
indicating a significant difference in the distribution of verbatim matches
between $D_{\rm pre}$ and $D_{\rm post}$. Furthermore, we investigate the
impact of factors such as the number of shots in MSR prompting and the
temperature settings of LLMs on the occurrence of verbatim reproduction. We
find that increasing the number of shots generally leads to higher frequencies
of verbatim matches, with 6 shots yielding the best results. Additionally,
lower temperature settings favor more deterministic and less diverse outputs,
leading to a higher likelihood of verbatim regurgitation.

The main contributions of this paper are as follows:

  * •

We introduce the Many-Shot Regurgitation (MSR) prompting technique, a novel
approach to investigate verbatim reproduction in LLMs across diverse domains
and dataset sources.

  * •

We propose a methodology that not only quantifies the occurrence of verbatim
matches but also determines whether the distribution of these matches is
statistically different from matches in documents that the LLMs are unlikely
to have been trained on, using statistical measures such as Cliff’s delta,
Kolmogorov-Smirnov distance, and the Kruskal-Wallis H test.

  * •

We conduct extensive experiments using the MSR prompting technique on
carefully curated dataset sources ($D_{\rm pre}$ and $D_{\rm post}$),
including Wikipedia articles and OER textbooks, and three state-of-the-art
LLMs (GPT-3.5, GPT-4, and LLAMA) to quantify the extent of verbatim
reproduction and analyze the influence of dataset availability on the
occurrence of verbatim reproduction.

  * •

We investigate the impact of factors such as the number of shots in MSR
prompting and the temperature settings of LLMs on the likelihood of verbatim
reproduction.

The remainder of this paper is organized as follows: Section 2 provides an
overview of related work on verbatim reproduction and memorization in LLMs.
Section 3 describes the proposed MSR prompting technique and the dataset
sources used in our experiments. Section 4 presents the experimental setup,
results, and analysis of our findings. Finally, Section 5 concludes the paper
and discusses the implications of our work for the development and usage of
LLMs in various domains.

##  2 Related Work

Membership Inference Attacks (MIAs) aim to determine whether a given data
sample was part of a model’s training data [13, 14]. MIAs have been
extensively studied in both computer vision [7, 8, 23] and natural language
processing domains [9, 10, 11, 12]. These attacks have implications for
quantifying privacy risks [24], measuring memorization [25], auditing models
[26, 27], and detecting issues like test set contamination [28] and
unauthorized use of copyrighted content [29, 30]. In the language modeling
domain, most MIA research focuses on the fine-tuning stage [12, 11, 10] rather
than pretraining. [14] demonstrated that language models can memorize verbatim
content from training data, but their method requires manually inspecting
model outputs. Subsequent work has explored more automated approaches. [9]
studied membership inference on embedding models, showing that they can leak
sensitive attributes. [31] proposed a method for auditing embedding models’
training data using gradient updates. For auto-regressive language models,
[14] introduced the exposure metric to measure data leakage but note that
Monte Carlo estimation of this metric is challenging for large models. [32]
showed that deduplicating training data can mitigate privacy risks, but doing
so for large web-scale corpora is difficult. Membership inference against
large pre-trained language models poses unique challenges compared to
conventional MIA settings, such as limited access to training data and model
scale. Assumptions like having shadow models trained on the same data
distribution often do not hold, as pretraining corpora tend to be proprietary
[33, 15]. The sheer scale of pretraining, involving massive datasets and
relatively few epochs, makes MIAs inherently more difficult.

Recent work has started exploring pretraining data detection for large
language models. [33] proposed the Min-K% method which uses the average log
probability of low-probability tokens as a score for pretraining data
detection. However, this approach relies on heuristics and access to token
probabilities which are no longer available for models like GPT-3. [15]
constructed the MIMIR benchmark and analyzed the challenges of pretraining
data detection, noting the need for better methods. Most prior MIA methods
assume some form of access to a model’s internals such as logits, gradients,
loss values or token probabilities [13, 14, 10, 26]. However, this poses
challenges for studying deployed language models in real-world settings. For
example, the GPT-3 API previously provided access to logits and token
probabilities but this functionality has been deprecated [33, 15], precluding
the use of methods that rely on these values. Access to gradients or internal
model activations is even more restricted. Therefore, there is a need for MIA
frameworks that can operate under black-box settings with only API-level
access.

Our work addresses this gap by introducing the Many-Shot Regurgitation (MSR)
technique, a black-box MIA framework that requires no access to model
internals. MSR exploits LLMs’ conversational and continuation abilities to
elicit verbatim regurgitation, which can then be precisely quantified and
statistically analyzed. This differentiates MSR from prior approaches that
rely on manual inspection [14], heuristics based on token probabilities [33],
or access to the full training dataset.

While most MIA research focuses on detecting individual training examples, our
work takes a distributional approach to assess a model’s overall propensity
for verbatim regurgitation. By comparing the frequency distributions of
verbatim matches between datasets that were likely seen during training
($D_{\rm pre}$) and those that were not ($D_{\rm post}$), we can quantify the
influence of dataset availability on regurgitation behavior. This comparative
analysis is particularly valuable in settings where verbatim reproduction is
rare but still concerning, such as identifying the presence of copyrighted
articles or book chapters in training data. Even if only a small fraction of a
document’s content is reproduced verbatim, it can serve as evidence of that
document’s inclusion in the training corpus.

##  3 Methodology

In this study, we introduce the Many-Shot Regurgitation (MSR) prompting
technique to investigate the extent of verbatim regurgitation in text
generated by language models. The primary objective of our methodology is to
quantify the occurrence of verbatim matches between the generated text and the
source material, and to determine if the distribution of these matches differs
significantly between datasets that the language models were likely trained on
and those that were released after the models’ training cutoff dates.

###  3.1 Dataset Curation

To achieve our research goals, we carefully curate two types of dataset
sources: Wikipedia articles and Open Educational Resource (OER) textbooks.
These sources are chosen for their high-quality, factual content and the
continuous addition of new material over time.

Wikipedia, being one of the largest and most comprehensive online
encyclopedias, covers a wide range of topics and is frequently updated with
new articles and information. By selecting Wikipedia articles written in
different time periods, we can effectively compare the verbatim regurgitation
behavior of language models on articles that were likely included in their
training data and those that were added after the models’ training cutoff
dates.

Similarly, OER textbooks provide a rich source of educational content across
various subjects. The release of new OER textbooks and the updating of
existing ones allow us to investigate the extent of verbatim regurgitation in
language models when prompted with textbook content that was available during
their training and content that was published later.

For each dataset source, we select a subset of documents that were likely
included in the language models’ training data ($D_{\rm pre}$) and a subset of
documents that were released after the models’ training cutoff dates ($D_{\rm
post}$). By creating this controlled setting, we aim to investigate the
influence of dataset age and availability on the extent of verbatim
regurgitation and draw insights into the models’ ability to generate novel
content.

###  3.2 Many-Shot Regurgitation (MSR) Prompting

The MSR prompting technique involves dividing the source text into multiple
segments (shots) and creating a single prompt with a series of alternating
turns between a user and the language model. The key steps of the MSR
prompting technique are as follows:

  1. 1.

Text Segmentation: The source text is divided into an even number of segments,
denoted as $T_{1}$, $T_{2}$, …, $T_{n}$, where $n$ is the total number of
segments.

  2. 2.

Prompt Construction: A single prompt is created by concatenating the segments
in an alternating manner, simulating a conversation between a user and the
language model. The user provides the odd-numbered segments ($T_{1}$, $T_{3}$,
…, $T_{n-1}$), and the language model is prompted to generate the even-
numbered segments ($T_{2}$, $T_{4}$, …, $T_{n}$).

System: complete the paragraph User: $T_{1}$ LLM: $T_{2}$ User: $T_{3}$ LLM:
$T_{4}$ … User: $T_{n-1}$ LLM: (generates $T_{n}^{\prime}$) single prompt with
alternating turns

It’s important to note that the parts labeled as LLM’s responses ($T_{2}$,
$T_{4}$, …, $T_{n-2}$) are not actually generated by the model but are
included in the prompt to create the illusion that LLM has generated those
parts. Only the last part ($T_{n}^{\prime}$) is the genuine output from the
LLM.

  3. 3.

Text Generation: The language model is tasked with generating the final
segment $T_{n}$ based on the constructed prompt. The generated segment is
denoted as $T_{n}^{\prime}$.

By presenting the source text in this format, the MSR prompting technique aims
to examine how language models respond when prompted to complete a passage
that closely resembles the original source material.

###  3.3 Verbatim Match Analysis

To quantify the occurrence of verbatim regurgitation, we analyze the generated
text ($T_{n}^{\prime}$) and compare it with the corresponding original segment
($T_{n}$) using the Longest Common Substring (LCS) algorithm. The LCS
algorithm identifies the longest contiguous substring that is present in both
the generated and original segments.

We define a verbatim match as a substring of a specified minimum length (e.g.,
$\ell_{\rm min}$ to $\ell_{\rm max}$ words) that is identical between the
generated and original segments. Formally, let $M_{k}$ be the set of all
substring matches of length $k$ between $T_{n}^{\prime}$ and $T_{n}$. The
frequency of exact verbatim matches of length $k$ is defined as:

$f_{k}=|M_{k}|$ (1)

For each language model and dataset source combination, we obtain an array of
frequencies $F=[f_{\ell_{\rm min}},f_{\ell_{\rm min}+1},...,f_{\ell_{\rm
max}}]$ corresponding to verbatim match length thresholds from $\ell_{\rm
min}$ to $\ell_{\rm max}$ words.

By counting the frequency of verbatim matches at different length thresholds,
we obtain a distribution of verbatim match frequencies for each combination of
language model and dataset source.

###  3.4 Statistical Analysis

To determine if the distribution of verbatim matches differs significantly
between $D_{\rm pre}$ and $D_{\rm post}$, we perform statistical analysis on
the arrays of frequencies $F$ obtained for each language model and dataset
source combination. We employ several statistical measures:

  * •

Cliff’s delta [20]: A non-parametric effect size measure that quantifies the
difference between two distributions. It ranges from -1 to +1, with 0
indicating no difference and ±1 indicating complete separation between the
distributions. We chose Cliff’s delta to robustly compare the verbatim match
frequencies between $D_{old}$ and $D_{new}$, as it is less sensitive to
outliers and does not assume a particular distribution shape.

  * •

Kolmogorov-Smirnov (KS) distance [21]: A non-parametric test that measures the
maximum distance between the cumulative distribution functions of two
distributions. It ranges from 0 to 1, with larger values indicating a greater
difference between the distributions. We used the KS distance to complement
Cliff’s delta in assessing the dissimilarity between the verbatim match
frequency distributions of $D_{old}$ and $D_{new}$.

  * •

Kruskal-Wallis H test [22]: A non-parametric test that determines whether
there are statistically significant differences between the distributions of
two or more groups. We employed the Kruskal-Wallis H test to assess the
overall significance of the differences in verbatim match frequencies between
$D_{old}$ and $D_{new}$ for each language model. This test is suitable for our
analysis because it does not assume normality or homogeneity of variances,
making it robust to the characteristics of our data.

By applying these statistical measures to the arrays of frequencies $F$, we
aim to quantify the significance of the differences in verbatim regurgitation
between the datasets that the language models were likely trained on and those
that were released after the models’ training cutoff dates.

The proposed methodology enables us to investigate the extent of verbatim
regurgitation in language models and shed light on the influence of dataset
age and availability on their ability to generate novel content. The insights
gained from this study can inform the development of strategies for mitigating
or detecting verbatim regurgitation in generated text and highlight the
importance of considering the potential for such regurgitation when using
language models for various applications.

##  4 Experiments

Figure 1: Distribution of verbatim match frequencies for different language
models on the Wikipedia dataset. The results show a consistent pattern across
all three models (GPT-3.5, GPT-4, and LLAMA), where the frequency of verbatim
matches is notably higher for articles published before the training cutoff
dates (Wikimedia) compared to those published after (April 2024 Wiki). This
difference is particularly evident for longer verbatim match lengths, with the
Wikimedia frequencies maintaining higher values even at substring lengths of
10 words or more. In contrast, the April Wiki frequencies exhibit a sharper
decline as the verbatim match length increases. Statistical tests (Cliff’s
Delta, KS Distance, and Kruskal-Wallis H Test) confirm the significance of
these differences, suggesting that the language models are more prone to
reproducing verbatim content when the input text is sourced from their
training data.

we now report on the results of a series of experiments to investigate the
extent of verbatim regurgitation in text generated by different language
models when prompted with text from various dataset sources. Our primary goal
was to quantify the occurrence of verbatim regurgitation and understand the
factors that influence this behavior, particularly focusing on the differences
between datasets that the language models were likely trained on and those
that were released after the models’ training cutoff dates.

###  4.1 Datasets and Language Models

In our experiments, we utilized two types of dataset sources: Wikipedia
articles and Open Educational Resource (OER) textbooks. For Wikipedia, we
curated two subsets: $D_{\rm pre}$, containing approximately 600 articles with
a length greater than 1,000 words, selected from a time period significantly
before the language models’ training cutoff dates, and $D_{\rm post}$,
consisting of around 600 articles with a length greater than 1,000 words,
crawled from the last few months, which are likely to be absent from the
language models’ training data. For OER textbooks, we selected $D_{\rm pre}$,
comprising textbooks with publication dates before the language models’
training cutoff dates, such as "OpenStax Biology 2e" [34], "OpenStax Economics
2e" [35], and "OpenStax Concepts of Biology" [36], and $D_{\rm post}$,
including recently released textbooks, such as two new OpenStax textbooks on
nutrition for nurses [37] and pharmacology, which are unlikely to be part of
the language models’ training data.

We evaluated the verbatim regurgitation behavior of three language models:
GPT-3.5 (engine: gpt-3.5-turbo-1106), GPT-4 (engine: gpt-4-turbo-2024-04-09),
and LLAMA (version: llama3-70B). For GPT-3.5 and GPT-4, we used a temperature
setting of 0.1 to encourage more deterministic and less diverse outputs, which
may increase the likelihood of verbatim regurgitation. LLAMA was used with
greedy sampling, as it does not support adjustable temperature settings.

By carefully selecting these dataset sources and language models, we aim to
create a controlled setting that allows us to compare the verbatim
regurgitation behavior of language models when prompted with text they were
likely exposed to during training ($D_{\rm pre}$) and text that they have not
encountered before ($D_{\rm post}$). This setup enables us to investigate the
influence of dataset age and availability on the extent of verbatim
regurgitation and draw insights into the models’ ability to generate novel
content.

Datasets | Language Model | Cliff’s Delta | KS Distance | KW H-statistic | KW p-value
---|---|---|---|---|---
Wikipedia | GPT-3.5 | $-1.0$ | $1.0$ | $11.31$ | $0.0008$
GPT-4 | $-0.828$ | $0.75$ | $7.77$ | $0.0053$
LLAMA | $-0.578$ | $0.50$ | $3.78$ | $0.0519$
OER | GPT-3.5 | $-0.984$ | $0.875$ | $11.12$ | $0.0009$
LLAMA | $-0.828$ | $0.625$ | $7.80$ | $0.0052$
Table 1: Statistical analysis results demonstrating a clear difference in the
distribution of verbatim matches between datasets published before ($D_{\rm
pre}$) and after ($D_{\rm post}$) the training cutoff dates for each language
model on the Wikipedia and OER datasets. The negative Cliff’s Delta effect
size values indicate a higher probability of verbatim matches in $D_{\rm
pre}$. The Kolmogorov-Smirnov (KS) test statistic measures the maximum
distance between the cumulative distributions of $D_{\rm pre}$ and $D_{\rm
post}$, with larger values suggesting a greater difference. The Kruskal-Wallis
(KW) test statistic assesses the overall difference in median verbatim match
frequencies across the datasets, with the corresponding p-values testing the
null hypothesis of equal medians. Significant p-values (< 0.05) reject this
null hypothesis. These results consistently demonstrate that verbatim match
frequencies are significantly higher for $D_{\rm pre}$ compared to $D_{\rm
post}$. Figure 2: Distribution of verbatim match frequencies for GPT-3.5 and
LLAMA on the OER textbook dataset. The frequency of verbatim matches is
consistently higher for the textbooks published before the training cutoff
dates (Old Bio & Econ) compared to the recently released textbooks (New Pharma
& Nursing). Statistical tests (Cliff’s Delta, KS Distance, and Kruskal-Wallis
H Test) confirm the significance of these differences.

###  4.2 Primary Experiments for Verbatim Regurgitation Analysis

In our base experiments, we applied the Many-Shot Regurgitation (MSR)
prompting technique to each combination of language model and dataset source.
We used the following many shot configurations: 6 shots (3 human and 3
assistant turns). For each source text, we obtained the generated text
($T_{n}^{\prime}$) and calculated the frequency of exact verbatim matches for
substring lengths ranging from 5 to 12 words.

To quantify the differences in verbatim regurgitation between the datasets
published before and after the training cutoff dates ($D_{\rm pre}$ and
$D_{\rm post}$), we employed several statistical measures: Cliff’s Delta,
Kolmogorov-Smirnov (KS) Distance, and the Kruskal-Wallis H Test. These
measures are described in detail in the Methodology section.

Figure 1 presents the distribution of verbatim match frequencies for different
language models on the Wikipedia dataset. The results show that the frequency
of verbatim matches is consistently higher for the articles published before
the training cutoff dates (Wikimedia) compared to those published after (April
2024 Wiki), across all three language models (GPT-3.5, GPT-4, and LLAMA). The
statistical analysis results, summarized in Table 1, confirm the significance
of these differences, with GPT-3.5 showing the most substantial difference
(Cliff’s Delta $=-1.0$, KS Distance $=1.0$, H-statistic $=11.31$, p
$=0.0008$), followed by GPT-4 (Cliff’s Delta $=-0.828$, KS Distance $=0.75$,
H-statistic $=7.77$, p $=0.0053$) and LLAMA (Cliff’s Delta $=-0.578$, KS
Distance $=0.5$, H-statistic $=3.78$, p $=0.0519$).

Similarly, figure 2 displays the distribution of verbatim match frequencies
for GPT-3.5 and LLAMA on the OER textbook dataset. The frequency of verbatim
matches is notably higher for the textbooks published before the training
cutoff dates (Old Bio & Econ) compared to the recently released textbooks (New
Pharma & Nursing). Table 1 shows that GPT-3.5 exhibits a highly significant
difference (Cliff’s Delta $=-0.984$, KS Distance $=0.875$, H-statistic
$=11.12$, p $=0.0009$), while LLAMA also shows a substantial difference
(Cliff’s Delta $=-0.828$, KS Distance $=0.625$, H-statistic $=7.80$, p
$=0.0052$).

The results of our verbatim regurgitation analysis demonstrate a clear
difference in the distribution of verbatim matches between the datasets
published before and after the training cutoff dates for both dataset sources.
The negative Cliff’s Delta values, substantial KS distances, and significant
Kruskal-Wallis H-statistics consistently indicate that verbatim match
frequencies are higher for $D_{\rm pre}$ compared to $D_{\rm post}$.

Figure 3: Effect of the number of shots on the occurrence of verbatim
regurgitation for GPT-3.5 and LLAMA models on the Wikipedia dataset. The
results show that increasing the number of shots generally leads to higher
frequencies of verbatim matches, with 6 shots (3 human and 3 assistant turns)
yielding the best results.

###  4.3 Ablation Experiments

To gain deeper insights into the factors that influence the occurrence of
verbatim regurgitation, we conducted two ablation experiments: analysis of
shot count and temperature control.

####  4.3.1 Analysis of Shot Count

In this experiment, we investigated the effect of the number of shots on the
occurrence of verbatim regurgitation. Drawing inspiration from the many-shot
jailbreaking approach, which suggests that using a higher number of shots can
lead to better results, we varied the number of shots from 2 to 8 while
keeping other parameters constant. We applied the MSR prompting technique to
the Wikipedia articles dataset using GPT-3.5 and LLAMA models.

Figure 3 shows the distribution of verbatim match frequencies for different
numbers of shots. We observed that increasing the number of shots generally
leads to higher frequencies of verbatim matches for both GPT-3.5 and LLAMA
models, with 6 shots (3 human and 3 assistant turns) yielding the best
results. This finding suggests that dividing the source text into an
appropriate number of segments is crucial for the effectiveness of the MSR
prompting technique in eliciting verbatim regurgitation.

####  4.3.2 Temperature Control

In this experiment, we explored the effect of temperature settings on verbatim
regurgitation. We conducted experiments with the GPT-3.5 model using two
temperature values: 0.1 and 0.7. The experiments were performed on the
Wikipedia dataset, and we measured the frequency of verbatim matches for
substring lengths ranging from 4 to 15 words.

The results show that increasing the temperature from 0.1 to 0.7 leads to a
decrease in the frequency of verbatim matches across all substring lengths.
For example, at a substring length of 4 words, the frequency of verbatim
matches decreases from 195 at a temperature of 0.1 to 162 at a temperature of
0.7. Similarly, for a substring length of 15 words, the frequency of verbatim
matches decreases from 57 at a temperature of 0.1 to 31 at a temperature of
0.7. To quantify the statistical significance of the differences in verbatim
match frequencies between the two temperature settings, we employed Cliff’s
Delta, Kolmogorov-Smirnov (KS) Distance, and the Kruskal-Wallis H Test. The
Cliff’s Delta value of 0.5625 indicates a moderate effect size, suggesting
that the verbatim match frequencies tend to be higher at a temperature of 0.1
compared to 0.7. The KS Distance of 0.5 further supports this observation,
indicating a notable difference between the cumulative distribution functions
of the two temperature settings.

####  4.3.3 Sensitivity of Verbatim Regurgitation Detection to Input Text
Length

In this experiment, we investigated the effect of input text length on the
occurrence of verbatim regurgitation. We selected 600 articles each from the
Wikipedia datasets published before and after the training cutoff dates
(Wikimedia and April Wiki, respectively) and truncated them to various
lengths: 75, 125, 250, 500, and 1000 words. We then applied the MSR prompting
technique with 6 shots (3 human and 3 assistant turns) and calculated the
frequency of exact verbatim matches for substring lengths ranging from 5 to 12
words.

It is important to note that the maximum substring length that can be analyzed
is limited by the ratio of the input text length ($L$) to the number of splits
($s$). The length of each split can be approximated as $L/s$, which determines
the upper limit of the substring length that can be meaningfully analyzed for
verbatim matches. As the input text length decreases while keeping the number
of splits constant, the length of each split also decreases, reducing the
maximum substring length that can be effectively analyzed.

Furthermore, as the input text length decreases, the distribution of verbatim
match frequencies becomes more sparse. With shorter input texts, there are
fewer opportunities for longer substring matches to occur, affecting the
statistical analysis. If the distribution becomes too sparse, it may be
necessary to adjust the analytical approach, such as focusing on shorter
substring lengths or increasing the input text length while keeping the number
of splits constant. For each language model and dataset source combination, we
obtain an array of frequencies $F=[f_{\ell_{\rm min}},f_{\ell_{\rm
min}+1},...,f_{l\ell_{\rm max}}]$ corresponding to verbatim match length
thresholds from $l\ell_{\rm min}$ to $\ell_{\rm max}$ words. In this
experiment, $\ell_{\rm min}$ is set to 5, and $\ell_{\rm max}$ is determined
by the ratio $L/s$. For example, with an input text length of 75 words and 6
splits, the average split length is approximately 12 words, so $\ell_{\rm
max}$ is set to 12.

Figure 4: Distribution of verbatim match frequencies for GPT-3.5 at different
input text lengths on the Wikipedia dataset. The results demonstrate that MSR
prompting consistently yields higher verbatim match frequencies for articles
published before the training cutoff dates (Wikimedia) compared to those
published after (April 2024 Wiki), even at shorter input text lengths.
However, as the input text length decreases, the maximum verbatim match length
that can be meaningfully analyzed is limited by the ratio of the input text
length ($L$) to the number of splits ($s$), approximated as $L/s$, resulting
in a more sparse distribution of verbatim match frequencies.

Input Text Length |  Cliff’s Delta |  KS Distance |  KW H-statistic |  KW p-value
---|---|---|---|---
$75$ |  $-1.0$ |  $1.0$ |  $11.48$ |  $0.0007$
$125$ |  $-0.91$ |  $0.75$ |  $9.33$ |  $0.0023$
$250$ |  $-1.0$ |  $1.0$ |  $11.46$ |  $0.0007$
$500$ |  $-1.0$ |  $1.0$ |  $11.29$ |  $0.0008$
$1000$ |  $-1.0$ |  $1.0$ |  $11.31$ |  $0.0008$

Table 2: Statistical analysis results comparing the distribution of verbatim
match frequencies between Wikimedia and April Wiki articles at different input
text lengths. The negative Cliff’s Delta values, high KS Distance values, and
significant Kruskal-Wallis H-statistic p-values (all $<0.05$) consistently
indicate that the Wikimedia articles have higher verbatim match frequencies
than the April 2024 Wiki articles across all input text lengths. These results
demonstrate the robustness of the MSR prompting technique in detecting
verbatim regurgitation across a range of input text lengths, while also
highlighting the limitations imposed by the relationship between the input
text length and the number of splits when analyzing the distribution of
verbatim match frequencies.

Figure 4 presents the distribution of verbatim match frequencies for GPT-3.5
at different input text lengths. The results show that the MSR prompting
technique consistently yields higher verbatim match frequencies for articles
published before the training cutoff dates (Wikimedia) compared to those
published after (April Wiki), even at shorter input text lengths.

To quantify the differences in verbatim regurgitation between the Wikimedia
and April Wiki datasets at various input text lengths, we employed Cliff’s
Delta, Kolmogorov-Smirnov (KS) Distance, and the Kruskal-Wallis H Test. Table
2 summarizes the statistical analysis results for each input text length.

The results demonstrate that the MSR prompting technique is effective in
eliciting verbatim regurgitation even at shorter input text lengths. The
Cliff’s Delta values of -1.0 or close to -1.0, along with the large KS
distances and significant Kruskal-Wallis H-statistics (p < 0.05), indicate
that the verbatim match frequencies are consistently higher for the Wikimedia
dataset compared to the April Wiki dataset across all input text lengths.

These findings suggest that the effectiveness of the MSR prompting technique
in detecting verbatim regurgitation is robust to variations in input text
length, at least down to a length of 75 words. This highlights the potential
of the MSR prompting technique as a tool for identifying instances of verbatim
regurgitation in language models, even when working with shorter input texts.

However, it is essential to consider the limitations imposed by the
relationship between the input text length and the number of splits when
analyzing the distribution of verbatim match frequencies. Future research
could explore the impact of varying the number of splits in conjunction with
the input text length to further investigate the boundaries and optimal
configurations for detecting verbatim regurgitation using the MSR prompting
technique.

##  5 Conclusion

In this paper, we have introduced Many-Shot Regurgitation (MSR) prompting, a
novel approach for investigating verbatim content reproduction in large
language models. MSR represents a new type of membership inference attack
(MIA) that operates in a black-box setting, requiring only the ability to
prompt the model and observe its outputs. We quantified the occurrence of
verbatim matches using the Longest Common Substring algorithm and devised a
novel evaluation strategy that compared the distribution of verbatim matches
between two carefully curated dataset sources: Wikipedia articles and Open
Educational Resource textbooks. The key innovation in our evaluation approach
was the inclusion of documents published after the LLMs’ training cutoff dates
($D_{\rm post}$), serving as a control group that the models could not have
encountered during training. Our experiments to quantify the extent of
verbatim regurgitation in state-of-the-art LLMs such as GPT-3.5, GPT-4, and
LLAMA revealed a striking difference in the distribution of verbatim matches
between datasets that the models were likely exposed to during training
($D_{\rm pre}$) and those published after the models’ training cutoff dates
($D_{\rm post}$). Robust statistical measures, including Cliff’s delta,
Kolmogorov-Smirnov distance, and the Kruskal-Wallis H test, consistently
indicated significantly higher verbatim match frequencies for $D_{\rm pre}$
compared to $D_{\rm post}$. Furthermore, our ablation experiments shed light
on the factors affecting the effectiveness of the MSR technique, such as the
number of shots and the temperature settings of LLMs. In sum, our work
highlights the effectiveness of MSR prompting as a new MIA framework for
studying LLMs in real-world settings where access to model internals is
limited.

## Acknowledgments

This work was supported by NSF grant 1842378, ONR grant N0014-20-1-2534, AFOSR
grant FA9550-22-1-0060, a Vannevar Bush Faculty Fellowship, OpenAI, and ONR
grant N00014-18-1-2047.

## References

  * [1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  Gpt-4 technical report.  arXiv preprint arXiv:2303.08774, 2023.
  * [2] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al.  Gemini: a family of highly capable multimodal models.  arXiv preprint arXiv:2312.11805, 2023.
  * [3] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288, 2023.
  * [4] AI@Meta.  Llama 3 model card.  2024\.
  * [5] Wikipedia contributors.  Ai alignment — Wikipedia, the free encyclopedia.  https://en.wikipedia.org/wiki/AI_alignment, 2023.  [Online; accessed 13-May-2024].
  * [6] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  Advances in neural information processing systems, 35:27730–27744, 2022.
  * [7] Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer.  Membership inference attacks from first principles.  In 2022 IEEE Symposium on Security and Privacy (SP), pages 1897–1914. IEEE, 2022.
  * [8] Nicholas Carlini, Matthew Jagielski, Chiyuan Zhang, Nicolas Papernot, Andreas Terzis, and Florian Tramer.  The privacy onion effect: Memorization is relative.  Advances in Neural Information Processing Systems, 35:13263–13276, 2022.
  * [9] Congzheng Song and Ananth Raghunathan.  Information leakage in embedding models.  In Proceedings of the 2020 ACM SIGSAC conference on computer and communications security, pages 377–390, 2020.
  * [10] Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schölkopf, Mrinmaya Sachan, and Taylor Berg-Kirkpatrick.  Membership inference attacks against language models via neighbourhood comparison.  arXiv preprint arXiv:2305.18462, 2023.
  * [11] Fatemehsadat Mireshghallah, Archit Uniyal, Tianhao Wang, David K Evans, and Taylor Berg-Kirkpatrick.  An empirical analysis of memorization in fine-tuned autoregressive language models.  In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1816–1826, 2022.
  * [12] Wenjie Fu, Huandong Wang, Chen Gao, Guanghua Liu, Yong Li, and Tao Jiang.  Practical membership inference attacks against fine-tuned large language models via self-prompt calibration.  arXiv preprint arXiv:2311.06062, 2023.
  * [13] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov.  Membership inference attacks against machine learning models.  In 2017 IEEE symposium on security and privacy (SP), pages 3–18. IEEE, 2017.
  * [14] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al.  Extracting training data from large language models.  In 30th USENIX Security Symposium (USENIX Security 21), pages 2633–2650, 2021.
  * [15] Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi.  Do membership inference attacks work on large language models?  arXiv preprint arXiv:2402.07841, 2024.
  * [16] J Schulman, B Zoph, C Kim, J Hilton, J Menick, J Weng, JFC Uribe, L Fedus, L Metz, M Pokorny, et al.  Chatgpt: Optimizing language models for dialogue, 2022.
  * [17] OpenAI.  Gpt-4 technical report, 2023.
  * [18] Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong.  Jailbreaking black box large language models in twenty queries, 2023.
  * [19] Cem Anil, Esin Durmus, Mrinank Sharma, Joe Benton, Sandipan Kundu, Joshua Batson, Nina Rimsky, Meg Tong, Jesse Mu, Daniel Ford, et al.  Many-shot jailbreaking.
  * [20] Norman Cliff.  Dominance statistics: Ordinal analyses to answer ordinal questions.  Psychological bulletin, 114(3):494, 1993.
  * [21] Kolmogorov An.  Sulla determinazione empirica di una legge didistribuzione.  Giorn Dell’inst Ital Degli Att, 4:89–91, 1933.
  * [22] William H Kruskal and W Allen Wallis.  Use of ranks in one-criterion variance analysis.  Journal of the American statistical Association, 47(260):583–621, 1952.
  * [23] Sajjad Zarifzadeh, Philippe Cheng-Jie Marc Liu, and Reza Shokri.  Low-cost high-power membership inference by boosting relativity.  2023\.
  * [24] Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Berg-Kirkpatrick, and Reza Shokri.  Quantifying privacy risks of masked language models using membership inference attacks.  arXiv preprint arXiv:2203.03929, 2022.
  * [25] Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang.  Quantifying memorization across neural language models.  arXiv preprint arXiv:2202.07646, 2022.
  * [26] Thomas Steinke, Milad Nasr, and Matthew Jagielski.  Privacy auditing with one (1) training run.  Advances in Neural Information Processing Systems, 36, 2024.
  * [27] Jin Yao, Eli Chien, Minxin Du, Xinyao Niu, Tianhao Wang, Zezhou Cheng, and Xiang Yue.  Machine unlearning of pre-trained large language models.  arXiv preprint arXiv:2402.15159, 2024.
  * [28] Yonatan Oren, Nicole Meister, Niladri Chatterji, Faisal Ladhak, and Tatsunori B Hashimoto.  Proving test set contamination in black box language models.  arXiv preprint arXiv:2310.17623, 2023.
  * [29] Matthieu Meeus, Shubham Jain, Marek Rei, and Yves-Alexandre de Montjoye.  Did the neurons read your book? document-level membership inference for large language models.  arXiv preprint arXiv:2310.15007, 2023.
  * [30] André V Duarte, Xuandong Zhao, Arlindo L Oliveira, and Lei Li.  De-cop: Detecting copyrighted content in language models training data.  arXiv preprint arXiv:2402.09910, 2024.
  * [31] Xudong Pan, Mi Zhang, Shouling Ji, and Min Yang.  Privacy risks of general-purpose language models.  In 2020 IEEE Symposium on Security and Privacy (SP), pages 1314–1331. IEEE, 2020.
  * [32] Nikhil Kandpal, Eric Wallace, and Colin Raffel.  Deduplicating training data mitigates privacy risks in language models.  In International Conference on Machine Learning, pages 10697–10707. PMLR, 2022.
  * [33] Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer.  Detecting pretraining data from large language models.  arXiv preprint arXiv:2310.16789, 2023.
  * [34] Mary Ann Clark, Matthew Douglas, and Jung Choi.  Biology 2e.  OpenStax, Houston, Texas, 2018.
  * [35] Steven A. Greenlaw and David Shapiro.  Principles of Economics 2e.  OpenStax, Houston, Texas, 2017.
  * [36] Samantha Fowler, Rebecca Roush, and James Wise.  Concepts of Biology.  OpenStax, Houston, Texas, 2013.
  * [37] Emerald Charity Bilbrew, Jody Vogelzang, and Kelli Whittington.  Nutrition for Nurses.  OpenStax, Houston, Texas, 2024.