# Grounding Large Language Model Generations to Pre-Training Data

Orion Weller∗, Marc Marone∗, Nathaniel Weir,
Dawn Lawrie, Daniel Khashabi, Benjamin Van Durme
Johns Hopkins University
<EMAIL_ADDRESS>
# Can Prompting Large Language Models Improve Grounding in their Pre-training
Data?

Orion Weller∗, Marc Marone∗, Nathaniel Weir,
Dawn Lawrie, Daniel Khashabi, Benjamin Van Durme
Johns Hopkins University
<EMAIL_ADDRESS>
# Grounding via Prompting

Orion Weller∗, Marc Marone∗, Nathaniel Weir,
Dawn Lawrie, Daniel Khashabi, Benjamin Van Durme
Johns Hopkins University
<EMAIL_ADDRESS>
#  _Grounding via Prompting:_
Prompting Language Model Improves Grounding in Pre-Training Data

Orion Weller∗, Marc Marone∗, Nathaniel Weir,
Dawn Lawrie, Daniel Khashabi, Benjamin Van Durme
Johns Hopkins University
<EMAIL_ADDRESS>
#  _Grounding via Prompting:_
Prompting Language Models Improves Quoting from Pre-Training Data

Orion Weller∗, Marc Marone∗, Nathaniel Weir,
Dawn Lawrie, Daniel Khashabi, Benjamin Van Durme
Johns Hopkins University
<EMAIL_ADDRESS>
#  _According to Wikipedia …_
Prompting Language Models Improves Quoting from Pre-Training Data

Orion Weller∗, Marc Marone∗, Nathaniel Weir,
Dawn Lawrie, Daniel Khashabi, Benjamin Van Durme
Johns Hopkins University
<EMAIL_ADDRESS>
#  _“According to …”_
Prompting Language Models Improves Quoting from Pre-Training Data

Orion Weller∗, Marc Marone∗, Nathaniel Weir,
Dawn Lawrie, Daniel Khashabi, Benjamin Van Durme
Johns Hopkins University
<EMAIL_ADDRESS>
###### Abstract

Large Language Models (LLMs) may hallucinate and generate fake information,
despite pre-training on factual data. Inspired by the journalistic device of
_“according to sources”_ , we propose _according-to_ prompting: directing LLMs
to ground responses against previously observed text. To quantify this
grounding, we propose a novel evaluation metric (QUIP-Score) that measures the
extent to which model-produced answers are directly found in underlying text
corpora. We illustrate with experiments on Wikipedia that these prompts
improve grounding under our metrics, with the additional benefit of often
improving end-task performance. Furthermore, prompts that ask the model to
decrease grounding (or to ground to other corpora) _decrease_ grounding,
indicating the ability of language models to increase or decrease grounded
generations on request.

##  1 Introduction

As the deployment of Large Language Models (LLMs) in real-world applications
continues to grow, their tendency to generate false content Ji et al. (2022)
poses significant risks to downstream users. Recent work has attempted to
address this issue by augmenting them with retrieval Shuster et al. (2021);
Sun et al. (2023); Borgeaud et al. (2022), however, these models still
struggle with hallucination problems in practice Liu et al. (2023).

Figure 1:  Prompting LLMs to respond with quotes directly from pre-training
data (shown in purple).

This work explores the intriguing possibility of _steering LLMs by prompting_
them to quote more of the curated sources of information they have memorized
during pre-training, thereby reducing their tendency to generate false
information. As illustrated in Figure 1, the hypothesis explores whether
adding phrases such as “According to Wikipedia” can guide LLMs to quote from
Wikipedia, which is presumably observed in the pre-training corpus. We find
empirical evidence that this is attainable using current LLMs (both open and
closed source).

Our study is inspired by two recent research areas. First, larger LLMs can be
more effectively guided using natural language prompts Ouyang et al. (2022);
Wan et al. (2023); Ganguli et al. (2023). Second, as LLMs grow in size, their
ability to remember facts and statements from pre-training improves Kandpal et
al. (2022); Tirumala et al. (2022); Carlini et al. (2023, 2020). Thus, we seek
to steer LLMs to use their memorization for a positive purpose: producing more
grounded outputs.

00footnotetext: * Authors contributed equally

A key step in this study is quickly determining whether generated outputs
overlap significantly with pre-training data; i.e., efficiently performing
membership testing via a Data Portrait Marone and Van Durme (2023). We design
a new metric called QUIP-Score, short for Quoted Information Precision, on top
of a data sketch that provides efficient membership queries within
milliseconds. QUIP-Score is an n-gram overlap measure that quantifies how much
a passage is formed of spans that are exactly contained in a text corpus.

We perform experiments111Code to reproduce our analysis is accessible at:
https://github.com/JHU-CLSP/according-to based on the task of open-domain
question answering (ODQA), for which provenance-grounded answers are of
particular importance. We collect human-authored prompts intended to steer
generations towards information grounded in Wikipedia, our target source for
grounding. We observe that across all human-authored prompts, we can increase
the amount of overlap with Wikipedia by 5-105% while maintaining or even
improving the downstream performance. We collect results across multiple
datasets and a large set of models, including both open- and closed-sourced to
ensure consistency.

Interestingly, we also observe the opposite phenomenon – it is possible to
discourage LLMs from grounding in Wikipedia via prompts that either discourage
grounding or encourage grounding to other corpora. This consequently decreases
the overlap with Wikipedia and also lowers performance on downstream tasks
that rely on Wikipedia content.

We conduct scaling experiments on different model sizes, which indicate that
as size increases, so does the effectiveness of our proposed approach. This
implies that the hallucination problem may diminish as a function of continued
scaling of LLMs.

In summary, we present _according-to_ prompting, a simple and effective
approach to improving an LLMs’ ability to generate more factual information.
Additionally, we introduce QUIP-Score, an efficient approach for measuring
groundedness of LLM generations against their pre-training corpus. We
experiment with various prompting strategies across models, datasets, and
scaling trends.

##  2 Related Work

#### Memorization in LLMs.

Large language models have been observed to memorize their training data
(Carlini et al., 2020; Chang et al., 2023, among others). This is problematic
when web-scraped training data contains sensitive personal data or low quality
information sources Dodge et al. (2021); Luccioni and Viviano (2021). However,
it can be beneficial for models to memorize content from _carefully curated
and trusted corpora_ , where careful de-duplication (Lee et al., 2022a) and
curation strategies (Feng et al., 2022) can improve language model quality.
Some language modeling datasets intentionally oversample trusted resources,
e.g., The Pile triples the effective weight of Wikipedia, oversampling it more
than any of the other 22 component subsets (Gao et al., 2020).

Work on analyzing memorization has proposed measuring overlap against Google
search results as a proxy for memorization. Carlini et al. (2020) manually
look for exact string matches in search results, while Levy et al. (2021)
automatically measure BLEU (Papineni et al., 2002) between model generations
and the first page of Google results.

Marone and Van Durme (2023) call for documenting large datasets through
membership testing tools, that they collectively label Data Portraits. As one
implementation, they illustrate using a Bloom Filter (Bloom, 1970) for storing
long n-grams. While other portrait implementations have been proposed, such as
based on BM-25 and suffix arrays (Piktus et al., 2023), Bloom Filters provide
both lightweight storage and access, enabling scalable overlap metrics against
large corpora.

#### Hallucination, grounding, and attribution.

Numerous studies De Cao et al. (2021); Li et al. (2022) have demonstrated that
LLMs struggle with both hallucination and factuality, leading to frequent
inaccuracies and outright falsehoods. Previous research has attempted to
alleviate this problem in various ways, including retrieving grounded
documents before generation Sun et al. (2023); Borgeaud et al. (2022); Mallen
et al. (2023), applying new decoding approaches He et al. (2022), post hoc
tuning of LLMs Menick et al. (2022); Lee et al. (2022b), and analyzing the
model’s output training data Han and Tsvetkov (2022); Park et al. (2023). Our
work is different from and complementary to this literature, as we investigate
a novel yet straightforward approach to steer LLMs towards generating more
factual responses.

Another similar line of work is that of LLM attribution Rashkin et al. (2021);
Bohnet et al. (2022) with several datasets designed to measured attribution to
identified sources (AIS). Our work is similar in that we seek to find
attribution for a language model’s output to a given collection but different
in that we define attribution more broadly: we find attribution for any text
that the language model generated, while AIS seeks to find attribution for the
semantic knowledge that the language model generated. Thus, our metric studies
grounding and attribution in a broader context than previously studied.

#### LLM Steerability via prompting.

The larger LMs become, the easier they are to steer with natural language
prompts Kandpal et al. (2022); Carlini et al. (2023); Mishra et al. (2022a);
Srivastava et al. (2023). Several works Mishra et al. (2022b); Chung et al.
(2022); Wang et al. (2022b); Wan et al. (2023) have shown that larger
instruction-tuned models are more easily steered than smaller and non-
instruction tuned models. This is desirable in our setting, as we seek to use
these capabilities of LLMs to quote more from a given corpus.

Prompt | TQA | NQ | Hotpot | ELI5
---|---|---|---|---
(appended after the question) | QUIP | EM | QUIP | EM | QUIP | F1 | QUIP | R-L
$\emptyset$ (no additional prompt) | 31.6 | 77.8 | 32.8 | 32.9 | 28.3 | 35.7 | 24.1 | 22.7
grounding prompts |  "Based on evidence from Wikipedia:" | 31.1 | 77.3 | 32.8 | 34.0 | 28.1 | 35.9 | 26.3 | 22.3
"As an expert editor for Wikipedia, I am confident in the following answer." | 31.7 | 73.2 | 33.0 | 30.2 | 28.7 | 35.3 | 25.5 | 22.7
"I found some results for that on Wikipedia. Here’s a direct quote:" | 31.7 | 70.1 | 33.8 | 27.6 | 28.1 | 33.1 | 27.2 | 21.0
"Reference Wikipedia when answering the following question." | 32.8 | 75.9 | 34.6 | 34.4 | 28.9 | 35.9 | 25.7 | 22.0
"Answer according to Wikipedia." | 33.6 | 78.8 | 34.3 | 34.8 | 29.2 | 36.6 | 26.5 | 21.7
"Go to https://www.wikipedia.org and find direct quotes to answer the question. Response: "" | 34.5 | 72.7 | 32.9 | 31.7 | 30.4 | 35.5 | 25.8 | 20.4
"Respond by using information from Wikipedia in your response." | 34.9 | 76.3 | 35.3 | 32.9 | 29.9 | 36.1 | 26.3 | 21.9
"Respond to this question using only information that can be attributed to Wikipedia." | 35.7 | 76.6 | 37.0 | 33.9 | 30.4 | 36.2 | 28.0 | 21.5
anti-grounding |  "Respond by using information from Reddit in your response." | 26.1 | 75.8 | 26.5 | 31.6 | 22.4 | 35.0 | 21.9 | 22.2
"Respond by using information from Github in your response." | 26.7 | 74.3 | 28.2 | 32.4 | 23.2 | 33.7 | 24.3 | 22.0
"Respond without using any information from Wikipedia in your response." | 30.4 | 76.9 | 32.0 | 32.0 | 26.8 | 32.9 | 24.7 | 22.1
Zero-Shot No-Retrieval SOTA | - | 68.2 | - | 24.9 | - | 44.6 | - | 22.7
Retreival-Augmented SOTA | - | 89.4 | - | 60.4 | - | 51.4 | - | 26.5

Table 1:  Impact of various prompts on the grounding (QUIP-Score) and
performance scores, using ChatGPT (Section 4). The top section is the null
prompt (no additional prompt other than the question), the middle section
includes anti-grounding prompts, and the last section includes grounding
prompts. We find that grounding prompts generally improve the QUIP-Score while
anti-grounding prompts generally reduce QUIP-Score. Colored cells indicate
changes (gains, losses, or the same) relative to the null row. Note that ELI5
Rouge-L is based on similarity to Reddit rather than Wikipedia. SOTA zero-shot
results are from LLaMA 33B, LLaMA 65B Touvron et al. (2023), PaLM 540B Wang et
al. (2022a), and BART Su et al. (2022) respectively. For retrieval-augmented
SOTA, Izacard et al. (2022) is for NQ, TriviaQA and HotpotQA, while Su et al.
(2022) is for ELI5.

##  3 Methodology

###  3.1 QUIP-Score: Measuring Grounding to Pre-Training Data

In order to understand whether models are able to ground to their pre-training
data, we first need to have a way of measuring this phenomena. We adopt a
narrow definition of grounding (quoting from source material) while
acknowledging that grounding is a broad term.

To enable fast and efficient measurement of quoting from pre-training data for
many language model generations across large corpora, we build off of a Data
Portrait Marone and Van Durme (2023), which allows for fast membership queries
for each n-gram in the output. This approach enables us to perform a one-time
indexing of a large corpus (e.g. Wikipedia) and at inference time simply
compute a constant time lookup operation (in milliseconds) for each n-gram in
the generation.

We build a Data Portrait on the version of Wikipedia included in the Pile,222
wikipedia/20200301.en as it allows for us to exactly test the pre-training
data included in many models like GPT-J and is similar to the training data
used in T5. However, we note that for some models evaluated in this paper
(e.g. OpenAI models) there is no public information about the Wikipedia
version in the models.

We use character based n-grams as opposed to a token-based n-gram as different
models have different tokenization schemes; furthermore, character-based
n-gram metrics have widespread usage in fields such as machine translation
with metrics like chrF and chrF++ Popović (2015, 2017). We use 25 character
grams for the sketch, approximately 5-gram words, as we found it empirically
gave meaningful results (not too small of an n-gram and not too large). The
Data Portrait checks for exact matches and is sensitive to orthographic
variation (e.g. case, whitespace). Therefore we view this as a lower-bound on
actual quoting performance.

We define our new metric QUIP-Score as the character n-gram precision of the
generated output compared to the pre-training corpus. More formally, for
generation $Y$ and text corpus $C$:

$\text{QUIP{}}(Y;C)=\dfrac{\sum_{\begin{subarray}{c}\text{gram}_{n}\in
Y\end{subarray}}\mathbb{1}_{C}(\text{gram}_{n})}{|\text{gram}_{n}\in
Y|},\vspace{-0.3em}$

where $\mathbb{1}(.)$ is an indicator function: 1 if $\text{gram}_{n}\in C$
else 0. Thus, a score of 0.5 would indicate that 50% of the generated text
$n$-grams are found in the pre-training corpus. We macro-average this quantity
over a set of generations to obtain a single performance number for a given
test dataset.333Early experiments indicated little difference between macro
and micro-averaging.

Model | TQA | NQ | Hotpot | ELI5
---|---|---|---|---
| QUIP | EM | QUIP | EM | QUIP | F1 | QUIP | R-L
Text-Davinci-003 | +14.7% | +5.3% | +14.7% | +20.6% | +14.4% | +7.2% | +16.5% | +22.8%
GPT-4 | - | - | - | - | - | - | +17.6% | -2.3%
GPT-J Instruct | +12.1% | - | +15.2% | - | +13.9% | - | +18.1% | +19.4%
Koala 7B | +5.1% | - | +6.3% | - | +5.0% | - | +35.5% | +22.8%
FLAN-T5 XXL | +43.3% | - | +41.5% | - | +20.7% | - | +105.2% | +18.7%
Table 2: Percentage improvement when using the grounding prompt vs the null
prompt. We find that the grounding prompt improves over the null prompt in
nearly every dataset and metric, typically by 5-15%. Smaller models struggled
to output both answer and explanation to the short form question with one
prompt; hence we use two step prompting (prediction then explanation) for
open-source models. Since the first step (prediction) uses the same prompt for
both grounding and null, the EM/F1 scores are the same and are omitted (c.f.
Section 3.4).

Figure 2:  Model size vs QUIP-Score performance, with FLAN-T5 models (left)
and OpenAI models (right). As model scale increases, so does performance. At
smaller model sizes, the grounding prompt is not more effective than the null
prompt, but gains efficacy with model size. Error bars indicate 1 standard
error.

###  3.2 Grounding via _according-to_ Prompting

We seek to improve knowledge grounding by causing LLMs to quote directly from
underlying trusted resources seen during training. This is equivalent to
_encouraging_ memorization of high quality or trusted documents. We test
whether models can do so by simply prompting them using wording that encourage
grounding such as “Respond by using information from Wikipedia in your
response" after444We tried appending, prepending, and their combinations in
early experiments and found that appending the grounding/anti-grounding
prompts performed the best. the given input. We call this strategy _according-
to_ prompting. We then measure the difference in their QUIP-Score by comparing
the output with the grounding prompt to one without the grounding prompt (i.e.
a null prompt).

To account for the difference in prompts lengths and to verify that prompts
can both increase and decrease grounding, we also include prompts that are
anti-grounding (e.g. “Respond by using information from [another source] in
your response" or “Respond without using any information from Wikipedia." This
allows us to test the hypothesis that models can ground (or not ground) to a
given corpus when asked because of the semantic meaning of the prompt, rather
than the length of the prompt. As prompting is notoriously brittle (e.g.
changing the phrasing can affect the results) we provide a number of grounding
and anti-grounding prompts to test whether these prompts provide consistent
gains or are merely prompting artifacts (see Table 1 for the list of prompts
used).

Figure 3: Impact of entity popularity on QUIP-Scores, showing that models are
better able to quote pre-training text about popular entities. The x-axis
shows how many times the given entity relationship was found co-occurring in
pre-training data. Bars indicate 1 standard error. We use the ranges following
Kandpal et al. (2022).

###  3.3 Datasets

We use a variety of datasets to test if LLMs are consistent and to check
whether grounding affects the end-task performance of a given dataset. To best
measure the grounding of the output however, the model generations must be
long enough to have many n-grams that can be measured. Thus, we test on long-
form question answering (QA), and for datasets that do not lend themselves
well to long-form output (e.g. short form QA) we ask the models to generate
both the answer and a corresponding explanation whose n-grams can be measured.

Note that our purpose is not to improve state-of-the-art performance on these
tasks, as our main research question is to analyze the grounding of these
model’s outputs. However, we do note that we have competitive performance
compared to other prompting baselines and sometimes even observe improved
performance from grounding, as it naturally correlates with the ability to
answer questions from the grounded material.

Thus we use the following datasets:

#### ELI5.

ELI5, or “Explain Like I’m 5” Fan et al. (2019) is a long form question
answering dataset composed of questions from the sub-Reddit r/ELI5, where
users ask questions and receive answers that are typically paragraph-sized.
The original ELI5 dataset was not grounded to any particular knowledge source,
which is not ideal for our experiments on grounding. Hence, we use the KILT
version Petroni et al. (2021) of ELI5, of which the development set contains
1507 instances of ELI5 questions, their answers, and provenances for those
answers grounded in Wikipedia. As this subset of the ELI5 dataset is the
grounded subset (with the non-grounded questions filtered out) it provides a
more suitable evaluation for our research question.

#### Natural Questions.

Natural Questions (NQ) Kwiatkowski et al. (2019) is a short form (e.g. less
than five words for the answer) QA dataset which consists of queries gathered
from real-world Google Searches. To better compare with previous work in
prompting on NQ, we evaluate on the full development set (3360 instances) and
report the exact match score.

#### TriviaQA.

TriviaQA (TQA) Joshi et al. (2017) was collected by scraping question and
answer pairs from trivia websites, and then matching the answers (short-form)
to Wikipedia passages. We use the filtered development set (7k instances),
following previous work and report the EM score.

#### HotpotQA.

HotpotQA Yang et al. (2018) is a multi-step short-form question answering
dataset that requires two-step reasoning to come to the correct answer. It was
gathered from crowdsourcing questions and answers from Amazon Mechanical Turk
using two-hop links on Wikipedia. We use the full dev set (7k instances) and
report the F1 score.

Figure 4: Comparing instructed-tuned FLAN-T5 XXL to non-instruction tuned
T5-v1.1-Adapt XXL. Note that grounding has a larger impact on instruction-
tuned models as compared to non-instruction tuned.

###  3.4 Models and Prompting

We test a wide array of models in our experiments including most OpenAI models
Wang et al. (2023), T5-based models (T5 adapted to language modeling Raffel et
al. (2020); Lester et al. (2021) and FLAN-T5 Chung et al. (2022)), GPT-J
instruction tuned555https://huggingface.co/nlpcloud/instruct-gpt-j-fp16 Wang
and Komatsuzaki (2021), and Koala Geng et al. (2023) (a Llama variant Touvron
et al. (2023)). By doing so, we provide both (1) results on both open and
closed source models, (2) results on instruction-tuned and non-instruction
tuned models, and (3) models ranging from 220 million parameters to 175B
models. Note that our experiments consist solely of providing prompts to the
models and do not include fine-tuning (as the goal is to see what these models
can do zero-shot).

For short-form QA datasets, we alter the prompt the have the model produce
both the answer and the explanation, allowing us to measure the explanation
overlap. However, smaller models (e.g. < 15B parameters) were not able to
follow instructions to provide both answer and explanation in a parseable
format from just one prompt. Thus, we do two-step prompting with them, asking
them to answer the question, then asking them to explain the answer (and
appending the grounding prompt, if used). We use the explanation to measure
the QUIP-Score. For detailed information and full text of the prompts used,
see Appendix A.

##  4 Results

We first analyze a wide range of _according-to_ prompts on ChatGPT, using this
as a cost-effective development stage to try a variety of grounding and anti-
grounding prompts. We later test the null prompt and the best performing
grounding prompt on a variety of other models for further analysis. Table 1
shows results for different prompts using the ChatGPT model. We see that the
QUIP-Score for the anti-grounding prompts generally performs the same or worse
than the null prompt (e.g. no additional text) and generally significantly
worse than the grounding prompts. There is also a clear trend in the grounding
prompts, with all grounding prompts performing similarly or improving on their
QUIP-Score compared to the null prompt.

Surprisingly, we find that grounding prompts perform similarly (and sometimes
even better than) the null prompt on end-task performance (e.g. up to a 6%
improvement on NQ, 2.5% on HotpotQA). We note that while this is not the case
for Rouge-L on ELI5, as that metric is measuring similarity to Reddit, rather
than similarity to Wikipedia.

We use these results on ChatGPT to inform our next experiments, using the null
prompt and the best grounding prompt (“Respond to this question using only
information that can be attributed to Wikipedia") in our future experiments
due to cost.

###  4.1 Results from Other Models

We show the relative difference of the grounding prompt over the null prompt
for more models in Table 2, which further confirms our findings (for the
absolute instead of relative numbers, see Appendix A). For example, using the
grounding prompt with Text-Davinci-003 improves over the null prompt by around
15% QUIP-Score and 5-20% for the specific task. For all models evaluated, the
grounding prompt improves in both end-task performance and QUIP-Score by
5-105%.

Thus, our findings hold for a wide variety of models and model-sizes – even
when prompts are not tuned for the specific model being prompted, indicating
the generality of our approach.

Prompt |  Generated Text
---|---
1\. Question (ELI5) |  Why is it that every time we put something down our throat toothbrush, finger etc. we start to gag but when we but food and water down our throat we don’t gag at all?,
Null Prompt (ChatGPT) |  This is because our gag reflex is triggered by objects that touch the back of our throat and activate sensory receptors called the pharyngeal receptors. When we eat or drink, the muscles in our throat automatically contract and move food and liquid down to the stomach, bypassing the pharyngeal receptors and avoiding the gag reflex. However, when we put non-food objects like a toothbrush or finger in our throat, they can activate these receptors and trigger the gag reflex.
Grounded Prompt (ChatGPT) |  The gag reflex is a natural protective mechanism of the body that can be triggered by stimulation of the soft palate, the back of the tongue or the back of the throat. It is a reflex contraction of the muscles of the back of the throat that can cause vomiting or choking. The reflex can be more easily triggered by foreign objects, such as toothbrushes or fingers, that are not natural to the body, compared to food and water which are familiar substances that we regularly ingest without triggering the reflex. Additionally, the shape and texture of the foreign object may also contribute to triggering the gag reflex.
2\. Question (NQ) |  In what part of the digestive tube do you expect the initial digestion of starch?
Null Prompt (ChatGPT) |  The initial digestion of starch occurs in the mouth; enzymes in saliva break down starch into smaller molecules.
Grounded Prompt (ChatGPT) |  The initial digestion of starch happens in the mouth; Salivary amylase, an enzyme found in the mouth, is responsible for breaking down starch into simpler sugars.
3\. Question (ELI5) |  Why do towels that feel so rough on the skin dry you so much better, but towels that feel soft don’t dry well at all?
Null Prompt (Davinci) |  Towels that feel rough on the skin are usually made of a coarser fabric, such as terry cloth, which is made of loops of cotton yarn. This fabric is more absorbent than softer fabrics, such as velour or microfiber, which are made of finer fibers. The loops of terry cloth fabric create more surface area for the towel to absorb water, which is why it is more effective at drying than softer towels.
Grounded Prompt (Davinci) |  Towels that feel rough on the skin are usually made of a material with a higher absorbency, such as cotton or linen. These materials are able to absorb more water than softer materials, such as microfiber or polyester. The absorbency of the material allows it to draw moisture away from the skin more effectively, resulting in better drying. Soft towels, on the other hand, are usually made of materials with lower absorbency, which means they are not able to absorb as much water and therefore do not dry as well.
4\. Question (TriviaQA) |  According to Smokey The Bear, Remember… only you can what?
Null Prompt (GPT-J-Instruct) |  “Only you can see me"; Because fire requires a fuel source to burn, and smoke is the byproduct of a fire’s burning process. Therefore, since smoke is created whenever a fire burns, the presence of smoke indicates that a fire is burning somewhere. “Only you can see me"; Because fire requires a fuel source to burn, and smoke is the byproduct of a fire’s burning process. Therefore, since smoke is created whenever a fire burns, the presence of smoke indicates that a fire is burning somewhere.
Grounded Prompt (GPT-J-Instruct) |  “Only you can remember"; According to Smokey The Bear, memory is an internal mental process that is unique to each person. It is the ability to remember past experiences, along with the ability to recognize and interpret the information that is stored in our memories. This explains why, despite the fact that memory techniques vary between individuals, memory is a universal quality that all humans share. Therefore, the answer to the question is “yes".
5\. Question (HotpotQA) |  Which magazine is based out of more countries, Adventist World or Baseball Digest?
Null Prompt (FLAN-T5 XXL) |  adventist world; Adventist World is a magazine that is based out of many countries.
Grounded Prompt (FLAN-T5 XXL) |  adventist world; Adventist World is a monthly magazine published by the Seventh-day Adventist Church. It is based in the United States, Canada, and many other countries.
Figure 5:  Example generations from various considered models. Purple text was
found in Wikipedia. Note that for non-ELI5 datasets, models were prompted to
generate the answer, a semicolon, and then the explanation (see Section 3.4).
Note that better grounding to Wikipedia does not always imply correct answers
(see Question 4).

###  4.2 Impact of Model Size

Does model size impact their ability to quote from their pre-training data? We
answer this question in Figure 2, which shows that smaller models perform the
same or worse with a grounding prompt as opposed to the null prompt. However,
larger models perform significantly better with the grounding prompt as
opposed to the null prompt, for both OpenAI models and FLAN-T5 models. Thus,
from this we can conclude that a model’s ability to quote from its pre-
training data improves with size.

###  4.3 Impact of Instruction Tuning

One potential reason for why these models can recall their pre-training data
on request is a better capability to instruction-follow. We test this
hypothesis in Figure 4 that compares T5-11B compared to FLAN-T5-11B.

We find that instruction-tuning does help, as the QUIP-Scores for
T5-v1.1-Adapt are similar between grounding and null prompts, while the
FLAN-T5 model has a large difference between the null and grounding prompt
(roughly 2x better).

###  4.4 Impact of Entity Popularity

Another potential factor influencing model memorization is the popularity of
the entities that it is asked questions about Kandpal et al. (2022); Carlini
et al. (2023). Previous work has shown that Question/Answer entity co-
occurrence (as measured by the count in the pre-training set of instances
where the entity in the question and the answer co-occur in the same passage)
is strongly correlated to the performance on the task Kandpal et al. (2022).
We use their code and data (from the Pile) to explore whether or not QUIP-
Score is correlated with the entity popularity/co-occurrence frequency.

Due to the imbalance between co-occurrence count, we sample 400 instances (or
as many as available) from each dataset and co-occurrence frequency bin. We
measure the QUIP-Score on these instances using the output generations from
ChatGPT on both grounding and null prompts.

We find in Figure 3 that QA entity popularity is positively correlated with
QUIP-Score for both grounding and null prompts (although more effective for
the grounding prompt) and that the model is better able to recall information
from Wikipedia when QA entities frequently co-occur.

###  4.5 Qualitative Examples

In Table 5 we show example output from a variety of models across the
evaluation datasets. We can see that when asked to be grounded the model
correctly generates much larger chunks of texts that occur in Wikipedia (shown
in purple).

It is also important to note that although the text may be present in
Wikipedia, it does not automatically make the output generation correct with
respect to the question. For example, the TriviaQA example shows that both
models predicted the incorrect quote for Smokey the Bear, although the
grounding prompt was better grounded in Wikipedia for the explanation of its
prediction.

However, overall, grounding prompts generate more quoted Wikipedia text than
the null prompt, showing the effectiveness of our approach.

##  5 Discussion and Future Implications

Our results strongly suggest that LLMs can be steered via prompting to
increase the amount by which they quote human-authored sources in their
training data. This finding has strong implications not just for our
considered task, ODQA, but also for a wide array of other task spaces in which
provenance grounding is important. We note that our _according-to_ prompting
strategy is domain agnostic, and constructing a Data Portrait for an arbitrary
corpus is straightforward and not limited to indices of Wikipedia or other
internet knowledge sources. For example, one might desire a scientific
reasoning system that quotes the findings of domain experts in areas such as
law or medicine.

Moreover, another appealing characteristic of a Data Portrait-based approach
to measuring overlap is that it preserves the privacy of source content. This
means that closed-source models trained on proprietary data can still use this
approach and release QUIP-Score performance, thus serving as a privacy-
preserving measure of provenance grounding without having to release the
private corpus. This can serve to add trust in systems such as ChatGPT that
are increasingly used for arbitrary reasoning tasks, by allowing them to
highlight quoted spans, without having to release the full data that was used
for training/fine-tuning.

##  6 Conclusion

Large language models struggle with hallucination, or generating incorrect
information, despite the large amount of factual pre-training data they were
trained on. To help alleviate this problem, we proposed according-to prompts,
asking language models to ground their output to their pre-training corpus. To
quantify the extent to which models achieve this goal, we introduced a new
metric, QUIP-Score, that efficiently and quickly measures the percent of the
model’s generation that exists as exact quotes in the pre-training corpus. We
showed that prompting models with grounding prompts greatly improves the QUIP-
Score while anti-grounding prompts reduces the QUIP-Score. Our analysis also
shows that QUIP-Score increases with instruction-tuning, popularity of the
entity in the question, and model size. We hope that this work brings more
attention to the positive aspects of LLM memorization and encourages more work
into understanding how and when language model output is grounded to its pre-
training data.

##  7 Limitations

Our proposed metric only accounts for exact lexical match and will miss other
types of grounded statements - thus we view QUIP-Score as a lower bound on
grounding where grounding is defined only by quoting from source material.

We also recognize the possibility of a discrepancy between the pre-training
data of private models like ChatGPT and the Wikipedia version we use for
analysis, due to limited information on their pre-training. However, this
might not be a significant concern, as although Wikipedia is not completely
static, a substantial part of the information in this knowledge source remains
consistent over a short span of years. Furthermore, our results with ChatGPT
are similar compared with models for which we do have the exact pre-training
data (like GPT-J).

## References

  * Bloom (1970) Burton H. Bloom. 1970.  Space/time trade-offs in hash coding with allowable errors.  _Communications of the ACM_ , 13(7):422–426.
  * Bohnet et al. (2022) Bernd Bohnet, Vinh Q Tran, Pat Verga, Roee Aharoni, Daniel Andor, Livio Baldini Soares, Jacob Eisenstein, Kuzman Ganchev, Jonathan Herzig, Kai Hui, et al. 2022\.  Attributed question answering: Evaluation and modeling for attributed large language models.  _arXiv preprint arXiv:2212.08037_.
  * Borgeaud et al. (2022) Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022.  Improving language models by retrieving from trillions of tokens.  In _International Conference on Machine Learning (ICML)_.
  * Carlini et al. (2023) Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2023.  Quantifying memorization across neural language models.  In _International Conference on Learning Representations (ICLR)_.
  * Carlini et al. (2020) Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B. Brown, Dawn Xiaodong Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. 2020.  Extracting training data from large language models.  In _USENIX Security Symposium (USENIX)_.
  * Chang et al. (2023) Kent K Chang, Mackenzie Cramer, Sandeep Soni, and David Bamman. 2023.  Speak, memory: An archaeology of books known to chatgpt/gpt-4.  _arXiv preprint arXiv:2305.00118_.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_.
  * De Cao et al. (2021) Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021.  Editing factual knowledge in language models.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6491–6506, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Dodge et al. (2021) Jesse Dodge, Maarten Sap, Ana Marasovic, William Agnew, Gabriel Ilharco, Dirk Groeneveld, and Matt Gardner. 2021.  Documenting large webtext corpora: A case study on the colossal clean crawled corpus.  In _Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Fan et al. (2019) Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019.  ELI5: Long Form Question Answering.  In _Annual Meeting of the Association for Computational Linguistics (ACL)_.
  * Feng et al. (2022) Yukun Feng, Patrick Xia, Benjamin Van Durme, and João Sedoc. 2022.  Automatic document selection for efficient encoder pretraining.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 9522–9530, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Ganguli et al. (2023) Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas Liao, Kamilė Lukošiūtė, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, et al. 2023.  The capacity for moral self-correction in large language models.  _arXiv preprint arXiv:2302.07459_.
  * Gao et al. (2020) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020.  The Pile: An 800gb dataset of diverse text for language modeling.  _arXiv preprint arXiv:2101.00027_.
  * Geng et al. (2023) Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023.  Koala: A dialogue model for academic research.  Blog post.
  * Han and Tsvetkov (2022) Xiaochuang Han and Yulia Tsvetkov. 2022.  ORCA: interpreting prompted language models via locating supporting data evidence in the ocean of pretraining data.  _arXiv preprint arXiv:2205.12600_.
  * He et al. (2022) Hangfeng He, Hongming Zhang, and Dan Roth. 2022.  Rethinking with retrieval: Faithful large language model inference.  _arXiv preprint arXiv:2301.00303_.
  * Izacard et al. (2022) Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022.  Atlas: Few-shot learning with retrieval augmented language models.  _arXiv preprint arXiv_.
  * Ji et al. (2022) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. 2022.  Survey of hallucination in natural language generation.  _ACM Computing Surveys_.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017.  TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension.  In _Annual Meeting of the Association for Computational Linguistics (ACL)_.
  * Kandpal et al. (2022) Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2022\.  Large language models struggle to learn long-tail knowledge.  _arXiv preprint arXiv:2211.08411_.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019.  Natural Questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics (TACL)_ , 7:453–466.
  * Lee et al. (2022a) Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2022a.  Deduplicating training data makes language models better.  In _Annual Meeting of the Association for Computational Linguistics (ACL)_.
  * Lee et al. (2022b) Nayeon Lee, Wei Ping, Peng Xu, Mostofa Patwary, Pascale N Fung, Mohammad Shoeybi, and Bryan Catanzaro. 2022b.  Factuality enhanced language models for open-ended text generation.  _Advances in Neural Information Processing Systems (NeurIPS)_.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.  The power of scale for parameter-efficient prompt tuning.  In _Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Levy et al. (2021) Sharon Levy, Michael Saxon, and William Yang Wang. 2021.  Investigating memorization of conspiracy theories in text generation.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 4718–4729, Online. Association for Computational Linguistics.
  * Li et al. (2022) Shaobo Li, Xiaoguang Li, Lifeng Shang, Zhenhua Dong, Chengjie Sun, Bingquan Liu, Zhenzhou Ji, Xin Jiang, and Qun Liu. 2022.  How pre-trained language models capture factual knowledge? a causal-inspired analysis.  In _Annual Meeting of the Association for Computational Linguistics (ACL)_.
  * Liu et al. (2023) Nelson F Liu, Tianyi Zhang, and Percy Liang. 2023.  Evaluating verifiability in generative search engines.  _arXiv preprint arXiv:2304.09848_.
  * Luccioni and Viviano (2021) Alexandra Sasha Luccioni and Joseph D Viviano. 2021.  What’s in the box? a preliminary analysis of undesirable content in the common crawl corpus.  In _Annual Meeting of the Association for Computational Linguistics (ACL)_.
  * Mallen et al. (2023) Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Hannaneh Hajishirzi, and Daniel Khashabi. 2023.  When not to trust language models: Investigating effectiveness and limitations of parametric and non-parametric memories.  In _Annual Meeting of the Association for Computational Linguistics (ACL)_.
  * Marone and Van Durme (2023) Marc Marone and Benjamin Van Durme. 2023.  Data portraits: Recording foundation model training data.  _arXiv preprint arXiv:2303.03919_.
  * Menick et al. (2022) Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, et al. 2022.  Teaching language models to support answers with verified quotes.  _arXiv preprint arXiv:2203.11147_.
  * Mishra et al. (2022a) Swaroop Mishra, Daniel Khashabi, Chitta Baral, Yejin Choi, and Hannaneh Hajishirzi. 2022a.  Reframing instructional prompts to gptk’s language.  In _Annual Meeting of the Association for Computational Linguistics (ACL) - Findings_.
  * Mishra et al. (2022b) Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022b.  Cross-Task Generalization via Natural Language Crowdsourcing Instructions.  In _Annual Meeting of the Association for Computational Linguistics (ACL)_.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022\.  Training Language Models to Follow Instructions with Human Feedback.  In _Advances in Neural Information Processing Systems (NeurIPS)_.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  BLEU: a method for automatic evaluation of machine translation.  In _Annual Meeting of the Association for Computational Linguistics (ACL)_.
  * Park et al. (2023) Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry. 2023.  TRAK: Attributing Model Behavior at Scale.  _arXiv preprint arXiv:2303.14186_.
  * Petroni et al. (2021) Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021.  KILT: A benchmark for knowledge intensive language tasks.  In _Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)_.
  * Piktus et al. (2023) Aleksandra Piktus, Christopher Akiki, Paulo Villegas, Hugo Laurençon, Gérard Dupont, Alexandra Sasha Luccioni, Yacine Jernite, and Anna Rogers. 2023.  The roots search tool: Data transparency for llms.
  * Popović (2015) Maja Popović. 2015.  chrF: character n-gram f-score for automatic mt evaluation.  In _Conference on Machine Translation (WMT)_.
  * Popović (2017) Maja Popović. 2017.  chrF++: words helping character n-grams.  In _Conference on Machine Translation (WMT)_.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research (JMLR)_.
  * Rashkin et al. (2021) Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and David Reitter. 2021\.  Measuring attribution in natural language generation models.  _arXiv preprint arXiv:2112.12870_.
  * Shuster et al. (2021) Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021.  Retrieval augmentation reduces hallucination in conversation.  In _Findings- Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Srivastava et al. (2023) Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2023.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.  _Transactions on Machine Learning Research (TMLR)_.
  * Su et al. (2022) Dan Su, Xiaoguang Li, Jindi Zhang, Lifeng Shang, Xin Jiang, Qun Liu, and Pascale Fung. 2022.  Read before generate! faithful long form question answering with machine reading.  In _Findings - Annual Meeting of the Association for Computational Linguistics (ACL)_.
  * Sun et al. (2023) Weiwei Sun, Zhengliang Shi, Shen Gao, Pengjie Ren, Maarten de Rijke, and Zhaochun Ren. 2023.  Contrastive learning reduces hallucination in conversations.  In _Conference on Artificial Intelligence (AAAI)_.
  * Tirumala et al. (2022) Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. 2022.  Memorization without overfitting: Analyzing the training dynamics of large language models.  _Advances in Neural Information Processing Systems_ , 35:38274–38290.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023.  LLaMA: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Wan et al. (2023) Alexander Wan, Eric Wallace, Sheng Shen, and Dan Klein. 2023.  Poisoning language models during instruction tuning.  _arXiv preprint arXiv:2305.00944_.
  * Wang and Komatsuzaki (2021) Ben Wang and Aran Komatsuzaki. 2021.  GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.  https://github.com/kingoflolz/mesh-transformer-jax.
  * Wang et al. (2022a) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. 2022a.  Rationale-augmented ensembles in language models.  _arXiv preprint arXiv:2207.00747_.
  * Wang et al. (2023) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023.  Self-Instruct: Aligning Language Model with Self Generated Instructions.  In _Annual Meeting of the Association for Computational Linguistics (ACL)_.
  * Wang et al. (2022b) Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Maitreya Patel, Kuntal Kumar Pal, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Shailaja Keyur Sampat, Savan Doshi, Siddhartha Mishra, Sujan Reddy, Sumanta Patro, Tanay Dixit, Xudong Shen, Chitta Baral, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi, and Daniel Khashabi. 2022b.  Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ Tasks.  In _Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018.  HotpotQA: A dataset for diverse, explainable multi-hop question answering.  In _Conference on Empirical Methods in Natural Language Processing (EMNLP)_.

Supplemental Material

##  Appendix A All Results

We show all results for the models in Table 2 that did not fit due to space.

#### Prompts for Short-Form QA

For short-form QA datasets, to help the models generate both the answer and
the explanation in a parseable format, we append the following prompt before
the question:

> You are a highly intelligent & complex question-answer generative model. You
> take a question as an input and answer it by imitating the way a human gives
> short answers with a corresponding explanation. You answer should be short -
> only a few words.\n\nYour output format should be the answer, then a
> semicolon, then the explanation.\n

For models that don’t respond well to the above prompt (or similar prompts
aimed at generating both answer and explanation from one generation), we use
the following prompts in a two step manner:

> Output the answer only. {Insert Question}\nAnswer string only:

> Question: {Insert Question}\nAnswer: {Insert Previous Output}\n\n\nGive a
> detailed explanation for why this is true. {Insert Grounding Prompt Here}
> \nExplanation:

#### Prompt for ELI5 for smaller models

For T5-v1.1-Adapt and GPT-J-Instruct evaluated on ELI5, we append “Answer:"
following the end of both the normal null and grounding prompts because
otherwise the model outputs very short (< 5 words) or nonsensical responses.
With this addition the model produces normal fluent text.

Model | Prompt | TQA | NQ | Hotpot | ELI5
---|---|---|---|---|---
|  | QUIP | EM | QUIP | EM | QUIP | F1 | QUIP | R-L
Text-Davinci-003 | Null | 35.9 | 68.2 | 38.7 | 24.3 | 34.6 | 29.2 | 27.7 | 23.7
Text-Davinci-003 | Grounded | 41.2 | 71.8 | 44.4 | 29.3 | 39.6 | 31.3 | 32.2 | 22.8
GPT-4 | Null | - | - | - | - | - | - | 21.0 | 21.5
GPT-4 | Grounded | - | - | - | - | - | - | 24.7 | 21.0
GPT-J-Instruct | Null | 28.1 | 2.2 | 28.2 | 0.9 | 29.2 | 7.0 | 22.8 | 19.9
GPT-J-Instruct | Grounded | 31.5 | 2.1 | 32.5 | 1.0 | 33.2 | 7.0 | 27.0 | 19.4
Koala | Null | 34.0 | 17.2 | 36.1 | 6.3 | 33.9 | 13.2 | 24.1 | 19.9
Koala | Grounded | 35.8 | 17.2 | 38.4 | 6.3 | 35.6 | 13.2 | 32.6 | 22.8
FLAN-T5 XXL | Null | 18.6 | 31.5 | 23.5 | 13.3 | 25.8 | 23.6 | 14.9 | 12.4
FLAN-T5 XXL | Grounded | 26.6 | 31.5 | 33.2 | 13.3 | 31.1 | 23.6 | 30.6 | 18.7
Table 3: Full results for other models. Note that the low EM scores for GPT-J
and Koala are due to the model failing to output short answers zero shot (e.g.
“the answer is…" instead of outputting the answer. Both used the same
instruction tuning dataset.). GPT-4 was only run on ELI5 due to cost.