#  SynthesizRR: Generating Diverse Datasets with Retrieval Augmentation

Abhishek Divekar$\spadesuit$$\diamondsuit$ Greg Durrett$\diamondsuit$
$\spadesuit$Amazon
$\diamondsuit$Department of Computer Science, The University of Texas at
Austin
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Large language models (LLMs) are versatile and can address many tasks, but for
computational efficiency, it is often desirable to distill their capabilities
into smaller student models. One way to do this for classification tasks is
via dataset synthesis, which can be accomplished by generating examples of
each label from the LLM. Prior approaches to synthesis use few-shot prompting,
which relies on the LLM’s parametric knowledge to generate usable examples.
However, this leads to issues of repetition, bias towards popular entities,
and stylistic differences from human text. In this work, we propose Synthesize
by Retrieval and Refinement (SynthesizRR), which uses retrieval augmentation
to introduce variety into the dataset synthesis process: as retrieved passages
vary, the LLM is “seeded” with different content to generate its examples. We
empirically study the synthesis of six datasets, covering topic
classification, sentiment analysis, tone detection, and humor, requiring
complex synthesis strategies. We find SynthesizRR greatly improves lexical and
semantic diversity, similarity to human-written text, and distillation
performance, when compared to standard 32-shot prompting and six baseline
approaches.

SynthesizRR: Generating Diverse Datasets with Retrieval Augmentation

Abhishek Divekar$\spadesuit$$\diamondsuit$ Greg Durrett$\diamondsuit$
$\spadesuit$Amazon $\diamondsuit$Department of Computer Science, The
University of Texas at Austin<EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Figure 1:  Synthetic examples from few-shot generation (middle) and
SynthesizRR (bottom). Few-shot generation leverages a frozen LLM’s parametric
knowledge as the source of content in the generated examples. SynthesizRR
incorporates a distinct content sourcing step by retrieving documents from a
corpus. For the task of detecting political bias, a news article is retrieved
and transformed in tone. The resulting synthesis procedure yields diverse
examples which more closely match human-written examples (top).  Figure 2:
Abstract depiction of the SynthesizRR pipeline. In the content sourcing stage,
we retrieve $K$ unique document $\\{r_{1},\dots,r_{K}\\}$ from a large corpus
for each in-context covariate $x_{\text{ICL}}$. The task-inversion stage of
synthesis uses a parameterized context refinement prompt $\mathcal{P}_{\tau}$,
which takes parameters $R_{inv}$ (inversion instruction), $r_{k}$ (a retrieved
document), and $\mathcal{V}(y_{\text{ICL}})$ (the verbalized target label). A
generalist teacher LLM autoregressively generates a synthetic covariate. Each
in-context example thus produces $K$ unique synthetic examples
$\\{\tilde{x}_{1},\dots,\tilde{x}_{K}\\}$, which we include in the dataset
with target $y_{\text{ICL}}$.

Large Language Models (LLMs) such as GPT-4 (OpenAI, 2023; Bubeck et al.,
2023), LLaMa (Touvron et al., 2023) and Claude (Bai et al., 2022) are powerful
generalist models, capable of solving multiple tasks without parameter tuning
via zero-shot or few-shot prompting. In comparison, previous approaches fine-
tuned variants of BERT (Devlin et al., 2019) on task-specific demonstrations,
producing specialist models. These smaller specialist models are more
economical at inference time, but typically require at least thousands of
examples to train.

Recent work has sought to avoid this reliance on annotation by fine-tuning
specialist models on synthetic datasets via teacher-student distillation (West
et al., 2022). While this has been employed in the context of classification
(Yu et al., 2023a; Ye et al., 2022a, b), human-preference alignment (Lee et
al., 2023; Bai et al., 2022), language understanding (Meng et al., 2022;
Schick and Schütze, 2021), and even tabular data (Borisov et al., 2022),
synthetic data has limitations. Yu et al. (2023a) examine the data-generation
process and discover that naive prompts generate text inputs that are low in
diversity and regurgitate the biases of the teacher LLMs, which proliferate
into the student model.

Figure 1 illustrates this: few-shot generation (Brown et al., 2020) is a
common approach that uses an LLM to perform a multitude of tasks without
modifying its parameters. Prior work has used few-shot generation to
synthesize a training dataset for fine-tuning a smaller model (Ye et al.,
2022a, b; Yehudai et al., 2024a). We refer to this type of dataset synthesis
as FewGen.

FewGen relies on a generalist LLM’s ability to perform task inversion. For
example, in Figure 1, we illustrate the task of detecting politically-biased
articles. By using the prompt “Write a news article using harsh political
language, a mocking tone and toxic commentary” and a few in-context learning
examples, sampling continuations from an LLM is often sufficient to generate
plausible news articles in the biased style we seek to detect. However, when
thousands of completions are sampled from this prompt to construct a large
dataset for distilling a specialist model, we observe repetition, bias towards
popular entities, and stylistic differences from human-written texts. If the
data is low diversity, a trained specialist model may not learn the task well.

In this work, we seek to alleviate the lack of diversity in synthetic data. We
make the observation that dataset synthesis may be decomposed as two distinct
LLM competencies: content sourcing, where the LLM obtains relevant information
for the task, and task inversion, where the LLM generates a synthetic input
example. Prior work has focused mainly on task inversion, i.e., generating
good covariates using a target-conditioned prompt, while implicitly using the
LLM’s parametric memory for content sourcing. In contrast, we investigate the
importance of an explicit content sourcing stage.

We propose Synthesize by Retrieval and Refinement (SynthesizRR111Pronounced
“synthesizer”.), a procedure by which a large corpus acts as the guiding
signal for the content sourcing stage. In the initial step, we repurpose each
in-context learning input as a retrieval query. We use this to retrieve
several documents from a large domain-specific corpus. We then use a
generalist LLM for task inversion on each retrieved document. As a unique
retrieved document is used in each prompt, the synthesis procedure yields more
diverse examples, containing a wider range of real-world entities and
assertions.

We benchmark SynthesizRR against FewGen on six text classification tasks,
selected carefully to measure a variety of different styles of dataset
synthesis. Experiments on an in-domain corpus (§5) reveal that SynthesizRR
significantly surpasses FewGen in diversity and resemblance to human-authored
texts, even though both procedures utilize the same frozen LLM. In §6, we see
that student classifiers fine-tuned on SynthesizRR-generated data perform
better than those fine-tuned on FewGen. We analyze the contributing factors in
the student’s training dynamics, finding that our approach generates examples
with superior class-separability, particularly for multiclass tasks.

Finally, in §7, we compare SynthesizRR to six prior approaches for synthesis
of classification datasets: AttrPrompt Yu et al. (2023a), ReGen Yu et al.
(2023b), SunGen Gao et al. (2023), ZeroGen Ye et al. (2022a), ProGen Ye et al.
(2022b), and S3 Wang et al. (2023a). In general, we find that SynthesizRR
gives higher diversity data, and in most cases, leads to higher student
accuracy as a result.

Our contributions are as follows: (1) we propose a new method for integrating
retrieval into example synthesis for teacher-student distillation; (2) we
perform a detailed analysis comparing our method with existing approaches
across both intrinsic data diversity metrics and downstream task accuracy.

##  2 Background and Task setup

The dataset generation tasks we focus on in this paper are in the domain of
text classification. Denote an example as consisting of an input $x$,
typically a natural language string, and output $y\in\mathcal{Y}$ for an
output space $\mathcal{Y}$ of $C$ classes.

We assume access to a _seed set_ of examples
$\mathcal{D}_{\textsc{Seed}}=\\{(x_{1},y_{1}),\ldots,(x_{n},y_{n})\\}$ which
is large enough to cover the classes but small enough that a user could
feasibly gather or construct these examples manually. In §5, we use the in-
context learning set as $\mathcal{D}_{\textsc{Seed}}$.

Our assumption is that $\mathcal{D}_{\textsc{Seed}}$ is not large enough to
train an effective student, and instead a larger synthetic dataset
$\mathbin{\mathbin{\mathcal{D}_{\textsc{Synth}}}=\left\\{(\mathbin{\tilde{x}^{i}},\mathbin{y^{i}})\right\\}^{m}_{i=1}}$
($m>>n$) will be needed. We repeatedly prompt a teacher language model
$\mathcal{M}_{\text{LM}}$ to generate synthetic covariates $\tilde{x}$ given
corresponding labels $y$ to produce $\mathbin{\mathcal{D}_{\textsc{Synth}}}$,
a process we refer to as task inversion. A student language model
$\mathcal{M}_{\text{S}}$ (e.g., a BERT-style pre-trained encoder model (Devlin
et al., 2019)) is trained on $\mathbin{\mathcal{D}_{\textsc{Synth}}}$. We
denote the student’s task (predicting $y$ from $x$) by $\tau$ and the
teacher’s task (generating $x$ given $y$) by $\tau_{inv}$.

SynthesizRR aims to address the lack of diversity by leveraging retrieval from
a large corpus $\mathcal{R}$ for the content sourcing step. Figure 2
illustrates how our method can be used to generate appropriate data. First, we
start by executing a retrieval step based on the examples
$\mathcal{D}_{\textsc{Seed}}$. Our assumption here is that we can find similar
examples in the corpus, which are domain-similar enough to
$\mathcal{D}_{\textsc{Seed}}$ to be usable as training examples. A context
refinement prompt is then constructed to perform task inversion on each
retrieved document. The LLM’s context window thus sees a unique and grounded
prompt for each example it generates. This overcomes the need to memorize a
corpus worth of information in the limited parameters of the teacher LLM.

We assume the existence of corpus $\mathcal{R}$, with each document containing
information relevant to the task. However, the documents do not need to be
unlabeled examples for our task; they can be more distantly related and still
provide useful information for generating examples. In §5 we successfully
generate customer reviews and questions about products, while retrieving from
a corpus where documents are product titles and descriptions. Appendix C looks
into how the corpus influences synthesis. Another reason raw retrieval
documents are unsuitable for direct use as examples is due to the chunking
process in retrieval, which can introduce coreference failures, among other
problems (Mialon et al., 2023)).

Due to the difficulty of the task inversion step as a result of this mismatch
between retrieved documents and test examples, we limit our investigation to
chat-tuned teacher LLMs which have been shown to demonstrate strong
instruction-following capabilities (Ouyang et al., 2022; Touvron et al., 2023;
Bai et al., 2022).

Algorithm 1 SynthesizRR Retr-ICL

Input A set of seed examples $\mathcal{D}_{\textsc{Seed}}$, retrieval corpus
$\mathcal{R}=\\{r_{k}\\}$, retrieval model $\mathcal{M}_{\mathrm{ret}}$,
expansion factor $K$, cosine-similarity criterion $(s_{\alpha},s_{\beta})$,
teacher model $\mathcal{M}_{\text{LM}}$, prompt template $\mathcal{P}_{\tau}$,
context refinement instruction $R_{inv}$, verbalizer
$\mathcal{V}:\\{y_{1},\dots,y_{C}\\}\rightarrow\\{v_{1},\dots,v_{C}\\}$.

Output Synthetic dataset $\mathbin{\mathcal{D}_{\textsc{Synth}}}{}$

Procedure $\textsc{SynthesizRR}(\mathcal{D}_{\textsc{Seed}},\mathcal{R})$:

$\mathcal{D}_{\textsc{Retr}}\leftarrow\emptyset$

$\mathcal{D}_{\textsc{ICL}}\leftarrow\emptyset$

$\mathbin{\mathcal{D}_{\textsc{Synth}}}{}\leftarrow\emptyset$

$\triangleright$ Content sourcing using retrieval:

for $(x,y)\in\mathcal{D}_{\textsc{Seed}}$ do

$[r_{1},\ldots,r_{K}]\leftarrow\mathcal{M}_{\mathrm{ret}}(x)$

$\mathcal{D}_{\textsc{Retr}}\leftarrow\mathcal{D}_{\textsc{Retr}}\cup\\{(x,y,[r_{1},\ldots,r_{K}])\\}$

$\triangleright$ In-context learning set construction:

for $(x,y,[r_{1},\ldots,r_{K}])\in\mathcal{D}_{\textsc{Retr}}$ do

for $r_{k}\in[r_{1},\ldots,r_{K}]$ do

$\mathcal{D}_{\textsc{ICL}}\leftarrow\mathcal{D}_{\textsc{ICL}}\cup\\{(r_{k},x)\\}$
if $s_{\alpha}\leq\cos(x,r_{k})\leq s_{\beta}$

$\triangleright$ Task inversion:

for $(x,y,[r_{1},\ldots,r_{K}])\in\mathcal{D}_{\textsc{Retr}}$ do

for $r_{k}\in[r_{1},\ldots,r_{K}]$ do

$\mathcal{D}_{\textsc{shots}}\underset{\raisebox{2.0pt}{$\scriptscriptstyle$}}{\sim}\mathcal{D}_{\textsc{ICL}}$

for $j\in[1,\dots]\text{ { until } }\mathbin{\tilde{x}}^{i}_{j}=\text{<eos>}$
do

$\mathbin{\tilde{x}}^{i}_{j}{}{\sim}{}\mathcal{M}_{\text{LM}}\left(\cdot|\mathbin{\tilde{x}}^{i}_{<j},\mathcal{P}_{\tau}(R_{inv},r_{k},\mathcal{V}(y)),\mathcal{D}_{\textsc{shots}}\right)$

$\mathbin{\mathcal{D}_{\textsc{Synth}}}{}\leftarrow\mathbin{\mathcal{D}_{\textsc{Synth}}}{}\cup\\{(\tilde{x}^{i},y)\\}$

return $\mathbin{\mathcal{D}_{\textsc{Synth}}}{}$

##  3 Method

Algorithm 1 shows our dataset generation method.

Step 1. Retrieval using $\mathcal{M}_{\mathrm{ret}}$: SynthesizRR uses each
in-context demonstration $(x_{{}_{\text{ICL}}},y_{{}_{\text{ICL}}})$ as a
query for information retrieval, in addition to its role as an in-context
demonstration for task inversion as discussed subsequently. When viewed as
queries, these demonstrations lead to an expansion as we retrieve $K$
documents $\\{r_{1},\dots,r_{K}\\}$ of decreasing similarity for each in-
context input $x_{{}_{\text{ICL}}}$. We rank contexts using retrieval model
$\mathcal{M}_{\mathrm{ret}}$ and retain contexts with cosine similarity in
(0.4, 0.9), where the upper limit helps avoid duplicates by excluding
retrieved contexts if they are too close to the ICL example. We thus append
each triplet $(x_{{}_{\text{ICL}}},y_{{}_{\text{ICL}}},[r_{1},\dots,r_{K}])$
to the set $\mathcal{D}_{\textsc{Retr}}$.

Step 2. Retrieval-augmented task inversion: The retrieved documents have
lexical and semantic overlap with the query, but are not necessarily suitable
examples for the final dataset. To accommodate this, we use a prompt to
perform task-inversion using content sourced from each retrieved document,
which we refer to as contextual refinement.

The minimum elements of the task inversion prompt $\mathcal{P_{\tau}}$ are the
instruction $\mathcal{I}_{inv}$ and target $y$. We use a verbalizer function
(Schick and Schütze, 2021; van de Kar et al., 2022) that provides a unique and
descriptive representation of each label, i.e.
$\mathcal{V}:\\{y_{1},\dots,y_{C}\\}\rightarrow\\{v_{1},\dots,v_{C}\\}$. We
follow prior work on classification-based task inversion (Schick and Schütze,
2021; Ye et al., 2022a, b; Yu et al., 2023b; Gao et al., 2023) and use
descriptive verbalizations to induce label-separability in the final dataset.

FewGen uses the standard causal language modeling objective to induce next-
token probabilities from $\mathcal{M}_{\text{LM}}$. Nucleus sampling Holtzman
et al. (2019) is used to autoregressively sample next tokens until the <eos>
token is generated. This becomes synthetic example $\mathbin{\tilde{x}}^{i}$.

$\mathbin{\tilde{x}}^{i}_{j}\underset{\raisebox{2.0pt}{$\scriptscriptstyle
p$}}{\sim}\mathcal{M}_{\text{LM}}\left(\cdot|\mathbin{\tilde{x}}^{i}_{<j},\mathcal{P}_{\tau}(I_{inv},\mathcal{V}(y))\right)$
(1)

For each label $y$, we sample $m/C$ times to generate
$\mathcal{D}_{\textsc{Synth}}=\bigcup\limits_{y\in\mathcal{Y}}{}\bigcup\limits_{i\in\\{1..m/C\\}}\left\\{(\mathbin{\tilde{x}^{i}},y)\right\\}$.

For SynthesizRR, the task inversion prompt is at minimum composed of the
contextual refinement instruction $\mathcal{R}_{inv}$ alongside the verbalized
target from the query example $\mathcal{V}(y_{{}_{ICL}})$. The LLM’s context
window thus sees a unique and grounded prompt when auto-regressively
generating each synthetic input $\mathbin{\tilde{x}}^{i}$:

$\displaystyle\mbox{$\mathbin{\tilde{x}}^{i}_{j}\underset{\raisebox{2.0pt}{$\scriptscriptstyle
p$}}{\sim}\mathcal{M}_{\text{LM}}\left(\cdot|\mathbin{\tilde{x}}^{i}_{<j},\mathcal{P}_{\tau}(R_{inv},r_{k},\mathcal{V}(y_{{}_{ICL}}))\right)$},$
(2) $\displaystyle\text{for all documents }r_{k}\in\\{r_{1},\cdots r_{K}\\}$

We continue to use nucleus sampling to get diverse generations. Each in-
context example thus produces $K$ unique synthetic examples
$\\{\tilde{x}_{1},\dots,\tilde{x}_{K}\\}$, where we refer to $K$ as the
“expansion factor”.

Step 3. Student distillation: The student is fine-tuned on
$\mathbin{\mathcal{D}_{\textsc{Synth}}}$ by passing the BERT [CLS] token
embedding through a feedforward layer. This produces a probability
distribution over the labelspace $C$. We optimize the cross-entropy loss of
the true label $y$ (which was used in the task-inversion prompt, thus is known
to us) when the student model is passed the generated example
$\mathbin{\tilde{x}}$. As we derive $\mathbin{\tilde{x}}$ from a teacher LLM,
this can be considered a form of symbolic knowledge distillation (West et al.,
2022) without the need for a critic model.

Unlike prior work, we make no changes to the student model to aid the out-of-
distribution performance and focus solely on improvements to the dataset
synthesis procedure.

In-context learning in SynthesizRR. Contextual refinement may also benefit
from in-context demonstrations. In this case, we face a challenge in
constructing an in-context learning set which accurately captures our context
refinement task $r_{k}^{i}\rightarrow\mathbin{\tilde{x}}^{i}$. A basic
exploration using handpicked demonstrations reveals that in-context learning
primarily influences the format, rather than content, of synthetic examples.
This finding is also supported by the literature on in-context learning (Min
et al., 2022). We thus develop two approaches to in-context learning (ICL).

1\. Retr-ICL: we use retrieval to construct a set of ICL examples
$\mathcal{D}_{\textsc{ICL}}$, such that each ICL example looks similar to the
SynthesizRR task-inversion prompt. We consider top-1 and top-2 retrieved
results from dense retriever $\mathcal{M}_{\mathrm{ret}}$, and use a cosine-
similarity criterion $s_{\alpha}\leq\cos(x_{{}_{\text{ICL}}},r_{k})\leq
s_{\beta}$ to determine whether there is a possible match between the
retrieved document $r_{k}$ and the ICL covariate $x_{{}_{\text{ICL}}}$. Within
a massive corpus, there are typically sufficient semantically-close retrieved
results to form an in-context pair ($r_{k},x_{{}_{\text{ICL}}}$). When used
with an appropriate prompt template from Appendix E these ICL examples
demonstrate the required format and approximately match in content.

2\. No-Retr-ICL: a baseline method, where we use retrieval for content
sourcing, but not for in-context learning. In No-Retr-ICL, for each generation
we select 32 ICL examples at random from $\mathcal{D}_{\textsc{Seed}}$. Each
ICL example is preceded by a prefix such as “News Article:” or “Product
details:”, but without any instruction. After these ICL examples, we append
the retrieved document $r_{k}$ and context refinement instruction $R_{inv}$ to
produce the final prompt. The resulting in-context learning prompt is very
similar to that used by FewGen, but also uses content-sourcing by including
$r_{k}$ and $R_{inv}$. This baseline highlights the importance of using
$\mathcal{D}_{\textsc{ICL}}$ in the Retr-ICL approach.

Dataset |  |  Class | Train, Test |  In-domain corpus |  |  Task inversion
---|---|---|---|---|---|---
AG News |  |  4 | $115\text{k},7.6\text{k}$ |  RN/Dom |  |  Easy
Hyperpartisan |  |  2 | $516,65$ |  RN/Dom |  |  Medium
ToI Headlines |  |  10 | $52\text{k},10\text{k}$ |  RN/Ind |  |  Easy
Category |  |  23 | $30\text{k}^{*},2.4\text{k}$ |  Products |  |  Medium
Humor |  |  2 | $15\text{k},3\text{k}$ |  Products |  |  Hard
Polarity |  |  2 | $72\text{k}^{*},7.2\text{k}^{*}$ |  Products |  |  Medium
Table 1:  Dataset statistics and our estimate of task inversion complexity.
∗Downsampled for convenience.

##  4 Experimental Setup

Tasks and complexity. We experiment on 6 datasets described in Table 1, which
are selected to give diverse task inversion steps. Previous work primarily
benchmarks sentiment and topic classification datasets like AG News (Zhang et
al., 2015), NYT (Meng et al., 2019), and Reddit (Geigle et al., 2021). We
broaden from topic classification, which primarily involves summarization
during the task inversion step, which LLMs are adept at (Goyal et al., 2022).
Hyperpartisan (Kiesel et al., 2019) detects bias in political news, so the
task inversion step includes a more substantial rewriting of neutral retrieved
articles to form biased examples. Task inversion for Humor (Ziser et al.,
2020), Category Blitzer et al. (2007) and Polarity (Ni et al., 2019) involves
generating humorous questions and reviews from retrieved product details,
which require additional skills.

Corpus | Domain | Docs | Item | Token
---|---|---|---|---
RealNews/Dom | US/EU News | $30.1\text{M}$ | Article | $27.1\text{B}$
RealNews/Reg | Regional News | $2.7\text{M}$ | Article | $2.1\text{B}$
RealNews/Ind | Indian News | $0.9\text{M}$ | Article | $0.6\text{B}$
Products | E-commerce | $15.0\text{M}$ | Product | $2.3\text{B}$
Table 2:  Corpus statistics. Used LLaMa2 tokenizer.

Each task in Table 1 has at least one corpus which can be considered in-domain
for the task. We ground the synthesis to this in-domain corpus during
SynthesizRR. Prompts for all tasks are in Appendix E.

In §7, we compare to six prior approaches on four popular tasks: Yelp Polarity
(Zhang et al., 2015), IMDb Maas et al. (2011), SST-2 Socher et al. (2013) and
AG News, when grounding to smaller corpora (42k-150k documents). We view these
tasks as less aligned with the goals our paper, so we do not focus our
evaluation on them here.

Method | Example
---|---
Gold |  There is decent bass, but the highs are a bit soft. A quick tweak to my equalizer, and they’re great. After reading several of the reviews on Amazon, I was a bit worried about the sound, but now that I have them I’m very happy. They’re a good price, and sooooo much better than the little ipod-like earbuds I’ve tried before. Those never stayed in my ear, and the bass never made me happy
FewGen |  I’ve been a very happy customer of this company for a long time. It is fast and does everything I need it to. I would definitely recommend it to anyone looking for a good external drive. However, I do have one issue with the product. The instructions that come with it are not very clear and I had a hard time figuring out how to properly use it.
(Retrieved Product) |  Portable Laptop Microphone. Connects to 1/8" mini microphone input on laptop. Right-angle shaped. Flat-frequency response.
SynthesizRR |  The portable laptop microphone is right-angled and has a flat-frequency response, making it easy to use for online meetings and interviews. It connects to the 1/8" mini microphone input on my laptop and has worked great for the past two months, but I have noticed some distortion in the audio when I move around too much. Overall, it’s a great value for the price and has made my remote work and video conferencing much more productive and efficient.
Table 3:  Real and synthetic examples from “electronics” class of Category.
Grey text indicates lack of specifics.

Retrieval corpora. Table 2 describes corpora used for retrieval. We consider
four corpora in different domains, each with millions of records:
RealNews/Dominant (US/EU News), RealNews/Regional (Regional News),
RealNews/India (Indian News), Products (Amazon products metadata (Ni et al.,
2019)).

Retrieval model. We use Contriever (Izacard et al., 2022) for dense retrieval
from each corpus. This performs a semantic match between the query and each
document using cosine-similarity. In Appendix B, we also perform an ablation
study using sparse retriever BM25, which does lexical matching between each
query-document pair.

Teacher LLMs. As teacher models, we primarily use a frozen LLaMa-2 Chat 13B
(Touvron et al., 2023) for the task inversion step in SynthesizRR and FewGen.
We also experiment with ClaudeV1 (Appendix H). For in-context learning (ICL)
Brown et al. (2020), we select examples randomly from the train set: 50 ICL
examples/class for multi-class and 100/class for binary tasks. We believe this
is a realistic number of examples that a system designer could source if they
were to put some effort into building a specialist model. We pick these
numbers to provide sufficient ICL diversity in 32-shot FewGen and SynthesizRR.

Student model. Specialization performance is measured on student LM
DeBERTa-v3-Large (435M params, He et al. (2021)).

Figure 3:  Self-BLEU $(\downarrow)$ for ngrams n=1-5. Comparison: Gold, FewGen
0-shot, FewGen 32-shot, SynthesizRR 0-shot, SynthesizRR 3-shot Retr-ICL,
SynthesizRR 32-shot No-Retr-ICL.

Evaluation criteria. Text generation can be challenging to evaluate
objectively in multi-task scenarios (Chang et al., 2024). Therefore in §5 we
evaluate synthetic text based on several criterion, to detect behaviours we
observe during synthesis as in Table 3. Self-BLEU (Papineni et al., 2002; Zhu
et al., 2018) measures lexical diversity of the dataset based on $n$-gram
overlap between pairs of examples. Entity entropy measures the diversity of
entities using the probability distribution of each of 16 entity-types
(inferred from a pretrained named entity recognition model). Dataset which
over-represent popular entities score lower on entropy. On the other hand,
Entity recall and Entity KL divergence compares the similarity of entities
compared to Gold, and datasets which reproduce entities frequently seen in
Gold data score higher. MAUVE (Liu et al., 2021) measure similarity to human-
written text using pretrained representations from a GPT2-XL model; this
indicates distributional differences that would be perceived by language
models.

Besides the final accuracy of the student model, we also consider label
preservation accuracy, which is obtained from an “oracle” model for the task.
We construct the oracle from Gold data by running a grid-search over
DeBERTa-v3-Large hyperparams, splitting 80% of the Gold train set for fine-
tuning and 20% for validation. Then, we measure the fraction of synthetic
examples which the oracle classifies to belong to the prompted target class.
This indicates the adherence of the generated example to the class it should
belong to, as per the prompt.

Figure 4:  Entity entropy $(\uparrow)$ on ToI (headlines) and Category
(reviews). Comparison: Gold, FewGen 32-shot, SynthesizRR 3-shot Retr-ICL and
SynthesizRR 32-shot No-Retr-ICL. Zero-shot results are similar for SynthesizRR
and worse for FewGen; we omit them.

##  5 Results: Intrinsic Evaluation

In this section we focus on evaluating intrinsic properties of the generated
datasets, including their diversity and entity coverage. We focus on a LLaMa-2
Chat 13B teacher LLM. We generate datasets of a fixed size comparable to the
number of Gold rows: $8$k rows (AG News, ToI Headlines, Category), $4$k rows
(Polarity) or $2$k rows (Hyperpartisan (516 originally), Humor). Examples of
generations on each dataset can be found in Appendix F.

RQ: Does retrieval augmentation improve lexical diversity? Figure 3 shows
lexical diversity within the dataset. Human-written texts (Gold) score high on
lexical diversity (low Self-BLEU). FewGen texts tend to reuse the same words
and phrases, leading to repeated text across generations (high Self-BLEU).
SynthesizRR text has lexical diversity approaching human text for all n-gram
values. We note in-context learning has an inconsistent effect; it improves
the lexical diversity for news corpora but not for products.

RQ: Does SynthesizRR address entity diversity? Popularity bias is a phenomenon
wherein LLM generations tend to over-represent popular “head” entities. It has
been studied for QA tasks (Mallen et al., 2023; Kandpal et al., 2023).

In Figure 4 we see how SynthesizRR eliminates popularity bias in several
entity types at once. Here, we use Spacy NER model en_core_web_lg (Honnibal et
al., 2020) to detect entity occurrences in each synthetic example. By sourcing
from the long-tail of retrieval results ($k=50$), the generated dataset has
much higher entity entropy, meaning the probability distribution of each
entity-type is closer to a uniform distribution compared to FewGen. This
positions SynthesizRR closer to Gold, which also shows high entity entropy.

Method | Norp | Org | Person | Gpe |  Recall $(\uparrow)$ |  KL div. $(\downarrow)$
---|---|---|---|---|---|---
Unique Entities
Gold | 319 | 3943 | 3952 | 712 |  - |  -
FewGen* | 43 | 480 | 400 | 73 |  0.05 |  -
SynthzRR† | 137 | 2718 | 1528 | 238 |  0.12 |  -
SynthzRR‡ | 109 | 1755 | 1012 | 178 |  0.10 |  -
Total Entities
Gold | 843 | 7233 | 6096 | 1558 |  - |  -
FewGen* | 94 | 775 | 506 | 96 |  0.23 |  3.10
SynthzRR† | 319 | 3991 | 1989 | 397 |  0.35 |  2.35
SynthzRR‡ | 314 | 2699 | 1464 | 363 |  0.32 |  2.52
Table 4:  Entity similarity in Category ($8$K). We show the counts of unique
and total entities for 4 entity-types. Entity recall measures the fraction of
Gold entities co-occuring in the synthetic data; in the bottom half, we
additionally weigh each entity by its frequency in Gold. Notation: *32-shot;
†3-shot Retr-ICL; ‡32-shot No-Retr-ICL.

RQ: How is entity similarity in synthetic data affected by grounding to an in-
domain corpus? Generating diverse entities is not beneficial if they are not
also present in Gold. We thus study entity similarity. For the Category task
we generate $8$k product reviews and randomly select $8$k Gold examples. In
Table 4, we measure entity recall, and find that the occurrence of Gold
entities is 100%-140% higher in SynthesizRR than FewGen. The KL divergence of
each entity-distribution is also lower. We finally consider the entity
coverage (unique entities) and entity density (total entities). Compared to
Gold, FewGen tends to produce fewer unique entities (places, events,
languages, currencies, etc). Each FewGen example also has a lower density of
entities, as visible in Table 3. SynthesizRR coverage and density more closely
match Gold.

RQ: How distributionally similar are our generated examples and human-written
examples? We calculate MAUVE score, which uses GPT-2XL (Radford et al., 2019)
to embed texts and compare them based on Type-1 and Type-2 errors (Liu et al.,
2021). We see from Table 5 that zero-shot generations are quite dissimilar in
both approaches compared to few-shot methods.

Method | AG. | Hyp. | ToI | Cat. | Hum. | Pol.
---|---|---|---|---|---|---
(Dataset size) | ($8$K) | ($4$K) | ($8$K) | ($8$K) | ($2$K) | ($4$K)
Zero shot
FewGen | $56.57$ | $53.68$ | $62.79$ | 63.2 | $75.6$ | $62.82$
SynthzRR | 90.3 | 59.2 | 63.0 | $61.06$ | 82.9 | 78.6
Few shot
FewGen* | $56.7$ | $65.39$ | $60.33$ | $65.8$ | $78.14$ | $69.21$
SynthzRR† | 92.0 | 72.8 | 87.9 | 75.2 | 87.5 | 89.9
SynthzRR‡ | $91.76$ | $67.86$ | $67.16$ | $75.14$ | $86.95$ | $83.2$
Table 5:  MAUVE similarity score ($\uparrow$) using GPT2-XL embeddings. Notation: *32-shot; †3-shot Retr-ICL; ‡32-shot No-Retr-ICL.  Method | Teacher LM |  | AG. | Hyper. | ToI | Categ. | Humor | Polar. | Avg
---|---|---|---|---|---|---|---|---|---
(Dataset size) |  | ($8$K) | ($4$K) | ($8$K) | ($8$K) | ($2$K) | ($4$K)
Gold |  |  | $91.015\,789$ | $93.230\,769$ | $82.488$ | $81.516\,667$ | $93.06$ | $95.2645$ | 89.43
Zero shot
FewGen | LLaMa2 |  | $69.457\,895$ | 72.6 | $32.09$ | $62.35$ | $74.433\,333$ | $80.996$ | 65.32
SynthesizRR | LLaMa2 |  | $83.510\,526$ | $69.846\,154$ | 74.4 | 68.9 | 82.5 | $84.707$ | 77.32
FewGen | ClaudeV1 |  | $75.031\,579$ | $57.538\,462$ | $23.298$ | $47.1$ | $49.866\,667$ | $87.481$ | 56.72
SynthesizRR | ClaudeV1 |  | 83.9 | $72.307\,692$ | $71.83$ | $66.783\,333$ | $62.113\,333$ | 88.7 | 74.29
Few shot
FewGen* | LLaMa2 |  | $84.2$ | $74.461\,538$ | 73.7 | $68.641\,667$ | $88.38$ | $90.899$ | 80.05
SynthesizRR† | LLaMa2 |  | $82.992\,105$ | $78.461\,538$ | $73.262$ | 72.4 | 90.2 | $91.0015$ | 81.38
SynthesizRR‡ | LLaMa2 |  | 85.2 | 79.1 | $72.826$ | $71.941\,667$ | $88.773\,333$ | $88.197$ | 81.00
FewGen* | ClaudeV1 |  | $75.852\,632$ | $58.461\,538$ | $72.224$ | $68.833\,333$ | $82.94$ | $91.2435$ | 74.93
SynthesizRR† | ClaudeV1 |  | $83.7394$ | $72.307\,692$ | $72.832$ | $65.408\,333$ | $83.393\,333$ | 91.3 | 78.16
SynthesizRR‡ | ClaudeV1 |  | $83.710\,526$ | $72.00$ | $72.45$ | $67.816\,667$ | $76.2066$ | $87.87$ | 76.68
Table 6:  Test Accuracy ($\uparrow$) after distilling DeBERTa-v3-Large student
from LLaMa-2 Chat 13B and Claude Instant-v1. We report mean of 5 runs and
rerun all 5 in cases of std $\geq$ 6.0. The top half considers zero-shot
synthesis and bottom half uses in-context learning, where we bold the best
result under each paradigm. Notation: *32-shot; †3-shot Retr-ICL; ‡32-shot No-
Retr-ICL.

Surprisingly, SynthesizRR generations are much more similar to human text than
FewGen. This is surprising as nothing in our content sourcing strategy
explicitly guides SynthesizRR generations to match the distribution of Gold.
We thus manually inspect generations and discover an interesting pattern which
can be attributed to content sourcing. As shown earlier, and in Table 3, the
density of entities is higher under SynthesizRR. FewGen produces generations
which obey the prompt, but are very bland and do not include specifics. On the
other hand, by obtaining information-rich documents, SynthesizRR is able to
ground the task inversion step in details of the retrieved article/product. We
hypothesise that this improves the MAUVE score towards Gold, which is
similarly grounded in specifics.

We further analyze the effect of the retrieval corpus in Appendix C.

##  6 Results: Student distillation

We have established that SynthesizRR generates more diverse datasets compared
to a baseline approach. Now, we return to the application of training a
specialist model based on these datasets.

Table 6 shows the results of training a DeBERTa-v3L student on generated
datasets across several settings. We report the mean of 5 runs and rerun if
the standard deviation is $\geq$6% (indicating one or more models failed to
train). In the zero-shot setting, we find that SynthesizRR performs much
better than FewGen, despite using the same frozen teacher LLM. Note that
SynthesizRR uses in-context examples for retrieval here whereas FewGen does
not; our method has some additional supervision here. However, in this
setting, we see clear gains during the task inversion stage ($\uparrow$12% for
LLaMa2 and $\uparrow$17.6% for ClaudeV1). Thus, having access to retrieval
yields a better final dataset, almost on par with the few-shot generation.

With in-context learning, 3-shot SynthesizRR using the No-Retr-ICL strategy
trains better students than 32-shot FewGen ($\uparrow$1.3% for LLaMa2 and
$\uparrow$3.2% for ClaudeV1). We conclude that naively adding ICL examples is
not an effective use of the LLM’s context window. Instead, a better content
sourcing strategy positively influences the student distillation, leading to
faster synthesis and better test performance.

Analysis of student distillation performance. Why does SynthesizRR improve
classification dataset synthesis? We take a closer look at the few-shot
generation strategies from a student training perspective.

We first look at the label preservation accuracy, obtained from an oracle
DeBERTa-v3L model tuned on the full Train dataset (Appendix H). We would
expect that better label preservation means a higher-fidelity training
dataset. However, Table 7 shows that FewGen datasets have very high label
preservation in spite of their lower test performance. Especially on
multiclass tasks (AG., ToI, Cat.), FewGen shows the highest label preservation
(exceeding Gold) but this does not translate into improved student
performance.

Figure 5:  Data maps from a DistilBERT training run on $8$K Category rows from
LLaMa2. FewGen (center) is skewed towards easy-to-learn examples (top-left)
while Gold (left) and SynthesizRR (right) have a higher density of ambiguous
examples.

To understand this, we conduct a deeper analysis of the student training
dynamics on multiclass datasets. We train a DistilBERT student for 6 epochs
and plot the corresponding data-maps Swayamdipta et al. (2020). As noted in
§4, label preservation can be exploited by only generating easy to classify
examples. From the Category data maps in Figure 5, it is clear that FewGen
generations tend to cluster around easy-to-learn examples (high confidence and
low variability). SynthesizRR contains more ambiguous examples (high
variability) which we hypothesize are essential to learning the nuances
between classes. To validate this, we filter out the least ambiguous examples
(bottom 17% variability) and retrain the students. In Table 8 we find that
FewGen performance degrades, whereas synthesizer improves (giving us new best
performances, despite using only 83% of rows). We conclude that SynthesizRR
generates more ambiguous examples, and this helps establish better class-
separability in multi-class data sets.

Method | AG. | Hyp. | ToI | Cat. | Hum. | Pol.
---|---|---|---|---|---|---
(Dataset size) | ($8$K) | ($4$K) | ($8$K) | ($8$K) | ($2$K) | ($4$K)
Gold | $93.81$ | $81.55$ | $85.23$ | $84.8$ | $95.5$ | $96.6$
LLaMa2 Few shot
FewGen* | 92.4 | $71.25$ | 85.9 | 88.1 | $71.7$ | $94.75$
SynthzRR† | $86.88$ | 78.6 | $74.29$ | $72.11$ | $90.65$ | $94.77$
SynthzRR‡ | $87.64$ | $75.5$ | $74.85$ | $74.46$ | 95.7 | 97.6
ClaudeV1 Few shot
FewGen* | 94.5 | $63.75$ | 87.4 | 89.4 | $85.85$ | $99.55$
SynthzRR† | $87.56$ | 72.8 | $74.79$ | $69.41$ | 90.7 | $99.32$
SynthzRR‡ | $87.36$ | $65.85$ | $73.24$ | $73.24$ | $77.35$ | 99.7
Table 7:  Few-shot label-preservation accuracy ($\uparrow$) using tuned oracle DeBERTa-v3L model. Gold row is accuracy on 20% validation split. Notation: *32-shot; †3-shot Retr-ICL; ‡32-shot No-Retr-ICL.  Method |  | AG. | ToI | Cat. |  Avg
---|---|---|---|---|---
(Dataset size) | ($6.6$K) | ($6.6$K) | ($6.6$K) |
LLaMa2 Few shot
FewGen* |  |  58.0 |  ${\downarrow}$26.2 |  37.6 |  ${\downarrow}$36.1 |  48.0 |  ${\downarrow}$20.6 |  ${\downarrow}$27.6
SynthzRR† |  |  85.7 |  ${\uparrow}$2.7 |  76.0 |  ${\uparrow}$2.7 |  74.3 |  ${\uparrow}$1.9 |  ${\uparrow}$2.4
SynthzRR‡ |  |  86.3 |  ${\uparrow}$1.1 |  75.0 |  ${\uparrow}$2.2 |  72.9 |  ${\uparrow}$1.0 |  ${\uparrow}$1.4
ClaudeV1 Few shot
FewGen* |  |  71.8 |  $\downarrow$4.1 |  72.1 |  $\downarrow$0.1 |  69.3 |  $\uparrow$0.5 |  ${\downarrow}$1.2
SynthzRR† |  |  86.2 |  $\uparrow$2.5 |  75.3 |  $\uparrow$2.5 |  69.0 |  $\uparrow$3.6 |  ${\uparrow}$2.9
SynthzRR‡ |  |  86.1 |  $\uparrow$2.4 |  74.6 |  $\uparrow$2.1 |  70.0 |  $\uparrow$2.2 |  ${\uparrow}$2.2
Table 8:  Test Accuracy $(\uparrow)$ after keeping 83% most-ambiguous
examples. We report improvements compared to Table 6. Notation: *32-shot;
†3-shot Retr-ICL; ‡32-shot No-Retr-ICL.

##  7 Comparison to additional baselines

Method |  Teacher / Retriever |  Data Size |  Yelp |  AG. |  IMDb |  SST-2
---|---|---|---|---|---|---
Gold |  - |  6k |  94.5 |  90.8 |  91.3 |  88.2
ZeroGen |  GPT2-XL |  6k |  $\otimes$ |  $\otimes$ |  $\bowtie$ |  82.4
(Reported) |  |  200k |  $\otimes$ |  $\otimes$ |  84.3 |  87.3
ProGen |  GPT2-XL |  6k |  $\bowtie$ |  $\bowtie$ |  $\bowtie$ |  $\bowtie$
(Reported) |  |  100k |  89.4 |  80.8 |  84.1 |  87.2
SunGen |  GPT2-XL |  6k |  $\bowtie$ |  $\bowtie$ |  84.9 |  $\bowtie$
(Reported) |  |  200k |  89.2 |  80.5 |  89.4 |  89.5
ReGen |  BERT |  6k |  $\bowtie$ |  82.7 |  $\bowtie$ |  $\bowtie$
(Reported) |  |  20k |  89.0 |  85.0 |  87.8 |  85.3
S3 |  GPT3.5-T |  6k |  $\otimes$ |  $\otimes$ |  87.1 |  $\otimes$
(Reported) |  |  21.2k |  $\otimes$ |  $\otimes$ |  89.0 |  $\otimes$
AttrPmt |  GPT3.5-T |  6k |  89.3 |  79.8 |  $\bowtie$ |  80.8
(Reported) |  |  6k |  90.6 |  82.7 |  85.9 |  83.6
Zero shot
SynthzRR |  LLaMa2 $+$ Contr. |  6k |  85.6 |  85.3 |  82.9 |  80.2
SynthzRR |  ClaudeV1 $+$ Contr. |  6k |  87.5 |  85.6 |  83.6 |  82.5
SynthzRR |  LLaMa2 $+$ BM25 |  6k |  83.2 |  84.3 |  74.1 |  84.4
SynthzRR |  ClaudeV1 $+$ BM25 |  6k |  86.7 |  84.2 |  79.1 |  82.6
3-shot Retr-ICL
SynthzRR |  LLaMa2 $+$ Contr. |  6k |  89.9 |  84.6 |  84.8 |  83.8
SynthzRR |  ClaudeV1 $+$ Contr. |  6k |  91.3 |  86.0 |  86.3 |  80.6
SynthzRR |  LLaMa2 $+$ BM25 |  6k |  90.7 |  84.3 |  84.7 |  84.4
SynthzRR |  ClaudeV1 $+$ BM25 |  6k |  89.4 |  84.1 |  81.6 |  82.3
Table 9:  Test accuracy of SynthesizRR Retr-ICL against baselines after fine-
tuning a DistilBERT student. For each released dataset, we subsample to 6k
examples to make a fair comparison. We train 5 student models and report the
mean accuracy (std. dev. was $\leq 2.0$ in all cases, thus we avoid reporting
it for brevity). We report the DistilBERT accuracy from the original paper (if
available) in italics. For each dataset, we bold the best result from the fair
evaluation. Tasks not evaluated by previous authors are denoted by

$\otimes$

, while those evaluated without dataset release are marked as

$\bowtie$

. GPT3.5-T is gpt-3.5-turbo.

Here, we explore how SynthesizRR directly compares to prior work on synthesis
of text classification datasets. We compare against six prior approaches:

ZeroGen Ye et al. (2022a): a simple approach which uses zero-shot generation
from GPT2-XL under different prompting schemes. For certain tasks like IMDb
and SST-2, the authors occasionally seed the prompt with a previously-
generated movie name (described subsequently).

ProGen Ye et al. (2022b): generates a small synthetic dataset, then uses a
student model along with an influence function to highlight high-quality
generations. These are used as in-context examples for another round of
generation.

SunGen Gao et al. (2023): uses the ZeroGen strategy to generate a large
synthetic dataset (200k rows). Then, uses a custom bi-level optimization
algorithm (involving the student model) to determine instance-weights of each
synthetic example.

ReGen Yu et al. (2023b): performs multi-round filtering of retrieved results
using 2 BERT models; one trained for retrieval, and one classifier. Use
consistency between these models to filter noisy data.

S3 Wang et al. (2023a): Constructs a “seed dataset” (different from ours) and
trains a student model. Then, extrapolate errors using an LLM and synthesizes
additional data. We combine this with the seed data and repeat the process.

AttrPrompt Yu et al. (2023a): a method focused on improving diversity and
unbiasedness of generated datasets. Prompts a powerful LLM like GPT3.5-Turbo
with different attributes, each along different dimensions. Attributes are
extracted from a human-in-the-loop analysis of task using GPT3.5-Turbo.

We note that zero shot generation and few-shot generation has already been
compared in Table 6, so we do not include it here.

We benchmark four classification tasks which are popular in prior work: Yelp
Polarity (Zhang et al., 2015), IMDb Maas et al. (2011), SST-2 Socher et al.
(2013) and AG News (Zhang et al., 2015). The first three tasks are binary
sentiment analysis, whereas the last is multi-class topic classification.

For the content sourcing stage of SynthesizRR, we retrieve documents from the
following corpora:

  * •

Yelp Polarity: to generate positive and negative reviews about businesses, we
retrieve from the Yelp Dataset corpus Yelp (2022) which contains details of
150k businesses. Each document contains the the name, address, and various
tags and boolean attributes for a single business.

  * •

IMDb and SST-2: to generate movie reviews, we retrieve from the CMU Movie
Summary corpus Bamman et al. (2013), which contains 42k plot summaries. Each
document is a plot summary for a single movie.

  * •

AG News: we use RealNews/Dominant from Table 2.

We note that the CMU Movie Summary and Yelp Dataset corpora are between one to
three orders of magnitude smaller than corpora in Table 2, thus it helps us
evaluate SynthesizRR with smaller corpora. We include two different retrieval
approaches: Contriever (semantic match) and BM25 (lexical match).

Following Yu et al. (2023a), we generate 6,000 rows using the SynthesizRR
Retr-ICL variant described in Algorithm 1. We generate an equal number of
examples for each class. We calculate intrinsic metrics Self-BLEU-5, MAUVE and
Entity Entropy as described in §4.

Prior work often generates much larger datasets (20k to 200k examples) and
uses different student model hyperparameters. Intrinsic evaluations of dataset
quality are also seldom reported. This makes it difficult to fairly compare
results from previous papers. Thus, we reproduce results ourselves by using
synthetic datasets released by the authors, after subsampling to 6,000
examples. We report distillation results in Table 9 and intrinsic metrics in
Table 10.

Method |  Teacher |  Retriever | Self-BLEU-5 $(\downarrow)$ | Entity Entropy $(\uparrow)$ | Mauve $(\uparrow)$
---|---|---|---|---|---
(Dataset) |  LM |  | Yelp | AG. | IMDb | SST-2 |  Yelp | AG. | IMDb | SST-2 | Yelp | AG. | IMDb | SST-2
Gold |  - |  - | 27.5 | 17.1 | 27.9 | 35.5 | 7.1 | 6.6 | 7.5 | 3.2 | - | - | - | -
ZeroGen |  GPT2-XL |  - |  $\otimes$ |  $\otimes$ |  $\bowtie$ | 15.3 |  $\otimes$ |  $\otimes$ |  $\bowtie$ | 4.9 |  $\otimes$ |  $\otimes$ |  $\bowtie$ | 52.0
ProGen |  GPT2-XL |  - |  $\bowtie$ |  $\bowtie$ |  $\bowtie$ |  $\bowtie$ |  $\bowtie$ |  $\bowtie$ |  $\bowtie$ |  $\bowtie$ |  $\bowtie$ |  $\bowtie$ |  $\bowtie$ |  $\bowtie$
SunGen |  GPT2-XL |  - |  $\bowtie$ |  $\bowtie$ | 15.4 |  $\bowtie$ |  $\bowtie$ |  $\bowtie$ | 4.9 |  $\bowtie$ |  $\bowtie$ |  $\bowtie$ | 68.7 |  $\bowtie$
ReGen |  - |  BERT |  $\bowtie$ | 56.5 |  $\bowtie$ |  $\bowtie$ |  $\bowtie$ | 8.1 |  $\bowtie$ |  $\bowtie$ |  $\bowtie$ | 68.1 |  $\bowtie$ |  $\bowtie$
S3 |  GPT3.5-T |  - |  $\otimes$ |  $\otimes$ | 62.2 |  $\otimes$ |  $\otimes$ |  $\otimes$ | 5.7 |  $\otimes$ |  $\otimes$ |  $\otimes$ | 62.0 |  $\otimes$
AttrPmt |  GPT3.5-T |  - | 70.3 | 39.8 |  $\bowtie$ | 71.5 | 3.4 | 6.0 |  $\bowtie$ | 3.4 | 55.9 | 52.8 |  $\bowtie$ | 50.0
Zero shot
SynthzRR |  LLaMa2 |  Contr. | 70.2 | 29.3 | 66.3 | 41.9 | 5.4 | 7.1 | 5.7 | 4.5 | 67.9 | 89.5 | 58.5 | 50.0
SynthzRR |  ClaudeV1 |  Contr. | 70.3 | 31.5 | 51.5 | 45.3 | 5.5 | 6.6 | 5.3 | 4.8 | 54.0 | 94.2 | 55.9 | 50.0
SynthzRR |  LLaMa2 |  BM25 | 70.6 | 28.7 | 62.2 | 36.5 | 4.7 | 7.0 | 5.6 | 5.1 | 64.7 | 90.3 | 60.5 | 50.0
SynthzRR |  ClaudeV1 |  BM25 | 72.9 | 30.9 | 50.4 | 36.9 | 4.5 | 6.5 | 5.1 | 5.4 | 54.6 | 90.8 | 53.2 | 50.0
3-shot Retr-ICL
SynthzRR |  LLaMa2 |  Contr. | 59.5 | 34.2 | 62.9 | 26.3 | 4.9 | 7.2 | 5.7 | 3.8 | 84.9 | 92.6 | 72.6 | 50.0
SynthzRR |  ClaudeV1 |  Contr. | 58.5 | 23.7 | 38.0 | 24.6 | 4.9 | 6.7 | 5.9 | 4.3 | 70.3 | 95.8 | 58.0 | 50.0
SynthzRR |  LLaMa2 |  BM25 | 66.6 | 32.0 | 59.7 | 25.3 | 5.1 | 7.2 | 5.6 | 4.8 | 81.9 | 92.5 | 78.7 | 50.0
SynthzRR |  ClaudeV1 |  BM25 | 59.9 | 24.6 | 41.9 | 26.8 | 5.0 | 6.7 | 5.4 | 4.9 | 63.3 | 96.0 | 58.5 | 50.0
Table 10:  Intrinsic evaluations of synthetic datasets released by prior work.
We subsample all to 6k examples before computing metrics as described in §4.
Within each dataset, we bold the best result. Tasks not evaluated by previous
authors are denoted by

$\otimes$

, while those evaluated without dataset release are marked as

$\bowtie$

. GPT3.5-T is gpt-3.5-turbo.

RQ: How does SynthesizRR compare to existing approaches in terms of distilled
student accuracy?

We measure accuracy of a DistilBERT Sanh et al. (2019) student model.
DistilBERT is a standard model used in prior dataset synthesis work Yu et al.
(2023a); Ye et al. (2022a); Gao et al. (2023); Wang et al. (2023a); Ye et al.
(2022b). We set the same model hyperparams as Yu et al. (2023a), i.e. training
for 5 epochs using using Adam optimizer with batch_size=32 , lr=2e-5 ,
weight_decay=1e-4 and epsilon=1e-6 . We do not use cartography-based filtering
for SynthesizRR to make a fair comparison. In addition to our results on 6k
examples, we report original numbers from prior work (which might be on much
larger datasets).

The results in Table 9 show that methods which rely on relatively weak LLM
teachers like GPT2-XL Radford et al. (2019) can perform well on sentiment and
topic tasks like IMDb, but require a very high data cost (15-30x more
synthetic data than SynthesizRR). When scaled down to 6k rows, the performance
of methods like ZeroGen and SunGen deteriorates.

Approaches which use strong instruction-following LLMs like AttrPrompt, S3,
and SynthesizRR can achieve similar or better performance with much smaller
datasets. These methods create high-quality datasets rather than modify the
student modeling process, as is done by SunGen and ProGen. SunGen performs an
iterative bi-level optimization over the ZeroGen datasets, jointly learning
instance-weights and improving the student. ProGen uses an iterative filtering
approach by training a student model and filtering partial generations based
on a noise-robust, to curate an in-context dataset. We hypothesize these
additions of the student model into the synthesis process also impact the
final classification accuracy, as the dataset becomes specialized to the
particular choice of student and its hyperparams. As we use a standard
student-distillation setup here, datasets from these approaches may not
perform as well.

More complex prompting techniques like Chain-of-Thought (Wei et al., 2022)
used by S3 can indeed improve the task-inversion step, though this requires
much higher API costs due to longer output lengths. Chain-of-Thought prompting
thus seems like a promising approach by which SynthesizRR’s task-inversion
step could be augmented.

RQ: do we find evidence that content sourcing promotes diversity and
similarity?

In Table 10, we measure the entity diversity (Entity Entropy), lexical
diversity (Self-BLEU), and similarity to Gold texts (MAUVE) compared to prior
approaches. Among prior approaches, only AttrPrompt (Yu et al., 2023a,
Appendix E) attempts to improve diversity of the generated text, by
templatizing the task inversion instruction with attributes such as style,
topic, length:min-words and more. ReGen is the only synthesis approach which
uses content sourcing (but not task inversion). We thus consider these the two
most relevant baselines for SynthesizRR.

We see that both ReGen and SynthesizRR achieve very high entity entropy
compared to AttrPrompt, thus underscoring the importance of a content sourcing
step. Unlike SynthesizRR, ReGen uses only retrieval, thus it has no explicit
task-inversion step to make the contexts similar to Gold texts. Thus, we
observe that ReGen suffers in terms of lexical diversity, Mauve and student
accuracy, compared to SynthesizRR.

On the other hand, Chain-of-Thought prompting (S3), despite generating a
strong classification dataset as we saw in Table 9, suffers from the lack of
lexical diversity and similarity to Gold texts. This is also seen in
AttrPrompt and previously in FewGen. This lends evidence to the hypothesis
that synthesis without content sourcing tends to produce datasets with lower
diversity, which cannot be overcome by complex prompting strategies alone.

Finally, we observe that ZeroGen shows very high diversity on SST-2, a task
which involves generating partial movie reviews having positive or negative
sentiment. However, as mentioned in Ye et al. (2022a, Section 4.6), ZeroGen is
not simply zero-shot generation; the authors first generate a movie using the
prompt Movie: and then insert the generated movie name into the prompt
template The movie review in positive sentiment for movie "<Movie>" is: to
generate an SST-2 example. We posit that the generated movie fulfils the same
purpose as a retrieved context in SynthesizRR. Similarly high diversity is
observed by SunGen which starts with ZeroGen-generated dataset and learns
instance-weights (Gao et al., 2023, Section 2.2). Such synthetic content
sourcing could be a fruitful direction for future work, to overcome the need
for a retrieval corpus in SynthesizRR.

##  8 Related Work

Dataset synthesis using LLMs. Using LLMs to perform task inversion for dataset
synthesis has been studied previously. Most use GPT-2XL without fine-tuning
(Ye et al., 2022b, a; Gao et al., 2023; Meng et al., 2022; Schick and Schütze,
2021; Jung et al., 2023). Only recent works have considering large teacher
LLMs such as GPT-3 (West et al., 2022; Honovich et al., 2023; Wang et al.,
2023b) and chat-tuned LLMs such as gpt-3.5-turbo (Yu et al., 2023a; Yehudai et
al., 2024b; Wang et al., 2023a).

For the generation of text classification datasets, class-conditioned
prompting is key. Prior approaches have investigated zero-shot (Ye et al.,
2022a), iterative few-shot prompting (Ye et al., 2022b), or synthesis using
seq2seq LLMs fine-tuned on a curated dataset (Lee et al., 2021), but only
recently has the importance of varying the prompt been established. Yu et al.
(2023a) use prompts which explicitly attribute variations (such as subtopics
and brands) and find this obtains greater diversity. We explore adding
retrieval contexts to the prompt as the source of diversity and knowledge-
grounding, and find it can significantly improve representation of tail
entities. This avoids issues related to memorization of parametric knowledge
(Roberts et al., 2020; Kandpal et al., 2023). We note that distillation for
generation tasks like paraphrasing and summarization (Jung et al., 2023) has
been demonstrated using off-the-shelf critic models; instead, our work focuses
on classification and does not need a filtering step to be effective (though,
as we show in §6, it does benefit from such steps in the multi-class case).

Retrieval-augmented generation. Our approach has many of the characteristics
of in-context retrieval-augmented generation (RAG) (Lewis et al., 2020; Ram et
al., 2023; Huang et al., 2023; Izacard et al., 2023). Previous studies show
how RAG bypasses numerous problems associated with generating solely from
parametric memory, i.e. heightened bias towards “head” entities (Mallen et
al., 2023), lower lexical diversity (Holtzman et al., 2019; Jentzsch and
Kersting, 2023), and hallucinated and confabulated information (Zhang et al.,
2023). To our best knowledge, using retrieval-augmented generation for
synthesis of classification tasks has not been explored at the instance level.
Yu et al. (2023b) studied the retrieval-only setting for creation of topic and
sentiment datasets; as described in §section 2 we consider tasks such as
Hyperpartisan, Humor, and ToI Headlines, which pose additional challenges.
Viswanathan et al. (2023) perform dataset-level retrieval and not instance-
level retrieval, additionally selecting the student model based on the task
whereas we focus on a fixed student. Finally, note that our application looks
very different from other RAG applications like question answering (Asai et
al., 2023) or counterfactual data augmentation (Paranjape et al., 2022); we
generate diverse outputs only loosely connected to the input query.

Diversity in dataset synthesis. An important distinction between RAG and
SynthesizRR is the desire for diverse generations. RAG has so far been used
primarily for knowledge-based tasks such as open-domain question-answering,
fact-checking and few-shot classification (Asai et al., 2023). These tasks
face the issue of multiple valid surface-forms, which only change the wording
of the generated output but not its meaning, making evaluation difficult
(Holtzman et al., 2021). Greedy sampling is used most often.

On the other hand, dataset synthesis is emboldened by diverse surface-forms,
as this produces diverse covariates linked to the same target, which
intuitively aids generalization; we thus use top-p sampling (Holtzman et al.,
2019). Another major difference from RAG is that we retain the long-tail of
retrieved results, as this allows us to create a diverse grounded dataset.

##  9 Conclusion

In this work we describe how a retrieval corpus can be used to aid the
synthesis of a text classification data set in specialized domains. We show
that the diversity of the generated data is enhanced by including retrieved
documents in a generation prompt. Compared to few-shot generation, we find
that SynthesizRR produces more diverse and representative text and leads to
better students.

## Limitations

Most principally, our work relies on the existence of a large corpus that is
close enough to the task at hand. This may be prohibitive for doing dataset
generation in low-resource languages, where a large corpus of related content
may not be available. It would be intriguing to explore cross-lingual transfer
of content sourcing, but this would require additional experimental
validation. By contrast, approaches like FewGen do not require this corpus.

The need for an explicit context sourcing step and increased prompt-length
causes an increase in the expenses and latency, especially when using LLM
APIs. Such increased expense may not be worth it in the presence of a poor
quality retrieval corpus. For one, if the in-context examples are not easily
reusable as queries, then SynthesizRR can retrieve irrelevant documents which
might not be suitable for task inversion. Furthermore, in the case of
factually dubious corpus documents, the student model may end up grounding in
factually incorrect information. This can be mitigated by a human-in-the-loop
step to remove such documents before task inversion.

Finally, we note that the scope of our experiments is restricted to a set of
classification tasks over a few English domains of text. While we believe our
approach can be applied to other languages, other domains, and tasks like
question answering that go beyond classification, we have not validated this
in this work.

## References

  * Asai et al. (2023) Akari Asai, Sewon Min, Zexuan Zhong, and Danqi Chen. 2023.  Retrieval-based language models and applications.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 6: Tutorial Abstracts)_ , pages 41–46, Toronto, Canada. Association for Computational Linguistics.
  * Bai et al. (2022) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, T. J. Henighan, Nicholas Joseph, Saurav Kadavath, John Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Christopher Olah, Benjamin Mann, and Jared Kaplan. 2022.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  _ArXiv_ , abs/2204.05862.
  * Bamman et al. (2013) David Bamman, Brendan O’Connor, and Noah A. Smith. 2013.  Learning latent personas of film characters.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 352–361, Sofia, Bulgaria. Association for Computational Linguistics.
  * Blitzer et al. (2007) John Blitzer, Mark Dredze, and Fernando Pereira. 2007.  Biographies, Bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification.  In _Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics_ , pages 440–447, Prague, Czech Republic. Association for Computational Linguistics.
  * Borisov et al. (2022) Vadim Borisov, Kathrin Sessler, Tobias Leemann, Martin Pawelczyk, and Gjergji Kasneci. 2022.  Language models are realistic tabular data generators.  In _The Eleventh International Conference on Learning Representations_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023.  Sparks of Artificial General Intelligence: Early experiments with GPT-4.  _arXiv e-prints_ , pages arXiv–2303.
  * Chang et al. (2024) Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2024.  A survey on evaluation of large language models.  _ACM Trans. Intell. Syst. Technol._
  * Chen et al. (2017) Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017.  Reading Wikipedia to answer open-domain questions.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1870–1879, Vancouver, Canada. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Gao et al. (2023) Jiahui Gao, Renjie Pi, LIN Yong, Hang Xu, Jiacheng Ye, Zhiyong Wu, Weizhong Zhang, Xiaodan Liang, Zhenguo Li, and Lingpeng Kong. 2023.  Self-guided noise-free data generation for efficient zero-shot learning.  In _The Eleventh International Conference on Learning Representations_.
  * Geigle et al. (2021) Gregor Geigle, Nils Reimers, Andreas Rücklé, and Iryna Gurevych. 2021.  TWEAC: Transformer with Extendable QA Agent Classifiers.  _arXiv preprint_ , abs/2104.07081.
  * Goyal et al. (2022) Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022.  News Summarization and Evaluation in the Era of GPT-3.  _arXiv preprint_.
  * He et al. (2021) Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021.  DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing.
  * Holtzman et al. (2019) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019.  The curious case of neural text degeneration.  In _International Conference on Learning Representations_.
  * Holtzman et al. (2021) Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. 2021.  Surface form competition: Why the highest probability answer isn’t always right.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7038–7051, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Honnibal et al. (2020) Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020.  spaCy: Industrial-strength Natural Language Processing in Python.
  * Honovich et al. (2023) Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2023.  Unnatural instructions: Tuning language models with (almost) no human labor.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 14409–14428, Toronto, Canada. Association for Computational Linguistics.
  * Huang et al. (2023) Jie Huang, Wei Ping, Peng Xu, Mohammad Shoeybi, Kevin Chen-Chuan Chang, and Bryan Catanzaro. 2023.  Raven: In-context learning with retrieval augmented encoder-decoder language models.  _ArXiv_ , abs/2308.07922.
  * Izacard et al. (2022) Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022.  Unsupervised dense information retrieval with contrastive learning.  _Transactions on Machine Learning Research_.
  * Izacard et al. (2023) Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2023.  Atlas: Few-shot learning with retrieval augmented language models.  _Journal of Machine Learning Research_ , 24(251):1–43.
  * Jentzsch and Kersting (2023) Sophie Jentzsch and Kristian Kersting. 2023.  ChatGPT is fun, but it is not funny! humor is still challenging large language models.  In _Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis_, pages 325–340, Toronto, Canada. Association for Computational Linguistics.
  * Jung et al. (2023) Jaehun Jung, Peter West, Liwei Jiang, Faeze Brahman, Ximing Lu, Jillian Fisher, Taylor Sorensen, and Yejin Choi. 2023.  Impossible distillation: from low-quality model to high-quality dataset & model for summarization and paraphrasing.  _arXiv preprint arXiv:2305.16635_.
  * Kandpal et al. (2023) Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023.  Large language models struggle to learn long-tail knowledge.  In _Proceedings of the 40th International Conference on Machine Learning_ , ICML’23. JMLR.org.
  * Kiesel et al. (2019) Johannes Kiesel, Maria Mestre, Rishabh Shukla, Emmanuel Vincent, Payam Adineh, David Corney, Benno Stein, and Martin Potthast. 2019.  SemEval-2019 task 4: Hyperpartisan news detection.  In _Proceedings of the 13th International Workshop on Semantic Evaluation_ , pages 829–839, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
  * Lee et al. (2023) Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. 2023.  RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback.
  * Lee et al. (2019) Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019.  Latent retrieval for weakly supervised open domain question answering.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6086–6096, Florence, Italy. Association for Computational Linguistics.
  * Lee et al. (2021) Kenton Lee, Kelvin Guu, Luheng He, Timothy Dozat, and Hyung Won Chung. 2021.  Neural data augmentation via example extrapolation.  _ArXiv_ , abs/2102.01335.
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020.  Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.  In _Advances in Neural Information Processing Systems_.
  * Liu et al. (2021) Lang Liu, Krishna Pillutla, Sean Welleck, Sewoong Oh, Yejin Choi, and Zaid Harchaoui. 2021.  Divergence Frontiers for Generative Models: Sample Complexity, Quantization Effects, and Frontier Integrals.  In _Advances in Neural Information Processing Systems_.
  * Liu et al. (2023) Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023.  Lost in the middle: How language models use long contexts.  ArXiv:2307.03172.
  * Maas et al. (2011) Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011.  Learning word vectors for sentiment analysis.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 142–150, Portland, Oregon, USA. Association for Computational Linguistics.
  * Mallen et al. (2023) Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023.  When not to trust language models: Investigating effectiveness of parametric and non-parametric memories.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 9802–9822, Toronto, Canada. Association for Computational Linguistics.
  * Meng et al. (2022) Yu Meng, Jiaxin Huang, Yu Zhang, and Jiawei Han. 2022.  Generating training data with language models: Towards zero-shot language understanding.  _Advances in Neural Information Processing Systems_ , 35:462–477.
  * Meng et al. (2019) Yu Meng, Jiaming Shen, Chao Zhang, and Jiawei Han. 2019.  Weakly-supervised hierarchical text classification.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 33(01):6826–6833.
  * Mialon et al. (2023) Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ramakanth Pasunuru, Roberta Raileanu, Baptiste Roziere, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, and Thomas Scialom. 2023.  Augmented language models: a survey.  _Transactions on Machine Learning Research_.  Survey Certification.
  * Min et al. (2022) Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022.  Rethinking the role of demonstrations: What makes in-context learning work? In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 11048–11064, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Ni et al. (2019) Jianmo Ni, Jiacheng Li, and Julian McAuley. 2019.  Justifying recommendations using distantly-labeled reviews and fine-grained aspects.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 188–197, Hong Kong, China. Association for Computational Linguistics.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 Technical Report.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.  In _Advances in Neural Information Processing Systems_.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  BLEU: A method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting on Association for Computational Linguistics_ , ACL ’02, page 311–318, USA. Association for Computational Linguistics.
  * Paranjape et al. (2022) Bhargavi Paranjape, Matthew Lamm, and Ian Tenney. 2022.  Retrieval-guided counterfactual generation for QA.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1670–1686, Dublin, Ireland. Association for Computational Linguistics.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.
  * Ram et al. (2023) Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023.  In-context retrieval-augmented language models.  _Transactions of the Association for Computational Linguistics_ , 11:1316–1331.
  * Roberts et al. (2020) Adam Roberts, Colin Raffel, and Noam Shazeer. 2020.  How much knowledge can you pack into the parameters of a language model? In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5418–5426, Online. Association for Computational Linguistics.
  * Robertson and Zaragoza (2009) Stephen Robertson and Hugo Zaragoza. 2009.  The probabilistic relevance framework: Bm25 and beyond.  _Found. Trends Inf. Retr._ , 3(4):333–389.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.  In _5th Workshop on Energy Efficient Machine Learning and Cognitive Computing @ NeurIPS 2019_.
  * Schick and Schütze (2021) Timo Schick and Hinrich Schütze. 2021.  Generating datasets with pretrained language models.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6943–6951, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics.
  * Swayamdipta et al. (2020) Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. 2020.  Dataset cartography: Mapping and diagnosing datasets with training dynamics.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9275–9293, Online. Association for Computational Linguistics.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv e-prints_ , pages arXiv–2307.
  * van de Kar et al. (2022) Mozes van de Kar, Mengzhou Xia, Danqi Chen, and Mikel Artetxe. 2022.  Don’t prompt, search! mining-based zero-shot learning with language models.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 7508–7520, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Viswanathan et al. (2023) Vijay Viswanathan, Chenyang Zhao, Amanda Bertsch, Tongshuang Wu, and Graham Neubig. 2023.  Prompt2Model: Generating deployable models from natural language instructions.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 413–421, Singapore. Association for Computational Linguistics.
  * Wang et al. (2023a) Ruida Wang, Wangchunshu Zhou, and Mrinmaya Sachan. 2023a.  Let’s synthesize step by step: Iterative dataset synthesis with large language models by extrapolating errors from small models.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 11817–11831, Singapore. Association for Computational Linguistics.
  * Wang et al. (2023b) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023b.  Self-instruct: Aligning language models with self-generated instructions.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 13484–13508, Toronto, Canada. Association for Computational Linguistics.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022.  Chain of thought prompting elicits reasoning in large language models.  In _Advances in Neural Information Processing Systems_.
  * West et al. (2022) Peter West, Chandra Bhagavatula, Jack Hessel, Jena Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. 2022.  Symbolic knowledge distillation: from general language models to commonsense models.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4602–4625, Seattle, United States. Association for Computational Linguistics.
  * Ye et al. (2022a) Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong. 2022a.  Zerogen: Efficient zero-shot learning via dataset generation.  _ArXiv_ , abs/2202.07922.
  * Ye et al. (2022b) Jiacheng Ye, Jiahui Gao, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. 2022b.  ProGen: Progressive zero-shot dataset generation via in-context feedback.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 3671–3683, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Yehudai et al. (2024a) Asaf Yehudai, Boaz Carmeli, Yosi Mass, Ofir Arviv, Nathaniel Mills, Assaf Toledo, Eyal Shnarch, and Leshem Choshen. 2024a.  Achieving human parity in content-grounded datasets generation.  In _International Conference on Learning Representations_.
  * Yehudai et al. (2024b) Asaf Yehudai, Boaz Carmeli, Yosi Mass, Ofir Arviv, Nathaniel Mills, Assaf Toledo, Eyal Shnarch, and Leshem Choshen. 2024b.  Genie: Achieving human parity in content-grounded datasets generation.  _ArXiv_ , abs/2401.14367.
  * Yelp (2022) Yelp. 2022.  Yelp Dataset - A trove of reviews, businesses, users, tips, and check-in data!
  * Yu et al. (2023a) Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang. 2023a.  Large language model as attributed training data generator: A tale of diversity and bias.  In _Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track_.
  * Yu et al. (2023b) Yue Yu, Yuchen Zhuang, Rongzhi Zhang, Yu Meng, Jiaming Shen, and Chao Zhang. 2023b.  ReGen: Zero-shot text classification via training data generation with progressive dense retrieval.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 11782–11805, Toronto, Canada. Association for Computational Linguistics.
  * Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019.  Defending against neural fake news.  In _Advances in Neural Information Processing Systems 32_.
  * Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015.  Character-level convolutional networks for text classification.  In _Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1_ , NIPS’15, page 649–657, Cambridge, MA, USA. MIT Press.
  * Zhang et al. (2023) Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. 2023.  Siren’s song in the ai ocean: A survey on hallucination in large language models.  _arXiv preprint arXiv:2309.01219_.
  * Zhu et al. (2018) Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018.  Texygen: A benchmarking platform for text generation models.  _SIGIR_.
  * Ziser et al. (2020) Yftah Ziser, Elad Kravi, and David Carmel. 2020.  Humor detection in product question answering systems.  In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’20, page 519–528, New York, NY, USA. Association for Computing Machinery.

##  Appendix A Risks

Although the main goal of our work is to improve text classification, our use
of LLMs to generate examples does carry some conceptual risks. By generating
news articles to train classifiers on, we run the risk of generating fake news
and other harmful content. However, we believe this risk is mitigated by the
fact that the final outcome of our system is a classifier: classification
models have relatively constrained failure modes (misclassification) compared
to text generation models that can mislead users. Furthermore, we do not
believe our approach uniquely advances the generation of fake news; our
advances are largely orthogonal to the technology that brings such risks.

##  Appendix B Dense vs sparse retrieval in SynthesizRR

Retriever |  AG. |  Hyp. |  ToI |  Cat. |  Hum. |  Pol. |  Avg.
---|---|---|---|---|---|---|---
(Size) |  ($8$K) |  ($4$K) |  ($8$K) |  ($8$K) |  ($2$K) |  ($4$K) |
Gold |  91.0 |  93.2 |  82.5 |  81.5 |  93.1 |  95.3 |  89.43
LLaMa2 Zero shot
Contr. |  83.5 |  69.8 |  74.5 |  68.9 |  82.5 |  84.7 |  77.32
BM25 |  83.2 |  74.2 |  70.7 |  57.6 |  78.5 |  85.4 |  74.93
ClaudeV1 Zero shot
Contr. |  83.9 |  72.3 |  71.8 |  66.8 |  62.1 |  88.7 |  74.29
BM25 |  83.2 |  57.2 |  69.8 |  53.7 |  73.9 |  91.8 |  71.60
LLaMa2 3-shot Retr-ICL
Contr. |  83.0 |  78.5 |  73.3 |  72.4 |  90.2 |  91.0 |  81.38
BM25 |  82.1 |  77.9 |  71.9 |  65.4 |  87.5 |  87.4 |  78.69
ClaudeV1 3-shot Retr-ICL
Contr. |  83.7 |  72.3 |  72.8 |  65.4 |  83.4 |  91.3 |  78.16
BM25 |  83.0 |  73.5 |  70.0 |  52.4 |  82.4 |  90.7 |  75.34
Table 11:  Test accuracy after distilling a DeBERTa-v3L student on a dataset
generated by SynthesizRR comparing BM25 and Contriever. We use the same seed
set and corpus as Table 2. We train 5 student models and report the mean
accuracy, rerunning all 5 in case of std $\geq$ 6.0. The top two subsections
consider zero-shot synthesis and bottom two considers 3-shot Retr-ICL variant.
We bold the best result in each subsection. Contriever numbers are reproduced
from Table 6.

So far, a single dense retriever (Contriever) has been used for the content
sourcing step by using a bi-encoder approach (Lee et al., 2019; Chen et al.,
2017). We embed both the input in-context covariate and each corpus document,
and then rank results based on cosine similarity. In §5, we retrieved $k=500$
documents for each in-context example and after filtering, randomly sampled
among these to produce a grounded set of documents on which we apply our task
inversion strategy Retr-ICL.

In this section we explore how changing the retrieval model affects the
content sourcing stage and its downstream effects. Table 11 compares the
DeBERTa-v3L accuracy of Contriever and BM25. Keeping other parts of the
process the same, we switch Contriever to BM25 Okapi (Robertson and Zaragoza,
2009), a popular sparse retrieval method which performs a lexical match based
on inverse term frequencies. Unlike dense retrieval models, BM25 outputs a
score which is an unbounded positive number. Thus we are unable to use
meaningful thresholds to bound the similarity in our Retr-ICL approach.
Instead, we construct the Retr-ICL in-context set using the top-2 retrieved
contexts for each ICL example and without applying the filter.

We observe in Table 11 that a powerful student model trained on the dense-
retrieved document set consistently outperforms the sparse-retrieved documents
from BM25. In Table 10, it is also observed that the intrinsic diversity from
BM25-retrieved documents is also not as good as that from Contriever.

Dense retrievers like Contriever perform a semantic match between the query
and document, whereas BM25 performs only a lexical match with no understanding
of semantics. Picking semantically similar information is more important in
SynthesizRR since we include a task inversion step, which can change the
lexical structure of the text without changing its semantics significantly.
Thus, we want contexts which are semantically related to Gold data, to which
we can apply stylistic or formatting transformations using a task-inversion
prompt to bring it closer to Gold. We might conclude that dense retrieval
models are more suitable for SynthesizRR.

##  Appendix C Influence of retrieval corpus on domain shift

AG News($4$K)
---
Corpus |  |  DeBERTa ($\uparrow$) |  |  Mauve ($\uparrow$) |  |  Self-BLEU-5 ($\downarrow$) |  |  Entity Ent. ($\uparrow$)
RN/Dom |  |  85.39 ± 0.8 |  |  92.58 |  |  0.23 |  |  6.72
RN/Rnd |  |  35.57 ± 6.1 |  |  83.39 |  |  0.22 |  |  7.07
RN/Reg |  |  84.17 ± 0.7 |  |  88.88 |  |  0.26 |  |  6.72
Hyperpartisan($2$K)
Corpus |  |  DeBERTa ($\uparrow$) |  |  Mauve ($\uparrow$) |  |  Self-BLEU-5 ($\downarrow$) |  |  Entity Ent. ($\uparrow$)
RN/Dom |  |  78.77 ± 2.8 |  |  66.94 |  |  0.35 |  |  6.11
RN/Rnd |  |  78.77 ± 3.5 |  |  61.45 |  |  0.25 |  |  7.40
RN/Reg |  |  72.00 ± 2.0 |  |  65.59 |  |  0.35 |  |  6.12
Table 12:  Effect of corpus-swapping for SynthesizRR 32-shot No-Retr-ICL. We
generate only $4$k rows for AG News to reduce costs.

Our expectation is that SynthesizRR can flexibly specialize students to
different domains by transparently changing the retrieval corpus, while
keeping a frozen LLM. To quantify how changing the retrieval corpus might
affect earlier metrics, we switch the news corpus for Hyperpartisan and AG
News. We had assumed RealNews/Dom was the most suitable corpus (in-domain),
and the others will cause domain-shift. In the following RQs, we validate the
degree to which this assumption holds and the importance of information
retrieval as the content sourcing mechanism in SynthesizRR.

RQ: Does modifying the corpus cause domain shift? Table 12 finds that the
retrieval corpus highly influences the test performance (both student and
intrinsic metrics). When grounding to a corpus with highly dissimilar entities
(such as RealNews/Reg), all metrics drop significantly. Thus, we can conclude
that an alternative content-source does indeed induce domain-shift. Mauve and
distillation accuracy are highest for the in-domain corpus, while Self-BLEU
and Entity entropy are highest for the random-retrieval results.

Figure 6:  Retrieval counts for Hyperpartisan and AG News. The red dashed line
represents the theoretical max, where all retrieved documents are unique. Note
that the Random histogram plot is always 1 hence shows up as a straight line.

RQ: is retrieval essential for content sourcing? We measure the importance of
retrieval by selecting top-k documents randomly from the in-domain corpus
RealNews/Dom. We observe in Table 12 that retrieval using in-context learning
queries plays a crucial role to the performance of AG News, as performance
drops significantly in a random setting. Hyperpartisandoes not face such a
drop. This matches our intuition in Table 1 that task-inversion is the more
challenging step for Hyperpartisan, and a powerful LLM we can apply stylistic
changes to most news articles. In both, Mauve suffers when entities no longer
match Gold.

RQ: Do in-context queries retrieve redundant results? Figure 6 measures the
overlap of top-50 retrieved documents from the 200 ICL queries, and finds that
in most cases, the retrieved documents are unique, especially when using a
large in-domain corpus. Thus, we can conclude that effective retrieval is
important for the diversity of the synthetic dataset.

RQ: Can SynthesizRR work effectively with relatively small corpora? In our
main results §section 5, we assumed the existence of a large corpus, with
minimum size of 0.9M documents. As noted, this corpus need not be unlabelled
examples for our task; we were able to successfully generate customer reviews
and product questions for Humor, Categoryand Polarity tasks, while retrieving
from a corpus of product information (title and description).

A potential problem with SynthesizRR is that corpuses of such massive size
might be few in number. Thus, we compare the performance of SynthesizRR on CMU
Movie Summary Bamman et al. (2013) and Yelp Dataset corpus Yelp (2022), which
are between one to three orders of magnitude smaller than corpora used in
Table 6. In Table 10 and Table 9, we see that SynthesizRR can perform suitably
even with relatively small corpora (42k movie plots and 150k businesses). From
the previous RQs, this suggests that the relevance of the corpus to the task
is more important than the size of the corpus for the performance of
SynthesizRR.

##  Appendix D Ablation: number of in-context examples

Figure 7:  Left: DeBERTa-v3L test accuracy $(\uparrow)$, center: entity
entropy $(\uparrow)$, right: Mauve $(\uparrow)$ for SynthesizRR Retr-ICL. We
vary the number of in-context examples from 0 to 8. Teacher LLMs LLaMa-2 Chat
13B and Claude Instant-v1 are compared on 6 tasks: AG News, Hyperpartisan, ToI
Headlines, Category, Humor and Polarity. We do not report Category 8-shot due
to model failures.

Figure 8:  Lexical diversity i.e. Self-BLEU $(\downarrow)$ ngrams n=1-5, when
varying the number of in-context examples for SynthesizRR Retr-ICL. We compare
of teacher LLMs LLaMa-2 Chat 13B (left) and Claude Instant-v1 (right).
Notation: 0-shot ($\bullet$), 1-shot (+), 3-shot ($\vartriangle$), 8-shot
($\bigstar$). Darker shade implies more in-context examples.

The use of in-context examples in the Retr-ICL variant of SynthesizRR leads to
significant improvements in intrinsic and distillation metrics, as we saw in
§5. Here, we do a deeper analysis on whether continually increasing the number
of in-context examples yields a positive benefit.

In Figure 7 we look at the DeBERTa-v3L accuracy, entity entropy and MAUVE for
our datasets with different numbers of in-context learning examples. We see
that adding even a single in-context example can greatly increase the
performance of all three metrics. However, no particular number of in-context
examples consistently outperforms. For ClaudeV1, adding more in-context
examples (up to 8) seems to always provide benefit, whereas with LLaMa2, we
observe a peak and then reduction. Thus, the optimal number of in-context
learning examples is a task dependent hyperparameter.

Figure 8 shows the lexical diversity i.e. Self-BLEU across datasets and number
of in-context examples. As in §5 we observed that using in-context examples is
neither positively nor negatively correlated with a lower Self-BLEU, despite
using nucleus sampling with $p=0.9$. This may be because for all number of
shots, task inversion is performed from a single source context and thus the
generation does not divert significantly from the unique n-grams of the
context. Thus we conclude that to affect lexical diversity, the number of in-
context learning examples has no effect and we must instead focus on changing
the retrieved contexts, perhaps by using a different retrieval model.

##  Appendix E Task inversion prompts and label verbalizations

Here we discuss the prompt templates and verbalizations that we use for the
task inversion step for both FewGen and SynthesizRR. We use descriptive
verbalizations as compared to the target label.

Additionally in the prompt, we place the retrieved document near the end, as
prior work indicates that intermediate placements degrade LLM recall (Liu et
al., 2023).

LLMs have a fixed window-size for conditional generation, so excessively long
documents are truncated (from the end) up to $r_{max}=500$ tokens. This
reserves the remaining window for in-context learning.

###  E.1 Hyperpartisan

Hyperpartisan is the task of detecting political bias in a news article. In
transforming the retrieved news article article_retr[k] to one with such bias,
typically there is the addition of mocking commentary and harsh political
language which deeply criticizes the subject such as a person, policy or
political event. On the other hand, articles in the opposite class gives a
well-rounded opinion with a neutral tone. We include a length-attribute to
ensure a long generation of one or two paragraphs.

Label | Verbalization
---|---
true |  harsh political language, using a mocking tone and toxic commentary
false |  neutral language, using a reasonable tone and politically correct commentary
Table 13:  Task-inversion verbalizations for Hyperpartisan.

Prompt E.1: Hyperpartisan FewGen In-context example:
“ Write a single news article using {label} . The written article should be 2
to 3 paragraphs long.
News Article: {icl[gold_text]} ”
Prompt:
“ Write a single news article using {label} . The written article should be 2
to 3 paragraphs long.
News Article: ”

Prompt E.2: Hyperpartisan SynthesizRR Retr-ICL In-context example:
“ News Article: {icl[article_retr]}
Rewrite the above news article using {label} . The rewritten article should be
2 to 3 paragraphs long.
Rewritten Article: {icl[gold_text]} ”
Prompt:
“ News Article: {article_retr[k]}
Rewrite the above news article using {label} . The rewritten article should be
2 to 3 paragraphs long.
Rewritten Article: ”

Prompt E.3: Hyperpartisan SynthesizRR No-Retr-ICL In-context example:
“ Rewritten Article: {icl[gold_text]} ”
Prompt:
“ News Article: {article_retr[k]}
Rewrite the above news article using {label} . The rewritten article should be
2 to 3 paragraphs long.
Rewritten Article: ”

###  E.2 ToI Headlines

ToI Headlines is a topic classification dataset of regional news headlines in
India. Here we attempt to refine the retrieved news article by summarizing it
into a short headline. We use verbalizations of the content of each class, as
example generation here involves summarizing the content. We add an “India”
location-attribute to guide the LLM generations to include regionalization to
the Indian subcontinent. A length-attribute is included to restrict the length
to one sentence.

Label | Verbalization
---|---
sports |  sports in India
life-style |  health and lifestyle trends in India
education |  Indian examinations and education
entertainment |  the Indian entertainment industry
business |  business-related developments in India
city |  ongoing matters in any Indian city
environment |  environment-related events in Indian cities
tech |  technology news and the tech industry in India
elections |  elections and politics in India
world |  international news and events outside of India
Table 14:  Task-inversion verbalizations for ToI Headlines.

Prompt E.4: ToI Headlines FewGen In-context example:
“ Write a headline for a news article about {label} . The headline should be a
single sentence.
Headline: {icl[gold_text]} ”
Prompt:
“ Write a headline for a news article about {label} . The headline should be a
single sentence.
Headline: ”

Prompt E.5: ToI Headlines SynthesizRR Retr-ICL In-context example:
“ News Article: {icl[article_retr]}
Write a headline for the above news article about {label} . The headline
should be a single sentence.
Headline: {icl[gold_text]} ”
Prompt:
“ News Article: {article_retr[k]}
Write a headline for the above news article about {label} . The headline
should be a single sentence.
Headline: ”

Prompt E.6: ToI Headlines SynthesizRR No-Retr-ICL In-context example:
“ Headline: {icl[article_retr]} ”
Prompt:
“ News Article: {article_retr[k]}
Write a headline for the above news article about {label} . The headline
should be a single sentence.
Headline: ”

###  E.3 AG News

We consider task inversion for the AG News dataset to be generation of news
summaries. We do not specify location modifiers as most Gold summaries are
from US news. We add a length-attribute to restrict the output one or two
sentences.

Label | Verbalization
---|---
Business |  companies, industries, markets, trade, investments, entrepreneurship, economic policies, and other business-related developments
World |  international news, such as politics, diplomacy, conflicts, global events, international relations, human rights issues, and significant global trends
Sci/Tech |  scientific discoveries, technological advancements, innovations, research breakthroughs
Sports |  professional sports leagues, major tournaments, athletes, teams, match results, player transfers, coaching changes, sports-related controversies
Table 15:  Task-inversion verbalizations for AG News.

Prompt E.7: AG News FewGen In-context example:
“ Write a summary for a news article about {label} . The summary should be one
or two short sentences.
Summary: {icl[gold_text]} ”
Prompt:
“ Write a summary for a news article about {label} . The summary should be one
or two short sentences.
Summary: ”

Prompt E.8: AG News SynthesizRR Retr-ICL In-context example:
“News Article: {icl[article_retr]}
Write a summary for the above news article about {label} . The summary should
be one or two short sentences.
Summary: {icl[gold_text]} ”
Prompt:
“ News Article: {article_retr[k]}
Write a summary for the above news article about {label} . The summary should
be one or two short sentences.
Summary: ”

Prompt E.9: AG News SynthesizRR No-Retr-ICL In-context example:
“ Summary: {icl[gold_text]} ”
Prompt:
“ News Article: {article_retr[k]}
Write a summary for the above news article about {label} . The summary should
be one or two short sentences.
Summary: ”

###  E.4 Category

In the Category dataset, we determine the product category from a review
written by a user. For task inversion in SynthesizRR we must retrieve a
product and prompt the frozen LLM to generate a user review within the same
product-category as the retrieval query. Thus, we include a style-attribute to
allow minor typos in the generation and restrict to a few sentences using a
length-attribute.

Label | Verbalization
---|---
magazines |  magazines or periodicals covering various topics
camera_photo |  photography gear including cameras, lenses, accessories, or photo editing tools
office_products |  office supplies or equipment for professional and home office setups
kitchen |  kitchenware, appliances, or culinary tools for cooking and dining
cell_phones_service |  cell phone service accessories or service plans for communication and connectivity
computer_video_games |  computers, gaming consoles, video games, or related accessories
grocery_and_gourmet_food |  groceries, fruits and vegetables, gourmet treats, or specialty food items
tools_hardware |  tools, hardware, or equipment for DIY projects and home repairs
automotive |  auto parts, accessories, or tools for vehicle maintenance and enhancements
music_album |  music albums spanning various genres and artists
health_and_personal_care |  healthcare products, personal care items, or wellness essentials
electronics |  electronic devices, gadgets, personal tech, or home electronics
outdoor_living |  products for outdoor activities, gardening, or patio living
video |  movies, TV shows, and documentaries spanning various genres and artists
apparel |  clothing including casual wear, formal attire, seasonal outfits, activewear, or fashion accessories for men, women, and children
toys_games |  fun or educational toys and games for kids of all ages
sports_outdoors |  products for various sports and outdoor activities
books |  books in various genres and formats
software |  computer software for productivity or gaming covering either personal or professional needs
baby |  baby essentials, gear, or toys for infants and toddlers
musical_and_instruments |  musical instruments, accessories, or music production equipment
beauty |  beauty products, cosmetics, or skincare essentials, makeup, hair care, fragrances, or grooming essentials
jewelry_and_watches |  watches or jewelry pieces such as necklaces, bracelets, earrings, or rings, crafted in precious metals or adorned with gemstones for special occasions
Table 16:  Task-inversion verbalizations for Category.

Prompt E.10: Category FewGen In-context example:
“ Write a product review about a product which is in the category of {label} .
Include relevant product details. The review should only be a single short
sentence, or a single paragraph of 3 to 4 sentences. Add very minor typos.
Review: {icl[gold_text]} ”
Prompt:
“ Write a product review about a product which is in the category of {label} .
Include relevant product details. The review should only be a single short
sentence, or a single paragraph of 3 to 4 sentences. Add very minor typos.
Review: ”

Prompt E.11: Category SynthesizRR Retr-ICL In-context example:
“ Product details: {icl[product_retr]}
Write a product review about the above product which is in the category of
{label} . Include relevant product details which are mentioned above. The
review should only be a single short sentence, or a single paragraph of 3 to 4
sentences. Add very minor typos.
Review: {icl[gold_text]} ”
Prompt:
“ Product details: {product_retr[k]}
Write a product review about the above product which is in the category of
{label} . Include relevant product details which are mentioned above. The
review should only be a single short sentence, or a single paragraph of 3 to 4
sentences. Add very minor typos.
Review: ”

Prompt E.12: Category SynthesizRR No-Retr-ICL In-context example:
“ Review: {icl[gold_text]} ”
Prompt:
“ Product details:
Write a product review about the above product which is in the category of
{label} . Include relevant product details which are mentioned above. The
review should only be a single short sentence, or a single paragraph of 3 to 4
sentences. Add very minor typos.
Review: ”

###  E.5 Humor

Asking humorous product questions is a challenge of the LLM’s task inversion
capabilities, as it must generate a question which is funny from the retrieved
product. Not all products have obvious humorous characteristics, thus the
generation requires some ingenuity. We restrict the output to only the
question to prevent explanations or extraneous product generations from the
LLM.

Label | Verbalization
---|---
humorous |  humorous
non_humorous |  solemn
Table 17:  Task inversion verbalizations for Humor.

Prompt E.13: Humor FewGen In-context example:
“ Write a short {label} question about a product on Amazon. Only include the
question.
Product Question: {icl[gold_text]} ”
Prompt:
“ Write a short {label} question about a product on Amazon. Only include the
question.
Product Question: ”

Prompt E.14: Humor SynthesizRR Retr-ICL In-context example:
“ Product details: {icl[product_retr]}
Write a short {label} question about the above product on Amazon. Only include
the question.
Product Question: {icl[gold_text]} ”
Prompt:
“ Product details: {product_retr[k]}
Write a short {label} question about the above product on Amazon. Only include
the question.
Product Question: ”

Prompt E.15: Humor SynthesizRR No-Retr-ICL In-context example:
“ Product Question: {icl[gold_text]} ”
Prompt:
“ Product details: {product_retr[k]}
Write a short {label} question about the above product on Amazon. Only include
the question.
Product Question: ”

###  E.6 Polarity

Polarity is a review-sentiment classification task. In SynthesizRR, the
difficulty is increased as we must generate a review from a product. For task
inversion, we prompt the LLM to generate a review which can have either
positive or negative sentiment and include details from the retrieved product.
As with Category, we allow typos and restrict the length to a few sentences
using a length-attribute in the prompt.

Label | Verbalization
---|---
positive |  what the reviewer liked about the product, how the reviewer found it easy to use the product, or the reviewer’s positive experience with the product
negative |  what the reviewer disliked about the product, how the reviewer found it challenging to use the product, or the reviewer’s negative experience with the product
Table 18:  Task inversion verbalizations for Polarity.

Prompt E.16: Polarity FewGen In-context example:
“ Write a review about a product on Amazon which discusses {label} . Include
relevant product details. The review should only be a single short sentence,
or a single paragraph of 3 to 4 sentences. Add very minor typos.
Review: {icl[gold_text]} ”
Prompt:
“ Write a review about a product on Amazon which discusses {label} . Include
relevant product details. The review should only be a single short sentence,
or a single paragraph of 3 to 4 sentences. Add very minor typos.
Review: ”

Prompt E.17: Polarity SynthesizRR Retr-ICL In-context example:
“ Product details: {icl[product_retr]}
Write a review about the above product on Amazon which discusses {label} .
Include relevant product details which are mentioned above. The review should
only be a single short sentence, or a single paragraph of 3 to 4 sentences.
Add very minor typos.
Review: {icl[gold_text]} ”
Prompt:
“ Product details: {product_retr[k]}
Write a review about the above product on Amazon which discusses {label} .
Include relevant product details which are mentioned above. The review should
only be a single short sentence, or a single paragraph of 3 to 4 sentences.
Add very minor typos.
Review: ”

Prompt E.18: Polarity SynthesizRR No-Retr-ICL In-context example:
“ Review: {icl[gold_text]} ”
Prompt:
“ Product details: {product_retr[k]}
Write a review about the above product on Amazon which discusses {label} .
Include relevant product details which are mentioned above. The review should
only be a single short sentence, or a single paragraph of 3 to 4 sentences.
Add very minor typos.
Review: ”

##  Appendix F Example generations

Here we showcase examples from the best-performing SynthesizRR approach
(3-shot No-Retr-ICL using LLaMa-2 Chat 13B) for each of our 6 tasks. For
brevity, we do not show the ICL examples, only the retrieved article and
generated text.

Class | Example
---|---
(Retrieved document) |  News Article: Gov. Andrew Cuomo Wednesdy afternoon staged a press conference to blast federal immigration officials, threatening to sue if agents don’t stop what he described as "reckless and unconstitutional enforcement" practices. Immigration and Customs Enforcement Deputy Director Thomas D. Homan responded hours later with a six-paragraph statement that accused Cuomo of "grandstanding." Cuomo cited a raid last week on a farm in Rome, N.Y. Immigration and Customs Enforcement agents entered John Collins’ property and detained a worker, Marcial DeLeon Aguilar, without presenting a warrant to Collins. That worker was later arrested and is still being detained. "The reckless and unconstitutional practices ICE is deploying in our communities violate everything we believe in New York and are an assault on our democracy," Cuomo said in a prepared statement. "I demand ICE immediately cease and desist this pattern of conduct, and if they fail to do so, I will pursue all available legal recourse and commit to doing everything in my power to protect the rights and safety of all New Yorkers." At about 7:30 p.m., Homan issued this response: "As a native New Yorker who began my law enforcement career as a police officer in the state, I was disappointed to learn about the Governor’s grandstanding today over the issue of immigration enforcement. The Governor’s comments were inaccurate and an insult to ICE’s sworn law enforcement officers who conduct their lawful mission professionally and with integrity. These brave men and woman leave the safety of their homes every day to protect this great nation and our communities.
hyperpartisan |  What a joke! Governor Andrew Cuomo had the audacity to call out ICE agents for doing their jobs, and now he’s threatening to sue them? Give me a break! The nerve of this guy, talking about "reckless and unconstitutional enforcement" practices when his own state is infested with illegal immigrants who are committing crimes left and right. Cuomo is just trying to grandstand and pander to his liberal base, but the fact is, he’s failing to protect the citizens of New York. ICE agents are just doing their jobs, and they deserve respect for their hard work and dedication. Cuomo should be ashamed of himself for attacking these brave men and women. And let’s not forget, this is the same guy who is protecting child rapists and allowing them to roam free in his state. What a disgrace! Cuomo needs to focus on the real problems in his state, like the rampant crime and corruption, instead of trying to score political points by attacking ICE agents. It’s pathetic.
(Retrieved document) |  News Article: This week on "Watters’ World", Jesse Watters caught up with several House Democrats who boycotted President Trump’s inauguration, including Rep. John Lewis, the Atlanta Democrat who called the Republican’s election "not legitimate". Approaching Houston Congresswoman Sheila Jackson Lee, Watters asked why she was absent from the festivities: "I have a statement on record, thank you so very much," Jackson Lee said. WATCH: Watters Tracks Down Rep. John Lewis to Ask About Trump Criticism ’Watters’ World’ EXCLUSIVE: Jesse Visits Scene of Chicago Hate Crime Attack Watters Confronts Man Who Harassed Ivanka Trump on JFK Flight As Watters introduced himself as a Fox News reporter to Rep. John Conyers of Michigan, the Detroiter kept walking toward an open elevator, "no, sir, not right now." "I didn’t boycott it, I just didn’t go to it," Conyers said. Rep. Jerrold Nadler of New York gave a similar answer when asked, "for the reasons I stated publicly; they’re on the record." Asked if he was being divisive, the Manhattan Democrat said that if anyone wanted to unite the country, they would not have voted for Trump. Lewis told Watters he was late and couldn’t speak with him as he walked with a quick gait down a hallway. An aide obstructed Watters’ cameraman as the Congressman continued toward his destination.
non-hyperpartisan |  In a recent segment on "Watters’ World", Jesse Watters spoke with several members of the House of Representatives about their absence from President Trump’s inauguration. While some members provided brief statements, others declined to comment or did not respond to Watters’ inquiries. Rep. John Lewis, an Atlanta Democrat who had publicly questioned the legitimacy of Trump’s election, was approached by Watters but did not stop to speak with him. Lewis had previously stated that he would not attend the inauguration due to his disagreement with Trump’s presidency. Other members, including Rep. John Conyers of Michigan and Rep. Jerrold Nadler of New York, also declined to comment or did not respond to Watters’ questions. When asked why he did not attend the inauguration, Conyers simply stated that he "didn’t go to it". Nadler, who had publicly stated that he would not attend the inauguration due to his disagreement with Trump’s policies, reiterated that he did not support the President’s election. Watters’ segment sparked a discussion on the perceived divisiveness of the President and the responses of those who did not attend his inauguration. While some argued that the boycott was a form of protest against Trump’s policies and actions, others felt that it was a divisive move that further polarized the country. Regardless of one’s political beliefs, it is important to approach discussions and debates with respect and civility, and to strive for understanding and cooperation in the pursuit of common goals.
Table 19:  Generated examples for Hyperpartisan task using SynthesizRR.  Class | Example
---|---
(Retrieved document) |  News Article: New Delhi: First reports of exit poll indicate history in West Bengal and regime change in Kerala while it’s too close to call in Tamil Nadu. An exit poll conducted by Star News television channel has projected Congress-led United Progressive Alliance (UPA)’s victory in West Bengal and Kerala. The survey projects Trinamool Congress - Congress alliance wiping out more than three decades old Left Front government in West Bengal. According to the exit polls telecast in the channel on Tuesday says the TMC and Congress will win 181 and 40 respectively while Left Front will be reduced to 62 from 227. In Kerala, the Congress-led United Democratic Front is expected to win 88 while CPM-led Left Democratic Front will win 49. In Tamil Nadu, AIADMK is projected as winning 110 and ruling DMK will have 124 seats in the 234 member assembly. In a close contest, Tarun Gogoi led Congress has a chance to create history in Assam by returning to power for a third consecutive term,post poll surveys predict. The Headlines Today poll predicts 44 of the 126 seats for Congress in Assam,with potential ally AUDF slated to win 14.
elections |  Exit Polls predict regime change in Kerala, West Bengal; Tight contest in Tamil Nadu
(Retrieved document) |  News Article: India’s teen sensation P V Sindhu stormed into the third round of the World Badminton Championship with a straight game victory over Olga Golovanova of Russia in women’s singles match here yesterday Copenhagen: India’s teen sensation PV Sindhu stormed into the third round of the World Badminton Championship with a straight game victory over Olga Golovanova of Russia in women’s singles match here yesterday. PV Sindhu The men’s doubles pair of Manu Attri and Sumeeth Reddy B stunned 15th seeded Japanese duo of Hirokatsu Hashimoto and Noriyasu Hirata 21-19 21-19 in 44 minutes to advance to the third round. Sindhu, seeded 11th, took 40 minutes to prevail over her Russian opponent 21-12 21-17 in the second round match at the Ballerup Super Arena here. She will next take on sixth seeded Yeon Ju Bae of Korea. Sindhu won a total of 42 points as compared to 29 by the Russian girl. The world No. 12 from Hyderabad looked a bit rusty to start with and was initially trailing in the opening game. She was playing the catching-up game till 10-10 before Sindhu managed to reel off four consecutive points and surge ahead to 14-10. There was no looking back after that, as Sindhu swiftly sealed the game in her favour with Golovanova earning just two more points. In the second game, the Russian got her act together and opened up a big lead, moving up to 11-6 at the break.
sports |  15-year-old PV Sindhu creates history, enters World Badminton Championship 3rd round
Table 20:  Generated examples for ToI Headlines task using SynthesizRR.  Class | Example
---|---
(Retrieved document) |  News Article: SANTIAGO Nov 14 A magnitude 6.1 earthquake hit central-north Chile on Wednesday, the U.S. Geological Survey said, causing buildings to sway in capital Santiago but causing no damage. The USGS said the quake struck on the coast, 54 miles (88 km) north of Coquimbo, at 4:02 p.m. (1902 GMT). It was initially reported as a magnitude 6.0 quake. Chile’s state emergency office, Onemi, said there were no reports of injuries, disruptions to basic services or infrastructure damage as a result of the tremor. The navy said the quake did not meet the conditions needed to trigger a tsunami. World No.1 copper producer Codelco said its operations were normal, and no mines in the metal-producing country reported damages. Seismically active Chile was battered in February 2010 by an 8.8 magnitude quake and ensuing tsunami, which caused billions in damage and killed hundreds.
World |  A magnitude 6.1 earthquake hit central-north Chile on Wednesday, causing buildings to sway in capital Santiago but causing no damage, the USGS said.
(Retrieved document) |  News Article: MONTREAL - Canadian astronaut David Saint-Jacques is bracing for some tense moments during next Monday’s launch of the Soyuz rocket that will send him and two others to the International Space Station. On Oct. 11, a rocket failure forced a Soyuz capsule carrying two astronauts to abort and make an emergency landing. Russia suspended all manned space launches pending an investigation before giving the green light Nov. 1. Saint Jacques spoke to reporters today from the launch site in Kazakhstan where he’s in quarantine. He said the most dangerous part of the six-month mission is the 10-minute Soyuz launch and the six hours that follow before docking. He said half of the last two-and-half years of training was dedicated to his role as Soyuz co-pilot for the trip to the space station. Saint-Jacques, 48, said once at the space station he’ll be able to focus on work and life aboard the outpost. The Canadian astronaut’s first space voyage, with NASA astronaut Anne McClain and Oleg Kononenkoof of the Russian space agency Roscosmos, was originally scheduled to launch Dec. 20. It was moved ahead after Russian authorities concluded their investigation into the failed launch.
Sci/Tech |  commentary Saint-Jacques says the Soyuz launch is the most dangerous part of the six-month mission but he’s been training for it for half of the last two-and-a-half years.
Table 21:  Generated examples for AG News task using SynthesizRR.  Class | Example
---|---
(Retrieved document) |  Product details: MSD ASY17296 Wiring Harness Factory Terminated MSD Ignition 6ALN (6430) Wiring Harness ASY17296 New harness with factory crimped connections a perfect item to have as an extra or replacement.
camera_photo |  I have a MSD Ignition 6ALN (6430) and this wiring harness is a great addition. The factory terminated connections are perfect and make installation a breeze. The harness is well made and will definitely last. I have been using it for a few weeks now and it has performed flawlessly. Highly recommend it to anyone looking for a high quality wiring harness.
(Retrieved document) |  Product details: Conair Travel Smart 3-Dial Combination Lock (6 Pack) Safely Locks Your Luggage With Tsa Recognized Locks Comes With A Red Inspection Status Feature Travel Sentry Approved
apparel |  I purchased this lock for a business trip to Europe. I’ve used it several times already and so far, so good. The TSA inspectors have been able to open it without a hitch, which is great!
Table 22:  Generated examples for Category task using SynthesizRR.  Class | Example
---|---
(Retrieved document) |  Product details: Keeping Up With The Kardashians - Season 5 Region E! brings viewers back into the sensational and completely unpredictable world of one of televisions favourite families with the fourth season of the hit series Keeping Up with the Kardashians. This season, Khlo and her husband, Los Angeles Lakers star, Lamar Odom, begin their new lives as a married couple. Meanwhile, Kourtney is pregnant and back together with Scott, and Kim and her boyfriend, NFL player Reggie Bush, work on their relationship. Through all this change, the Kardashian family must find ways to evolve and keep their close-knit ties.
humorous |  Does anyone else notice that this is the only show where you can hear people say things like, "OMG I am totally exhausted, I need to take a nap.? And it’s not considered weird?
(Retrieved document) |  Product details: Coleman Camper’s Storage Tote Securely organize, store and carry camping gear 23" x 14" x 17" Lightweight 600D polyester construction Multiple pockets and compartments Convenient carry strap The Coleman Camper’s Storage Tote is a great way to store, organize and carry your gear – use it in the garage, in the trunk or RV, and at the campsite. Constructed of reliable, lightweight 600D polyester, the Camper’s Storage measures 23" long, 14" wide and 17" tall. It features multiple pockets and compartments for organizing all your gear, and a convenient carry strap so it’s easy to bring along.
non_humorous |  is this product big enough for all the camping gear for a family of four?
Table 23:  Generated examples for Humor task using SynthesizRR.  Class | Example
---|---
(Retrieved document) |  Product details: Birth Of Soul 1 No half measures here, but a real statement of intent, particularly with that Volume One tag. 28 cuts that helped define that thing called soul - you can argue the point, but you can’t argue the music. Sublime.
positive |  This album was the beginning of my love affair with soul music. These tracks are timeless and still sound amazing today.
(Retrieved document) |  Product details: Beware! the Lies of Satan It’s high time we pull the covers off the enemy and expose him for who he really is, the murderer, destroyer, their, deceiver and biggest liar of all time. In this book , you will begin to discover the truth about our heavenly Father and how you can stand in victory over the devil.
negative |  The book does not live up to it’s promise. There is no revelation of truth about our heavenly father, or any insight into Satan’s lies. It is simply a polemic diatribe against Satan, with no concrete solutions to any of life’s problems.
Table 24:  Generated examples for Polarity task using SynthesizRR.

##  Appendix G Data Preprocessing

###  G.1 Datasets

  * •

AG News: We use https://huggingface.co/datasets/zapsdcn/ag

  * •

ToI Headlines: we pick examples in following 10 topics: {sports, life-style,
education, entertainment, business, city, environment, tech, elections, world}
and use the

  * •

Category: We get the data from:
https://www.cs.jhu.edu/~mdredze/datasets/sentiment/. We merge both positive
and negative examples and subsample as mentioned in Table 1.

  * •

Humor: We use https://registry.opendata.aws/humor-detection/.

  * •

Polarity: We use https://huggingface.co/datasets/amazon_polarity and
subsample.

Other datasets are not pre-processed.

###  G.2 Corpora

  * •

RealNews: we use the article text field and download the data from
https://github.com/rowanz/grover/tree/master/realnews.

  * •

RealNews/Regional is a subset of RealNews (Zellers et al., 2019). It includes
2.7M articles from non-US and non-EU websites. We manually check RealNews
websites and identified 141 regional-news websites with headquarters in 56
non-US and non-EU countries: India, Pakistan, Nigeria, Philippines, etc.

  * •

RealNews/India is further filtered to only include Indian news websites.

  * •

RealNews/Dominant is the remaining 30.1M articles from 1063 news websites
headquartered in 20 countries (of which over $75\%$ are US-based).

  * •

Products: We pull the data from
https://nijianmo.github.io/amazon/index.html#complete-data and concatenate
title and description

##  Appendix H Teacher and Student hyperparameters

###  H.1 Teacher LLM hyperparams

For LLaMa-2 Chat 13B, we use the implementation from HuggingFace:
https://huggingface.co/TheBloke/Llama-2-13B-fp16 and run it at half-precision.

For Claude Instant-v1, we use Claude Instant v1.2:
https://www.anthropic.com/news/releasing-claude-instant-1-2

We use a batch size of 1 for all generations as we have long contexts and
encountered failures with higher batch sizes. We use nucleus sampling with
top-p=0.9.

###  H.2 Student LM hyperparams

We use DeBERTa-v3-Large and DistilBERT models from HuggingFace:
https://huggingface.co/microsoft/deberta-v3-large,
https://huggingface.co/distilbert/distilbert-base-uncased

We use the same hyperparameters for DeBERTa-v3L and DistilBERT as (Yu et al.,
2023a):

  * •

DistilBERT: Learning rate of 5e-5, gradient_accumulation_steps of 1,
batch_size 32. We use the Adam optimizer with weight_decay of 1e-4 and epsilon
of 1e-6. We use max_sequence_length of 512.

  * •

DeBERTa-v3L: Learning rate of 2e-5, gradient_accumulation_steps of 8,
batch_size 4. We use the Adam optimizer with weight_decay of 1e-4 and epsilon
of 1e-6. We use max_sequence_length of 512.

We train all students for 6 epochs. Following (Yu et al., 2023a), we use
warmup for 6% of the training steps.

###  H.3 Oracle model hyperparams

To train the DeBERTa-v3-Large oracle model for Label Preservation, we use a
grid search over 9 combinations: 3 learning rates {2e-5, 5e-5, 1e-4} by 3
batch-sizes {1, 4, 16} (with same graident accumulation). We train on 80% of
the Gold training data and use the remaining 20% as validation.

###  H.4 Retriever

We use Contriever from HuggingFace library:
https://huggingface.co/facebook/contriever.

We pass a batch-size of 512 for embedding.

##  Appendix I Computational budget

We run all our models on AWS Elastic Cloud
Compute222https://aws.amazon.com/ec2/ using 20 p3dn.24xlarge machines.

###  I.1 Information Retrieval

The corpora was embedded by us and the trivial was done using the Faiss
library.333https://faiss.ai/index.html We orchestrate 80 copies of Contriever
using the Ray distributed framework444https://docs.ray.io/en/latest/index.html
to embed the RealNews and Products corpus in $\sim$ 3 hours each.

###  I.2 Dataset synthesis

For LLaMa-2 Chat 13B, we create 48 copies of the model and orchestrate it
using the Ray distributed framework. Generation is done in roughly 6 hours per
dataset of 8k rows.

To use Claude Instant-v1, we invoke AWS
Bedrock555https://docs.aws.amazon.com/pdfs/bedrock/latest/APIReference/bedrock-
api.pdf using the boto3
library666https://boto3.amazonaws.com/v1/documentation/api/latest/index.html.
Generation from Claude was done at an AWS-account level RPM of 1600 and takes
roughly 4 hours per dataset on 8k rows.

###  I.3 Student distillation

Each DeBERTa-v3-Large student model trains for between 1 to 3 hours on a
single V100 GPU on 8k rows. Each DistilBERT student model trains in 1 hour to
generate the data-map.

##  Appendix J Licensing

We use datasets that have been released in prior work with various open
licenses. Specifically:

###  J.1 Datasets

  * •

AG News: custom license, described at
http://groups.di.unipi.it/~gulli/AG_corpus_of_news_articles.html

  * •

ToI Headlines: uses Creative Commons CC0 1.0 Universal Public Domain
Dedication licence as per
https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/DPQMQH

  * •

Hyperpartisan: taken from SemEval 2019 Task 4, this is licensed under a
Creative Commons Attribution 4.0 International License as per
https://zenodo.org/records/1489920

  * •

Category: Blitzer et al. (2007) does not specify a licence but has made the
data available for research at:
https://www.cs.jhu.edu/~mdredze/datasets/sentiment/

  * •

Humor: Community Data License Agreement – Sharing – Version 1.0 licence as per
https://registry.opendata.aws/humor-detection/

  * •

Polarity: Apache License 2.0 as per
https://huggingface.co/datasets/amazon_polarity

###  J.2 Corpora

  * •

RealNews: custom licence as per
https://docs.google.com/forms/d/1LMAUeUtHNPXO9koyAIlDpvyKsLSYlrBj3rYhC30a7Ak/viewform?edit_requested=true.
The code repository is Apache Licence 2.0 as per
https://github.com/rowanz/grover/blob/master/LICENSE

  * •

Products: (Ni et al., 2019) does not specify a licence but has made the data
available for research at:
https://nijianmo.github.io/amazon/index.html#complete-data.