# Few-shot Mining of Naturally Occurring Inputs and Outputs

Mandar Joshi† Terra Blevins† Mike Lewis§
Daniel S. Weld†ϵ Luke Zettlemoyer†§
† Allen School of Computer Science & Engineering, University of Washington,
Seattle, WA
<EMAIL_ADDRESS>
ϵAllen Institute of Artificial Intelligence, Seattle
<EMAIL_ADDRESS>
§ Facebook AI Research, Seattle
<EMAIL_ADDRESS>
###### Abstract

Creating labeled natural language training data is expensive and requires
significant human effort. We mine input output examples from large corpora
using a supervised mining function trained using a small seed set of only 100
examples. The mining consists of two stages – (1) a biencoder-based recall-
oriented dense search which pairs inputs with potential outputs, and (2) a
crossencoder-based filter which re-ranks the output of the biencoder stage for
better precision. Unlike model-generated data augmentation, our method mines
_naturally occurring_ high-quality input output pairs to mimic the style of
the seed set for multiple tasks. On SQuAD-style reading comprehension,
augmenting the seed set with the mined data results in an improvement of 13 F1
over a BART-large baseline fine-tuned only on the seed set. Likewise, we see
improvements of 1.46 ROUGE-L on Xsum abstractive summarization.

##  1 Introduction

Gathering high-quality labeled training data has been one of the most reliable
ways of achieving empirical progress for a range of natural language
processing tasks. However, creating natural language training data often
involves significant human effort – in carefully designing the data collection
tasks as well as getting contributors to perform them. The complexity of this
process in turn drives up the cost of creating training data. In this paper,
we mine naturally occurring examples from large corpora using supervision from
a small seed set of only 100 labeled examples.

Summary: The Scottish city of Edinburgh is looking to crack down on so-called
”silent disco” walking tours as residents complain they make too much noise.
---
Passage: Silent disco tours in Edinburgh are ’too loud’ and could face
clampdown Silent discos in Edinburgh could be facing a clampdown - with locals
upset that the crowds of boogying tourists are too loud. The ”discos” involve
people wearing their own headphones and dancing along as they follow a guided
walking tour of the Scottish capital’s most famous spots. …
Question: How long does it take for a crab to get full grown?
Passage: 10-13 moults the crab will reach maturity. This usually takes _3-4
years_ , but when food is limited it can take longer to reach maturity.
Figure 1: Examples of mined input output pairs for summarization and reading
comprehension. Answer spans are indicated via italics.

Our method provides a way to collect large amounts of training data using a
small set of labeled seed examples, and allows for more direct control over
what the model learns compared to relatively brittle prompts Lu et al. (2021).
Yet, unlike model generated data augmentation, we mine high-quality human-
authored data which is less susceptible to the limitations of synthetic data
(Section 6). While similar unsupervised methods have been used for mining
parallel data for machine translation Tran et al. (2020), we show that they
can be extended to other tasks using minimal human supervision.

One of the main challenges in mining high-quality data is the large search
space which could be quadratic in the size of the corpora as inputs and
outputs could be spread across multiple documents. Our approach consists of a
two-stage pipeline which first mines data efficiently from large corpora and
then filters out low-quality examples. More specifically, we use supervision
from an initial seed set and train a dot-product similarity function over
separately encoded fixed length dense representations of inputs and outputs
(i.e. a dual-encoder or a biencoder). This function efficiently searches top-k
candidate outputs (e.g. summary sentences) for each candidate input (e.g. news
document) from the output corpus. We refer to this stage as biencoder-based
search. It relies on maximal inner product search (MIPS) over dense encodings
on inputs and outputs for efficiency. The initial list of paired inputs and
outputs from this stage is geared towards recall and often contains examples
with subtle errors. For example, it might link a document about World Cup 2006
with a summary sentence about World Cup 2010.

For the second stage, we train a precision-oriented crossencoder-based filter
which re-ranks the output of the first stage for better quality control. The
crossencoder is trained using positive samples from the original seed set and
negative examples from the biencoder-based stage. Jointly encoding inputs and
outputs provides fine grained interaction between them. The crossencoder is
able to filter out more subtle errors such as the one described earlier to
give a high quality mined dataset set.

We apply our method to reading comprehension and summarization, gathering high
quality question-answer-passage and document-summary tuples respectively. The
supervision from the small seed set helps embed relevant outputs closer to
inputs across corpora for each task. This is in contrast to MT where cross-
lingual cosine similarity functions (over mean pooled output representations)
are readily available without additional human supervision – since self-
supervised pre-training embeds sentences across languages in the same space.

Figure 2: A pipeline of bi- and cross-encoder models for mining high-quality
data from large corpora. The bincoder search encodes the input and output
corpora separately, and produces input-output pairs. The crossencoder filter
then re-ranks these pairs to produce high-quality data.

Our method is able to add up to 5x as many examples as the seed set. On SQuAD
Rajpurkar et al. (2016), augmenting the seed set with the mined data results
in an improvement of 13 F1 over a BART-large Lewis et al. (2020b) baseline
fine-tuned only on the seed set. Likewise, we see improvements of 1.46 ROUGE-L
on XSum abstractive summarization Narayan et al. (2018). Our analysis shows
that, compared to model generated data, mined data better matches several
characteristics of the gold data. For example, mined document-summary pairs
are more abstractive and are topically closer to XSum. Overall, mined data
augmentation is based on the intuition that learning similarity functions
between contexts may be an easier problem that generating one context from
another Khandelwal et al. (2020).

##  2 Mining

We describe our general pipeline for mining examples, and then describe how it
is applied to reading comprehension and summarization. We assume access to a
small labeled seed set of 100 examples which can be used to train components
of the pipeline. The final output is a (ranked) list of input output pairs
which can be augmented to the original seed set to train better-performing
models.

The key idea consists of a two stage approach for mining input output pairs
$(x,y)$ from their respective corpora $C_{x}$ and $C_{y}$ – (a) a biencoder-
based efficient similarity function which retrieves top-k similar candidates
from $C_{y}$ for every $x\in C_{x}$ and (b) a crossencoder-based filtering
stage which reranks top scoring candidate pairs mined in the first stage.

###  2.1 Biencoder Search

The biencoder search relies on dense retrieval to construct a candidate list
of input-output pairs. We first train a function $f_{c}(\cdot,\cdot)$ to
maximize similarity between input output pairs in the seed data, and then use
the similarity function to mine the candidate input output pairs.

####  2.1.1 Training Biencoder Similarity

The aim is to create a vector space where the similarity between relevant
input output pairs is higher than that between irrelevant pairs. Specifically,
we use cosine similarity as our similarity function to enable efficient mining
(Section 2.1.2).

##### Objective

Let $D=\\{(x_{i},y_{i}^{+},y_{i,1}^{-}...y_{i,n}^{-})\\}_{i=1}^{i=m}$ be the
training data that consists of $m$ seed instances. Each instance contains one
input $x_{i}$ and one output $y_{i}^{+}$ along with $n$ sampled negative
outputs $y_{i,*}^{-}$. Our biencoder first encodes inputs and outputs
separately as $\mathbf{x}_{*}$ and $\mathbf{y}_{*}$. The training objective
maximizes the negative log likelihood of the positive output.

$L(x_{i},y_{i}^{+},y_{i,1}^{-}...y_{i,n}^{-})=\frac{e^{\mathbf{x}_{i}\cdot\mathbf{y}_{i}^{+}}}{e^{\mathbf{x}_{i}\cdot\mathbf{y}_{i}^{+}}+\sum_{j=1}^{n}e^{\mathbf{x}_{i}\cdot\mathbf{y}_{i,j}^{-}}}$

We discuss details about negative samples in Section 3.

####  2.1.2 Mining Candidate Pairs

To mine input output pairs, we follow previous work on unsupervised machine
translation and employ the margin function formulation Artetxe and Schwenk
(2019); Tran et al. (2020) based on K nearest neighbors (KNN). Let
$\mathbf{x}$ and $\mathbf{y}$ be the vector representations of a candidate
input output pair $(x,y)$. We score the similarity of $x$ and $y$ using a
ratio margin function defined as the following.

$score(x,y)=\frac{cos(\mathbf{x},\mathbf{y})}{\sum_{z\in
N_{x}}\frac{cos(\mathbf{x},\mathbf{z})}{2k}+\sum_{z\in
N_{y}}\frac{cos(\mathbf{z},\mathbf{y})}{2k}}$

where $N_{x}$ is the KNN neighborhood of $x$ in $C_{y}$, the corpus of $y$;
and $N_{y}$ is the KNN neighborhood of $y$ in $C_{x}$. The margin scoring
function is interpreted as a cosine score normalized by average distances to
the margin regions established by the KNN neighborhoods of the input and
outputs. The KNN distance metrics are defined by $cos(x,y)$. The margin score
is designed to take into account scale inconsistencies in cosine similarity,
and has been shown to perform better than approaches which use a hard
threshold over cosine similarity. We use the distributed dense vector
similarity search library FAISS Johnson et al. (2019) to search for all
neighborhoods efficiently.

###  2.2 Crossencoder Filtering

We found the quality of the biencoder-mined data to be rather low for
summarization and reading comprehension. Top ranked pairs for summarization
were mostly extractive and those for reading comprehension often had very
short passages (Section 7). We hypothesize that separate encoding of inputs
and outputs combined with cosine based scoring makes it much harder to control
characteristics like abstraction for summarization. The crossencoder filtering
stage re-ranks the outputs of the previous stage using a crossencoder based
ranking function to remove noisy input-output pairs. A crossencoder jointly
encodes the input and output text enabling more fine grained interaction
between them. Specifically, we encode a pair $(x,y)$ as sequence $[CLS]$ $x$
$[SEP]$ $y$ $[SEP]$ where $[CLS]$ and $[SEP]$ are special tokens indicating
beginning and end of sequences. The same input is fed into the encoder and
decoder. The pair is scored using a 2-layer MLP on top of the final hidden
state of the final decoder token. A more detailed comparison of the data
obtained from the two stages is presented in Section 7.

##  3 Application to Tasks

One of the main challenges in mining data is the scale of web corpora. In this
section, we describe additional techniques which make our method scalable when
applied to downstream tasks.

###  3.1 Reading Comprehension

##### Inputs and Outputs

For reading comprehension, the input sequence $X$ consists of the question and
the output sequence $Y$ consists of a concatenation of the answer and the
passage. Our input corpus consists of 20M random questions crawled from the
popular community QA website _answers.com_. The output corpus consists of
passages from Wikipedia with named entities111We use spaCy NER. as candidate
answers resulting in 200M passage-answer pairs. Our input and output corpora
were chosen to match the downstream evaluation data from SQuAD.

##### Biencoder and Binary Classifier

To mitigate noise in the question corpus and to reduce the search space, we
multitask a binary classifier with the biencoder to filter out questions which
don’t look like those in SQuAD. The binary classifier is trained using
questions from the seed set as positive examples with negatives sampled from
the question corpus. Furthermore, for faster processing, we filter out
examples from the biencoder stage, where the answer spans are found verbatim
in the question.

Apart from in-batch negatives, we also add synthetically generated negative
samples for each question by (a) pairing incorrect candidate answer spans
(identified using NER) with correct passages, and (b) pairing correct answers
with incorrect passages. The aim of adding synthetic negatives is to avoid
overfitting to passage-only and answer-only features.

##### Crossencoder

The crossencoder for SQuAD is a simple binary classifier which uses the seed
set as positive examples and the negative samples from the bicencoder as
negatives.

###  3.2 Summarization

##### Inputs and Outputs

For summarization, the input sequence $X$ consists of the document and the
output sequence $Y$ consists of the summary. The input corpus consists of
documents from CC-News; we remove documents with less than four sentences to
reduce noise. The output corpus consists of sentences from the same corpus.
Following MARGE Lewis et al. (2020a), we divide CC-News into shards based on
document publishing dates, and restrict the space of candidate summaries for a
given document to be from the same shard.

##### Biencoder and Binary Classifier

Like reading comprehension, we multitask a binary classifier with the
biencoder to remove sentences which don’t resemble XSum summaries. This allows
us to filter out more than 80% of the sentences and makes the biencoder based
mining a lot more scalable.

Finally, for faster processing, we filter out examples from the biencoder
stage, where the summary sentences are found verbatim in the document. Put
together, this allows use to efficiently search over 40M documents and 550M
sentences. We use in-batch summary sentences along with randomly sampled
sentences from the output corpus to construct negative summary samples.

##### Crossencoder

Unlike reading comprehension, summarization has a larger space of _valid_
outputs. For example, it is possible to alter the style of the given summary
without changing the core content. We tweaked the crossencoder training
objective for summarization to take into account stylistic differences.
Specifically, we encoded the documents in the seed set using the biencoder and
retrieved summary sentences from the corpus to be used as negatives. Each
document summary pair was then scored by the cross-encoder. For a given
document, the training objective maximized the score of the correct document
summary pair over the incorrect one. The inference stage re-ranking document
summary pairs obtained from the biencoder stage using their unnormalized
score.

##  4 Downstream Task Model

We fine-tune BART on the seed set and the mined data for each task separately.

##### Reading Comprehension

For SQuAD, we feed the complete document into the encoder and decoder, and use
the top hidden state of the decoder as a representation for each word. Start
and end classifiers use this representation to classify the token. We simply
add the mined data to the initial seed set during fine-tuning.

##### Summarization

For XSum, we use sequence-to-sequence training with beam decoding during
inference. While adding the mined data to the seed set, we denote mined
documents using a special token to account for stylistic differences in the
documents and summaries.

##  5 Experiments

###  5.1 Benchmarks

We perform experiments on SQuAD v1.1 for reading comprehension and XSum for
summarization – two competitive benchmarks which are used to evaluate state of
the art pre-trained models. For each dataset, we randomly pick 100 examples
from the training set as our seed set; we call this subset $X_{100}$.

##### SQuAD

SQuAD Rajpurkar et al. (2016) is an extractive reading comprehension benchmark
containing Wikipedia passages and crowdsourced questions.

##### Xsum

Xsum Narayan et al. (2018) is an abstractive news summarization dataset
collected by harvesting online articles from the British Broadcasting
Corporation (BBC).

###  5.2 Baselines

##### BART

Our main baseline for each dataset is a BART model finetuned on $X_{100}$ and
is referred to as BART in the subsequent tables. The fine-tuning differs from
our methods (Section 3) only in the data used.

For summarization, we also compare our method to WikiTransfer Fabbri et al.
(2021) and Pegasus Zhang et al. (2020) – both of which use data augmentation
techniques tailored towards summarization.

##### WikiTransfer

WikiTransfer uses an intermediate finetuning stage with 400K synthetic
examples from Wikipedia along with additional auxiliary losses before
finetuning on Xsum.

##### Pegasus

Pegasus, a self-supervised pretraining method, simulates the summarization
tasks during pre-training by masking (and then generating) important
heuristically identified sentences from documents.

###  5.3 Implementation Details

We implemented all models in fairseq Ott et al. (2019) and initialized them
with BART. The biencoders were trained for 100 (XSum) or 200 (SQuAD) steps on
8 Volta 32GB GPUs using learning rates (LRs) in the range 5e-6 and
{1,2,3,4,5}e-5. The crossencoders were trained for 200 (XSum) or 1000 (SQuAD)
steps on 8 GPUs using LRs in the same range. We used the accuracy on the dev
set for model selection. Both downstream task models are fine-tuned for 200
steps on 8 GPUs using the same learning rates.

###  5.4 Main Results

Method | EM | F1
---|---|---
BART | 35.27 | 49.43
Us | 49.39 | 62.5
Table 1: Performance on the SQuAD dev set after training on the seed set of 100 examples, and additionally in case our method, 500 mined examples. Method | R-1 | R-2 | R-L
---|---|---|---
BART | 36.28 | 14.07 | 27.98
Us | 37.67 | 14.86 | 29.44
BART* | 35.17 | 13.29 | 27.20
WikiTransfer* | 37.26 | 14.20 | 28.85
Pegasus* | 39.07 | 16.44 | 31.27
Table 2: Performance on the XSum test set after training on 100 examples (and
an additonal 500 mined examples for our method). Both BART and Us use the same
training examples. * indicates that results have been taken from Fabbri et al.
(2021) and use a different seed set.

##### SQuAD

Table 1 shows our performance on the SQuAD v1.1 dev set. The BART baseline
reaches an F1 of 48.86; adding 500 mined examples gives us a boost of 13 F1
points.

##### XSum

Table 2 shows our performance on the Xsum test set. Augmenting the 100 example
training set with 500 mined examples increases the ROUGE-L score of the
finetuned BART baseline from 27.98 (BART) to 29.44 (Us).

BART* refers to BART results taken from WikiTransfer.222We posit that our
numbers for BART are slightly higher due to a different seed set. Our gains
over the BART baseline are comparable to WikiTransfer despite using only 500
augmented examples indicating that our examples are high-quality. It is harder
to make more direct controlled comparisons to Pegasus which uses a larger
model and a tailored pre-training scheme compared to BART.

Our approach is complementary to both WikiTransfer and Pegasus; in contrast to
pre-training (or intermediate fine-tuning) on synthetic or heuristic-collected
examples, we focus on mining fewer high quality examples aimed at mirroring
the seed set so that they can be used directly during finetuning instead of
pre-training.

Figure 3: Performance (ROUGE-L) on the XSum dev set of various augmentation
techniques with varying amounts of augmented data added to $X_{100}$. Figure
4: Performance (F1) on the SQuAD dev set of various augmentation techniques
with varying amounts of augmented data added to $X_{100}$.

##  6 Ablations

In this section, we investigate the impact of each part of our pipeline and
compare it to model-generated data augmentation (Figures 3 and 4). All methods
involve fine-tuning BART on the same seed set plus the exact same number of
augmented examples, but differ on the source of the augmented data.

##### Gold

The Gold method augments gold examples sampled uniformly at random from the
full training set of each dataset.

##### Full

Full refers to examples obtained by running our full pipeline including the
crossencoder stage.

##### Biencoder

This refers to post-processed examples from the biencoder which are
subsequently fed to the crossencoder (in our full model).

##### Gen

Gen refers to self-training on model generated examples. Specifically, we take
source documents (or answer passage pairs in case of reading comprehension)
from the full/crossencoder stage and generate summaries (or questions) using
the BART baseline trained on $X_{100}$. We then retrain the model by
augmenting $X_{100}$ with the generated examples. This allows us to make a
controlled comparison to the full model which uses naturally occurring mined
examples.

###  6.1 Amount of Mined Data

For both SQuAD and XSum, we see an increase in performance for the Full model
as we increase the number of mined examples up to 5x. There is still a
considerable difference between our method and Gold indicating that there is
still room for improvement in quality. We discuss these qualitative
differences in Section 7.

###  6.2 Bi vs Crossencoder Differences

For SQuAD, input output pairs from the biencoder provide some improvement with
50 examples but performance quickly plateaus with more examples. For XSum,
adding examples from the biencoder hurts performance. On the other hand,
performance continues to improve until 5x examples from the crossencoder are
added for both tasks. Section 7 contains a qualitative analysis of the bi and
crossencoder examples.

###  6.3 Naturally Occurring vs. Model Generated Data

To show the value of mining naturally occurring data, we compare our method to
the _Gen_ baseline described in Section 6. For both XSum and SQuAD, adding
model-generated examples provides some gains over BART, but performance
quickly degrades with more examples. While model generated data augmentation
has been shown to help for QA Alberti et al. (2019), our results suggest that
it isn’t particularly useful when training data is limited.

##  7 Discussion

We conducted a case study of the top ranking examples from the bi and cross
encoders for both SQuAD and XSum (Table 3).

Biencoder |  Crossencoder
---|---
Summary: BERLIN (AP) — A 44-pound minute hand has fallen off a clock on a Hamburg church tower, plunging 130 feet onto the sidewalk below. |  Summary: Blackpool’s three historic piers have been put on an international preservation list amid fears their future could be under threat from climate change.
Document: Minute hand plunges from Hamburg church tower; no one hurt BERLIN (AP) — A 20-kilogram (44-pound) minute hand has fallen off a clock on a Hamburg church tower, plunging 40 meters (130 feet) onto the sidewalk below… |  Document: …The council is now completing an action plan to work with the World Monuments Fund which was launched in 1996 with founding partner American Express and issues a list every two years. Blackpool is unique in being the only UK seaside resort with three piers…
Summary: NEW LONDON, Conn. (AP) — The U.S. Coast Guard Academy is now offering an academic program in cyber systems, its first new major in a quarter century. |  Summary: Holidaymakers are being reminded to drink responsibly at Gatwick Airport as an annual campaign to tackle disruptive passengers is launched.
Document: Coast Guard Academy to offer new major in cyber systems NEW LONDON, Conn. (AP) — The U.S. Coast Guard Academy is offering a new major in cyber systems. It’s the first new academic program at the school in New London since the addition of mechanical engineering as a major in 1993… |  Document: The force, working closely with the airport, its pubs and bars, and airlines, will carry out increased patrols as part of Operation Disrupt…”You could be refused carriage or sent on the next plane home if you are considered to be drunk, disorderly or disruptive…
Question: What is most times one actor has been nominated? |  Question: When was Regina Lobiondo born?
Answer/Passage: The actress has also been nominated _five_ times (most |  Answer/Passage: Other notable appearances include a recurring role in L.A. Law, a regular role in the 1993 The Untouchables television series, and starring in the 1996 film It’s My Party. Regina was born on _October 25, 1956_ , in Brooklyn, New York.
Question: Where was Jesus baptised? |  Question: When was Gouverneur Morris born?
Answer/Passage: He was baptised at _St Jude’s_ |  Answer/Passage: Gouverneur Morris Jr. Gouverneur Morris II (_February 9, 1813_ – August 20, 1888) was an American railroad executive and the son of a founding father of the United States, Gouverneur Morris. Gouverneur Morris was born on February 9, 1813, Morrisania, Bronx, New York.
Table 3: Mined examples for summarization and reading comprehension after the
bi and crossencoders. Answer spans are indicated in _italics_. Biencoder
outputs focus on word overlap leading to (a) abstractive summaries which do
not resemble XSum those from XSum in style and topicality, and (b) short noisy
passages with with greater proportion of question words for reading
comprehension.

###  7.1 XSum

After examining outputs from the bi and crossencoder, we found that both
stages were generally able to match documents to sentences from the same news
story, but with the sentences taken from a parallel document. Exceptions to
this included cases where similar events (e.g. World Cup games) occuring over
different time frames. However, we found three clear differences which could
explain why data from the crossencoder is more helpful. First, summaries from
the crossencoder were more abstractive than the biencoder. We measured
abstractiveness using ROUGE precision scores with respect to the _source_ ;
higher scores being indicative of higher extractiveness and lower
abstractiveness. ROUGE-1/2/L precision scores for the crossencoder summaries
(64.91 / 15.42 / 55.59) were closer to the XSum validation (66.85 / 17.94 /
59.82) compared to those from the biencdoer (82.21 / 51.66 / 75.92). Second,
the distribution of topics and locations from the top ranked crossencoder
documents better matched XSum with most stories centered around UK politics
and sports. On the other hand, top ranked pairs from the biencoder were more
diverse topically. Lastly, the summaries mined by the biencoder rarely matched
the style of XSum and often contained source markers (e.g. CNN or AP News).
Our analysis suggests that the biencoder uses word overlap as it’s primary
signal for scoring document summary pairs while the crossencoder is able to
focus on finer details.

###  7.2 SQuAD

We found mining high-precision QA pairs to be a much harder problem in part
due to the large search space of over 200M (answer, passage) tuples. Like
summarization, biencoder outputs for QA focused on word overlap which resulted
in the retrieval of very short passages (see Table 3). While the crossencoder
was more successful in selecting longer passages, we still found pairs with
subtle errors. For example, for the question, “When was Regina Lobiondo
born?”, the retrieved passage actually refers to Paul Regina and not Regina
Lobiondo. We hypothesize the negative impact of such noise is mitigated in the
reading comprehension setting where the given evaluation passage always answer
the question. Another limitation of data from both stages was the lack of
diversity in question types with a significant proportion of questions seeking
date or numerical answers.

##  8 Related Work

Perhaps most related to our work is CRISS Tran et al. (2020), an unsupervised
method for mining parallel sentences across languages. Similar to our work,
they use a biencoder to mine pairs. However, multilingual pre-training enables
the use of cross-lingual cosine similarity functions (over mean pooled output
representations) without additional biencoder training. In contrast, for tasks
where such high-precision similarity functions aren’t readily available, we
need (a) a seed set to train biencoders, and (b) a (trained) crossencoder for
removing more subtle errors. We divide other related work into the following
categories.

##### Data augmentation

Data augmentation is a popular technique with a large body of work Wang and
Yang (2015); Jia and Liang (2016) among others. Recent work has explored model
generated data augmentation for a range of tasks including text classification
Anaby-Tavor et al. (2020); Schick and Schütze (2021a), question answering
Alberti et al. (2019), common-sense reasoning Yang et al. (2020), and machine
translation Sennrich et al. (2016). A common problem with model-generated data
augmentation is the quality of the synthetic data. Attempts to remedy this
have focused either on regularization He et al. (2020) or on variations of
consistency Xie et al. (2019); Sohn et al. (2020) for a given task–such as
round-trip consistency of question generation and answer prediction Alberti et
al. (2019); Puri et al. (2020) for QA or between source and targets in
summarization Fabbri et al. (2021). Work on retrieval-based data augmentation
shows that self-training captures signal complementary to pre-training but has
focused on classification tasks Du et al. (2021). Ex2 Lee et al. (2021) focus
on generating synthetic data for underrepresented or few-shot slices using a
model learned by simulating the example generation procedure on data-rich
slices of the data. Task augmentation Vu et al. (2021) generates data in the
target domain by using a model trained on the auxiliary task of natural
language inference. In contrast, we focus on a general framework for mining
_naturally occurring_ data for multiple tasks using supervision from a small
labeled seed set.

##### Improving fine-tuning

Work in this area has looked at improving finetuning particularly for small
datasets either via better optimization or by using auxiliary tasks. Careful
design and hyperparameter choices have a significant impact on performance and
stability in limited data settings Mosbach et al. (2021); Zhang et al. (2019).
Likewise, regularization based approaches have also been shown to improve
performance Jiang et al. (2020); Aghajanyan et al. (2020). While we use
lessons about careful design choices, our basic models use standard fine-
tuning for simplicity. Efforts on the data side have focused on intermediate
fine-tuning either by using unlabeled target domain data Karouzos et al.
(2021); Gururangan et al. (2020) or via labeled data from other tasks Phang et
al. (2018); Aghajanyan et al. (2021); Vu et al. (2020).

##### Few-shot learning

GPT-3 Brown et al. (2020) is perhaps the most prominent recent work in this
area which shows that massive language models can perform a variety of tasks
if prompted with few input output pairs. Other work has shown that good few-
shot performance can also be obtained by combining gradient based optimization
with textual prompts Schick and Schütze (2021b); Gao et al. (2021); Tam et al.
(2021); Wang et al. (2021). Other work on pre-training focuses on tailored
masking schemes such as recurrent span masking Ram et al. (2021) for question
answering and salient sentence masking for summarization Zhang et al. (2020).
Our approach is complementary to this line of work in that it is agnostic to
pre-training schemes or textual prompts, and focuses instead on supervised
automated collection of high-quality data.

##  9 Conclusion

We presented a method to mine input output pairs from large corpora using
supervision from a small seed set of labeled examples. Our approach consisted
of two stages – an efficient cheap biencoder search to identify promising
candidate pairs from the corpus and an expensive crossencoder filter for re-
ranking the output of the first stage. Our method achieved consistent
improvements on XSum and SQuAD, two popular NLP benchmarks, from adding up to
5x examples to the seed set. Our analysis shows that, compared to model
generated data, mined data better matches several characteristics of the gold
data (e.g. abstractiveness in summarization).

## References

  * Aghajanyan et al. (2021) Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. 2021.  Muppet: Massive multi-task representations with pre-finetuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 5799–5811, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Aghajanyan et al. (2020) Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, and Sonal Gupta. 2020.  Better fine-tuning by reducing representational collapse.  _arXiv preprint arXiv:2008.03156_.
  * Alberti et al. (2019) Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins. 2019\.  Synthetic QA corpora generation with roundtrip consistency.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6168–6173, Florence, Italy. Association for Computational Linguistics.
  * Anaby-Tavor et al. (2020) Ateret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, and Naama Zwerdling. 2020.  Do not have enough data? deep learning to the rescue! _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34(05):7383–7390.
  * Artetxe and Schwenk (2019) Mikel Artetxe and Holger Schwenk. 2019.  Margin-based parallel corpus mining with multilingual sentence embeddings.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3197–3203, Florence, Italy. Association for Computational Linguistics.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Du et al. (2021) Jingfei Du, Edouard Grave, Beliz Gunel, Vishrav Chaudhary, Onur Celebi, Michael Auli, Veselin Stoyanov, and Alexis Conneau. 2021.  Self-training improves pre-training for natural language understanding.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5408–5418, Online. Association for Computational Linguistics.
  * Fabbri et al. (2021) Alexander Fabbri, Simeng Han, Haoyuan Li, Haoran Li, Marjan Ghazvininejad, Shafiq Joty, Dragomir Radev, and Yashar Mehdad. 2021.  Improving zero and few-shot abstractive summarization with intermediate fine-tuning and data augmentation.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 704–717, Online. Association for Computational Linguistics.
  * Gao et al. (2021) Tianyu Gao, Adam Fisch, and Danqi Chen. 2021.  Making pre-trained language models better few-shot learners.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3816–3830, Online. Association for Computational Linguistics.
  * Gururangan et al. (2020) Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020.  Don’t stop pretraining: Adapt language models to domains and tasks.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8342–8360, Online. Association for Computational Linguistics.
  * He et al. (2020) Junxian He, Jiatao Gu, Jiajun Shen, and Marc’Aurelio Ranzato. 2020.  Revisiting self-training for neural sequence generation.  In _Proceedings of ICLR_.
  * Jia and Liang (2016) Robin Jia and Percy Liang. 2016.  Data recombination for neural semantic parsing.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 12–22, Berlin, Germany. Association for Computational Linguistics.
  * Jiang et al. (2020) Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. 2020.  SMART: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2177–2190, Online. Association for Computational Linguistics.
  * Johnson et al. (2019) Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019.  Billion-scale similarity search with GPUs.  _IEEE Transactions on Big Data_ , 7(3):535–547.
  * Karouzos et al. (2021) Constantinos Karouzos, Georgios Paraskevopoulos, and Alexandros Potamianos. 2021\.  UDALM: Unsupervised domain adaptation through language modeling.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2579–2590, Online. Association for Computational Linguistics.
  * Khandelwal et al. (2020) Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020\.  Generalization through Memorization: Nearest Neighbor Language Models.  In _International Conference on Learning Representations (ICLR)_.
  * Lee et al. (2021) Kenton Lee, Kelvin Guu, Luheng He, Timothy Dozat, and Hyung Won Chung. 2021.  Neural data augmentation via example extrapolation.  _ArXiv_ , abs/2102.01335.
  * Lewis et al. (2020a) Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettlemoyer. 2020a.  Pre-training via paraphrasing.  _ArXiv_ , abs/2006.15020.
  * Lewis et al. (2020b) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020b.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Lu et al. (2021) Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021\.  Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity.  _ArXiv_ , abs/2104.08786.
  * Mosbach et al. (2021) Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. 2021.  On the stability of fine-tuning bert: Misconceptions, explanations, and strong baselines.  _ArXiv_ , abs/2006.04884.
  * Narayan et al. (2018) Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018.  Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1797–1807, Brussels, Belgium. Association for Computational Linguistics.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.  fairseq: A fast, extensible toolkit for sequence modeling.  In _Proceedings of NAACL-HLT 2019: Demonstrations_.
  * Phang et al. (2018) Jason Phang, Thibault Févry, and Samuel R. Bowman. 2018.  Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks.  _ArXiv_ , abs/1811.01088.
  * Puri et al. (2020) Raul Puri, Ryan Spring, Mohammad Shoeybi, Mostofa Patwary, and Bryan Catanzaro. 2020\.  Training question answering models from synthetic data.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5811–5826, Online. Association for Computational Linguistics.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Ram et al. (2021) Ori Ram, Yuval Kirstain, Jonathan Berant, Amir Globerson, and Omer Levy. 2021.  Few-shot question answering by pretraining span selection.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3066–3079, Online. Association for Computational Linguistics.
  * Schick and Schütze (2021a) Timo Schick and Hinrich Schütze. 2021a.  Generating datasets with pretrained language models.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6943–6951, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Schick and Schütze (2021b) Timo Schick and Hinrich Schütze. 2021b.  It’s not just size that matters: Small language models are also few-shot learners.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2339–2352, Online. Association for Computational Linguistics.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Improving neural machine translation models with monolingual data.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 86–96, Berlin, Germany. Association for Computational Linguistics.
  * Sohn et al. (2020) Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. 2020.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  _arXiv preprint arXiv:2001.07685_.
  * Tam et al. (2021) Derek Tam, Rakesh R. Menon, Mohit Bansal, Shashank Srivastava, and Colin Raffel. 2021.  Improving and simplifying pattern exploiting training.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 4980–4991, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Tran et al. (2020) C. Tran, Y. Tang, Xian Li, and Jiatao Gu. 2020.  Cross-lingual retrieval for iterative self-supervised training.  _ArXiv_ , abs/2006.09526.
  * Vu et al. (2021) Tu Vu, Minh-Thang Luong, Quoc Le, Grady Simon, and Mohit Iyyer. 2021.  STraTA: Self-training with task augmentation for better few-shot learning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 5715–5731, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Vu et al. (2020) Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarella-Micke, Subhransu Maji, and Mohit Iyyer. 2020.  Exploring and predicting transferability across NLP tasks.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7882–7926, Online. Association for Computational Linguistics.
  * Wang et al. (2021) Sinong Wang, Han Fang, Madian Khabsa, Hanzi Mao, and Hao Ma. 2021.  Entailment as few-shot learner.  _ArXiv_ , abs/2104.14690.
  * Wang and Yang (2015) William Yang Wang and Diyi Yang. 2015.  That’s so annoying!!!: A lexical and frame-semantic embedding based data augmentation approach to automatic categorization of annoying behaviors using #petpeeve tweets.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 2557–2563, Lisbon, Portugal. Association for Computational Linguistics.
  * Xie et al. (2019) Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. 2019.  Unsupervised data augmentation for consistency training.  _arXiv preprint arXiv:1904.12848_.
  * Yang et al. (2020) Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji-Ping Wang, Chandra Bhagavatula, Yejin Choi, and Doug Downey. 2020\.  Generative data augmentation for commonsense reasoning.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1008–1025, Online. Association for Computational Linguistics.
  * Zhang et al. (2020) Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2020.  Pegasus: Pre-training with extracted gap-sentences for abstractive summarization.  _ArXiv_ , abs/1912.08777.
  * Zhang et al. (2019) Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q. Weinberger, and Yoav. Artzi. 2019\.  Revisiting few-sample bert fine-tuning.  _arXiv preprint arXiv:2006.05987_.