# Data Augmentation is Dead, Long Live Data Augmentation

###### Abstract

Textual data augmentation (DA) is a prolific field of study where novel
techniques to create artificial data are regularly proposed, and that has
demonstrated great efficiency on small data settings, at least for text
classification tasks. In this paper, we challenge those results, showing that
classical data augmentation is simply a way of performing better fine-tuning,
and that spending more time fine-tuning before applying data augmentation
negates its effect. This is a significant contribution as it answers several
questions that were left open in recent years, namely : which DA technique
performs best (all of them as long as they generate data close enough to the
training set as to not impair training) and why did DA show positive results
(facilitates training of network). We furthermore show that zero and few-shot
data generation via conversational agents such as ChatGPT or LLama2 can
increase performances, concluding that this form of data augmentation does
still work, even if classical methods do not.

Keywords: NLP, Classification, Data Augmentation, ChatGPT, Llama2

Data Augmentation is Dead, Long Live Data Augmentation

Frédéric Piedboeuf, Philippe Langlais
---
Université de Montréal, RALI
<EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

Data augmentation (DA) consists in generating artificial data points with the
hope of improving the training of a model. More specifically, interpretable
textual DA takes interest into whether new sentences can increase
performances, with the most common task for testing algorithms being text
classification. Research on the subject generally finds that DA provides a
great increase on small data settings Chen et al. (2021); Kumar et al. (2021),
a small increase in classification with medium datasets (up to 1000 examples)
Karimi et al. (2021); Liesting et al. (2021), and almost no increase on large
datasets Kobayashi (2018); Yang et al. (2020).

DA algorithms proposed in the literature have an important effect not only
because they guide practitioners towards one technique or the other, but also
because they contribute to the understanding of how neural networks learn.
Showing, for example, that a word substitution algorithm helps a pretrained
model like BERT has important implications about its inner working.

In this paper, we show that inadequate fine-tuning of the baselines (model
trained without DA) from previous studies lead to an overestimation of the
impact of textual DA. In fact, we show that by simply playing with the
patience parameter as well as occasionally adding label smoothing Szegedy et
al. (2015), we can raise the performance of the baseline classifier to be as
good as performances with DA, and that when the classifier is trained
correctly DA does not bring improvements. We further compare novel DA methods
using ChatGPT and Llama2 and show that the only thing that can consistently
increase classification performance is generating data akin to external data
(with zero or few-shot data generation), rather than generating new data from
the current training distribution (by paraphrasing or modifying a sentence
from the training set), as classical DA approaches do. Finally, we consider
more realistic approaches to DA, where we either don’t have access to clean
validation data, or where we adjust the training/validation split to better
reflect real-world conditions.

Overall, this paper covers several important contributions. The first and most
important of them lies in showing that classical data augmentation (DA methods
from before the advent of Very Large Language Models, or VLLMs) does not work
on textual classification, and we also answer why previous studies showed
positive results. These positive results have been a big question in recent
years, as nobody could explain with satisfaction why generated sentences
helped transformers learn better, the best hypothesis being that it brought
some kind of regularization to the network (Feng et al., 2021; Queiroz
Abonizio and Barbon Junior, 2020).

The second contribution is showing that by using VLLMs to generate novel
sentences (and not just paraphrase or modify existing ones), the performance
increases, but that this is still not as efficient as collecting and
annotating novel data. We furthermore analyze two strategies for generating
novel data, namely zero-shot data generation (using the task description) and
3-shot data generation (giving three examples to the VLLM), showing that in
most case the 3-shot strategy largely outperforms zero-shot generation.

The third contribution joins a small but growing literature comparing VLLMs in
showing that ChatGPT outperforms Llama2 (Liu et al., 2023; Plátek et al.,
2023; Guo et al., 2023). To our knowledge, we are the first to study this in
the context of data augmentation.

Finally, our last contribution is a questioning of the use of the validation
data in data augmentation for small data settings. DA research often supposes
the availability of clean validation samples, but its usage is not realistic
in most cases. With the splits used in those studies, one finds itself with a
few data points for training the model (generally less than 30), and a few
hundreds for validation. In this paper, we attempt to consider better
splitting and more realistic use of data.

The paper is organized as such. In Section 2, we go over the literature on
interpretable textual DA. Sections 3 and 4 explain the strategies, protocol,
and datasets used in this paper. Then Section 5 presents the results and
Section 6 go over analysis, including the evaluation of more realistic uses of
the validation data. We finally include a discussion about the meaning of our
work in Section 7.

## 2\. Related Work

We may categorize DA techniques in three broad families : word-level
operations, paraphrasing, and generative DA. Furthermore, we refer to DA
techniques that were developed before VLLMs as classical DA.

The first family creates variations of existing sentences by affecting
individual words. The most seminal technique of this category is EDA, which
considers four operations : word substitution, word deletion, word swapping,
and insertion of related vocabulary (Wei and Zou, 2019; Liesting et al.,
2021). For insertion and substitution, EDA uses WordNet to find synonyms of
words from the input sentences. Another simple but efficient word-level
technique is AEDA Karimi et al. (2021), where random punctuation is inserted
in between words of the sentence.

Other ways of replacing words have also been considered, such as the use of a
pre-trained neural network to predict masked words Kobayashi (2018), using
pre-trained embeddings to find words close in the embedded space Marivate and
Sefara (2020), or using BERT/BART conditioned on the class to predict masked
words or spans Wu et al. (2019); Kumar et al. (2021). These techniques are
generally easy to implement but bring little diversity to the dataset.

The second family of techniques acts at the sentence level, by considering the
whole sentence for creating paraphrases. The seminal technique representing
this family of techniques is Back-translation (BT), in which a sentence is
translated to another language and then back into English (Hayashi et al.,
2018; Yu et al., 2018; Corbeil and Ghadivel, 2020; Edunov et al., 2018). The
use of generative models for paraphrasing has also been considered, for
example by encoding and decoding a sentence through a VAE (Mesbah et al.,
2019; Yerukola et al., 2021; Nishizaki, 2017). Some other strategies that have
been considered are modifying syntactic trees Coulombe (2018), fine-tuning
BART on an in-domain corpus of paraphrases Okur et al. (2022), using an off
the shelf T5 model for paraphrasing (T5-Tapaco) Piedboeuf and Langlais (2023),
or using VLLMs such as ChatGPT for paraphrasing Fang et al. (2023).

Finally, generative methods aim to generate novel sentences from the same
distribution as the training data. With an ideal generative strategy, this is
equivalent to collecting new data, without the annotation cost associated to
the collection process. Early studies of these techniques used GPT-2 (Kumar et
al., 2020; Liu et al., 2020; Yang et al., 2020; Bayer et al., 2023) or VAEs
Qiu et al. (2020); Piedboeuf and Langlais (2022), but recently focus has
shifted to the use of VLLMs, first with GPT3 (Yoo et al., 2021; Sahu et al.,
2022), and then with ChatGPT for zero or small data generation (Møller et al.,
2023; Ubani et al., 2023; Shushkevich and Cardiff, 2023; Sharma and Feldman,
2023).

Some recent works pertinent to this paper are the research of Kim et al.
(2022); Zhu et al. (2023). Kim et al. (2022) question the training-validation
splitting methodology in semi-supervised learning, showing that it is more
efficient to fine-tune on augmented samples (created with DA) and use the
original training sentences as validation data, instead of using a classical
training/validation split. In Zhu et al. (2023), the authors note that weakly
supervised learning approaches are evaluated by assuming the availability of
clean validation samples, which is not often the case when working with small
data. They notably develop novel methods for testing training under more
realistic small data learning settings. In Section 6, we take inspiration of
those papers to define our new strategies.

## 3\. Data augmentation strategies

Kumar et al. (2021); Piedboeuf and Langlais (2023) compared several DA
methods, from which we select strategies that have been shown to be efficient.
Concretely, we test EDA, AEDA, CBERT, CBART, T5, BT, and three strategies with
VLLMs : paraphrasing, zero-shot generation, and k-shot generation. All
strategies are illustrated in Figure 1.

EDA (multiple operations on the words) and AEDA (insertion of punctuation) can
be implemented fairly simply. Following experiments and results from the
literature, we affect 10% of the words of a sentence in EDA, and use the
formula given in Karimi et al. (2021) to calculate the number of punctuation
signs to insert for AEDA.

CBERT and CBART both use the task of masked words prediction to generate new
words conditionally on the class. Concretely, we mask words, prepend the class
to the sentence (with a separation token), and fine-tune the model to predict
these masked words. For generating new sentences, we follow the same process.
The difference between CBERT and CBART is that CBART can predict spans of
words instead of just individual tokens, allowing it more flexibility in the
generated sentences.

Both BT and T5-Tapaco aim to produce paraphrases of the original sentences to
bring diversity to the training set. In BT, paraphrases are generated by
translating the sentence into a second language and then back into English,
and we use WMT111https://huggingface.co/facebook/wmt19-De-en with German as a
pivot language, which has shown good performances in the past Edunov et al.
(2018). T5-Tapaco makes use of T5-small-
Tapaco222https://huggingface.co/hetpandya/t5-small-tapaco, which is a T5 model
fine-tuned on the TaPaCo paraphrase corpus (Scherrer, 2020), allowing us to
directly generate paraphrases of the sentences.

In terms of VLLMs, we test three strategies, which are the three standards
strategies we could find in the literature : paraphrasing existing sentences,
zero-shot generation by giving the VLLMs a description of the task, and a
k-shot generation strategy where we give examples of the given class. We give
the exact prompts used in appendix A. Furthermore, we test the use of Llama2
(using LLama-2-13B-Chat) with the same prompts. We denote in this paper these
strategies by the letters P (paraphrase), ZS (zero-shot), and 3S (3-shot
generation). We opted on 3-shot for the K-shot generation strategy based on
results from exploratory experiments.

Figure 1: Strategies tested in this paper.

Finally, we implement three simple baselines. The first one consists in
training the network without the use of DA (denoted “Baseline”), and the
second one is an idealized strategy (denoted “‘Perfect”) where we fetch
additional unused data from the training set to act as generated data. This
gives an idea of the results obtainable should one collect and annotate data
instead of generating new data.

The last baseline is a strategy which we denote “Copy”, and which artificially
inflates the size of the training set by copying multiple times the original
data, without modification. If the only effect of DA is truly to help fine-
tune the network better, and the modification brought by the DA strategies are
not helpful for that, then the copy strategy should be just as efficient as
all other classical data augmentation strategies. We give the code with all
hyperparameters in the associated github.333Github will be made available upon
acceptance of the paper.

## 4\. Datasets and protocol

We use five datasets to test the various data augmentation strategies : SST-2,
Irony, IronyB, TREC6, and SNIPS. SST-2 (Socher et al., 2013) is a dataset of
movie review classification with the binary classes of positive or negative.
Irony and IronyB (Van Hee et al., 2018) are the binary and multiclass version
of an Irony detection dataset. In the binary task (Irony), one must detect if
tweets are ironic or not, and in the multiclass version one must as well
determine which type of Irony the tweet represents, if the tweet is ironic
(between polarity clash, situational irony, and other irony). TREC6 (Li and
Roth, 2002) is a task of question classification, where questions must be
separated into six classes (abbreviation, description, entities, human beings,
locations, and numeric values). Finally, SNIPS Coucke et al. (2018) is a
dataset of intent classification, where short commands have to be classified
into different intent (PlayMusic, GetWeather, RateBook, SearchScreeningEvent,
SearchCreativeWork, AddToPlaylist, BookRestaurant). Some characteristics of
the datasets are available in Table 1.

Table 1: Characteristics of the tasks tackled in this study. The length of the sentences is defined by the number of white-space separated tokens. Name | SST2 | Irony | IronyB | TREC6 | SNIPS
---|---|---|---|---|---
|classes| | 2 | 2 | 4 | 6 | 7
|train| | 6920 | 2683 | 2681 | 5452 | 13084
len. sents. | 19.3 | 13.7 | 13.7 | 10.2 | 9.3
|val| | 872 | 460 | 460 | 500 | 700
|test| | 1821 | 3834 | 3832 | 492 | 700

To define our training protocol, we first look at the literature to understand
what is usually done. Piedboeuf and Langlais (2023) use variable patience for
fine-tuning, depending on the dataset and dataset size, between 5 and 20
epochs, and Kumar et al. (2021) use 100 epochs of warmup followed by 8 epochs
of fine-tuning from which they select the best model. Wei and Zou (2019) use
early stopping with patience of 3 epochs on CNN and RNNs.

In contrast, we use a patience parameter of 50, using the validation set to
find when to stop, and we fine-tune the learning rate and the label smoothing
(a mean of introducing noise to the labels for regularization) before applying
data augmentation, using grid search. We define here the patience as the
number of epochs during which the validation performance doesn’t increase
before we stop the training process, and we then select the best model from
previous epochs. We report accuracy for binary tasks and the macro-F1 for
multiclass tasks.

Our focus in this paper is data augmentation for small data learning, for
which we use starting size of 10 and 20, but we also report results for medium
training sizes (500 and 1000) in Appendix B, a setting which is commonly
explored in the literature. When subsampling to create the training set, we
make sure to choose an equal number of data points for each class, by going
over the fixed dataset size if needed (for example, the actual dataset size
for TREC6 and starting size of 10 is 12; two sentences per class).

The final parameter is the ratio of generated-to-genuine examples. There seems
to be no strict rules about this ratio, but we found in both the literature
and experiments that smaller sizes of datasets benefited more from larger
ratios, while larger starting sizes did not show improvements with ratios
larger than one. Therefore, we use a ratio of 10 for the small data settings
and of one for the larger dataset sizes.444Based on our main results, we can
hypothesize that this is simply due to pretrained models needing more time to
be fine-tuned correctly on smaller datasets than medium or large ones, and the
larger ratio leads to a larger number of batches, meaning more training time.

Something worth noting is that all of these protocols (including ours) use the
validation data to fine-tune both the DA algorithm and the model to train.
This is somewhat questionable, as keeping only 10 or 20 data points out of a
few hundreds might not be the best way to utilize the data. While we keep this
protocol in the main section to stay in accordance with the DA literature, we
propose more realistic evaluation protocol of DA in Section 6.

The norm in data augmentation literature is to give the average and standard
deviation over a fixed number of runs. However, to confirm whether the
occasional improvements are due to randomness of the model or the effect of
DA, we include in this paper statistical tests.

Table 3: Average metric over 15 runs for the training set sizes of 10 (left) and 20 (right) with a ratio of 1. We report accuracy for binary tasks and macro-f1 for multiclass ones. STDs are between 0.6 and 3.0, depending on the dataset. Stars represent results for which the difference with the baseline was found to be statistically significant. | SST2 | Irony | IronyB | Trec6 | SNIPS | Average
---|---|---|---|---|---|---
Baseline | 62.4/67.5 | 54.1/61.9 | 25.8/29.8 | 34.4/43.3 | 78.5/80.4 | 51.1/56.6
Perfect | 82.3*/85.8* | 62.4*/58.7 | 19.3/33.7 | 72.5*/80.6* | 94.0*/94.9* | 66.1/70.7
Copy | 61.0/68.1 | 54.4/57.1 |  25.5/30.3 | 36.4/49.3* | 77.7/83.4* | 51.0/57.6
EDA | 63.4/69.0 | 54.9/58.9 | 23.8/30.1 | 35.8/48.0* | 76.6/83.2* | 50.9/57.8
AEDA | 62.4/67.1 | 55.4/56.9 | 23.4/27.4 | 33.2/45.7 | 78.0/84.0* | 50.5/56.2
BT | 60.2/66.8 | 55.1/58.0 | 24.7/27.9 | 29.1/51.5* | 75.0/82.6* | 48.8/57.4
CBERT | 62.5/65.4 | 56.3*/58.5 | 23.2/28.3 | 36.4/44.5 | 76.3/83.4* | 50.9/56.0
CBART | 62.5/65.6 | 54.5/57.7 | 23.6/29.0 | 36.9/47.2 | 77.0/84.5* | 50.9/56.8
T5 | 64.1/67.4 | 56.3/58.3 | 24.1/28.1 | 34.3/48.5* | 76.7/84.1* | 51.1/57.2
GPT3.5-P | 65.0*/68.3 |  56.6*/57.5 | 22.7/28.3 | 34.0/40.5 | 80.2/84.7* | 51.7/55.9
GPT3.5-ZS | 82.0*/78.7* | 56.4*/56.1 | 23.5/24.6 | 40.0*/48.1* | 84.8*/88.7* | 57.3/59.2
GPT3.5-3S |  87.7*/76.1* | 49.2/53.6 | 20.9/24.6 |  58.4*/63.9* |  87.2*/89.6* |  60.7/61.5
Llama2-P | 66.3*/68.6 | 55.3/57.1 | 24.6/24.6 | 33.4/41.6 | 78.1/82.8* | 51.6/55.0
Llama2-ZS | 74.8*/74.0* | 54.3/57.2 | 22.2/24.1 | 37.3/47.3 | 78.1/84.1* | 53.4/57.3
Llama2-3S | 63.3/64.4 | 54.1/54.8 | 21.0/27.2 | 48.6*/55.8* | 83.3*/88.2* | 54.0/58.1

For each starting size (10, 20) and dataset, we perform a two-tailed paired
t-test, comparing the algorithms to each other555We choose to do this instead
of a Hotelling’s T-square to take into account the fact the DA might work on
some datasets but not others. For example, one would intuitively expect
synonym replacement to work better on sentiment classification than on Irony
detection.. As is standard, we assess the two distributions to be different if
the p-value is lower than 0.05.

## 5\. Results

We first show the inefficiency of the fine-tuning protocol in past DA papers
by reporting the results they obtain without data augmentation and comparing
it to our results, without data augmentation as well. We attempt to replicate
the starting sizes, changing only the fine-tuning protocol to see the
difference a longer patience, label smoothing, and better fine-tuning does.

| Reported | Ours
---|---|---
Kumar + Ubani SST-2 | 52.9 | 60.6
Kumar + Ubani SNIPS | 48.6 | 66.8
Kumar + Uabni TREC6 | 79.4 | 91.6
Piedboeuf SST-2 | 87.7 | 87.7
Piedboeuf Irony | 65.6 | 67.0
Piedboeuf IronyB | 42.4 | 43.9
Piedboeuf TREC6 | 81.0 | 84.5
Table 2: Fine-tuning comparison from the literature and in this paper. All
results are from training BERT without data augmentation, showing the
difference made by using a longer patience as well as label smoothing.

Table 2 presents the results of our experiments compared to results from the
literature. As we can see, having a larger patience parameter and applying
label smoothing increase the performance significantly, gaining between 2 and
10%. This is the core of our argument, as our hypothesis is that the gains
observed in previous studies stems only from data augmentation, giving more
time for the model to learn correctly. Furthermore, if the DA strategies
involve simple transformations of the sentences, it is likely that BERT is not
learning new information from these transformations, and that the same result
will be achieved with the Copy strategy.

We now turn to DA algorithms, showing that using the updated training protocol
does not result in higher gain, and that indeed the supposed gains from DA
that were shown in the literature can more likely be explained by inadequate
training of the classifier. Results are presented in Table 3.

For the small data setting, we notice that, on average, only two strategies
really outperforms the baseline : zero-shot and 3-shot generation, both using
VLLMs. Of these two strategies, the 3-shot generation outperforms the zero-
shot one, but not on all datasets. We explore this phenomenon in Section 6.2.

An important thing to note is that while some algorithms perform statistically
better than the baseline at times, so does the “Copy” strategy, which is a
simple duplication of the data. As the duplication of data should bring no new
information, this suggests that the training protocol was suboptimal and that
the difference between the baseline and the DA results is due to this rather
than a real effect of the DA algorithm. One avenue that could be explored to
bridge the gap would be to use sampling with replacement while fine-tuning
BERT, which should be equivalent to duplication of data666There would still be
minor differences as sampling with replacement would not represent each sample
equally, but this difference should reasonably not have a large impact.. The
results for medium sizes (500 and 1000) are presented in Appendix B, and are
concordant with more recent literature showing a lack of improvement on such a
setting.

We give in Figure 4 the results of the statistical tests. To simplify reading,
we regroup together all tests between the same algorithms, meaning all tests
between algorithm A and algorithm B are compiled together. Each entry in the
heatmap is therefore composed of 10 t-tests (5 datasets * 2 starting sizes),
and we report the percentage of t-tests for which distributions were found to
be statistically different, counting only the entries for which the row
algorithm performs better than the column algorithm. This gives, for each row,
the direct measure of how many times it beats the algorithm in the column by a
statistically significant margin.777We provide in the code the full saved
results for all experiments, as well as the code necessary to run all
statistical tests.

|  Baseline |  Perfect |  Copy |  EDA |  AEDA |  BT |  CBERT |  CBART |  T5 |  GPT3.5-P |  GPT3.5-ZS |  GPT3.5-3S |  Llama2-P |  Llama2-ZS |  Llama2-3S
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Baseline |  0 |  20 |  10 |  10 |  10 |  30 |  10 |  10 |  10 |  20 |  20 |  40 |  10 |  30 |  30
Perfect |  70 |  0 |  70 |  70 |  70 |  70 |  70 |  70 |  70 |  70 |  70 |  80 |  80 |  80 |  80
Copy |  20 |  10 |  0 |  0 |  10 |  20 |  10 |  0 |  0 |  30 |  10 |  40 |  20 |  20 |  30
EDA |  20 |  10 |  20 |  0 |  10 |  20 |  0 |  0 |  0 |  10 |  20 |  40 |  20 |  10 |  20
AEDA |  10 |  10 |  0 |  0 |  0 |  20 |  0 |  0 |  0 |  10 |  0 |  30 |  20 |  0 |  0
BT |  20 |  10 |  0 |  0 |  10 |  0 |  10 |  10 |  0 |  10 |  10 |  30 |  10 |  10 |  20
CBERT |  20 |  10 |  10 |  0 |  10 |  10 |  0 |  10 |  0 |  10 |  0 |  20 |  10 |  0 |  20
CBART |  10 |  10 |  0 |  0 |  0 |  10 |  0 |  0 |  0 |  10 |  0 |  30 |  10 |  10 |  10
T5 |  20 |  10 |  0 |  0 |  0 |  20 |  10 |  0 |  0 |  10 |  0 |  40 |  10 |  10 |  20
GPT3.5-P |  30 |  10 |  10 |  20 |  10 |  40 |  20 |  20 |  10 |  0 |  10 |  30 |  10 |  20 |  20
GPT3.5-ZS |  70 |  10 |  60 |  50 |  50 |  50 |  60 |  50 |  50 |  60 |  0 |  40 |  60 |  50 |  30
GPT3.5-3S |  60 |  10 |  60 |  60 |  60 |  60 |  60 |  60 |  60 |  60 |  40 |  0 |  60 |  60 |  50
Llama2-P |  20 |  10 |  10 |  0 |  10 |  20 |  20 |  10 |  0 |  0 |  0 |  30 |  0 |  0 |  10
Llama2-ZS |  30 |  10 |  20 |  20 |  20 |  30 |  20 |  20 |  20 |  30 |  0 |  20 |  40 |  0 |  30
Llama2-3S |  40 |  0 |  40 |  40 |  40 |  40 |  40 |  40 |  40 |  40 |  20 |  20 |  40 |  40 |  0
Figure 2: Percent of times the row algorithm performs statistically better
than the column algorithm, with a p-value threshold of 0.05 and using a two-
tails paired t-test, and across the two small data settings (10/20).

From the statistical tests, we note that only three methods outperform the
others by a significant and consistent margin : the perfect strategy, in which
we simulate collecting and annotating external data, and the zero/k-shot
generation with ChatGPT. All others perform equivalently and do not beat the
baseline most of the time, or at least not by a significant margin. This
suggests that the slight random variations we observe, which could be taken as
a sign that some algorithm performs better than other, are due to randomness
of the network and not a superior performance.

## 6\. Analysis

The preceding section evaluated data augmentation algorithms using a standard
training protocol. While our main result is that the efficiency of DA is
generally over-estimated, this does not imply data augmentation is never
useful. In this section, we aim to test more realistic settings with the
validation data, inspired by Kim et al. (2022); Zhu et al. (2023), who study
the same problems but in different settings (semi-supervised learning and
weakly supervised learning) and then discuss what our results mean for textual
DA. While Kim et al. (2022); Zhu et al. (2023) findings are relevant to ours,
our interest in the results differ. In both papers, the authors attempt to
find the best way to use the available data. In our case, we test several
settings we think are more realistic, but the point of interest is to find if
data augmentation is useful when there is little or no validation data for
fine-tuning the classifier. Indeed, there is little doubt that using
validation data as training data will be more efficient than the current
accepted setting.

### 6.1. Evaluating small data learning with more realistic use of data

In academic papers, DA is often evaluated with the assumption that we have
access to clean validation data, often in larger quantity than the training
data. This validation data is used to fine-tune both the DA algorithm and the
classifier, which results in unrealistic settings for practitioners, who may
not have access to validation data or want to use their data in better ways.
In this section, we evaluate the use of DA in various low-data settings in
which we redefine the training and validation split, using different
strategies.

We design experiments using two setups, electing to test the setups with the
starting size of 10. This is motivated by the long time it takes to fine-tune
and run all algorithms888Depending on the final training set size and the DA
algorithms, running one experiment takes between 30 and 75 minutes, excluding
the grid-search for the hyperparameters, on a RTX 3090., and the fact that the
efficiency of DA was the same for both starting sizes, making duplication of
the experiments unnecessary. An open question is how these settings would work
if we had more initial training data, but we leave this for future work.

Firstly, we assume the validation data is not available (noVal), and we only
have training data. In this setting, we train for a random number of epochs
between 50 and 150, which is the range for which our models performed best in
small data learning.

Secondly, we assume the presence of validation data but redefine the
training/validation split, by either 1- using it all as training data and
training for a random number of epochs between 4 and 8 (valAsTrain), or 2-
keeping some of it as validation data (10, 250) to fine-tune both DA
algorithms and the classifier (10 val/250 val).

Overall, this gives us four settings to test, illustrated in Figure 3. We
select the following algorithms : Copy, Perfect, EDA, AEDA, BT, CBERT, BART,
T5, and GPT3.5-ZS. We use a limited selection for efficiency and because the
goal of these experiments is simply to establish whether DA helps at all,
rather than which one works best.

Figure 3: Graphic representation of the four settings we test for data
augmentation on small data learning. Blue represent the original training set,
purple the validation set, and yellow, the test set.

Finally, to keep in line with other data augmentation studies, we use a ratio
of 10 when the final training size is small (noVal) and of 1 for the other
three settings.

| noVal | valAsTrain | 10 val | 250 val
---|---|---|---|---
Baseline | 47.9 | 75.8 | 64.6 | 64.7
Perfect | 68.8 | 82.4 | 69.4 | 69.9
Copy | 49.6 | 78.6 | 66.3 | 65.8
EDA | 49.1 | 77.8 | 66.0 | 64.9
AEDA | 49.4 | 77.6 | 66.0 | 66.3
BT | 44.8 | 75.9 | 64.6 | 65.0
CBERT | 49.7 | 76.8 | 65.3 | 63.9
BART | 50.6 | 76.6 | 65.3 | 63.9
T5Par | 49.4 | 77.2 | 65.3 | 64.6
GPT3.5-ZS | 55.9 | 76.9 | 65.6 | 64.4
GPT3.5-3S | 57.2 | 71.6 | 67.2 | 64.8
Table 3: Average metrics over the five datasets for different settings and DA
strategies. The best result for each setting is in bold.

Results are presented in table 3, and several important observations can be
made. First, in almost all cases and contrary to the main results of this
paper, data augmentation reveals itself to be useful. However, the fact that
the “Copy” strategy often outperforms the other reinforces our belief that the
use of data augmentation is simply to facilitate fine-tuning the network.

Second, there is no significant difference in performance between having 10
and 250 data points as validation data, with the rest as training set. This
could be related to diminished increase in model performance as dataset size
augment, or point that there is a trade-off in validation/training size and
that a validation set of 10 data points is too small to be of use.

Third, we confirm here that methods adding something akin to external data
(GPT3.5-ZS and GPT3.5-2S) are the most useful techniques, but only on small
training sizes. On larger training sizes, they don’t bring any improvement
whatsoever to the model, most likely due to the general inefficiency of data
augmentation in this setting.999This is concordant with the literature and our
own experiments. We also note that while the performance of GPT3.5-ZS is
higher than GPT3.5-3S for two settings, this is due exclusively to its poor
performance on the Irony datasets, which we discuss in Section 6.2. For the
other three datasets, GPT3.5-3S outperforms GPT3.5-ZS by a large margin.

Finally, it seems that the best strategy overall when the total amount of
available data is only a few hundred is to use all of them as training data,
with the Copy strategy. This outperforms by a large margin the strategies
where we keep some data as validation data, most likely because these
strategies overfit. As mentioned, however, there is a large gap between 10 and
250 data and it is possible that some amount of data in the middle would
perform best. We leave this for future work.

### 6.2. Analysis of the VLLMs

Our main results showed that in a small data setting, 3-shot generation was
often more efficient than zero-shot generation, by as much as 18% in some
cases. However, this does not hold for the two irony datasets, for which zero-
shot outperforms 3-shot. In this section, we analyze the output of the
algorithms, in the hope of understanding better why some prompt mechanisms
work well on some datasets but not on others.

We first show examples of generated sentences in Tables 4 and 5. For the sake
of space, we only show generated sentences for the classes of SST-2 and Irony,
but we give all sentences generated by ChatGPT in the associated Github.

Negative | Dataset |  apparently reassembled from the cutting-room floor of any given daytime soap.
---|---|---
GPT3.5-P |  It appears to have been pieced together from discarded scenes of any typical daytime soap opera.
GPT3.5-ZS |  The film was a major disappointment, lacking any coherent plot or engaging characters.
GPT3.5-3S |  It’s a shame that such a promising concept was wasted on a poorly executed film.
Positive | Dataset |  béart and berling are both superb , while huppert … is magnificent.
GPT3.5 - P |  Both Béart and Berling deliver exceptional performances, while Huppert… she is truly magnificent.
GPT3.5-ZS |  This film is a refreshing breath of fresh air, offering a unique and original take on familiar themes.
GPT3.5-3S |  This film is a delightful and heartwarming celebration of friendship, full of laughter and touching moments.
Table 4: Examples of generated sentences for the SST-2 dataset. Non-Ironic | Dataset |  Couldn’t have asked for a better night.
---|---|---
GPT3.5-P |  Could not have hoped for a more perfect evening.
GPT3.5-ZS |  Wow, I absolutely love it when my phone battery dies in the middle of an important call. Just what I needed!
GPT3.5-3S |  Enjoying a relaxing day at the beach with clear skies and warm sunshine. #PerfectDay
Ironic | Dataset |  work should be fun today :unamused_face:
GPT3.5 - P |  Today at work is expected to be unenjoyable :unamused_face:
GPT3.5-ZS |  Oh, how wonderful! Another day of unexpected surprises and setbacks. It’s like the universe’s way of keeping me on my toes!
GPT3.5-3S |  Declared I’d be more organized and lost my keys for the umpteenth time. I’m the queen of tidiness! :key::mag: #OrganizationQueen
Table 5: Examples of generated sentences for the Irony dataset.

By observing generated sentences, we see that ChatGPT has difficulty
understanding the concept of Irony, as well as sticking to the more familiar
language of Twitter. As such, adding more sentences doesn’t help because the
new sentences are too far from the training distribution to bring valuable
information. Nevertheless, the 3-shot strategy does seem to bring the
generated sentences closer to the training distribution, as we can at least
observe that the new sentences contain hashtags and emojis.

## 7\. Discussion

In this paper, we test data augmentation for sentence classification and show
that in both medium and small data learning, performances of DA had been
overestimated by the use of inadequate training protocols. Furthermore, while
we have shown that the classical methods of DA are inefficient, we have also
demonstrated that simulating external data collection with VLLMs does improve
the performance, and future work should therefore focus on this rather than
techniques modifying the sentences.

We want to emphasize that our results are only looking at balanced
classification task, and furthermore only at short text classification using
BERT as a classifier. While this seems a small field of study, it is one of
the most popular in textual DA literature, making our findings significant.

Future work should also evaluate whether the protocols are adapted for fields
close to interpretable DA. Notably non-interpretable DA, or DA for unbalanced
data, might still be an efficient mean of increasing performances. Other
textual tasks might also still benefit from DA, due to different goals. As an
example, data augmentation for the task of keyphrase generation (Ray Chowdhury
et al., 2022; Garg et al., 2023) aims to encourage the network to generate
more keyphrases that are absent from the input. It is more than plausible that
data augmentation is useful for this field. As another example, data
augmentation is sometime used to improve explainability (Chen and Ji, 2020;
Ansari et al., 2023), and protocol should therefore be checked thoroughly in
independent fields to ensure that it works as intended.

## 8\. Conclusion

In this paper, we look at the protocol surrounding data augmentation, and
demonstrate that data augmentation is mainly used for easier fine-tuning, an
effect that can be replicated by simply giving more care to the fine-tuning of
the network, or by using a simple strategy where we duplicate the data. We
also consider other, more realistic applications of DA, where validation data
is either not available or used as training data, instead of keeping hundreds
of examples for validation and only a dozen for training the model.

These findings are important as they answer many questions in textual DA which
have been standing for years, namely which data augmentation algorithms is
best (they all perform similarly except for data generation), why does it help
(gives more time to the network to learn), and what makes a generated sentence
informative (they don’t bring contradictory information to the network). As
noted, data generation remains one of the best way to perform data
augmentation, if one can get the VLLMs to generate data adequate for the
training distribution.

Future work on textual DA should explore ways to bridge this gap and generate
data which is more diverse as well as correspond more closely to the real
task. Another pertinent exploration that remains to be done is to thoroughly
study the difference between Llama2 and the GPT family. In this paper, we used
Llama2-13B-Chat and concluded that it was less efficient than the use of
ChatGPT, but it remains to be seen if this is true for all sizes of the model.
We also used default parameters of ChatGPT, to stay in line with Piedboeuf and
Langlais (2023). It is very probable however that other configurations are
more efficient, and this should be studied as well. Finally, we attempted to
evaluate DA in more realistic settings, by redefining training/validation
split, but the testing should be taken further. For example, it is very likely
that a jackknife approach would be a better evaluation method for DA in small
data setting, and this should be considered in future research on data
augmentation.

## 9\. Limitations

As stated on the paper, this study as a limited, although important, scope of
study. We didn’t look at the impact of data augmentation on other languages
than English, or other textual task, and while we performed fine-tuning to the
best of our knowledge, it is possible that some combinations of
hyperparameters that were not explored change the conclusions here. We also
omitted an ablation study due to time limitation, but this would provide more
insight into the mechanism of training BERT. Other limitations about the
models are the limited hyperparameters explored for ChatGPT, the limited
models used for Llama2, and the fact that we did not check the results with
other classifiers than BERT.

## 10\. Bibliographical References

##

  * Ansari et al. (2023) Gunjan Ansari, Parmeet Kaur, and Chandni Saxena. 2023.  Data Augmentation for Improving Explainability of Hate Speech Detection.  _Arabian Journal for Science and Engineering_ , pages 1–13.  Publisher: Springer.
  * Bayer et al. (2023) Markus Bayer, Marc-André Kaufhold, Björn Buchhold, Marcel Keller, Jörg Dallmeyer, and Christian Reuter. 2023.  Data augmentation in natural language processing: a novel text generation approach for long and short text classifiers.  _International journal of machine learning and cybernetics_ , 14(1):135–150.  Publisher: Springer.
  * Chen and Ji (2020) Hanjie Chen and Yangfeng Ji. 2020.  Improving the Explainability of Neural Sentiment Classifiers via Data Augmentation.  ArXiv:1909.04225 [cs].
  * Chen et al. (2021) Jiaao Chen, Derek Tam, Colin Raffel, Mohit Bansal, and Diyi Yang. 2021.  An Empirical Survey of Data Augmentation for Limited Data Learning in NLP.  _arXiv:2106.07499 [cs]_.  ArXiv: 2106.07499.
  * Corbeil and Ghadivel (2020) Jean-Philippe Corbeil and Hadi Abdi Ghadivel. 2020.  BET: A Backtranslation Approach for Easy Data Augmentation in Transformer-based Paraphrase Identification Context.  ArXiv:2009.12452 [cs].
  * Coucke et al. (2018) Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, Maël Primet, and Joseph Dureau. 2018.  Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces.  ArXiv:1805.10190 [cs].
  * Coulombe (2018) Claude Coulombe. 2018.  Text Data Augmentation Made Simple By Leveraging NLP Cloud APIs.  ArXiv:1812.04718 [cs].
  * Edunov et al. (2018) Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018.  Understanding Back-Translation at Scale.  ArXiv:1808.09381 [cs].
  * Fang et al. (2023) Yihao Fang, Xianzhi Li, Stephen W. Thomas, and Xiaodan Zhu. 2023.  ChatGPT as Data Augmentation for Compositional Generalization: A Case Study in Open Intent Detection.  ArXiv:2308.13517 [cs].
  * Feng et al. (2021) Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy. 2021.  A Survey of Data Augmentation Approaches for NLP.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 968–988, Online. Association for Computational Linguistics.
  * Garg et al. (2023) Krishna Garg, Jishnu Ray Chowdhury, and Cornelia Caragea. 2023.  Data Augmentation for Low-Resource Keyphrase Generation.  ArXiv:2305.17968 [cs].
  * Guo et al. (2023) Yue Guo, Zian Xu, and Yi Yang. 2023.  Is ChatGPT a Financial Expert? Evaluating Language Models on Financial Natural Language Processing.  ArXiv:2310.12664 [cs].
  * Hayashi et al. (2018) T. Hayashi, S. Watanabe, Y. Zhang, T. Toda, T. Hori, R. Astudillo, and K. Takeda. 2018.  Back-Translation-Style Data Augmentation for end-to-end ASR.  In _2018 IEEE Spoken Language Technology Workshop (SLT)_ , pages 426–433.
  * Karimi et al. (2021) Akbar Karimi, Leonardo Rossi, and Andrea Prati. 2021.  AEDA: An Easier Data Augmentation Technique for Text Classification.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 2748–2754, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Kim et al. (2022) Hazel H. Kim, Daecheol Woo, Seong Joon Oh, Jeong-Won Cha, and Yo-Sub Han. 2022.  ALP: Data Augmentation Using Lexicalized PCFGs for Few-Shot Text Classification.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 36(10):10894–10902.  Number: 10.
  * Kobayashi (2018) Sosuke Kobayashi. 2018.  Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 452–457, New Orleans, Louisiana. Association for Computational Linguistics.
  * Kumar et al. (2020) Varun Kumar, Ashutosh Choudhary, and Eunah Cho. 2020.  Data Augmentation using Pre-trained Transformer Models.  In _Proceedings of the 2nd Workshop on Life-long Learning for Spoken Language Systems_ , pages 18–26, Suzhou, China. Association for Computational Linguistics.
  * Kumar et al. (2021) Varun Kumar, Ashutosh Choudhary, and Eunah Cho. 2021.  Data Augmentation using Pre-trained Transformer Models.  ArXiv:2003.02245 [cs].
  * Li and Roth (2002) Xin Li and Dan Roth. 2002.  Learning question classifiers.  In _Proceedings of the 19th international conference on Computational linguistics -_ , volume 1, pages 1–7, Taipei, Taiwan. Association for Computational Linguistics.
  * Liesting et al. (2021) Tomas Liesting, Flavius Frasincar, and Maria Mihaela Truşcă. 2021.  Data augmentation in a hybrid approach for aspect-based sentiment analysis.  In _Proceedings of the 36th Annual ACM Symposium on Applied Computing_ , SAC ’21, pages 828–835, New York, NY, USA. Association for Computing Machinery.
  * Liu et al. (2020) Ruibo Liu, Guangxuan Xu, Chenyan Jia, Weicheng Ma, Lili Wang, and Soroush Vosoughi. 2020.  Data Boost: Text Data Augmentation Through Reinforcement Learning Guided Conditional Generation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9031–9041, Online. Association for Computational Linguistics.
  * Liu et al. (2023) Zhengliang Liu, Yiwei Li, Peng Shu, Aoxiao Zhong, Longtao Yang, Chao Ju, Zihao Wu, Chong Ma, Jie Luo, Cheng Chen, Sekeun Kim, Jiang Hu, Haixing Dai, Lin Zhao, Dajiang Zhu, Jun Liu, Wei Liu, Dinggang Shen, Tianming Liu, Quanzheng Li, and Xiang Li. 2023.  Radiology-Llama2: Best-in-Class Large Language Model for Radiology.  ArXiv:2309.06419 [cs].
  * Marivate and Sefara (2020) Vukosi Marivate and Tshephisho Sefara. 2020.  Improving Short Text Classification Through Global Augmentation Methods.  In _Machine Learning and Knowledge Extraction_ , Lecture Notes in Computer Science, pages 385–399, Cham. Springer International Publishing.
  * Mesbah et al. (2019) Sepideh Mesbah, Jie Yang, Robert-Jan Sips, Manuel Valle Torre, Christoph Lofi, Alessandro Bozzon, and Geert-Jan Houben. 2019.  Training Data Augmentation for Detecting Adverse Drug Reactions in User-Generated Content.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2349–2359, Hong Kong, China. Association for Computational Linguistics.
  * Møller et al. (2023) Anders Giovanni Møller, Jacob Aarup Dalsgaard, Arianna Pera, and Luca Maria Aiello. 2023.  Is a prompt and a few samples all you need? Using GPT-4 for data augmentation in low-resource classification tasks.  ArXiv:2304.13861 [physics].
  * Nishizaki (2017) H. Nishizaki. 2017.  Data augmentation and feature extraction using variational autoencoder for acoustic modeling.  In _2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)_ , pages 1222–1227.
  * Okur et al. (2022) Eda Okur, Saurav Sahay, and Lama Nachman. 2022.  Data Augmentation with Paraphrase Generation and Entity Extraction for Multimodal Dialogue System.  ArXiv:2205.04006 [cs].
  * Piedboeuf and Langlais (2022) Frédéric Piedboeuf and Philippe Langlais. 2022.  Effective Data Augmentation for Sentence Classification Using One VAE per Class.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 3454–3464, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Piedboeuf and Langlais (2023) Frédéric Piedboeuf and Philippe Langlais. 2023.  Is ChatGPT the ultimate Data Augmentation Algorithm.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_.
  * Plátek et al. (2023) Ondřej Plátek, Vojtěch Hudeček, Patricia Schmidtová, Mateusz Lango, and Ondřej Dušek. 2023.  Three Ways of Using Large Language Models to Evaluate Chat.  ArXiv:2308.06502 [cs].
  * Qiu et al. (2020) Siyuan Qiu, Binxia Xu, Jie Zhang, Yafang Wang, Xiaoyu Shen, Gerard de Melo, Chong Long, and Xiaolong Li. 2020.  EasyAug: An Automatic Textual Data Augmentation Platform for Classification Tasks.  In _Companion Proceedings of the Web Conference 2020_ , pages 249–252. Association for Computing Machinery, New York, NY, USA.
  * Queiroz Abonizio and Barbon Junior (2020) Hugo Queiroz Abonizio and Sylvio Barbon Junior. 2020.  Pre-trained Data Augmentation for Text Classification.  In _Intelligent Systems_ , Lecture Notes in Computer Science, pages 551–565, Cham. Springer International Publishing.
  * Ray Chowdhury et al. (2022) Jishnu Ray Chowdhury, Seo Yeon Park, Tuhin Kundu, and Cornelia Caragea. 2022.  KPDROP: Improving Absent Keyphrase Generation.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 4853–4870, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Sahu et al. (2022) Gaurav Sahu, Pau Rodriguez, Issam H. Laradji, Parmida Atighehchian, David Vazquez, and Dzmitry Bahdanau. 2022.  Data Augmentation for Intent Classification with Off-the-shelf Large Language Models.  ArXiv:2204.01959 [cs].
  * Scherrer (2020) Yves Scherrer. 2020.  TaPaCo: A Corpus of Sentential Paraphrases for 73 Languages.
  * Sharma and Feldman (2023) Ashwyn Sharma and David I Feldman. 2023.  Team Cadence at MEDIQA-Sum 2023: Using ChatGPT as a Data Augmentation Tool for Classifying Clinical Dialogue.  CLEF.
  * Shushkevich and Cardiff (2023) Elena Shushkevich and John Cardiff. 2023.  Tudublin at CheckThat! 2023: Chatgpt for data augmentation.  _Working Notes of CLEF_.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013.  Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics.
  * Szegedy et al. (2015) Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. 2015.  Rethinking the Inception Architecture for Computer Vision.  ArXiv:1512.00567 [cs].
  * Ubani et al. (2023) Solomon Ubani, Suleyman Olcay Polat, and Rodney Nielsen. 2023.  ZeroShotDataAug: Generating and Augmenting Training Data with ChatGPT.  ArXiv:2304.14334 [cs].
  * Van Hee et al. (2018) Cynthia Van Hee, Els Lefever, and Veronique Hoste. 2018.  SemEval-2018 Task 3: Irony Detection in English Tweets.  In _Proceedings of The 12th International Workshop on Semantic Evaluation_ , pages 39–50, New Orleans, Louisiana. Association for Computational Linguistics.
  * Wei and Zou (2019) Jason Wei and Kai Zou. 2019.  EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 6382–6388, Hong Kong, China. Association for Computational Linguistics.
  * Wu et al. (2019) Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, and Songlin Hu. 2019.  Conditional BERT Contextual Augmentation.  In _Computational Science – ICCS 2019_ , Lecture Notes in Computer Science, pages 84–95, Cham. Springer International Publishing.
  * Yang et al. (2020) Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji-Ping Wang, Chandra Bhagavatula, Yejin Choi, and Doug Downey. 2020.  Generative Data Augmentation for Commonsense Reasoning.  _arXiv:2004.11546 [cs]_.  ArXiv: 2004.11546.
  * Yerukola et al. (2021) Akhila Yerukola, Mason Bretan, and Hongxia Jin. 2021.  Data Augmentation for Voice-Assistant NLU using BERT-based Interchangeable Rephrase.  _arXiv:2104.08268 [cs]_.  ArXiv: 2104.08268.
  * Yoo et al. (2021) Kang Min Yoo, Dongju Park, Jaewook Kang, Sang-Woo Lee, and Woomyoung Park. 2021.  GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 2225–2239, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Yu et al. (2018) Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V. Le. 2018.  QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension.  _arXiv:1804.09541 [cs]_.  ArXiv: 1804.09541.
  * Zhu et al. (2023) Dawei Zhu, Xiaoyu Shen, Marius Mosbach, Andreas Stephan, and Dietrich Klakow. 2023.  Weaker Than You Think: A Critical Look at Weakly Supervised Learning.  ArXiv:2305.17442 [cs].

##  Appendix A Prompts

In this section, we describe the prompts we used for ChatGPT and LLama2. For
the paraphrasing strategy, we copy Piedboeuf and Langlais (2023), who simply
asks for paraphrases in batch if the ratio is 1 and for multiple paraphrases
if the ratio is larger than one. We refer to their paper for more detail on
the prompting.

For the zero and three-shot generation, we use the template shown in Table 6.

DATASET_DESC | CLASS_DESC
---|---
movie reviews | negative or somewhat negative
positive or somewhat positive
headline Fake/Real news classification | Real
Fake
Ironic tweet detection | Non-Ironic Tweets
Ironic Tweets
Ironic tweet detection |  Tweets that are not ironic
Tweets ironic by polarity contrast, where the polarity is inverted between the
literal and intended evaluation
Tweets ironic by Situational Irony, where a situation fails to meet some
expectation
Tweets ironic by Other type of Irony, where the Irony is neither by Polarity
Contrast or by Situational Irony
Question classification |  Questions about an abbreviation
Questions about an entity (event, animal, language, etc)
Question concerning a description (of something, a definition, a reason, etc)
Questions about a human (description of someone, an individual, etc)
Questions about a location
Questions about something numerical (weight, price, any other number)
Table 6: Prompt patterns for the zero-shot strategies for ChatGPT and Llama.
The prompt is of the form “Here are some examples of $CLASS_DESC from a
dataset of $DATASET_DESC: $EXAMPLES. Can you create 10 more sentences of that
class that would fit the dataset ”.

We referred to the description given in the original papers of each dataset to
craft informative prompts.

##  Appendix B Supplementary Results

Table 8: Average metric over 15 runs for the training set sizes of 500 (left) and 1000 (right) with a ratio of 1. We report accuracy for binary tasks and macro-f1 for multiclass ones. STDs are between 0.6 and 3.0, depending on the dataset. Stars represent results for which the difference with the baseline was found to be statistically significant. | SST2 | Irony | IronyB | TREC6 | SNIPS | Average
---|---|---|---|---|---|---
Baseline | 87.7/88.8 | 67.0/68.1 | 43.9/45.9 | 84.5/87.9 | 95.6/96.2 | 75.8/77.4
Perfect | 88.9*/89.7* | 67.3/71.1* | 46.0*/48.5* | 87.6*/89.3 | 96.1*/96.9* | 77.2/79.1
Copy | 87.7/88.8 |  66.2/69.2 | 43.7/46.7 | 83.8/86.3 | 95.5/96.4 | 75.4/77.5
EDA | 88.1/88.9 | 65.9/69.4 | 43.6/47.2* | 82.5/86.1 | 95.8/96.3 | 75.2/77.6
AEDA | 88.0/89.0 | 65.1/69.5 | 42.8/46.7 | 82.7/87.4 |  96.1*/96.3 | 74.9/77.8
BT | 88.0/89.2* | 64.9/68.4 | 44.2/45.8 | 83.1/87.1 | 95.5/96.3 | 75.1/77.3
CBERT | 87.6/88.6 | 64.8/69.5 | 43.6/46.1 | 82.9/84.7 | 95.4/96.3 | 74.9/77.0
CBART | 87.7/88.7 | 66.0/68.8 | 43.6/45.1 | 81.2/82.5 | 95.9/96.2 | 74.9/76.3
T5 | 87.5/88.8 | 64.9/67.7 | 43.6/45.6 | 82.9/84.8 | 95.8/96.3 | 75.0/76.6
GPT3.5-P |  88.2*/89.3* | 65.1/69.5* |  44.3/47.5 |  84.5/86.8 | 95.5/96.1 |  75.5/77.8
GPT3.5-ZS | 87.7/88.8 | 64.2/67.4 | 42.0/45.1 | 81.9/86.0 | 95.5/95.8 | 74.3/76.6
GPT3.5-3S | 88.1*/88.6 | 64.9/69.0 | 40.4/44.4 | 83.0/86.8 | 95.9/96.2 | 74.5/77.0
Llama2-P | 87.8/88.9 | 65.4/68.1 | 44.1/46.1 | 83.3/85.9 | 95.7/96.4 | 75.3/77.1
Llama2-ZS | 88.1*/88.8 | 65.0/69.2 | 42.5/45.7 | 82.2/86.0 | 95.6/96.0 | 74.7/77.2
Llama2-3S | 87.8/88.4 | 65.0/67.9 | 42.8/46.2 | 83.8/85.3 | 95.7/96.4* | 75.0/76.8
|  Baseline |  Perfect |  Copy |  EDA |  AEDA |  BT |  CBERT |  CBART |  T5 |  GPT3.5-P |  GPT3.5-ZS |  GPT3.5-3S |  Llama2-P |  Llama2-ZS |  Llama2-3S
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Baseline |  0 |  0 |  0 |  20 |  10 |  10 |  30 |  20 |  20 |  0 |  40 |  30 |  20 |  30 |  30
Perfect |  80 |  0 |  90 |  70 |  70 |  90 |  100 |  80 |  90 |  80 |  100 |  90 |  90 |  100 |  90
Copy |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  30 |  10 |  0 |  40 |  20 |  0 |  10 |  20
EDA |  10 |  0 |  0 |  0 |  0 |  0 |  0 |  20 |  20 |  0 |  50 |  20 |  10 |  10 |  10
AEDA |  10 |  0 |  0 |  10 |  0 |  10 |  20 |  20 |  20 |  20 |  40 |  30 |  20 |  20 |  30
BT |  10 |  0 |  0 |  0 |  0 |  0 |  20 |  20 |  10 |  0 |  20 |  20 |  0 |  20 |  10
CBERT |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  10 |  10 |  0 |  30 |  20 |  10 |  0 |  10
CBART |  0 |  0 |  0 |  0 |  0 |  0 |  10 |  0 |  0 |  0 |  20 |  10 |  0 |  0 |  0
T5 |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  10 |  0 |  0 |  20 |  10 |  0 |  10 |  0
GPT3.5-P |  30 |  0 |  20 |  20 |  10 |  10 |  30 |  40 |  50 |  0 |  60 |  30 |  0 |  40 |  20
GPT3.5-ZS |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  10 |  0 |  0 |  0 |  10 |  0 |  0 |  0
GPT3.5-3S |  10 |  0 |  10 |  0 |  0 |  0 |  10 |  10 |  10 |  10 |  30 |  0 |  0 |  0 |  0
Llama2-P |  0 |  0 |  0 |  0 |  0 |  0 |  10 |  10 |  0 |  0 |  30 |  20 |  0 |  10 |  10
Llama2-ZS |  10 |  0 |  10 |  0 |  0 |  0 |  0 |  10 |  10 |  0 |  20 |  10 |  10 |  0 |  20
Llama2-3S |  10 |  0 |  0 |  0 |  0 |  0 |  10 |  10 |  0 |  10 |  20 |  20 |  0 |  10 |  0
Figure 4: Percentage of times the row algorithm performs statistically better
than the column algorithm, with a p-value threshold of 0.05 and using a two-
tails paired t-test and with the starting sizes of 500/1000.


# SECP: A SPEECH ENHANCEMENT-BASED CURATION PIPELINE FOR SCALABLE ACQUISITION
OF CLEAN SPEECH

###### Abstract

As more speech technologies rely on a supervised deep learning approach with
clean speech as the ground truth, a methodology to onboard said speech at
scale is needed. However, this approach needs to minimize the dependency on
human listening and annotation, only requiring a human-in-the-loop when
needed. In this paper, we address this issue by outlining Speech Enhancement-
based Curation Pipeline (SECP) which serves as a framework to onboard clean
speech. This clean speech can then train a speech enhancement model, which can
further refine the original dataset and thus close the iterative loop. By
running two iterative rounds, we observe that enhanced output used as ground
truth does not degrade model performance according to $\Delta_{PESQ}$, a
metric used in this paper. We also show through comparative mean opinion score
(CMOS) based subjective tests that the highest and lowest bound of refined
data is perceptually better than the original data.

Index Terms—  speech enhancement, text-to-speech, deep learning, data mining,
signal processing

##  1 Introduction

As speech-based technologies rely on deep learning, the need for clean speech
rises proportionally. Onboarding clean speech data is a non-trivial process
which generally requires many manual hours to listen and annotate clean
speech. However, this proves to be both an expensive and intensive process.
Having an automated process is needed to address both scaling up speech
acquisition while simultaneously allowing manual annotation to have an
outsized impact.

There are annotation processes in other modalities such as computer vision
which enable annotation on the pixel basis [1]. Furthermore, speech
recognition tasks (which partially exists in the audio modality) allow for
direct editing of low-performing transcribed speech which can then be used to
retrain the model [2, 3]. Despite both annotation methods working well in
their respective domain, there is very little overlap in what can be done to
“listen” and locate clean speech, which is imperative for audio-based
supervised learning tasks like speech enhancement (SE) and text-to-speech
(TTS).

Fig. 1: A high level overview of the proposed iterative process in which both
the data and speech enhancement model improve each other.

In this paper, we aim to outline SECP – the Speech Enhancement-based Curation
Pipeline. The pipeline has two main objectives: cleaning speech in large sets
of audio files reliably, and formatting said speech for further downstream use
cases. The goal of these two objectives is to develop a process in which the
acquisition of clean speech and the training of a speech enhancement model
improve each other symbiotically and iteratively.

SE and TTS architectures described in research papers are generally trained in
a supervised fashion with the ground truth being anechoic clean speech. These
tasks often utilize datasets such as LJSpeech [4] and Voice Bank [5]. While
these datasets serve as a useful benchmark, they are not sufficient for
generalization.

For the development of data-driven models tailored to speech-oriented
applications, it is ideal to construct a substantial dataset consisting of a
large speech corpus to mitigate potential bias. A lack of generalization, in
these cases, can be looked at as favoring one language over another, removing
speech or noise when not necessary, as well as discriminating against speech
across factors such as gender, age, accent, and emotions.

Algorithm 1 Overview of the SECP Algorithm

$f_{s}$ $\triangleright$ Specify sampling rate

$N$ $\triangleright$ Duration of saved curated sample (in seconds)

$w_{l}$ $\triangleright$ Duration of frame size (in seconds)

$s_{th}$ $\triangleright$ $\hat{\rho}$ threshold to filter out high noise
frames

$b_{w}$ $\triangleright$ Frequency bandwidth threshold to filter out low-
bandwidth frames

SE $\triangleright$ Speech Enhancement Model

VAD $\triangleright$ Voice Activity Detection Model

${\mathbf{x}}\in\mathbb{R}^{1\times(f_{s}\cdot L)}$

1:for number of files in dataset do

2: ${\mathbf{\hat{x}}}\leftarrow\texttt{SE}({\mathbf{x}})$, where
${\mathbf{\hat{x}}}\in\mathbb{R}^{1\times(f_{s}\cdot L)}$

3: ${\mathbf{v}}\leftarrow\texttt{VAD}({\mathbf{\hat{x}}})$, where
${\mathbf{v}}\in\mathbb{B}^{1\times(f_{s}\cdot L)}$ and
$\mathbb{B}\coloneqq\\{0,1\\}$

4: ${\mathbf{X}},{\mathbf{\hat{X}}},\mathbf{V}\leftarrow\text{Reshape
}{\mathbf{x}},{\mathbf{\hat{x}}},{\mathbf{v}}$, where
$\\{{\mathbf{X}},{\mathbf{\hat{X}}},\mathbf{V}\\}\in\mathbb{R}^{(f_{s}\cdot
w_{l})\times\frac{L}{w_{l}}}$

5:
$\tilde{\rho}\leftarrow\hat{\rho}({\mathbf{X}}_{*,l},{\mathbf{\hat{X}}}_{*,l},\mathbf{V}_{*,l})$,
for all $l=1,\dots,\frac{L}{w_{l}}$ and
$\tilde{\rho}\in\mathbb{R}^{1\times\frac{L}{w_{l}}}$

6: ${\mathbf{s}}\leftarrow\hat{\rho}>s_{th},\text{where
}{\mathbf{s}}\in\mathbb{B}^{1\times\frac{L}{w_{l}}}$

7: ${\mathbf{b}}\leftarrow f_{c}\geq b_{w}$ for all frames in
${\mathbf{\hat{X}}}$, where
${\mathbf{b}}\in\mathbb{B}^{1\times\frac{L}{w_{l}}}$.

8: ${\mathbf{a}}\leftarrow{\mathbf{s}}\dot{\wedge}{\mathbf{b}}$, where
${\mathbf{a}}\in\mathbb{B}^{1\times\frac{L}{w_{l}}}$

9: Locate all $N$ seconds of contiguous, non-overlapping, and approved frames
in ${\mathbf{a}}$.

10:end for

The main objective of this pipeline is to obtain more high-quality clean
speech which allows us to improve the quality and distribution of our speech
data. The improvement in our datasets can then be used to train a new model,
thus reprocessing the dataset once again and onboard more clean data. As
opposed to a scalable data augmentation engine with a fixed speech corpus
proposed by [6], we propose a methodology aimed to strictly increase the clean
speech corpus available for research teams to utilize for SE and TTS research
tasks.

##  2 Methodology

###  2.1 Neural Network Models

In the SECP, we leverage speech enhancement and voice activity detection (VAD)
neural networks to ensure our processing is done on clean speech. In this
subsection, we aim to outline the design of these two networks and the
function they serve in the pipeline.

Our speech enhancement model is a U-Net based Convolutional Neural Network [7,
8, 9, 10], operating on sampling rate $f_{s}$. The network learns to produce a
magnitude mask in the time-frequency domain and we found a non-causal network
to serve us best for the objective of curation.

We computed the STFT, and subsequent ISTFT, using window size $w_{s}$ with a
hop size of $0.25w_{s}$. The mask generated takes in context from both past
and future frames to make a prediction on the center frame. Although the model
is non-causal, we developed it such that it is able to process files that are
tens of minutes and even over an hour long. This allows us to run enhancement
in an offline process without worrying about real-time constraints. This
network is the model used for experimentation in this paper.

Furthermore, we develop a convolutional-based non-causal VAD neural network
that is trained on noisy speech [11, 12]. Our VAD takes in both left and right
context to predict the speech presence probability (SPP) on window length
$w_{v}$ as shown in [13]. After determining the binary decision of speech
presence in the window, we extrapolate the decision to all samples in the
window before continuing on to the next one. By doing this, we ensure the VAD
decision is shape aligned with the input and enhanced audio. The VAD is not
retrained throughout rounds and is only used for inference.

Fig. 2: The number of accepted curated hours between rounds (a) with the
comparison of $\Delta_{PESQ}$ scores between model training rounds across
various noise types (b).

###  2.2 SECP Algorithm

When designing the pipeline, we needed to ensure that the criteria of the
formatted speech was aligned with the criteria of publicly available TTS [14,
15] and SE datasets [16] as the ground truth outputs for both problems require
clean speech. Upon investigating datasets for both problems, full band data
with a high SNR served as a minimum criterion for acceptance.

SECP leverages two machine learning models to curate data, a VAD model and SE
model, both outlined in Section 2.1. Furthermore, the pipeline requires design
parameters for formatting the curated data. We aim to utilize these models as
modular components to the pipeline to ensure flexibility.

The algorithm takes in audio file ${\mathbf{x}}$, where
${\mathbf{x}}\in{\mathbb{R}}^{1\times(f_{s}\cdot L)}$ where $L$ is the
duration of the audio file in seconds. We first enhance the audio to produce
${\mathbf{\hat{x}}}$ and run the VAD on the enhanced file to get the sample-
wise binary decisions, ${\mathbf{v}}$, where
${\mathbf{\hat{x}}}\in{\mathbb{R}}^{1\times(f_{s}\cdot L)}$ and
${\mathbf{v}}\in{\mathbb{B}}^{1\times(f_{s}\cdot L)}$. We define
${\mathbb{B}}$ as the set of binary numbers, or
${\mathbb{B}}\coloneqq\\{0,1\\}$.

Once we have ${\mathbf{x}},{\mathbf{\hat{x}}},\text{and }{\mathbf{v}}$, we
then reshape the vectors into frames of length $w_{l}\cdot f_{s}$, where
$w_{l}$ is a parameter for specifying the frame length. In our experiments,
$w_{l}$ is chosen to be 1 second long.
${\mathbf{x}},{\mathbf{\hat{x}}},\text{and }{\mathbf{v}}$ then become
${\mathbf{X}},{\mathbf{\hat{X}}},\text{and }\mathbf{V}$, respectively where
$\\{{\mathbf{X}},{\mathbf{\hat{X}}}\\}\in{\mathbb{R}}^{(w_{l}\cdot
f_{s})\times\frac{L}{w_{l}}}$ and $\mathbf{V}\in{\mathbb{B}}^{(w_{l}\cdot
f_{s})\times\frac{L}{w_{l}}}$.

By framing our vectors, we are able to perform our $SNR$ calculation on each
window, narrowing the search for high $SNR$ speech. To find high $SNR$ speech
segments, we formulate $\hat{\rho}$ as an estimate of $SNR$. We calculate and
define $\hat{\rho}$ as the following:

$\hat{\rho}_{l}({\mathbf{x}}_{l},{\mathbf{\hat{x}}}_{l},{\mathbf{v}}_{l})=\\\
\begin{cases}RMSdB({\mathbf{\hat{x}}}_{l})-RMSdB({\mathbf{x}}_{l}-{\mathbf{\hat{x}}}_{l}),&\text{if
$\mathbb{E}({\mathbf{v}}_{l})\geq 0.5$}\\\
-\infty,&\text{otherwise}\end{cases}$ (1)

$RMSdB({\mathbf{x}})=20\log_{10}\left(\sqrt{\frac{1}{n}\sum_{i=0}^{n}{x_{i}^{2}}}\text{
}\right)$ (2)

Where ${\mathbf{\hat{x}}}_{l}$ represents the frame of enhanced speech,
${\mathbf{x}}_{l}$ represents the frame of the corresponding unprocessed
speech, and ${\mathbf{v}}_{l}$ corresponds to the sample-wise VAD decision of
the entire frame, where
$\\{{\mathbf{x}}_{l},{\mathbf{\hat{x}}}_{l},{\mathbf{v}}_{l}\\}\in\mathbb{R}^{1\times\frac{L}{w_{l}}}$.
Simply we perform the $SNR$ estimation using each column of
${\mathbf{X}},{\mathbf{\hat{X}}},$ and $\mathbf{V}$.

The core assumption in our formulation is that the speech enhancement model
will remove noise and preserve speech, and thus the difference will only
contain speech. Because $\hat{\rho}$ serves as an estimate to the actual
$SNR$, investigating its performance was necessary for robustness. To do this,
we tested it on various situations to ensure that we would not get undesirable
segments labelled as clean speech.

We ran internal listening and annotation to determine that an $SNR$ threshold,
notated as $s_{th}$ in Algorithm 1, of 20 dB was a sufficient threshold to use
to locate clean speech with adequate noise removal. More specifically, we
found that when $\hat{\rho}\geq 20$, the enhanced speech files removed noise
to a degree that not only was unperceivable by human annotators, but also was
not visible in spectrogram visualizations. For our experiments, we find and
calculate ${\mathbf{s}}$, a vector of all binary decisions of segments
containing $\hat{\rho}\geq 20$ dB, where
${\mathbf{s}}\in\mathbb{B}^{1\times\frac{L}{w_{l}}}$.

To detect and filter out low-bandwidth content, we leverage simple signal
processing techniques to analyze binned spectrograms to allow for time-
windowed analysis of the content. The cut-off frequency, $f_{c}$, per each
time window is calculated and used to determine the bandwidth of the frame. If
$f_{c}\geq b_{w}$, where $b_{w}$ is the specified bandwidth, the frame is
preserved. This is used to filter out low-bandwidth content and preserve high-
bandwidth content. We assign all binary results of bandwidth detection to
${\mathbf{b}}$, where ${\mathbf{b}}\in\mathbb{B}^{1\times\frac{L}{w_{l}}}.$

After calculating vectors ${\mathbf{s}}$ and ${\mathbf{b}}$, we use an entry
wise logical AND operation, notated as $\dot{\wedge}$. We assign the entry-
wise logical AND output as ${\mathbf{a}}$, which determines all of our
approved, high-quality segments in our input audio file. Every non-overlapping
$N$ seconds of contiguous accepted segments would then be considered as one
curated sample. In our experiments, $N$ is set to 12 seconds. All the curated
sample’s relevant metadata will be saved, and this process is repeated until
the entire dataset is fully curated.

Fig. 3: $\hat{\rho}$ distribution of approved one second segements between
rounds (a), as well as subjective test results comparing lowest bound of
performance (b) and the highest bound of performance (c) to the original
unprocessed files.

###  2.3 Iterative Training and Data Onboarding

The methodology of implementing SECP aims to be a systematic one. This means
that even if a research team does not have a large corpus of data, starting
with a small seed round to train the first Speech Enhancement model is
sufficient. To show this, we first began by training our model on a small set
of data that we annotated as clean. This model then processes four internal
dataset fed into SECP. This newly curated data serves as ground truth for the
training of a second model which then formats the same unprocessed dataset and
repeats the workflow all over again.

##  3 Results

We use our non-causal wideband model outlined in Section 3.1 for two rounds of
curation and training with no architectural changes. We aim to evaluate our
model objectively, for the direct comparison between models in denoising
performance, and subjectively, to ensure that objective measurement is
correlated with human perception.

###  3.1 Objective Model Performance

Upon investigating the curation performance outlined in Figure 2a, we see an
improvement across all four internal sets. Aside from the 72.4% increase in
Dataset 1, the other three sets increased by 21.01%, 26.30%, and 20.03%,
respectively.

Upon analysis of our denoising model, we studied its performance in removing
noise from speech across 21 noise categories shown in Figure 2b. Each noise
set contains 1000 clean speech files with noise injected in them. The degree
of noise that is added to the speech file is determined by a Rayleigh
distribution. The test sets in 2b are fixed between rounds.

To measure the models’ performances in denoising, we define $\Delta_{PESQ}$ as
the following:

$\Delta_{PESQ}=PESQ({\mathbf{\hat{x}}},{\mathbf{x}})-PESQ(\tilde{{\mathbf{x}}},{\mathbf{x}})$
(3)

Where $\tilde{{\mathbf{x}}}$ represents the noisy speech, ${\mathbf{x}}$
represents the clean speech, ${\mathbf{\hat{x}}}$ represents the enhanced
noisy speech files, and $PESQ$ is defined in [17]. The higher the
$\Delta_{PESQ}$ performance, the larger the difference between the noisy and
enhanced files are. In Figure 2b, we see a general improvement in
$\Delta_{PESQ}$ between rounds.

###  3.2 Subjective Model Performance

When observing the $\hat{\rho}$ distribution in Figure 3 (a), we notice a
general decrease in $SNR_{c}\leq 30$ dB as well as an increase in $SNR_{c}\geq
45$ dB. This behavior is ideal as we aim to use the pipeline to shift the
$\hat{\rho}$ distribution towards cleaner speech, or higher $\hat{\rho}$
estimations.

However, to show that the data is of equal, or better, quality at these lower
and upper bounds, we utilize subjective testing to verify a perceptual
difference between processing rounds. To do this, we leveraged $\hat{\rho}$
and time stamp metadata saved from the pipeline to see if the how the lowest
($SNR_{c}\leq 30$ dB) and highest ($SNR_{c}\geq 45$ dB) bounds of both rounds
compare to the original unprocessed files.

We can filter curated files by $\hat{\rho}$ estimations. Because each curated
file was set to $N=12$ and $w_{l}=1$, each curated file has a $\hat{\rho}$
array of length 12. We can filter on the lowest and highest $\hat{\rho}$
segments to locate files that satisfy this criteria. Furthermore, by
leveraging the timestamp metadata, we are able to snip out the same curated
segment from the original file, thus enabling a direct A/B comparison between
the unprocessed and processed file.

We construct our crowd-sourced subjective tests in an A/B format and utilize
Comparative Mean Opinion Scores (CMOS) [18] as the metric of comparison. We
utilize Amazon MTurk to crowdsource the results and polled 1300 listeners.

In Figure 3 (b) and 3 (c), a negative subjective score favors the unprocessed
file, a positive subjective score favors the enhanced file, and 0 does not
favor either. We show that the lowest bound of $\hat{\rho}$ is either
identical or in favor of the enhanced data, while also improving between
rounds. Furthermore, the highest bound $\hat{\rho}$ is either identical or in
favor of enhanced data while staying approximately equal between rounds. These
results indicate to us that the model did not lose performance in transparency
and was able to perceptually improve upon “clean speech” after two rounds of
curation.

##  4 Conclusion

In this paper, we outlined the Speech Enhancement-based Curation Pipeline
(SECP). This pipeline enables any SE model to refine speech data in an
iterative fashion to increasing the clean speech corpus size. We also showed
how this iterative process produced perceptually favorable speech that
improves an SE neural network trained on it.

## References

  * [1] U.A. Usmani, J. Watada, J. Jaafar, I. Abdul Aziz, and A. Roy,  “A reinforced active learning algorithm for semantic segmentation in complex imaging,”  IEEE Access, vol. 9, pp. 168415–168432, 2021.
  * [2] G. Riccardi and D. Hakkani-Tur,  “Active learning: theory and applications to automatic speech recognition,”  IEEE Transactions on Speech and Audio Processing, vol. 13, no. 4, pp. 504–511, 2005.
  * [3] D. Hakkani-Tur, G. Tur, M. Rahim, and G. Riccardi,  “Unsupervised and active learning in automatic speech recognition for call classification,”  in 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2004, vol. 1, pp. I–429.
  * [4] K Ito and L. Johnson,  “The lj speech dataset,” https://keithito.com/LJ-Speech-Dataset/, 2017\.
  * [5] C. Veaux, J. Yamagishi, and S. King,  “The voice bank corpus: Design, collection and data analysis of a large regional accent speech database,”  in 2013 International Conference Oriental COCOSDA held jointly with 2013 Conference on Asian Spoken Language Research and Evaluation (O-COCOSDA/CASLRE), 2013, pp. 1–4.
  * [6] C. KA Reddy, E. Beyrami, J. Pool, R. Cutler, S. Srinivasan, and J. Gehrke,  “A scalable noisy speech dataset and online subjective test framework,”  Proc. Interspeech 2019, pp. 1816–1820, 2019.
  * [7] O. Ronneberger, P. Fischer, and T. Brox,  “U-net: Convolutional networks for biomedical image segmentation,”  in Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi, Eds., Cham, 2015, pp. 234–241, Springer International Publishing.
  * [8] S. Mirsamadi and I. Tashev,  “Causal speech enhancement combining data-driven learning and suppression rule estimation,”  09 2016, pp. 2870–2874.
  * [9] S. Rim Park and J. Lee,  “A fully convolutional neural network for speech enhancement,”  CoRR, vol. abs/1609.07132, 2016.
  * [10] M. P.V., N. Adiga, V. Tsiaras, and Y. Stylianou,  “A non-causal FFTNet architecture for speech enhancement,”  in Interspeech 2019. sep 2019, ISCA.
  * [11] A. Esmaeili, S. M. Ahadi, and M. A. Fassihi,  “A non-causal approach to voice activity detection in adverse environments using a novel noise estimator,”  in 2007 15th European Signal Processing Conference, 2007, pp. 2065–2069.
  * [12] S. Braun and I. Tashev,  “On training targets for noise-robust voice activity detection,” 2021\.
  * [13] I. Tashev and S. Mirsamadi,  “Dnn-based causal voice activity detector,”  in Information Theory and Applications Workshop, 2016.
  * [14] E. Bakhturina, V. Lavrukhin, B. Ginsburg, and Y. Zhang,  “Hi-fi multi-speaker english tts dataset,” 2021.
  * [15] H. Zen, V. Dang, R. Clark, Y. Zhang, R. Weiss, Y. Jia, Z. Chen, and Y. Wu,  “Libritts: A corpus derived from librispeech for text-to-speech,” 2019\.
  * [16] H. Dubey, A. Aazami, V. Gopal, B. Naderi, S. Braun, R. Cutler, H. Gamper, M. Golestaneh, and R. Aichner,  “Deep speech enhancement challenge at icassp 2023,”  in ICASSP, 2023.
  * [17] A.W. Rix, J.G. Beerends, M.P. Hollier, and A.P. Hekstra,  “Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs,”  in 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.01CH37221), 2001, vol. 2, pp. 749–752 vol.2.
  * [18] ITUT-T Recommendation P.808,  “Subjective evaluation of speech quality with a crowdsourcing approach,”  Geneva: International Telecommunication Union, 2021.