# Evaluation Metrics for Text Data Augmentation in NLP

Marcellus Amadeus, William Alberto Cruz Castañeda
Alana AI Research
Brazil
{marcellus<EMAIL_ADDRESS>

###### Abstract

Recent surveys on data augmentation for natural language processing have
reported different techniques and advancements in the field. Several
frameworks, tools, and repositories promote the implementation of text data
augmentation pipelines. However, a lack of evaluation criteria and standards
for method comparison due to different tasks, metrics, datasets,
architectures, and experimental settings makes comparisons meaningless. Also,
a lack of methods unification exists and text data augmentation research would
benefit from unified metrics to compare different augmentation methods. Thus,
academics and the industry endeavor relevant evaluation metrics for text data
augmentation techniques. The contribution of this work is to provide a
taxonomy of evaluation metrics for text augmentation methods and serve as a
direction for a unified benchmark. The proposed taxonomy organizes categories
that include tools for implementation and metrics calculation. Finally, with
this study, we intend to present opportunities to explore the unification and
standardization of text data augmentation metrics

_K_ eywords text data augmentation $\cdot$ natural language processing $\cdot$
unified metrics $\cdot$ taxonomy

##  1 Introduction

Recent surveys on data augmentation (DA) for natural language processing (NLP)
have reported different classifications, techniques, and advancements in the
field. An overview exposed by [1] presents more than 100 DA methods for text
classification organized into 12 different groups and provides state-of-the-
art references that expound what kind of methods are promising by relating
them to each other. In addition, a list of frameworks and practical tools
promotes DA implementation pipelines and raises the necessity of comprehensive
evaluation criteria and standards for method comparison. The survey of [2]
describes methodologies and DA techniques for NLP categorized into rule-based,
example interpolation-based, or model-based. It discussed the usefulness of DA
techniques, including low-resource languages, mitigating bias, fixing class
imbalance, few-shot learning, and adversarial examples. Moreover, a lack of
methods unification exists, and popular ones are present in an auxiliary
fashion. Further, text DA research would benefit from unified benchmark tasks
and datasets to compare different augmentation techniques.

To stimulate the use of strategies for selecting and constructing text DA
techniques, [3] divide into three categories according to the diversity of
their augmented data: paraphrasing, noising, and sampling. Paraphrasing-based
methods are thesauruses, semantic embeddings, language models, rules, machine
translation, and model generation. The noising-based methods are swapping,
deletion, insertion, and substitution. The sampling-based methods are rules,
non-pretrained models, pretrained models, self-training, and mixups. For
industrial applications, [4] presents text DA techniques such as back-
translation, lexical substitution, and conditional DA. [5] presents a list of
augmentation frameworks for text data implementing symbolic or neural methods.
Symbolic methods use rules or discrete data structures to form synthetic
examples. Alternatively, the neural method implements a deep neural network
trained on a different task to augment data. However, to compare the
performance of DA techniques with its characteristics, the experimental
settings make direct comparisons meaningless. Thus, the work contribution is
to provide a taxonomy of evaluation metrics for text augmentation methods and
serves as a direction for a unified benchmark. This work exposes the metrics
found in the NLP literature between 2018 to 2023. The taxonomy includes tools
and repositories to implement and calculate the metrics. Following is
described each of the categories of the taxonomy. Finally, with this study, we
intend to expose opportunities to explore the unification and standardization
of DA metrics.

##  2 Recent Development on Data Augmentation for NLP

The problem of generating labeled sentences with generative adversarial
networks (GANs) for DA is studied by [6]. Combine reinforcement learning (RL),
GANs, and recurrent neural networks (RNN) to build a category sentence GAN
(CS-GAN) model. The proposed model generates categorized sentences that
increase the original dataset and improve its generalization capability during
supervised training. Setup and training with conditional GAN (cGAN) are
described by [7] for DA in sentiment classification. Moreover, it addressed
the training methodology and the architecture of the cGAN on low-resource
sentiment analysis datasets. A conditional BERT with contextual augmentation
is proposed by [8] for labeled sentences. BERT allows sentence augmentation
without breaking the label compatibility and introducing a new conditional
masked language model task. The proposed method is an unidirectional language
model and can be applied to enhance contextual augmentation. Working with hate
speech classification, [9] proposes three techniques to reduce the degree of
class imbalance and maximize the amount of information extracted from limited
resources. The first technique is synonym replacement on word embedding vector
closeness. The second uses warping techniques of the word tokens along the
sequence or class-conditional, and the third uses recurrent neural language
generation. All the techniques follow the process of text acquisition, pre-
processing, word embedding, 1D convolutional neural network (CNN), max
pooling, gated recurrent unit/long short-term memory (LSTM), max pooling,
dense layer, and softmax activation.

Inspired by the capabilities of pre-trained word embedding models, [10]
propose a word embedding-based parallel text augmentation technique, which
increases the size of the limited text for low-resource neural machine
translation (NMT). Improving NMT robustness, [11] proposes DA techniques to
extend limited noisy data, injecting synthetic noise samples and adversarial
attacks. Results show that techniques could benefit from data injected with
noise through manual transcriptions of spoken language. The automatic speech
recognition (ASR) generated data don’t improve robustness, while the human
transcripts from speech proved helpful in translating noisy texts. Easy DA
(EDA) is a set of techniques developed by [12] that boost performance on text
classification tasks. EDA consists of four operations: synonym replacement,
random insertion, random swap, and random deletion. Results show that EDA
improves performance with CNN and RNN and demonstrates strong results for
smaller datasets. It highlighted the lack of standardized DA in NLP by
introducing a set of simple operations that serve as a baseline for future
investigation. At the word and sentence level, [13] proposed the Hierarchical
DA (HDA) method for augmenting texts. HDA implements an attention mechanism to
distill important contents from hierarchical texts as summaries. As a
complement, training in the hierarchical attention networks model allows
attention values of all documents. The experiments reveal that HDA is a
promising technique when compared with EDA. The proposed approach of [14] is
based on text DA for product reviews of online opinions in the Vietnamese
language to substitute large amounts of human-labeled data that is costly to
obtain. The word embedding implements cosine distance for measuring the
similarity. Stopwords are created manually based on their TF-IDF score.
Techniques are applied to generate comments by random insertions,
substitutions, word swapping, word deletion based on word2vec, and word
replacement by synonyms or similar words for Vietnamese text. The method of
[15] combines EDA and pseudo-labeling to increase the size of the initial
train set. A pre-trained BERT model is used as a text classifier and fine-
tuned the data.

[16] evaluates the influence of the parameters of a DA technique in a textual
dataset and the use of neural networks for emotion analysis of Brazilian
Portuguese texts. Implements a multi-lingual fine-tuning method, which applies
the concept of transfer learning. The text augmentation approach consists of
synonymous words instead of original ones without losing the sentence’s sense.
The study of [17] presents MixText, a semi-supervised learning method with a
DA technique called TMix. TMix takes two text instances and interpolates them
in their corresponding hidden space. Since the combination is continuous, TMix
has the potential to create an infinite amount of augmented samples and avoid
overfitting.

Comparing different DA techniques is a non-trivial task that requires
incorporating different architectures and model implementation. To address
these challenges, [18] develop EasyAug, a DA platform that provides several
augmentation approaches. EasyAug incorporates all prominent textual DA
approaches. Provide three methods: random resampling, word-level
transformations, and VAE-based text generation models. [19] propose and
evaluate various augmentation methods, including GPT-2 fine-tunned version.
Explore semantic text exchange, synthetic noise, synonym replacement, hyponym
replacement, hypernym replacement, and random insertion. The quality of the
generated text is similar to the original data. [20] study GPT-2, BERT, and
BART for conditional DA. These methods integrate text content manipulation as
co-training the data generator and augmentation techniques combined with
latent space augmentation.

Data Boost, described by [21], is an RL-guided text DA framework built on off-
the-shelf GPT-2. Requires collecting extra data and training a task-specific
language model from scratch. The generated samples serve as augmentation data
similar to the original. A nonlinear Mixup, proposed by [22], addresses
limitations in DA with Mixup. The nonlinear mixing policy mixes sample pairs
on the word embedding level and deploys a separate mixing policy in the word
dimensions of a given sentence. [23] presents, Unsupervised DA (UDA), a method
for effectively noise unlabeled examples. This procedure forces the model to
be insensitive to the noise and hence smoother concerning changes in the
hidden space. From another perspective, minimizing the consistency loss
propagates label information from labeled examples to unlabeled ones. For
text, UDA combines well with representation learning as BERT. The contribution
of [24] is a short survey of several DA methods. The work categorizes text
augmentation techniques in text source and text representation. Besides, an
alternative way of augmenting low-resource language is with unsupervised word
embedding models. The mixup-transformer architecture proposed by [25] applies
mixup into transformer-based pre-trained models. Unlike static mixup, the
architecture can dynamically construct inputs for text classification. Multi-
granularity text-oriented DA technologies proposed by [26] generate large-
scale artificial data into a hybrid neural network model. The model combined
two channels at the feature level and was co-trained to obtain the final
feature representation for the input sentences.

The DA technique introduced by [27] avoids involving noisy data with a
keywords-oriented DA (KDA) method that extracts keywords based on category
labels and augments them based on the keywords. The proposed technique selects
the essential information and expands the selected data. KDA is inspired by
the EDA approach, extracting the keywords from the text and then applying the
DA technique to the data. A text augmentation method, presented by [28],
scales the size of an emotion corpus by including only relevant positive
examples. Tweets labeled with happiness are used as gold standard seeds to
augment the training data to include similar distant supervision. [29]
presents a technique called Augmented SBERT, where the BERT cross-encoder
label input pairs to fine-tune the bi-encoder and augment the training data.
Based on LSTM networks, [30] proposes three text DA techniques: permutation,
antonym, and negation. Sentence permutations expand an initial dataset while
retaining the statistical properties, applied multiple times per dataset
without duplicating entries, thus avoiding overfitting. Antonym replacement
and negation insertion reverse the classification of each augmented example
but require mutually exclusive classes.

[31] generate sentences for five regional Arabic dialects. The objective is to
overcome the scarcity of richly annotated dialectal Arabic datasets. The
dataset has five regions, represented by classes corresponding to a different
dialect. The Sentimental GAN (SentiGAN) model is used to populate low-resource
dialects. However, one generator per class instead of multiple generators has
five generators/discriminators working independently on a specific dialect
without cross-group mixing. The main obstacle in training neural models for
data-to-text generation is the lack of training data. To address this problem,
[32] proposes a few-shot approach to augment the data in three ways. Train to
generate new text samples based on replacing specific values with alternative
ones from the same category. Based on GPT-2, generate text samples and propose
an automatic method for pairing the new text samples with data samples.

Most text GANs are unstable in training due to inefficient generator
optimization and rely on maximum likelihood pre-training. The work of [33]
addresses the above problems by proposing an augmentation method with a
Sentence Generator (SG) and Sentence Discriminator (SD) for Iterative
Translation-based DA (ITDA). The ITDA SG provides universal multiple-language
support by generating comprehensive augmented sentences through serial and
parallel iterations of an existing translator. Implementing BERT for Persian
tasks, [34] uses the pre-trained language model (PLM) ParsBERT to augment the
input data and then use it for text classification. The augmentation enables
the generalization of unseen and new words well. For patient outcomes
prediction, [35] proposes an architecture using fine-tuned GPT-2 for Medical
text Augmentation (MedAug) to synthesize labeled text with the original
training data. The study of [36] proposes a sentiment analysis framework
powered by two performance boosters: 1) a DA technique based on a transformer-
based GAN and RoBERTa, and 2) an optimized BERT to improve the prediction
accuracy. The GAN-based DA technique can generate high-quality synthetic
samples to increase the size and diversity of the training set. [37] present
an enhanced UDA (EUDA) by mixing DA strategies and using a problem-related
prefilter. The DA strategies for UDA enhancement are back translation, TF-IDF,
and EDA techniques. [38] compare a variational autoencoder (VAE) with EDA to
increase the number of training samples for malware detection. [39] modify the
EDA technique with sophisticated text editing operations powered by masked
language models to analyze the benefits or setbacks of creating a linguistical
quality augmentation.

Virtual DA (VDA) is a general framework for fine-tuning PLM developed by [40].
The idea is to generate DA at the embedding layer of PLM. To guarantee
semantic relevance, consider a multinomial mixture of the original token
embeddings as the augmented embedding. There are two advantages to this
approach. First, the original token embeddings are the representation basis,
and the augmented embeddings stay close to the existing embeddings. Second,
the injected Gaussian noise generates variations for augmentations.

[41] works with the DA technique to improve the performance of the BERT
classification model for fake news in the Portuguese language. The DA
technique consists of manipulating the dataset with modified copies or
creating synthetically from existing data. The built process applies DA before
translation takes place. [42] proposes a text augmentation based on attention
score (TABAS). The proposed approach recognizes that a criterion for selecting
a replacement word rather than a random selection is necessary. The attention
score processes only words with the same entity and part-of-speech tags to
consider informational aspects. TABAS has two steps. The first step is dataset
preparation, training an attention score, and building a word dictionary. In
the second step are tokenized sentences in the dataset using the attention
model and the word dictionary.

Studying the advantages and drawbacks of text augmentation methods [43]
evaluated strategies of text DA techniques on datasets for sentiment analysis
and related tasks using four different classifiers. The proposed dual text DA
strategy of [44] uses a mix of four pre/post-augmentation operations in two
different settings on a classification model based on RNN. Considering a
dynamic policy scheduling, [45] design a search space over augmentation
policies by integrating generic augmentation operations. It adopted a
population-based training method to search for the best augmentation schedule.
Tailored Text Argumentation (TTA) proposed by [46] has two operations. The
first is the probabilistic word sampling for synonym replacement based on the
discriminative power and relevance of the word to sentiment. The second is the
word identification and the zero masking or contextual replacement of these
words. A two-step DA process developed by [47] consists of a former stage with
a method for preparing a comprehensive list of identity pairs with word
embeddings. The next step prepares an identity pairs list to enhance the
training instances by applying three simple operations (namely identity pair
replacement, identity term blindness, and identity pair swap). Thus, an
Identity Information DA method (IIDA) bolsters text classification models
against unintended bias by automatically generating diverse instances. [48]
presents a DA using lexicalized probabilistic context-free grammar that
generates augmented samples with diverse syntactic structures with plausible
grammar. Another contribution is to the train-validation splitting
methodologies, where the traditional splitting of training and validation sets
is sub-optimal compared to our novel augmentation-based splitting strategies.
[49] presents a combination of DA techniques to boost the performance of
state-of-the-art transformer-based language models for Patronizing and
Condescending Language (PCL) detection and multi-label PCL classification
tasks. The approaches rely on fine-tuning pre-trained RoBERTa and GPT3 models
with extra-enriched PCL datasets.

Inspired by AutoAugment, [50] proposes an automated DA for language processing
and understanding with twofold novelties. First, an adaptive augmentation
policy optimizes individual data points, sentences, or documents. Second, an
efficient evaluation strategy designs training for auto augmentation policy
networks. [51] proposes an alternative DA technique to improve the robustness
of poor tokenization by using multiple tokenizations. The developed technique
for the Thai language uses different and possibly errorful tokenizations to
enhance the Thai text classification.

NEO Natural Language DA (NEO-NDA) is a comprehensive tool to address data
generation and rebalancing datasets proposed by [52] that work with multiple
languages, create new samples, boost the performance of ML models, and handle
multilingual DA. Offer a hybrid (rule-based and model-based) group of
techniques for DA in seven different languages: English, French, German,
Italian, Portuguese, Romanian, and Spanish. The study of [53] proposes a
hybrid framework that implements a GAN and Shapley algorithm based on EDA to
obtain balanced data for classifier training, generates an initial training
sentence, and uses it as the seed data for the next GAN-based model. TreeMix
is a compositional DA approach for natural language understanding proposed by
[54]. Leverages constituency parsing trees to decompose sentences into
constituent sub-structures and the Mixup DA technique to recombine them to
generate new sentences. Exploring the potential of using text-to-text language
models, [55] proposes T5 and BART integration to construct a two-phase
framework for augmentation. First, a fine-tuning phase where PLMs are well
adapted to downstream classification. Second, a generation phase where the
fine-tuned models create samples for performance lifting.

[56] proposes DA with Generation and Modification (DAGAM), which combines DA
techniques using a generation model (DAG) and character order changing (DAM)
for a boosted performance. DAG uses a language generation model for DA. DAM
exploits a psychological phenomenon called the word superiority effect on text
data. Introduce three DA schemes that help to reduce the underfitting problems
of large-scale language models. To support effective text classification, [57]
presents a plug-in DA framework (EPiDA). EPiDA employs two mechanisms to
control the diversity and quality of augmented data: relative entropy
maximization (REM) and conditional entropy minimization (CEM) for data
generation control. REM enhances augmented data diversity, and CEM ensures
semantic consistency.

[58] combines the advantages of Dropout and Mixup to develop a new approach
for DA and regularization called DropMix to mitigate the overfitting problem
in text learning. DOUBLEMIX, proposed by [59], is a simple interpolation-based
DA approach to improve the robustness of neural models in text classification
by mixing up the original text with its perturbed variants in hidden space.
[60] implements several DA techniques in Korean corpora with PLMs (SKT-koBERT,
SKT-GPT2, KrBERT, ENLIPLE-v2, KoELECTRA-Base-v3, BERT-Base).

[61] proposes a mixing-based text augmentation approach based on explainable
artificial intelligence (XAI) to consider the importance of mixed-up words in
labeling the augmented data. To improve the robustness of PLMs (BERT,
DistilBERT), [62] proposes two DA techniques for text classification: cognate-
based and antipode-based. A DA technique ChatAug proposed by [63], based on
ChatGPT, generates auxiliary samples for few-shot text classification. ChatGPT
is trained on data with unparalleled linguistic richness and employs a
reinforcement training process with large-scale human feedback, which endows
the model with affinity to the naturalness of human language. The ChatAug
rephrases each sentence in training into semantically different samples. [64]
proposes an XAI-based mix-up approach that explicitly derives the importance
of manipulated words and reflects this importance in labeling augmented data.

##  3 Taxonomy of Evaluation Metrics

A taxonomy organizes categories that represent mappings of the text DA
techniques. The result is shown in Figure 1 with ten categories as follows:
human evaluation, machine translation quality, text generation quality (split
up into novelty, diversity, and fluency), character n-gram matches; prediction
quality for classification; datasets relationship; ASR Performance; training;
language model robustness, and manipulated words importance. As a fundamental
concern in this work, each metric in a category describes procedures,
computational tools, or repositories to promote their calculation and
implementation.

Figure 1: Taxonomy of evaluation metrics for text augmentation methods.

###  3.1 Human Evaluation

This category checks the text generation quality from the human point of view.
No computational implementation allows its use. The main objective is to
obtain a fluency score and count of information missed and wrong information.
Thus, the generated text maintains the information’s completeness and
correctness. Table 1 explains the procedure.

Reference | Procedure
---|---
[32] | The data and generated text are evaluated by individuals. For each data-text pair, the annotator is instructed to evaluate: a) if the text is fluent (score 0-5 with 5 being fully fluent), b) if it misses information contained in the source data, and c) if it includes wrong information.
Table 1: Human evaluation for text generation quality.

###  3.2 Machine Translation Quality

Human assessment is the best option for evaluating translation quality and is
considered ground truth. However, as human evaluation is expensive, it is
impossible to have every new output of the model evaluated by a group of
individuals. Thus, this category assesses machine translation through the
Bilingual Evaluation Understudy (BLEU) metric score. BLUE is a corpus-level
metric based on the modified-gram precision measure with a length penalization
for the candidate sentences. Table 2 presents the common computational
implementations of this metric in Python language. Metrics 1, 2, 3, and 4
represent the precision of the n-gram sequence while calculating the BLEU
score.

Metric | Python-based Tools
---|---
BLEU score, Macro BLEU-2 |  NLTK
Table 2: Metrics for machine translation quality involved in text data
augmentation.

###  3.3 Text Generation Quality

This category joins five aspects required to define text generation quality
metrics involved in text DA: novelty, diversity, fluency, semantic content
preservation, and sentiment consistency. Novelty, an independent metric,
verifies text consistency created by GANs and is not measured by the fluency
or logic in the sentence. Ensure the generator creates new sentences instead
of copying the ones in the dataset. No computational implementation exists and
is defined as follows:

$Novelty(S_{i})=1-max\\{\varphi(S_{i},C_{j})\\}_{j=1}^{j=|C|}$

where $C$ is the sentences set, $\varphi$ is the Jaccard similarity function
and $S_{i}$ is a generated sentence. The diversity assesses the difference
between generated and original sentences in the same dataset. The metrics to
measure it are self-bleu, unique trigrams (UTR), type-token ratio (TTR), and
rare words (RWORDS). Self-BLEU evaluates the variety of sentences, measuring
the BLEU score for each generated sentence by considering other generated
sentences as references. UTR measures the ratio of unique to total trigrams in
a population of generations, high UTR represents excellent diversity. TTR is
the ratio of unique to total tokens in a piece of text and serves as a measure
of intra-continuation diversity. The higher the TTR, the more varied the
vocabulary in a continuation. In RWORDS, lower values indicate the usage of
more rare words (less frequent in the corpus) and higher diversity.

Perplexity (PPL) is a metric for evaluating language models. Measure the
probability of a sentence produced by the model trained on a dataset. With a
lower perplexity value, the model predicts a sample better. Syntactic log-odds
ratio (SLOR) modifies PPL by normalizing individual tokens. Higher SLOR
represents higher fluency. SPELLCHECK measures two spelling-related metrics
for synthetic noise. SPELLWORDS is the average number of misspelled words per
continuation, and SPELLCHARS is the average number of character-level mistakes
per continuation. Table 3 presents computational implementations in Python
language for these metrics.

Metric | Python-based Tools
---|---
Diversity Self-BLEU, UTR, TTR, RWORDS |
Fluency PPL, SLOR, SPELLCHECK |
Semantic Content Preservation BPRO |
Sentiment Consistency SENTSTD, SENTDIFF |
Table 3: Metrics for machine translation quality involved in text data
augmentation.

###  3.4 Character n-gram matches

Word-level n-gram-based metrics, including BLEU, relied on surface form
matching, and they are not appropriate for subword and character-level neural
machine translation systems. If the target side language is morphologically
rich, these metrics are not preferable because similar words present several
forms. The character n-gram F-score (CHRF) metric is language and
tokenization-independent with good correlations in human judgments. Table 4
presents the Python language implementations for this metric.

Metric | Python-based Tools
---|---
CHRF |  NLTK
Table 4: Metric for character n-gram matches in text data augmentation.

###  3.5 Prediction quality for classification

Prediction quality is a set of metrics computed in the augmentation evaluation
process. They are measurements that show how close the text augmentation
model’s predictions are to the actual values. Table 5 shows the different
metrics used to measure the prediction quality.

Metric | Python-based Tools
---|---
Accuracy |
Precision |
Macro-F1 Score |
Recall score |
AUC |
Jaccard similarity |
Matthew’s correlation, similarity score |
Confusion matrix |
F1 standard deviation |
AUPRC |
CTF |
NLL |
Table 5: Metrics for prediction quality for classification in text data
augmentation.

###  3.6 Datasets relationship

The set of metrics of this category allows us to measure the following
aspects. First, the relationship between the two datasets is based on
Spearman’s rank. If positive correlations exist, data in datasets one and two
increases. Negative correlations imply that as dataset one increases, dataset
two decreases. Correlations of -1 or +1 indicate an exact monotonic
relationship. Second, dataset transformation enhances the data by rescaling
and transforming its features. Table 6 shows the metrics involved in this
category.

Metric | Python-based Tools
---|---
Spearman’s rank correlation, Cosine similarity |
Table 6: Metric for a relationship between two datasets.

###  3.7 ASR Performance

The metric used for ASR systems is Word Error Rate (WER), which compares the
predicted output and the target transcript word by word to resolve the number
of differences between them. On the other hand, the Word Recognition Rate
(WRR) measures the noise level of ASR transcripts. Table 7

Metric | Python-based Tools
---|---
WER, WRR |
Table 7: Metric used for ASR performance systems for text augmentation
techniques.

###  3.8 Training

The metrics of this category give insight into the behavior of the
augmentation algorithms and their impact on model performance. An error rate,
defined as $1-accuracy$, refers to the degree of prediction error of a model.
Table 8 shows Cross-entropy loss that gives information on how correct a
particular prediction is.

Metric | Python-based Tools
---|---
Cross-entropy loss |
Table 8: Metric to measure prediction performance of the model after training.

###  3.9 Pre-trained language model robustness

The selection of a PLM and the layers for transfer learning improve the
performance on a target task and prevent negative transfer. Thus, TransRate
measures the transferability as the mutual information between features of
target examples extracted by a PLM and their labels. From the perspective of
feature representation, TransRate evaluates both the completeness and
compactness of pre-trained features. A set of metrics allows the improvement
of PLM. Attack accuracy is the core metric to reflect better robustness, where
a larger query number indicates that the vulnerability of the target model is
harder to detect. The perturbed percentage is the ratio of perturbed word
numbers to the text length, and a larger percentage reveals more difficulty in
successfully attacking the model. Table 9 shows metrics to select and improve
the robustness of PLMs.

Metric | Python-based Tools
---|---
TransRate | $TrR_{T_{s}}\rightarrow T_{t}(g,\in)=R(\hat{Z},\in)-R(\hat{Z},\in|Y)$
Attack accuracy, Query number, Perturbation Ratio |
Table 9: Metrics to select and improve robustness of pre-trained language
models.

###  3.10 Manipulated words importance

The word importance is measured by the explainability derived from an XAI
model. Subsequently, the explainability score of the word manipulated in the
augmentation process reflects the final labeling process of the augmented data
used to derive the soft labeling result. No computational implementation was
found, in that case, the explainability score ($S_{c}$) is calculated as
follows $S_{c}=[S_{c}(X^{1}_{1}),\dots,S_{c}(X^{n}_{m})]$.

##  4 Conclusion

This study investigates opportunities to explore and establish unified
evaluation metrics categories for improving text data augmentation evaluation.
The characteristics of the metrics and their applications in NLP tasks help to
understand and measure text augmentation good practices. In addition, it was
proposed a selection of computational tools that researchers and practitioners
can implement and use. The tools introduce various strategies to measure
quality augmented text data to enhance data augmentation work and make it more
accessible and reproducible. Is expected that the results of this work serve
as a guide for NLP researchers to spread the use and implementation of the
metric categories and computational tools to start inspiring additional works
in this area.

## References

  * [1] Markus Bayer, Marc-André Kaufhold, and Christian Reuter.  A survey on data augmentation for text classification.  ACM Comput. Surv., 55(7), 2022.
  * [2] Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy.  A survey of data augmentation approaches for NLP.  In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 968–988. Association for Computational Linguistics, 2021.
  * [3] Bohan Li, Yutai Hou, and Wanxiang Che.  Data augmentation approaches in natural language processing: A survey.  AI Open, 3:71–90, 2022.
  * [4] Pei Liu, Xuemin Wang, Chao Xiang, and Weiye Meng.  A survey of text data augmentation.  In 2020 International Conference on Computer Communication and Network Security (CCNS), pages 191–195, 2020.
  * [5] Connor Shorten, Taghi M. Khoshgoftaar, and Borko Furht.  Text data augmentation for deep learning.  Journal of Big Data, 8, 2021.
  * [6] Yang Li, Quan Pan, Suhang Wang, Tao Yang, and Erik Cambria.  A generative model for category text generation.  Information Sciences, 450:301–315, 2018.
  * [7] Rahul Gupta.  Data augmentation for low resource sentiment analysis using generative adversarial networks.  In ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7380–7384, 2019.
  * [8] Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, and Songlin Hu.  Conditional bert contextual augmentation, 2018.
  * [9] Georgios Rizos, Konstantin Hemker, and Björn Schuller.  Augment to prevent: Short-text data augmentation in deep learning for hate-speech classification.  In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, New York, NY, USA, 2019. Association for Computing Machinery.
  * [10] Tulu Tilahun Hailu, Junqing Yu, and Tessfu Geteye Fantaye.  Pre-trained word embedding based parallel text augmentation technique for low-resource nmt in favor of morphologically rich languages.  In Proceedings of the 3rd International Conference on Computer Science and Application Engineering, New York, NY, USA, 2019. Association for Computing Machinery.
  * [11] Zhenhao Li and Lucia Specia.  Improving neural machine translation robustness via data augmentation: Beyond back-translation.  In Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019). Association for Computational Linguistics, 2019.
  * [12] Jason Wei and Kai Zou.  Eda: Easy data augmentation techniques for boosting performance on text classification tasks.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6382–6388, Hong Kong, China, 2019. Association for Computational Linguistics.
  * [13] Shujuan Yu, Jie Yang, Danlei Liu, Runqi Li, Yun Zhang, and Shengmei Zhao.  Hierarchical data augmentation and the application in text classification.  IEEE Access, 7:185476–185485, 2019.
  * [14] Thien Ho Huong and Vinh Truong Hoang.  A data augmentation technique based on text for vietnamese sentiment analysis.  In Proceedings of the 11th International Conference on Advances in Information Technology, New York, NY, USA, 2020. Association for Computing Machinery.
  * [15] Heereen Shim, Stijn Luca, Dietwig Lowet, and Bart Vanrumste.  Data augmentation and semi-supervised learning for deep neural networks-based text classifier.  In Proceedings of the 35th Annual ACM Symposium on Applied Computing, page 1119–1126. Association for Computing Machinery, 2020.
  * [16] Vinícius Veríssimo and Rostand Costa.  Using data augmentation and neural networks to improve the emotion analysis of brazilian portuguese texts.  In Proceedings of the Brazilian Symposium on Multimedia and the Web, page 13–20, New York, NY, USA, 2020. Association for Computing Machinery.
  * [17] Jiaao Chen, Zichao Yang, and Diyi Yang.  MixText: Linguistically-informed interpolation of hidden space for semi-supervised text classification.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2147–2157. Association for Computational Linguistics, 2020.
  * [18] Siyuan Qiu, Binxia Xu, Jie Zhang, Yafang Wang, Xiaoyu Shen, Gerard de Melo, Chong Long, and Xiaolong Li.  Easyaug: An automatic textual data augmentation platform for classification tasks.  In Companion Proceedings of the Web Conference 2020, page 249–252, New York, NY, USA, 2020. Association for Computing Machinery.
  * [19] Steven Y. Feng, Varun Gangal, Dongyeop Kang, Teruko Mitamura, and Eduard Hovy.  Genaug: Data augmentation for finetuning text generators, 2020.
  * [20] Varun Kumar, Ashutosh Choudhary, and Eunah Cho.  Data augmentation using pre-trained transformer models.  In Proceedings of the 2nd Workshop on Life-long Learning for Spoken Language Systems, pages 18–26, Suzhou, China, 2020. Association for Computational Linguistics.
  * [21] Ruibo Liu, Guangxuan Xu, Chenyan Jia, Weicheng Ma, Lili Wang, and Soroush Vosoughi.  Data boost: Text data augmentation through reinforcement learning guided conditional generation.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9031–9041. Association for Computational Linguistics, 2020.
  * [22] Hongyu Guo.  Nonlinear mixup: Out-of-manifold data augmentation for text classification.  Proceedings of the AAAI Conference on Artificial Intelligence, 34(04):4044–4051, Apr. 2020.
  * [23] Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le.  Unsupervised data augmentation for consistency training.  In Advances in Neural Information Processing Systems, volume 33, pages 6256–6268. Curran Associates, Inc., 2020.
  * [24] Vukosi Marivate and Tshephisho Sefara.  Improving short text classification through global augmentation methods.  In Andreas Holzinger, Peter Kieseberg, A Min Tjoa, and Edgar Weippl, editors, Machine Learning and Knowledge Extraction, pages 385–399, Cham, 2020. Springer International Publishing.
  * [25] Lichao Sun, Congying Xia, Wenpeng Yin, Tingting Liang, Philip Yu, and Lifang He.  Mixup-transformer: Dynamic data augmentation for NLP tasks.  In Proceedings of the 28th International Conference on Computational Linguistics, pages 3436–3440, Barcelona, Spain (Online), 2020. International Committee on Computational Linguistics.
  * [26] Xiao Sun and Jiajin He.  A novel approach to generate a large scale of supervised data for short text sentiment analysis.  Multimedia Tools and Applications, 79, 2020.
  * [27] Fang Yuan, Xianbin Hong, Cheng Yuan, Xiang Fei, Sheng-Uei Guan, Dawei Liu, and Wei Wang.  Keywords-oriented data augmentation for chinese.  In 2020 IEEE 6th International Conference on Computer and Communications (ICCC), pages 2006–2012, 2020.
  * [28] Yong Kuan Shyang and Jasy Liew Suet Yan.  A text augmentation approach using similarity measures based on neural sentence embeddings for emotion classification on microblogs.  In 2020 IEEE 2nd International Conference on Artificial Intelligence in Engineering and Technology (IICAIET), pages 1–6, 2020.
  * [29] Nandan Thakur, Nils Reimers, Johannes Daxenberger, and Iryna Gurevych.  Augmented SBERT: Data augmentation method for improving bi-encoders for pairwise sentence scoring tasks.  In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 2021.
  * [30] Giannis Haralabopoulos, Mercedes Torres Torres, Ioannis Anagnostopoulos, and Derek McAuley.  Text data augmentations: Permutation, antonyms and negation.  Expert Systems with Applications, 177:114769, 2021.
  * [31] Xavier A. Carrasco, Ashraf Elnagar, and Mohammed Lataifeh.  A generative adversarial network for data augmentation: The case of arabic regional dialects.  Procedia Computer Science, 189:92–99, 2021.  AI in Computational Linguistics.
  * [32] Ernie Chang, Xiaoyu Shen, Dawei Zhu, Vera Demberg, and Hui Su.  Neural data-to-text generation with LM-based text augmentation.  In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 758–768. Association for Computational Linguistics, 2021.
  * [33] Sangwon Lee, Ling Liu, and Wonik Choi.  Iterative translation-based data augmentation method for text classification tasks.  IEEE Access, 9:160437–160445, 2021.
  * [34] Mohammadreza Varasteh and Arefeh Kazemi.  Using parsbert on augmented data for persian news classification.  In 2021 7th International Conference on Web Research (ICWR), pages 78–81, 2021.
  * [35] Qiuhao Lu, Dejing Dou, and Thien Huu Nguyen.  Textual data augmentation for patient outcomes prediction.  In 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pages 2817–2821, 2021.
  * [36] Yu Shang, Xiaohui Su, Zhifeng Xiao, and Zidong Chen.  Campus sentiment analysis with gan-based data augmentation.  In 2021 13th International Conference on Advanced Infocomm Technology (ICAIT), pages 209–214, 2021.
  * [37] Xiaomeng Liu, Fei Long, Kun Huang, and Qiang Ling.  Enhanced unsupervised data augmentation for emergency events detection and classification.  In 2021 33rd Chinese Control and Decision Conference (CCDC), pages 2367–2371, 2021.
  * [38] Jangseong Bae and Changki Lee.  Easy data augmentation for improved malware detection: A comparative study.  In 2021 IEEE International Conference on Big Data and Smart Computing (BigComp), pages 214–218, 2021.
  * [39] Mohammad Amin Rashid and Hossein Amirkhani.  The effect of using masked language models in random textual data augmentation.  In 2021 26th International Computer Conference, Computer Society of Iran (CSICC), pages 1–5, 2021.
  * [40] Kun Zhou, Wayne Xin Zhao, Sirui Wang, Fuzheng Zhang, Wei Wu, and Ji-Rong Wen.  Virtual data augmentation: A robust and general framework for fine-tuning pre-trained models.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3875–3887, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics.
  * [41] Wilton O. Júnior, Mauricio S. da Cruz, Andre Brasil Vieira Wyzykowski, and Arnaldo Bispo de Jesus.  The use of data augmentation as a technique for improving neural network accuracy in detecting fake news about covid-19, 2022.
  * [42] Yeong Jae Yu, Seung Joo Yoon, So Young Jun, and Jong Woo Kim.  Tabas: Text augmentation based on attention score for text classification model.  ICT Express, 8(4):549–554, 2022.
  * [43] Hugo Queiroz Abonizio, Emerson Cabrera Paraiso, and Sylvio Barbon.  Toward text data augmentation for sentiment analysis.  IEEE Transactions on Artificial Intelligence, 3(5):657–668, 2022.
  * [44] Sridevi Bonthu, Abhinav Dayal, M. Sri Lakshmi, and S. Rama Sree.  Effective text augmentation strategy for nlp models.  In Ramesh Chandra Poonia, Vijander Singh, Dharm Singh Jat, Mario José Diván, and Mohammed S. Khan, editors, Proceedings of Third International Conference on Sustainable Computing, pages 521–531, Singapore, 2022. Springer Nature Singapore.
  * [45] Shuokai Li, Xiang Ao, Feiyang Pan, and Qing He.  Learning policy scheduling for text augmentation.  Neural Networks, 145:121–127, 2022.
  * [46] Zijian Feng, Hanzhang Zhou, Zixiao Zhu, and Kezhi Mao.  Tailored text augmentation for sentiment analysis.  Expert Syst. Appl., 205(C), nov 2022.
  * [47] Mohit Wadhwa, Mohan Bhambhani, Ashvini Jindal, Uma Sawant, and Ramanujam Madhavan.  Fairness for text classification tasks with identity information data augmentation methods, 2022.
  * [48] Hazel H. Kim, Daecheol Woo, Seong Joon Oh, Jeong-Won Cha, and Yo-Sub Han.  Alp: Data augmentation using lexicalized pcfgs for few-shot text classification.  Proceedings of the AAAI Conference on Artificial Intelligence, 36(10):10894–10902, 2022.
  * [49] Daniel Saeedi, Sirwe Saeedi, Aliakbar Panahi, and Alvis C.M. Fong.  CS/NLP at SemEval-2022 task 4: Effective data augmentation methods for patronizing language detection and multi-label classification with RoBERTa and GPT3.  In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 503–508, Seattle, United States, 2022. Association for Computational Linguistics.
  * [50] Chang-Ting Chu, Mahdin Rohmatillah, Ching-Hsien Lee, and Jen-Tzung Chien.  Augmentation strategy optimization for language understanding.  In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7952–7956, 2022.
  * [51] Patawee Prakrankamanant and Ekapol Chuangsuwanich.  Tokenization-based data augmentation for text classification.  In 2022 19th International Joint Conference on Computer Science and Software Engineering (JCSSE), pages 1–6, 2022.
  * [52] Lucas Z. Ladeira, Frances Santos, Lucas Cléopas, Pieter Buteneers, and Leandro Villas.  Neo-nda: Neo natural language data augmentation.  In 2022 IEEE 16th International Conference on Semantic Computing (ICSC), pages 99–102, 2022.
  * [53] Jheng-Long Wu and Shuoyen Huang.  Application of generative adversarial networks and shapley algorithm based on easy data augmentation for imbalanced text data.  Applied Sciences, 12(21), 2022.
  * [54] Le Zhang, Zichao Yang, and Diyi Yang.  TreeMix: Compositional constituency-based data augmentation for natural language understanding.  In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5243–5258, Seattle, United States, 2022. Association for Computational Linguistics.
  * [55] Yanan Chen and Yang Liu.  Rethinking data augmentation in text-to-text paradigm.  In Proceedings of the 29th International Conference on Computational Linguistics, pages 1157–1162, Gyeongju, Republic of Korea, 2022. International Committee on Computational Linguistics.
  * [56] Byeong-Cheol Jo, Tak-Sung Heo, Yeongjoon Park, Yongmin Yoo, Won Ik Cho, and Kyungsun Kim.  Dagam: Data augmentation with generation and modification, 2022.
  * [57] Minyi Zhao, Lu Zhang, Yi Xu, Jiandong Ding, Jihong Guan, and Shuigeng Zhou.  Epida: An easy plug-in data augmentation framework for high performance text classification, 2022.
  * [58] Fanshuang Kong, Richong Zhang, Xiaohui Guo, Samuel Mensah, and Yongyi Mao.  DropMix: A textual data augmentation combining dropout with mixup.  In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 890–899, Abu Dhabi, United Arab Emirates, 2022. Association for Computational Linguistics.
  * [59] Hui Chen, Wei Han, Diyi Yang, and Soujanya Poria.  DoubleMix: Simple interpolation-based data augmentation for text classification.  In Proceedings of the 29th International Conference on Computational Linguistics, pages 4622–4632, Gyeongju, Republic of Korea, 2022. International Committee on Computational Linguistics.
  * [60] Dang Thanh Vu, Gwanghyun Yu, Chilwoo Lee, and Jinyoung Kim.  Text data augmentation for the korean language.  Applied Sciences, 12(7), 2022.
  * [61] Jinyi Yu, Jinhae Choi, and Younghoon Lee.  Mixing approach for text data augmentation based on an ensemble of explainable artificial intelligence methods.  Neural Processing Letters, 55, 2023.
  * [62] Huidong Tang, Sayaka Kamei, and Yasuhiko Morimoto.  Data augmentation methods for enhancing robustness in text classification tasks.  Algorithms, 16(1), 2023.
  * [63] Haixing Dai, Zhengliang Liu, Wenxiong Liao, Xiaoke Huang, Zihao Wu, Lin Zhao, Wei Liu, Ninghao Liu, Sheng Li, Dajiang Zhu, Hongmin Cai, Quanzheng Li, Dinggang Shen, Tianming Liu, and Xiang Li.  Chataug: Leveraging chatgpt for text data augmentation, 2023.
  * [64] Soonki Kwon and Younghoon Lee.  Explainability-based mix-up approach for text data augmentation.  ACM Trans. Knowl. Discov. Data, 17(1), feb 2023.