# Stethoscope-guided Supervised Contrastive Learning for Cross-domain
Adaptation on Respiratory Sound Classification

###### Abstract

Despite the remarkable advances in deep learning technology, achieving
satisfactory performance in lung sound classification remains a challenge due
to the scarcity of available data. Moreover, the respiratory sound samples are
collected from a variety of electronic stethoscopes, which could potentially
introduce biases into the trained models. When a significant distribution
shift occurs within the test dataset or in a practical scenario, it can
substantially decrease the performance. To tackle this issue, we introduce
cross-domain adaptation techniques, which transfer the knowledge from a source
domain to a distinct target domain. In particular, by considering different
stethoscope types as individual domains, we propose a novel stethoscope-guided
supervised contrastive learning approach. This method can mitigate any domain-
related disparities and thus enables the model to distinguish respiratory
sounds of the recording variation of the stethoscope. The experimental results
on the ICBHI dataset demonstrate that the proposed methods are effective in
reducing the domain dependency and achieving the ICBHI Score of 61.71%, which
is a significant improvement of 2.16% over the baseline.

Index Terms—  Cross-domain adaptation, stethoscope-guided supervised
contrastive learning, adversarial training, respiratory sound classification,
domain-shift problem

##  1 Introduction

An electronic stethoscope is a device utilized by medical professionals to
analyze sounds within the human body, including those produced by the heart,
lungs, and other organs. In general, the design and construction of the
electronic stethoscope can influence its efficacy in capturing and recording
respiratory sounds [1, 2]. Despite the significant advancements originating
from artificial intelligence and signal-processing domain in respiratory sound
analysis [3, 4, 5, 6, 7], the type and quality of the stethoscope used, as
well as the characteristics of the surrounding environment, can introduce bias
into the recorded breath sounds and affect their interpretation. This may lead
to a shift in data distribution [8], resulting in biased representations
within neural network models and hindering their ability to generalize [9,
10].

Fig. 1: Overview of the proposed stethoscope-guided cross-domain adaptation.
The proposed method attempts to minimize the dependency or distribution shift
between respiratory sounds recorded by different stethoscopes. Fig. 2: Overall
illustration of proposed stethoscope-guided supervised contrastive learning
for cross-domain adaptation.

To address this, we focus on the domain adaptation method that uses data
collected from the source domain to train the model for the target domain, as
in Fig. 1. The domain adaptation techniques facilitate the training of
effective models capable of extracting domain-invariant features, thereby
overcoming the challenges posed by domain dependencies in the data
distribution. Therefore, in the context of respiratory sound classification,
we consider each type of stethoscope as an individual domain for cross-domain
adaptation. Although there are differences in the perception of respiratory
sounds across different stethoscopes, we aim to consistently learn the
characteristics of both abnormal and normal sounds, even in the presence of
mixed samples in data distribution.

In this paper, we introduce two methods for cross-domain adaptation in a
supervised manner. First, we present a straightforward methodology that
leverages domain adversarial training inspired by [11]. And, for more
effective cross-domain adaptation, we propose a novel approach that
simultaneously considers both domain adversarial training and stethoscope-
guided supervised contrastive learning [12]. As shown in Fig. 2, the extracted
representations of a multi-viewed batch [12] are computed using supervised
contrastive loss, and their gradients are multiplied by a negative constant
during the back-propagation process to reverse the gradients. The gradient
reversal feature extractor forces the model to reduce the distribution shift
between different stethoscope classes while maintaining equivalence in the
same class.

Experimental results demonstrate that the proposed method achieves the ICBHI
[13] Score of 61.71% which represented 2.16% improvement over the baseline and
exhibited performance comparable to the state-of-the-art on the ICBHI dataset.
This suggests that the proposed method can be an effective approach for
addressing domain dependencies in terms of domain adaptation111Code is
available at https://github.com/kaen2891/stethoscope-
guided_supervised_contrastive_learning.

##  2 Preliminaries

###  2.1 Details of ICBHI Dataset

As shown in Table 1, the ICBHI [13] dataset was constructed using four
distinct stethoscope types: Meditron, LittC2SE, Litt3200, and AKGC417L, each
with its own unique input sensor. The Litt3200 uses a piezoelectric sensor to
capture low-frequencies like heart murmurs, while the AKGC417L and Meditron
use condenser microphones which are suitable for high-frequencies like
wheezes, with superior resolution for wider frequencies. The LittC2SE is an
analog stethoscope with a dynamic microphone ideal for mid-frequencies to
strike a balance for general auscultation.

In lung sound labels, there are four classes in train/test sets: normal,
crackle, wheeze, and both (crackle + wheeze). In stethoscope labels, however,
the absence of LittC2SE data in the test set as well as the dependency on data
quantity between the training and test sets suggests that conventional
training methods might be biased, potentially leading to performance
degradation.

Table 1: Overall details of the ICBHI lung sound dataset. We report two
perspectives: lung sound and device (stethoscope).

|  label |  train |  test |  sum
---|---|---|---|---
lung |  Normal |  2,063 |  1,579 |  3,642
Crackle |  1,215 |  649 |  1,864
Wheeze |  501 |  385 |  886
Both |  363 |  143 |  506
device |  Meditron |  997 |  459 |  1,459
LittC2SE |  594 |  0 |  594
Litt3200 |  41 |  461 |  502
AKGC417L |  2,510 |  1,836 |  4,346

###  2.2 Preprocessing Details

We followed the pretrained Audio Spectrogram Transformer [14] (AST) settings
for the ICBHI dataset with SpecAugment [15] as presented in [7]. Specifically,
we resampled all lung sound samples to 16 kHz, fixed their lengths to 8
seconds, and extracted 128-dim Fbank features with window/overlap sizes of
25/10ms, respectively. We also normalized the features using the mean and
standard deviation of –4.27 and 4.57 [14].

###  2.3 Training Details

We used the official split of the ICBHI dataset (train-test split as 60-40%),
and the default experimental setup for the AST and other architectures as
described in [7]. We utilized a weighted cross-entropy (CE) loss [3, 7, 16] to
alleviate the imbalanced data classification problem, except when employing
the proposed domain adaptation methods. For the domain classifier in domain
adversarial training, we used a 4-class linear head. For stethoscope-guided
supervised contrastive learning, we utilized a projector that consists of two
768-dimensional MLP layers, followed by ReLU and BN layers. We set the
temperature parameter $\tau=0.06$, and the domain adaptation parameter
$\lambda$ is initiated at 0.0096 and is gradually increased to 1.0 until
reaching 200 training steps with a batch size of 8 and 50 epoch settings. We
reported results using a fixed set of five seeds for our experiments.

###  2.4 Evalutation Metrics

We used Sensitivity ($S_{e}$), Specificity $(S_{p})$, and Score as the
performance metrics for the ICBHI dataset [13]. Sensitivity, also known as the
true positive rate, indicates the percentage of actual disease cases that are
accurately detected. Specificity, on the other hand, indicates the percentage
of healthy individuals who are rightly classified as disease-free. We used the
Score metric that is calculated by the arithmetic mean of Specificity and
Sensitivity to achieve a balance between them.

##  3 Proposed Method

###  3.1 Baselines

The AST [14] has shown superior performance on a variety of audio-related
tasks, owing to its pretrained weights derived from large-scale datasets such
as ImageNet [17] and AudioSet [18]. AST has also been used in studies of lung
sound classification, achieving state-of-the-art performance using a simple
fine-tuning approach [7]. Therefore, we use the AST model setting described in
[7] as a baseline for our respiratory sound classification task.

###  3.2 Domain Adversarial Training

To address the variations in data distribution between different stethoscope
domains, we propose a simple stethoscope Domain Adversarial Training (DAT)
approach motivated by [11]. The proposed DAT is comprised of two losses:

$\displaystyle\mathcal{L}_{\text{CE}}\\!=-\sum_{i=1}^{n}\\!\,y_{i}\\!\,\log\,\\!(\hat{y_{i}}),\quad\mathcal{L}_{\text{DA}}\\!=-\sum_{i=1}^{n}\\!\,d_{i}\\!\,\log\,\\!(\hat{d_{i}}).$
(1)

where $\mathcal{L}_{\text{CE}}$ and $\mathcal{L}_{\text{DA}}$ are CE loss with
label $y$ and stethoscope domain label $d$ (division by $N$ is omitted), and
the predicted probabilities $\hat{y}$ and $\hat{d}$ are obtained by passing
through the class and domain classifiers, respectively. We formulate the
proposed DAT as:
$\mathcal{L}_{\text{DAT}}=\mathcal{L}_{\text{CE}}+\lambda\,\\!\mathcal{L}_{\text{DA}}$
where $\lambda$ is a domain regularization parameter drawn from [11]. In other
words, the goal of DAT is to minimize the classification error for the
respiratory sound class and to ensure that the learned features cannot
distinguish between the stethoscope domains.

###  3.3 Proposed Method: SG-SCL

For more effective learning, we propose a novel approach that simultaneously
considers both domain adversarial training and Stethoscope-Guided Supervised
Contrastive Learning (SG-SCL) for cross-domain adaptation. In detail, the loss
for the proposed SG-SCL can be formulated as
$\mathcal{L}_{\text{CE}}\,+\,\\!\,\lambda\\!\,\mathcal{L}_{\text{CL}}$, where
$\mathcal{L}_{\text{CL}}$ is as follows:

$\displaystyle\sum_{i\in I}-\log\Big{\\{}\\!\frac{1}{|P(i)|\\!}\sum_{p\in
P(i)}\\!\frac{e\,\\!(h(z_{i})\\!\cdot\texttt{sgd}\,(h(z_{p}))\\!\,/\tau)}{\sum_{a\in
A(i)}\\!e\,\\!(h(z_{i})\cdot\texttt{sgd}\,(h(z_{a}))\\!\,/\tau)}\Big{\\}}$ (2)

where the index $i$ is the anchor index from $A(i)\equiv\,I\setminus\\{i\\}$,
and $P(i)\equiv\,\\{p\in\\!A(i):d_{p}=d_{i}\\}$ represents the collection of
all positive samples within the multi-viewed batch that corresponds to the
$i$-th sample [12]. $e$ and sgd denote the exponential function and stop-
gradient222is a technique employed to selectively prevent the backpropagation
algorithm from updating the weights of certain layers during training.
operation, respectively. $h$ is the projector described in Section 2.3, which
takes the feature extractor outputs $z$ as input. Both $z$ and $h$ have the
same dimension, and all $z$ are normalized before the dot product. The $\tau$
controls the sharpness of the cosine similarity.

In other words, the augmented samples from the multi-viewed batch are first
fed into the encoder and then computed with Eq. 2. This encourages the model
to reduce the dependencies between distinct classes while maintaining
equivalence in the same class. Note that the gradient reversal in Fig. 2
depicted that the gradients are multiplied by a negative constant during the
back-propagation process.

Table 2: SG-SCL performance based on two factors: anchor representation $z_{i}$ and that of target $z_{p}$. sgd indicate the stop-gradient operation. Bold denotes the best result. The final implementation choices were highlighted.  anchor | target |  $S_{p}$ (%) |  $S_{e}$ (%) |  Score (%)
---|---|---|---|---
$z_{i}$ | $z_{p}$ | $\text{\bf 89.84}_{\pm 3.92}$ | $\text{{13.61}}_{\pm 5.67}$ | $\text{51.73}_{\pm 1.53}$
$z_{i}$ | $h(z_{p})$ | $\text{76.30}_{\pm 1.55}$ | $\text{\bf 44.60}_{\pm 2.20}$ | $\text{60.45}_{\pm 0.44}$
$h(z_{i})$ | $z_{p}$ | $\text{81.87}_{\pm 3.20}$ | $\text{39.83}_{\pm 1.05}$ | $\text{60.85}_{\pm 1.60}$
$h(z_{i})$ | $h(z_{p})$ | $\text{{77.25}}_{\pm 3.43}$ | $\text{{36.35}}_{\pm 17.97}$ | $\text{{60.78}}_{\pm 0.85}$
$h(z_{i})$ | sgd$(z_{p})$ | $\text{{76.31}}_{\pm 6.35}$ | $\text{{43.79}}_{\pm 4.38}$ | $\text{60.05}_{\pm 1.19}$
$h(z_{i})$ | sgd$(h(z_{p}))$ | $\text{79.87}_{\pm 8.89}$ | $\text{{43.55}}_{\pm 5.93}$ | $\text{\bf 61.71}_{\pm 1.61}$
Table 3: Respiratory sound classification performance of different
architectures using CE, DAT, and SG-SCL. Here, IN and AS refer to ImageNet
[17] and AudioSet [18], respectively. Bold denotes the best result.

architecture | method | pretrain |  $S_{p}$ (%) |  $S_{e}$ (%) |  Score (%)
---|---|---|---|---|---
EfficientNet | CE | IN | $\text{73.48}_{\pm 5.93}$ | $\text{\bf 39.24}_{\pm 2.43}$ | $\text{{57.46}}_{\pm 1.05}$
DAT | $\text{\bf 89.99}_{\pm 7.15}$ | $\text{11.78}_{\pm 7.05}$ | $\text{50.89}_{\pm 0.69}$
SG-SCL | $\text{{81.58}}_{\pm 3.38}$ | $\text{{33.85}}_{\pm 3.75}$ | $\text{\bf 57.72}_{\pm 1.32}$
ResNet18 | CE | IN | $\text{74.72}_{\pm 3.43}$ | $\text{{33.95}}_{\pm 3.88}$ | $\text{{54.33}}_{\pm 0.91}$
DAT | $\text{\bf 91.26}_{\pm 6.32}$ | $\text{12.03}_{\pm 5.86}$ | $\text{51.51}_{\pm 0.34}$
SG-SCL | $\text{{75.58}}_{\pm 6.36}$ | $\text{\bf 34.63}_{\pm 5.98}$ | $\text{\bf 55.10}_{\pm 1.18}$
CNN6 | CE | IN | $\text{{80.13}}_{\pm 2.64}$ | $\text{{35.91}}_{\pm 3.52}$ | $\text{{58.10}}_{\pm 0.59}$
DAT | $\text{\bf 88.57}_{\pm 5.66}$ | $\text{13.73}_{\pm 6.47}$ | $\text{51.15}_{\pm 0.45}$
SG-SCL | $\text{78.16}_{\pm 3.49}$ | $\text{\bf 38.05}_{\pm 4.41}$ | $\text{\bf 58.11}_{\pm 0.64}$
AST | CE | IN + AS | $\text{{77.14}}_{\pm 3.35}$ | $\text{{41.97}}_{\pm 2.21}$ | $\text{59.55}_{\pm 0.88}$
DAT | $\text{77.11}_{\pm 7.20}$ | $\text{{41.99}}_{\pm 5.00}$ | $\text{{59.81}}_{\pm 1.25}$
SG-SCL | $\text{\bf 79.87}_{\pm 8.89}$ | $\text{{\bf 43.55}}_{\pm 5.93}$ | $\text{\bf 61.71}_{\pm 1.61}$

Our experimental results in Table 2 show that the most promising results were
obtained when the stop-gradient was applied to the target representations,
$z_{p}$, which are the second augmented samples from the same source sample in
the multi-view batch. We found that allowing gradient flows through both
anchor and target representations, $z_{i}$ and $z_{p}$, simultaneously did not
improve the performance. This suggests that the stop-gradient applied to the
target representations $z_{p}$ is more effective in the latent space for
cross-domain adaptation.

##  4 Results

Table 4: Comprehensive comparison of the ICBHI dataset for the respiratory
sound classification task. We compared previous studies that adhered to the
official split of the ICBHI dataset (60-40% split for the train/test set).
Scores marked with $*$ denote the previous state-of-the-art performance. Best
and second best results.

| method | architecture | pretrain | venue |  $S_{p}$ (%) |  $S_{e}$ (%) |  Score (%)
---|---|---|---|---|---|---|---
4-class eval. | CNN-MoE [19] | C-DNN | - | JBHI‘21 | 72.40 | 21.50 | 47.00
RespireNet [3] (CBA+BRC+FT) | ResNet34 | IN | EMBC‘21 | 72.30 | 40.10 | 56.20
Ren et al. [4] | CNN8-Pt | - | ICASSP‘22 | 72.96 | 27.78 | 50.37
Chang et al. [20] | CNN8-dilated | - | INTERSPEECH‘22 | 69.92 | 35.85 | 52.89
Chang et al. [20] | ResNet-dilated | - | INTERSPEECH‘22 | 50.22 | 51.83 | 51.02
Wang et al. [5] (Splice) | ResNeSt | IN | ICASSP‘22 | 70.40 | 40.20 | 55.30
Nguyen et al. [6] (CoTuning) | ResNet50 | IN | TBME‘22 | 79.34 | 37.24 | 58.29
Moummad et al. [16] (SCL) | CNN6 | AS | arXiv‘22 | 75.95 | 39.15 | 57.55
Bae et al. [7] (Fine-tuning) | AST | IN + AS | INTERSPEECH‘23 | 77.14 | 41.97 | 59.55
Bae et al. [7] (Patch-Mix CL) | AST | IN + AS | INTERSPEECH‘23 | 81.66 | 43.07 | $\text{\bf 62.37}^{\textbf{*}}$
DAT [ours] | AST | IN + AS | ICASSP‘24 | $\text{77.11}_{\pm 7.2}$ | $\text{42.50}_{\pm 5.39}$ | $\text{59.81}_{\pm 1.25}$
SG-SCL [ours] | AST | IN + AS | ICASSP‘24 |<EMAIL_ADDRESS>8.89}$ | $\text{\bf{43.55}}_{\pm 5.93}$ |<EMAIL_ADDRESS>1.61}$
2-class eval. | CNN-MoE [19] | C-DNN | - | JBHI‘21 | 72.40 | 37.50 | 54.10
Nguyen et al. [6] (CoTuning) | ResNet50 | IN | TBME‘22 | 79.34 | 50.14 | 64.74
Bae et al. [7] (Fine-tuning) | AST | IN + AS | INTERSPEECH‘23 | 77.14 | 56.40 | 66.77
Bae et al. [7] (Patch-Mix CL) | AST | IN + AS | INTERSPEECH‘23 | 81.66 | 55.77 |<EMAIL_ADDRESS>
DAT [ours] | AST | IN + AS | ICASSP‘24 | $\text{77.11}_{\pm 7.2}$ |<EMAIL_ADDRESS>7.42}$ | $\text{67.04}_{\pm 1.29}$
SG-SCL [ours] | AST | IN + AS | ICASSP‘24 |<EMAIL_ADDRESS>8.89}$ | $\text{\bf 57.97}_{\pm 8.96}$ | $\text{\bf 68.93}_{\pm 1.47}$

###  4.1 Effectiveness of Proposed SG-SCL

To validate the effectiveness of the proposed methods, we trained CE, DAT, and
SG-SCL methods on different architectures with the ICBHI dataset under the
same conditions without additional learning techniques. As shown in Table 3,
the proposed SG-SCL method achieved the best Score in all architectures, while
the DAT method showed the lowest performance except for the AST architecture.
We conjecture that these outcomes are due to potential underfitting issues
inherent to the DAT domain classifier, which may come out from the limitation
of the CNN-based architectural designs.

###  4.2 Overall ICBHI Dataset Results

As summarized in Table 4, we conducted an overall comparison of the ICBHI
dataset for the lung sound classification task, including our proposed
methods. In 4-class evaluation, the proposed SG-SCL achieved a 61.71% Score,
which outperforms the AST fine-tuning model [7] by 2.16%. Interestingly, our
SG-SCL achieved a state-of-the-art Score in 2-class evaluation, with a value
of 68.93%. Moreover, our SG-SCL obtained the highest Sensitivity $S_{e}$ in
both 4-class and 2-class evaluations, with the values of 43.55%, 57.97%. These
results suggest that our SG-SCL is the most accurate model for actually
classifying abnormal lung sounds. We would like to highlight that the proposed
stethoscope-guided supervised contrastive learning for cross-domain adaptation
methodology achieved a significant improvement of 2.16% over the AST baseline,
without any additional learning techniques such as device-specific fine-tuning
[3], co-tuning [6], or Patch-Mix augmentation [7].

###  4.3 Qualitative Analysis

To analyze the extent to which our SG-SCL mitigates domain dependencies, Fig.
3 provides the t-SNE results generated by the outputs of the feature extractor
in terms of stethoscope labels from the ICBHI test set for both the AST fine-
tuning and SG-SCL. The AST fine-tuning results in Fig. 3(a) show that the
representations are clustered according to the stethoscopes, while our SG-SCL
results in Fig. 3(b) are well mixed regardless of the stethoscope type.

(a) AST fine-tuning

(b) Proposed SG-SCL

Fig. 3: T-SNE results of AST fine-tuning and SG-SCL on ICBHI test set for
stethoscope labels.

##  5 Conclusion

The scarcity of data limits the learning ability of neural networks, and
potential biases from various electronic devices for obtaining medical data
cause domain dependencies, which can lead to overfitting and poor
generalization performance. In this paper, we addressed these challenges by
introducing domain adversarial training techniques and proposing a novel
stethoscope-guided supervised contrastive learning approach for cross-domain
adaptation. Experimental results on the ICBHI dataset demonstrated the
effectiveness of the proposed methods, achieving the ICBHI Score of 61.71%,
which is a significant improvement of 2.16% over the baseline. Consequently,
this suggested that our method can effectively alleviate domain-related
mismatches by learning to distinguish respiratory sounds regardless of the
recording devices.

## References

  * [1] Valerie Rennoll, Ian McLane, Dimitra Emmanouilidou, James West, and Mounya Elhilali,  “Electronic stethoscope filtering mimics the perceived sound characteristics of acoustic stethoscope,”  IEEE journal of biomedical and health informatics, vol. 25, no. 5, pp. 1542–1549, 2020.
  * [2] Jun Jie Seah, Jiale Zhao, De Yun Wang, and Heow Pueh Lee,  “Review on the advancements of stethoscope types in chest auscultation,”  Diagnostics, vol. 13, no. 9, pp. 1545, 2023.
  * [3] Siddhartha Gairola, Francis Tom, Nipun Kwatra, and Mohit Jain,  “Respirenet: A deep neural network for accurately detecting abnormal lung sounds in limited data setting,”  in 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). IEEE, 2021, pp. 527–530.
  * [4] Zhao Ren, Thanh Tam Nguyen, and Wolfgang Nejdl,  “Prototype learning for interpretable respiratory sound analysis,”  in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 9087–9091.
  * [5] Zijie Wang and Zhao Wang,  “A domain transfer based data augmentation method for automated respiratory classification,”  in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 9017–9021.
  * [6] Truc Nguyen and Franz Pernkopf,  “Lung sound classification using co-tuning and stochastic normalization,”  IEEE Transactions on Biomedical Engineering, vol. 69, no. 9, pp. 2872–2882, 2022.
  * [7] Sangmin Bae, June-Woo Kim, Won-Yang Cho, Hyerim Baek, Soyoun Son, Byungjo Lee, Changwan Ha, Kyongpil Tae, Sungnyun Kim, and Se-Young Yun,  “Patch-Mix Contrastive Learning with Audio Spectrogram Transformer on Respiratory Sound Classification,”  in Proc. INTERSPEECH 2023, 2023, pp. 5436–5440.
  * [8] Hao Guan and Mingxia Liu,  “Domain adaptation for medical image analysis: a survey,”  IEEE Transactions on Biomedical Engineering, vol. 69, no. 3, pp. 1173–1185, 2021.
  * [9] Joaquin Quinonero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence,  Dataset shift in machine learning,  Mit Press, 2008.
  * [10] Antonio Torralba and Alexei A Efros,  “Unbiased look at dataset bias,”  in CVPR 2011. IEEE, 2011, pp. 1521–1528.
  * [11] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky,  “Domain-adversarial training of neural networks,”  The journal of machine learning research, vol. 17, no. 1, pp. 2096–2030, 2016.
  * [12] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan,  “Supervised contrastive learning,”  Advances in neural information processing systems, vol. 33, pp. 18661–18673, 2020.
  * [13] BM Rocha, Dimitris Filos, L Mendes, Ioannis Vogiatzis, Eleni Perantoni, E Kaimakamis, P Natsiavas, Ana Oliveira, C Jácome, A Marques, et al.,  “A respiratory sound database for the development of automated classification,”  in Precision Medicine Powered by pHealth and Connected Health: ICBHI 2017, Thessaloniki, Greece, 18-21 November 2017. Springer, 2018, pp. 33–37.
  * [14] Yuan Gong, Yu-An Chung, and James Glass,  “AST: Audio Spectrogram Transformer,”  in Proc. Interspeech 2021, 2021, pp. 571–575.
  * [15] Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le,  “Specaugment: A simple data augmentation method for automatic speech recognition,”  Interspeech 2019, 2019.
  * [16] Ilyass Moummad and Nicolas Farrugia,  “Supervised contrastive learning for respiratory sound classification,”  arXiv preprint arXiv:2210.16192, 2022.
  * [17] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei,  “Imagenet: A large-scale hierarchical image database,”  in 2009 IEEE conference on computer vision and pattern recognition. Ieee, 2009, pp. 248–255.
  * [18] Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter,  “Audio set: An ontology and human-labeled dataset for audio events,”  in Proc. IEEE ICASSP 2017, New Orleans, LA, 2017.
  * [19] Lam Pham, Huy Phan, Ramaswamy Palaniappan, Alfred Mertins, and Ian McLoughlin,  “Cnn-moe based framework for classification of respiratory anomalies and lung disease detection,”  IEEE journal of biomedical and health informatics, vol. 25, no. 8, pp. 2938–2947, 2021.
  * [20] Yi Chang, Zhao Ren, Thanh Tam Nguyen, Wolfgang Nejdl, and Björn W. Schuller,  “Example-based Explanations with Adversarial Attacks for Respiratory Sound Analysis,”  in Proc. Interspeech 2022, 2022, pp. 4003–4007.


# Data Generation Using Large Language Models for Text Classification:
An Empirical Case Study

Yinheng Li  Rogerio Bonatti  Sara Abdali  Justin Wagle  Kazuhito Koishida

###### Abstract

Using Large Language Models (LLMs) to generate synthetic data for model
training has become increasingly popular in recent years. While LLMs are
capable of producing realistic training data, the effectiveness of data
generation is influenced by various factors, including the choice of prompt,
task complexity, and the quality, quantity, and diversity of the generated
data. In this work, we focus exclusively on using synthetic data for text
classification tasks. Specifically, we use natural language understanding
(NLU) models trained on synthetic data to assess the quality of synthetic data
from different generation approaches. This work provides an empirical analysis
of the impact of different factors and offers recommendations for better data
generation practices.

Synthetic Data Generation, Large Language Model, Data Augmentation, Text
Classification

##  1 Introduction

Data augmentation is a method that utilize existing data to generate
additional training data without collecting more data (Feng et al., 2021). It
is an effective solution to improve model performance when limited data is
available (Xie et al., 2020). With the emergence of large language models,
data augmentation becomes even more accessible and has been successfully
applied in training language models (Gunasekar et al., 2023; Liu et al.,
2024).

Using LLM to generate or annotate data is a cost-efficient alternative to
human-labeled data. While human-labeled data tends to have higher quality,
leveraging LLM with well-designed prompts can also generate data that achieves
comparable model performance at a much lower cost. As estimated in (Ding et
al., 2023), labeling 3000 samples for SST-2 task (Socher et al., 2013) would
cost between 221 to 300 USD and take around 1000 minutes. In contrast,
generating the same amount of data using GPT-3 only costs 14.37 USD and takes
46 minutes. With only 6000 samples generated by GPT-3, the model is able to
achieved 76% accuracy, compared to 88% from human-curated data.

Our research focuses on synthetic data generation using large language models
(LLMs) for text classification tasks, specifically tasks uses natural language
understanding models with transformer encoder architecture. In the scope of
this study, we use the terms data augmentation and data generation
interchangeably, as LLMs often require a few in-context samples to generate
data. The data produced in this way can be considered augmented from these in-
context samples. Meanwhile, we focus solely on tasks that have limited or no
data at all, as our experiments have shown that tasks with sufficient data
receive minimal improvements from additional synthetic data. Numerous studies
have proposed various frameworks to improve the quality of synthetic data
generation (Wang et al., 2023; Gao et al., 2023; Gupta et al., 2023). However,
to the best of our knowledge, few works have addressed the fundamental
questions associated with LLM for data generation. These questions include:

  * •

What is the optimal amount of data to generate, and does increasing the volume
of synthetic data improve model performance?

  * •

Can in-context learning (generation) enhance the quality of synthetic data,
would providing a few examples lead to higher quality data than zero-shot
generation?

  * •

Does the LLM’s performance on a particular task directly influence the quality
of the generated synthetic data for this task?

  * •

Is combining synthetic data with raw data beneficial for model training?

  * •

Is the synthetic data diversity an important factor for model performance?

We experimented with six common NLP tasks (Table 1) with different data
generation methods. We found it is very challenging to pinpoint a definitive
answer to the questions above that applies universally to all NLP tasks due to
their inherent differences. Nevertheless, the findings from 6 tasks offer
valuable insights into practical data generation techniques.

##  2 Related Work

#### Data Augmentation

The goal of data augmentation is to increase diversity of existing data by
exposing the model to unseen data. This method has been applied to many
domains in computer vision (Yang et al., 2023) and natural language processing
(Li et al., 2022). In (Feng et al., 2021), augmentation techniques are
categorized into rule based generation and model based generation. Rule based
generation are used in computer vision tasks including image transformations,
such as rotation, flipping, and cropping(Mikołajczyk & Grochowski, 2018),
while model based generation has been widely used in natural language
processing tasks, such as rephrasing and back translation (Kumar et al., 2019;
Yang et al., 2020; Cai et al., 2020; Ye et al., 2022; Okur et al., 2022b).

#### Large Language models (LLMs)

With the development of large language models, model based data augmentation
for NLP becomes trivial (Zhou et al., 2024). By instructing LLM with proper
prompt, it is able to generate a new example in human like text. While it is
easy to implement, the synthetic data generated from LLM is usually noisy and
has a different distribution compared with raw data, which hampers the
training performance. Lots of work has explored ways to deal with this issue.
The work from (Veselovsky et al., 2023) uses techniques like grounding,
providing taxonomy and filtering to ensure the quality of synthetic data by
LLM. Synthesis Step by Step (Wang et al., 2023) uses an iterative step to
create prompt based on misclassified golden data to reduce the gap between the
synthesized data distribution and gold distribution. SunGen (Gao et al., 2023)
uses weighted loss to reduce the impact of noise from synthetic data during
training.

##  3 Methods

Figure 1: Pipeline for Data Augmentation using LLM

We follow the workflow in Figure 1 for our experiment. We explore the
following in-context data generation methods. The term ”in-context generation”
refers to using an LLM to generate data for training given a specific context,
similar to in-context learning (Brown et al., 2020). The methods we
investigate can be categorized as follows:

  * •

Zero-shot in-context generation: Provide the task description in the prompt
and ask the LLM to generate a similar example.

  * •

One-shot in-context generation: Provide the task description and one example,
prompting the LLM to generate a similar example.

  * •

Few-shot in-context generation: Provide the task description and a few
examples, prompting the LLM to generate a similar example.

Inspired by the work from (Yu et al., 2023), we also experiment with an
additional method called zero-shot topic in-context generation:

  * •

Zero-shot topic in-context generation: Use the LLM to generate a list of
topics (see Appendix A). Provide the task description and sample one topic
from the list to prompt the LLM to generate a similar example.

To evaluate the success of synthetic data generation, we train a NLU model on
the synthetic data and assess its performance on the task’s validation set. We
then compare the performance of the model trained on synthetic data with that
of the model trained on the original data. Following the practice established
in previous works (Li et al., 2023), we consider the generated data is better
if it results in better model performance.

##  4 Experiments

In our experiment, GPT-3.5 turbo111 GPT-3.5 version: 2024-02-15 preview
accessed from Azure OpenAI Studio is selected for all data generation process
except for topic generation (see appendix A). Although more powerful models
like GPT-4 is available, we decided to use GPT-3.5 turbo due to the resource
constrain, especially we need to run the large number of inferences for our
data generation experiment. Overall, GPT-3.5 turbo is a well-rounded model
with competitive performance across multiple benchmarks (Liang et al., 2023).
It would be interesting to compare the quality of synthetic data generated
from different LLMs, which we plan to explore in the future.

Existing work (Gupta et al., 2023) have utilized common NLP benchmarks, such
as SuperGLUE (Wang et al., 2019), as tasks for evaluation or employ a
customized selection of existing benchmarks(Gao et al., 2023; Ye et al.,
2022).

We select six common tasks for evaluation: SST-2 (Socher et al., 2013; Wang et
al., 2019), Twitter Emotion Classification (EMO) (Saravia et al., 2018), New
York Times News Classification (NYT)(Stefano, 2021), Review (Amazon Review
Classification) (Keung et al., 2020), RTE (Recognizing Textual Entailment)
(Bentivogli et al., 2009; Wang et al., 2019) and BoolQ (Clark et al., 2019;
Wang et al., 2019). The goal is to select diverse tasks that represent a wide
range of popular NLP corpora (Table 1). Additionally, we try to include
challenging tasks for which current NLU models do not perform well when
provided with limited training data. Therefore, we do not use the entire GLUE
benchmark, as models like BERT (Devlin et al., 2019) or RoBERTa(Liu et al.,
2019) can easily achieve high accuracy on such tasks. We also do not use the
complete SuperGLUE task collection since some of its tasks require token-level
classification. In this work, we focus on sequence-to-sequence and sequence
pair classification tasks. The six selected tasks cover common web data, such
as news and Wikipedia, as well as popular user data, like Twitter, movie
reviews, and product reviews. They cover binary classification, multi-class
classification, and question-answering tasks.

For the evaluation metric, the default metric is accuracy, but we use F1 or
Macro-F1 to calculate the performance since these metrics provide a more
balanced and comprehensive assessment of classification performance, taking
into account both precision and recall, especially in cases of multi-class
classification tasks. In our experiment, RoBERTa is selected as the NLU model
for all tasks, as it is a commonly used model for benchmark on these tasks.

We experiment five in-context generation methods for each task: zero-shot,
zero-shot topic, one-shot, few-shot with 3 examples, few-shot with 5 examples.
Prompt used in the generation can be found in Appendix C.

Corpus | Training Size | Test Size |  Task | Metrics |  Domain
---|---|---|---|---|---
SST-2 | 67k | 1.8k |  Binary Classification | F1 |  Movie Reviews
EMO | 16k | 2k |  Multi-class Classification | Macro-F1 |  Twitter
NYT | 256k |  3k222The official dataset does not provide testset, we randomly sample 3k data as testset |  Multi-class Classification | Macro-F1 |  News
Review | 200k | 5k |  Multi-class, Ordinal Regression | Macro-F1 |  Amazon Review
RTE | 2.5k | 3k |  Pair Classification, Question Answering | Macro-F1 |  News, Wikipedia
BoolQ | 16k | 3.2k |  Pair Classification, Question Answering | Macro-F1 |  News, Wikipedia, Web Query
Table 1: Summary of datasets and tasks.

In our experiment, we generate 1,000 synthetic data points per task, as we
found the benefit of additional synthetic data diminishes after that. To
simulate a low-resource setting, we allow only 100 raw examples to be used for
one-shot and few-shot generation. For zero-shot topic generation, we generate
500 random topics related to the task domain. Details can be found in Appendix
A.

##  5 Key Findings

In this section, we present the key findings from our experiments.

###  5.1 Mixing Raw Data is Necessary

Figure 2: Performance of different prompting methods with and without
augmentation. Synthetic only: use 1000 synthetic data only. Augmented: 1000
synthetic data plus 100 raw data

To assess the effectiveness of data augmentation, we train models with pure
synthetic data and augmented data. For the augmented setting, 100 raw data
points are mixed with 1000 synthetic data. In the data generation stage, we
use only the same 100 raw data points used for in-context generation to
prevent the model from accessing additional data. As shown in Figure 2, we
observe significant improvements across all tasks for most prompting methods
when incorporating raw data into training. Even as few as 100 data points can
boost synthetic data performance compared to using only synthetic data.

###  5.2 Impact of Bias

In the BoolQ task, we found that the zero-shot generation method outperforms
other methods, which contrasts with the results obtained for the rest of the
tasks. This finding is intriguing since zero-shot data exhibits the highest
repetition rate, which is detrimental to model training. Upon further
examination, we noticed that only in the datasets generated using one-shot or
few-shot methods, terms like ”not,” ”significant,” ”only,” ”just,” ”few,” and
”little” frequently appear in the generated questions. These terms create a
tone that can be used to imply the answer to the question (which is often
False). Table 2 provides an example of such trivial question. Table 3 provides
statistics for such questions from different prompting method.

We hypothesize that this pattern introduces bias in model training by
encouraging the model to search for specific keywords in the question rather
than reading the passage. To test this hypothesis, we instruct the LLM to
rephrase the questions like ”what people would search online” for each
synthetic example (see Appendix B). We found that performance significantly
improved for zero-shot topic and one-shot method after rephrasing. The work
(Okur et al., 2022a) has also shows the effectiveness of paraphrasing in other
data augmentation techniques.

Although we only detected synthetic bias in the BoolQ task, it remains an
important factor to consider during data generation. The technique of
rephrasing might not be applicable to other cases, but ensuring that synthetic
data does not contain unwanted patterns is necessary.

For all the rest experiments, the results for BoolQ task are all under the
question rephrasing setting unless otherwise specified.

Did the Mars Exploration Rover mission only involve one rover? – False
---
Did scientists in the 20th century make no significant discoveries or
advancements? – False
Table 2: Examples of Trivial Questions – questions contain terms ”not,”
”significant,” ”only,” ”just,” ”few,” and ”little”.
| Trivial Q. Count | F1 Score
---|---|---
| Raw | Rephrased | Raw (SD) | Raw (SD) | Raw (AD) | Rephrased (AD)
Zero-Shot Topic | 230 | 208 | 0.19 | 0.77 | 0.75 | 0.77
One-Shot | 131 | 74 | 0.38 | 0.74 | 0.76 | 0.77
Few-Shot (3 ex.) | 90 | 30 | 0.55 | 0.51 | 0.70 | 0.72
Few-Shot (5 ex.) | 57 | 28 | 0.53 | 0.48 | 0.75 | 0.73
Zero-Shot | 11 | - | 0.71 | - | 0.73 | -
Raw Data | 31 | - | - | 0.768 | - | -
Table 3: BoolQ Trivial Questions and F1 score comparison. SD: use 1000
synthetic data. AD: use 100 raw data plus 1000 synthetic data. raw data: model
uses 1000 raw data only without question rephrase, this score is used as a
baseline

###  5.3 Relationship between LLM Performance and Data Quality

While it may seem intuitive that the effectiveness of using LLMs to generate
data for model training depends on the LLM’s knowledge of a specific task, our
research has shown that this is not always the case. The zero-shot or few-shot
performance of an LLM on a task does not necessarily determine the performance
of a model (specifically, the RoBERTa model used in our experiment) trained
with data generated by the LLM. In other words, the fact that an LLM performs
well on a task does not guarantee that models finetuned with data generated by
the LLM will also perform well. Additionally, for tasks where the LLM performs
poorly, models finetuned on the synthetic data generated by the LLM could
actually outperform the LLM itself. The former scenario could be due to the
fact that the ability of an LLM to generate good examples for a task does not
always correspond to its ability to solve the task itself. The latter scenario
is also plausible, as an LLM may be proficient at generating examples with a
given label, but not as good at predicting the label given the task itself.

The results of our experiment can be found in Table 4. For each task, we
prompted the LLM (GPT3.5-turbo) with zero/one/three/five-shot learning and
reported the best performance achieved across all in-context learning methods.
We did not optimize the prompt or use any advanced prompting methods in our
evaluation of the LLM. It is possible that the LLM could achieve better
performance with more advanced prompting techniques. However, the results
obtained from the most basic in-context learning method (see Appendix D) do
provide valuable insights into this problem.

For SST-2, BoolQ, NYT, and Review tasks, we found a performance gap of 10-15%
between the LLM’s in-context learning performance on the task and the fine-
tuned language model (RoBERTa model) using synthetic data. For RTE and EMO
tasks, the LLM does not perform well, but the data generated by the LLM leads
to much better performance. Therefore, even for tasks that LLMs struggle to
solve, using LLM-generated synthetic data can still be helpful.

| GPT3.5-turbo | RoBERTa on Synthetic Data | RoBERTa on Augmented Data
---|---|---|---
SST-2 | 0.956 | 0.845 | 0.874
BoolQ | 0.870 | 0.641 | 0.742
NYT | 0.729 | 0.604 | 0.742
Review | 0.603 | 0.475 | 0.527
RTE | 0.345 | 0.574 | 0.653
Emo | 0.300 | 0.404 | 0.568
Table 4: LLM performance vs model trained by synthetic data on 6 tasks.
Average f1 score from 5 prompting method under (1) Synthetic Data (1000
synthetic data) (2) Augmented data (1000 synthetic data + 100 raw data)

###  5.4 Synthetic Data is Helpful Mostly in Low-Resource Settings

Figure 3: Improvement on Different Raw Data Amount. raw data (x) is only using
X number of raw data points. augmented (x) is using X amount raw data points
plus 100 synthetic data. For augmented f1 score, it is the average model
performance on the data generated by 5 different prompting methods Figure 4:
Synthetic Data Similarity Figure 5: Impact on Synthetic Data Quantity

Previous work has shown that it is challenging for models trained with
synthetic data to perform as well as models trained with the same amount of
original data (Li et al., 2023; Ding et al., 2023). However, when human-
annotated data is limited, synthetic data augmentation can improve model
performance. In fact, this technique is most effective in low-resource
settings. For all tasks with 100 raw data points, we found that synthetic data
augmentation yields improvements from 3% to 26%. When the raw training data
increases from 100 to 1,000, only four tasks show improvements, where
improvements are less than 5% (Figure 3). There are no established rules for
determining the amount of raw data as low-resource. For all six tasks in our
experiment, 1,000 data points represent a small portion of training data. We
found the model continues to improve as we increase the number of raw data for
training. However, the amount of performance gain obtained from increasing
training data is also dependent on other factors such as task and model
complexity. Based on this observation, we consider 100 raw data points as low-
resource tasks, which will be used as the default augmented setting in all
experiments.

###  5.5 A Comparison Between Different Prompting Methods

In the synthetic data only setting, one-shot or zero-shot topic methods rank
in the top two for all tasks except the Review task (Figure 2). In the
augmented setting, few-shot generation and zero-shot topic generation methods
demonstrate good performance across all tasks. In BoolQ, EMO, and RTE tasks,
zero-shot topic methods outperform other prompting methods. In SST-2 and NYT
tasks, few-shot generation methods perform best. The performance of zero-shot
methods is sub-optimal across all tasks.

In the five prompting methods we experimented with, zero-shot topic generation
typically produces the most diverse dataset because different topics are
sampled each time during generation. Zero-shot methods generate the least
diverse dataset, as the prompt remains the same for each generation. One-shot
and few-shot methods also generate repeated examples due to the limitation of
in-context examples. We found for most tasks, a diversity dataset tends to
benefit model training.

As shown in (Figure 2) in non-augmented setting, zero-shot generation shows
the worst performance for RTE, EMO, Review and SST-2, while zero-shot topic
generation outperforms other methods (or at least is comparable to other
methods) for BoolQ, NYT, RTE and EMO task. This effect does not appear on all
tasks as there might be other factors that impact the model performance.
Meanwhile, the effect of diversity diminishes when we mix synthetic data with
raw data. Therefore, training with both raw data and synthetic data could help
when synthetic data is less diverse.

While not generating the most optimally diverse dataset, one-shot or few-shot
generation methods typically helps LLMs better understand the task description
and generate examples similar to the original examples (Li, 2023; Song et al.,
2022). In EMO and Review tasks, we observe the advantage of few-shot learning
over other prompting methods. We suspect this is because both tasks are more
subjective compared to the rest of the tasks, as the EMO contains twitter
posts and Review task are made up of customer reviews and ratings.

###  5.6 Synthetic Data Diversity and Similarity to Raw Data

In this section, we examine the diversity of our training data using inter-
sample semantic similarity. To calculate this similarity, we use vector
embedding proposed in (Reimers & Gurevych, 2019) and average the similarity
score across all examples pairs following (Yu et al., 2023). Figure 4 displays
the inter-sample similarity for each task, comparing data generated by five
prompting methods. On the x-axis, we show the performance of the finetuned
model using the 1000 synthetic data only. Figure 4 shows that for BoolQ, NYT,
and SST-2, a lower inter-sample diversity results in a better F1 score.
However, for other tasks, the correlation is weak due to the existence of
outliers, especially for RTE, and the possible impact of other factors, such
as task complexity. We also calculated the similarity between the synthetic
data and the actual raw data using the same method and found that the
synthetic data generated from five different prompting methods had similar
similarity scores with the raw data. However, it is not clear whether
synthetic data that closely resembles the raw data would lead to better model
performance. This could be due to the limitations of our similarity measuring
method, which only considers semantic similarity, as discussed in (Steck et
al., 2024). Many NLP tasks rely on subtle contextual cues and nuanced
wordings, such as in the SST-2 task, where changes to wording can affect the
sentiment of the text more than contextual semantics. Our measurement does not
account for other aspects of similarity, such as structural or lexical
similarity, as discussed in (Wang et al., 2020; Ayeldeen et al., 2014).
Lastly, due to the limited number of data points and the potential variation
in synthetic data, it needs to be cautious to generalize our findings to our
tasks or domains.

###  5.7 Synthetic Data Quantity

We have found that increasing the amount of synthetic data in our model
training improves its performance. Figure 5 shows the relationship between the
model’s performance (measured by the f1 score) on the y-axis and the total
number of training data on the x-axis. In the augmented scenario, we mixed 100
raw data points with varying amounts of synthetic data. The performance is the
average of the model’s f1 score over 5 prompting methods for different data
amount. For the raw data scenario, only real-world data was used in model
training. Our graph indicates that model performance from raw data serves as
an upper bound for the augmented setting in almost all tasks. Moreover, we
observe that the marginal effect of performance gain with increasing training
data is present in both raw and synthetic data. For BoolQ and SST-2 tasks, we
observed this phenomenon at the same data size. As such, the raw data size at
which marginal improvement of model performance appears can be used as a
reference point when increasing the number of synthetic data.

##  6 Data Generation Techniques in Practice

In the process of using LLM to generate data for this study, we identified
several useful techniques. These practices lack sufficient theoretical support
and the effectiveness of these techniques can be subject to the choice of
large language models or the requirements of a specific task.

###  6.1 Condition on Label

Typically, there are two ways to generate a classification dataset: Condition
on the Label and Left-to-Right (see Table 5). It is recommended to use
Condition on the Label for each generation as it saves effort in parsing the
label and avoids LLM generating unknown labels. It also provides the user
control over the label distribution in the synthetic dataset.

Left-to-right prompt: generate an example text first and then generate its
class label.
---
Class-conditioned prompt: generate an example text where the label must be
Class X.
Table 5: Left-to-right prompt vs. class-conditioned prompt.

It is worth noting that class-conditioned generations are more likely to
introduce bias and reduce the difficulty of the synthetic example. When the
class label is visible, LLM might leak the label information during content
generation. In the BoolQ example, LLM hints the answer ”FALSE” via the use of
certain words in the question it generates (e.g. the word ”only”). In this
case, rephrasing the question with the class label hidden improves the
performance, which is essentially performing left-to-right generation.

###  6.2 Generation on Target Corpus

It is critical to provide topics or descriptions closely related to the use
case when generating examples. Ensuring that the topics are relevant to the
use case significantly improves the quality of generated data. For example,
when creating examples from Twitter, it is beneficial to first generate common
topics found on Twitter. On the other hand, when generating Amazon customer
reviews, it is effective to generate an Amazon product catalog as a list of
potential topics. This approach ensures that the synthetic data is more
closely aligned with the target corpus, leading to better performance in
classification tasks.

###  6.3 Iterative Data Generation and Prompt Refinement

Generating synthetic data can be both time-consuming and resource-intensive.
To maximize efficiency and ensure high-quality data, it is recommended to
adopt an iterative approach. Initially, generate a small number of examples
and evaluate their quality. If the quality of these initial data points is
low, refine the prompt before generating more data. It is unlikely that simply
generating more data points with the same prompt will magically produce high
quality data.

##  7 Conclusion

In this work, we analyzed different factors that influences the data
generation using LLMs. We found data generation is most effective in low
resourced settings. Increasing the amount of synthetic data does not
necessarily lead to continuous improvements in model performance. It is
beneficial to combine synthetic data with raw data during training.
Additionally, it is crucial to be vigilant for patterns or biases in synthetic
data that may hinder model training. Overall, using LLM for data augmentation
has great potential in model training. With a carefully tuned prompt, the data
generated by LLM is able to obtain comparable performance with human annotated
data, but at a much lower cost.

The domain of data generation for classification tasks is highly complex. Due
to the diversity of NLP tasks, it is challenging to find rules that generalize
well across all tasks. However, our findings could still serve as valuable
resources for researchers and practitioners looking to use synthetic data for
training classification models. For future work, it would be valuable to study
the effects of more advanced prompting methods, such as the Chain of Thought
(Wei et al., 2023), or LLM hyperparameters, such as temperature, on the
quality of synthetic data.

## Impact Statement

This paper presents work whose goal is to advance the field of Machine
Learning. There are many potential societal consequences of our work, none
which we feel must be specifically highlighted here.

## References

  * Ayeldeen et al. (2014) Ayeldeen, H., Hassanien, A. E., and Fahmy, A. A.  Lexical similarity using fuzzy euclidean distance.  In _2014 International Conference on Engineering and Technology (ICET)_ , pp. 1–6, 2014.  doi: 10.1109/ICEngTechnol.2014.7016801.
  * Bentivogli et al. (2009) Bentivogli, L., Dagan, I., Dang, H. T., Giampiccolo, D., and Magnini, B.  The fifth PASCAL recognizing textual entailment challenge.  2009\.
  * Brown et al. (2020) Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D.  Language models are few-shot learners, 2020.
  * Cai et al. (2020) Cai, H., Chen, H., Song, Y., Zhang, C., Zhao, X., and Yin, D.  Data manipulation: Towards effective instance learning for neural dialogue generation via learning to augment and reweight.  In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J. (eds.), _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 6334–6343, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.564.  URL https://aclanthology.org/2020.acl-main.564.
  * Clark et al. (2019) Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K.  BoolQ: Exploring the surprising difficulty of natural yes/no questions.  In _Proceedings of NAACL-HLT 2019_ , 2019.
  * Devlin et al. (2019) Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K.  Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
  * Ding et al. (2023) Ding, B., Qin, C., Liu, L., Chia, Y. K., Joty, S., Li, B., and Bing, L.  Is gpt-3 a good data annotator?, 2023.
  * Feng et al. (2021) Feng, S. Y., Gangal, V., Wei, J., Chandar, S., Vosoughi, S., Mitamura, T., and Hovy, E.  A survey of data augmentation approaches for nlp, 2021.
  * Gao et al. (2023) Gao, J., Pi, R., Lin, Y., Xu, H., Ye, J., Wu, Z., Zhang, W., Liang, X., Li, Z., and Kong, L.  Self-guided noise-free data generation for efficient zero-shot learning, 2023.
  * Gunasekar et al. (2023) Gunasekar, S., Zhang, Y., Aneja, J., Mendes, C. C. T., Giorno, A. D., Gopi, S., Javaheripi, M., Kauffmann, P., de Rosa, G., Saarikivi, O., Salim, A., Shah, S., Behl, H. S., Wang, X., Bubeck, S., Eldan, R., Kalai, A. T., Lee, Y. T., and Li, Y.  Textbooks are all you need, 2023.
  * Gupta et al. (2023) Gupta, H., Scaria, K., Anantheswaran, U., Verma, S., Parmar, M., Sawant, S. A., Baral, C., and Mishra, S.  Targen: Targeted data generation with large language models, 2023.
  * Keung et al. (2020) Keung, P., Lu, Y., Szarvas, G., and Smith, N. A.  The multilingual amazon reviews corpus.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_ , 2020.
  * Kumar et al. (2019) Kumar, A., Bhattamishra, S., Bhandari, M., and Talukdar, P.  Submodular optimization-based diverse paraphrasing and its effectiveness in data augmentation.  In Burstein, J., Doran, C., and Solorio, T. (eds.), _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pp. 3609–3619, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.  doi: 10.18653/v1/N19-1363.  URL https://aclanthology.org/N19-1363.
  * Li et al. (2022) Li, B., Hou, Y., and Che, W.  Data augmentation approaches in natural language processing: A survey.  _AI Open_ , 3:71–90, 2022.  ISSN 2666-6510.  doi: https://doi.org/10.1016/j.aiopen.2022.03.001.  URL https://www.sciencedirect.com/science/article/pii/S2666651022000080.
  * Li (2023) Li, Y.  A practical survey on zero-shot prompt design for in-context learning.  In _Proceedings of the Conference Recent Advances in Natural Language Processing - Large Language Models for Natural Language Processings_ , RANLP. INCOMA Ltd., Shoumen, BULGARIA, 2023.  doi: 10.26615/978-954-452-092-2˙069.  URL http://dx.doi.org/10.26615/978-954-452-092-2_069.
  * Li et al. (2023) Li, Z., Zhu, H., Lu, Z., and Yin, M.  Synthetic data generation with large language models for text classification: Potential and limitations, 2023.
  * Liang et al. (2023) Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., Newman, B., Yuan, B., Yan, B., Zhang, C., Cosgrove, C., Manning, C. D., Ré, C., Acosta-Navas, D., Hudson, D. A., Zelikman, E., Durmus, E., Ladhak, F., Rong, F., Ren, H., Yao, H., Wang, J., Santhanam, K., Orr, L., Zheng, L., Yuksekgonul, M., Suzgun, M., Kim, N., Guha, N., Chatterji, N., Khattab, O., Henderson, P., Huang, Q., Chi, R., Xie, S. M., Santurkar, S., Ganguli, S., Hashimoto, T., Icard, T., Zhang, T., Chaudhary, V., Wang, W., Li, X., Mai, Y., Zhang, Y., and Koreeda, Y.  Holistic evaluation of language models, 2023.
  * Liu et al. (2024) Liu, R., Wei, J., Liu, F., Si, C., Zhang, Y., Rao, J., Zheng, S., Peng, D., Yang, D., Zhou, D., and Dai, A. M.  Best practices and lessons learned on synthetic data for language models, 2024.
  * Liu et al. (2019) Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V.  Roberta: A robustly optimized bert pretraining approach, 2019.
  * Mikołajczyk & Grochowski (2018) Mikołajczyk, A. and Grochowski, M.  Data augmentation for improving deep learning in image classification problem.  In _2018 International Interdisciplinary PhD Workshop (IIPhDW)_ , pp. 117–122, 2018.  doi: 10.1109/IIPHDW.2018.8388338.
  * Okur et al. (2022a) Okur, E., Sahay, S., and Nachman, L.  Data augmentation with paraphrase generation and entity extraction for multimodal dialogue system.  In Calzolari, N., Béchet, F., Blache, P., Choukri, K., Cieri, C., Declerck, T., Goggi, S., Isahara, H., Maegaard, B., Mariani, J., Mazo, H., Odijk, J., and Piperidis, S. (eds.), _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pp. 4114–4125, Marseille, France, June 2022a. European Language Resources Association.  URL https://aclanthology.org/2022.lrec-1.437.
  * Okur et al. (2022b) Okur, E., Sahay, S., and Nachman, L.  Data augmentation with paraphrase generation and entity extraction for multimodal dialogue system, 2022b.
  * Reimers & Gurevych (2019) Reimers, N. and Gurevych, I.  Sentence-bert: Sentence embeddings using siamese bert-networks, 2019.
  * Saravia et al. (2018) Saravia, E., Liu, H.-C. T., Huang, Y.-H., Wu, J., and Chen, Y.-S.  CARER: Contextualized affect representations for emotion recognition.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pp. 3687–3697, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.  doi: 10.18653/v1/D18-1404.  URL https://www.aclweb.org/anthology/D18-1404.
  * Socher et al. (2013) Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., and Potts, C.  Recursive deep models for semantic compositionality over a sentiment treebank.  In Yarowsky, D., Baldwin, T., Korhonen, A., Livescu, K., and Bethard, S. (eds.), _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pp. 1631–1642, Seattle, Washington, USA, October 2013. Association for Computational Linguistics.  URL https://aclanthology.org/D13-1170.
  * Song et al. (2022) Song, Y., Wang, T., Mondal, S. K., and Sahoo, J. P.  A comprehensive survey of few-shot learning: Evolution, applications, challenges, and opportunities, 2022.
  * Steck et al. (2024) Steck, H., Ekanadham, C., and Kallus, N.  Is cosine-similarity of embeddings really about similarity?  _arXiv preprint arXiv:2403.05440v1_ , 2024.  arXiv.org perpetual non-exclusive license.
  * Stefano (2021) Stefano, D. D.  New york times topics.  https://huggingface.co/datasets/dstefa/New_York_Times_Topics, 2021.
  * Veselovsky et al. (2023) Veselovsky, V., Ribeiro, M. H., Arora, A., Josifoski, M., Anderson, A., and West, R.  Generating faithful synthetic data with large language models: A case study in computational social science, 2023.
  * Wang et al. (2019) Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R.  SuperGLUE: A stickier benchmark for general-purpose language understanding systems.  _arXiv preprint 1905.00537_ , 2019.
  * Wang et al. (2023) Wang, R., Zhou, W., and Sachan, M.  Let’s synthesize step by step: Iterative dataset synthesis with large language models by extrapolating errors from small models, 2023.
  * Wang et al. (2020) Wang, Z., Zhang, Y., and Wu, H.  Structural-aware sentence similarity with recursive optimal transport, 2020.
  * Wei et al. (2023) Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D.  Chain-of-thought prompting elicits reasoning in large language models, 2023.
  * Xie et al. (2020) Xie, Q., Dai, Z., Hovy, E., Luong, M.-T., and Le, Q. V.  Unsupervised data augmentation for consistency training, 2020.
  * Yang et al. (2023) Yang, S., Xiao, W., Zhang, M., Guo, S., Zhao, J., and Shen, F.  Image data augmentation for deep learning: A survey, 2023.
  * Yang et al. (2020) Yang, Y., Malaviya, C., Fernandez, J., Swayamdipta, S., Le Bras, R., Wang, J.-P., Bhagavatula, C., Choi, Y., and Downey, D.  Generative data augmentation for commonsense reasoning.  In Cohn, T., He, Y., and Liu, Y. (eds.), _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pp. 1008–1025, Online, November 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.findings-emnlp.90.  URL https://aclanthology.org/2020.findings-emnlp.90.
  * Ye et al. (2022) Ye, J., Gao, J., Li, Q., Xu, H., Feng, J., Wu, Z., Yu, T., and Kong, L.  Zerogen: Efficient zero-shot learning via dataset generation, 2022.
  * Yu et al. (2023) Yu, Y., Zhuang, Y., Zhang, J., Meng, Y., Ratner, A., Krishna, R., Shen, J., and Zhang, C.  Large language model as attributed training data generator: A tale of diversity and bias, 2023.
  * Zhou et al. (2024) Zhou, Y., Guo, C., Wang, X., Chang, Y., and Wu, Y.  A survey on data augmentation in large model era, 2024.

##  Appendix A Appendix

Prompt for topic generation for zero-shot with topics and LLM output examples.
GPT-4 is used to generate 500 random topics per task:

Task | Role |  Message
---|---|---
BoolQ, RET, NYT, SST-2, Emo | System |  You are an AI assistant that generates random topics. There is no limit on the number of topics you can generate.
BoolQ, RET, NYT | User |  Please generate 500 topics
BoolQ, RET, NYT | LLM |  Output example: The world’s most beautiful sculptures, The role of technology in modern education …
SST-2, Emo | User |  Please generate 500 twitter post topics
SST-2, Emo | LLM |  Output example: Lunch break, Online dating …
Review | System |  You are an AI assistant that knows Amazon product categories. The user will ask you to generate a list of categories. It is your responsibility to generate the entire list of categories.
Review | User |  Please generate 500 amazon different product categories
Review | LLM |  Output example: Baby Products, Clothing, Jewelry …

##  Appendix B Appendix

Prompt for Question Rephrasing in Section 5.2

Please rephrase the question as if you are typing it in a search engine. Make
sure the answer can only be true or false, Input: question Output:
---

##  Appendix C Appendix

Prompt used for data generation for each task:

Task | Prompt Type |  Prompt
---|---|---
BoolQ | zero-shot |

#### Step 1

Please generate a random short passage. Passage:

#### Step 2

Please generate a True or False question based on the passage. The answer to
the question must be [random([True, False])] Passage: [passage from step 1]
Question:
BoolQ | zero-shot topic |

#### Step 1

Please generate a short passage about this topic: [topic sampled from a topic
list] Passage:

#### Step 2

Please generate a True or False question based on the passage. The answer to
the question must be [random([True, False])] Passage: [passage from step 1]
Question:
BoolQ | one-shot |

#### Step 1

Please generate a Passage, a Question and the Label to the question following
this example: [example from raw data: Passage, Question, Label] Please
generate a similar passage. Passage:

#### Step 2

Please generate a True or False question based on the passage. The answer to
the question must be [label from example in Step 1] Passage: [passage
generated in Step 1] Question:
BoolQ | few-shot (3 or 5) |

#### Step 1

Please generate a Passage, a Question and the Label to the question. Here are
some examples: [examples from raw data: Passage, Question, Label] Please
generate a similar example. Make sure the question is a True or False question
and the answer to the question is [random([True, False])]. Passage:
EMO | zero-shot |

#### Step 1

Please generate a twitter post with the emotion of [random(label)]. Text:
EMO | zero-shot topic |

#### Step 1

Please consider this topic for generation: [topic sampled from a topic list].
Please generate a twitter post with the emotion of [random(label)]. Text:
EMO | one-shot |

#### Step 1

The task is to predict the emotion of a twitter post. The emotion contains six
categories: sadness, joy, love, anger, fear, surprise. Here is an example.
Text: [example from raw data] Emotion: [example label from raw data] Please
generate another example for the same emotion. Text:
EMO | few-shot (3 or 5) |

#### Step 1

The task is to predict the emotion of a twitter post. The emotion contains six
categories: sadness, joy, love, anger, fear, surprise. Here are some examples:
[examples: Text, Emotion] Please generate a twitter post with the emotion of
[first label from examples]. Text:
Task | Prompt Type |  Prompt
---|---|---
NYT | zero-shot |

#### Step 1

Please generate a news title for [random(label)] category. Headline:
NYT | zero-shot topic |

#### Step 1

Please consider this sentence for generation: [topic sampled from topic list].
Please generate a news headline for [random(label)] category. Headline:
NYT | one-shot |

#### Step 1

The task is to predict the topic of a news headline. The topics contain
’sports’, ’arts, culture and entertainment’, ’business and finance’, ’health
and wellness’, ’lifestyle and fashion’, ’science and technology’, ’politics’,
’crime’. Here is an example News: [example news] Topic: [example topic] Please
generate another news on [example topic]. Headline:
NYT | few-shot (3 or 5) |

#### Step 1

The task is to predict the topic of a news headline. The topics contain
’sports’, ’arts, culture and entertainment’, ’business and finance’, ’health
and wellness’, ’lifestyle and fashion’, ’science and technology’, ’politics’,
’crime’. Here are some examples: [examples: Headline, Topic] Please generate a
news headline for [first topic from examples] category. News:
Review | zero-shot |

#### Step 1

The Amazon customer review has a rating ranges from 1 to 5, 1 being the lowest
and 5 being the highest. Please generate a customer review with a rating of
[random(label)]. Content:
Review | zero-shot topic |

#### Step 1

The Amazon customer review has a rating ranges from 1 to 5, 1 being the lowest
and 5 being the highest. Please generate a customer review with a rating of
[random(label)] for a specific product under [a product category sampled from
topic list]. Please use a fake product name. Content:
Review | one-shot |

#### Step 1

The task is to predict the rating of an Amazon customer review based on the
content. The rating ranges from 1 to 5, 1 being the lowest and 5 being the
highest. Here is a review example. Content: [example content] Rating: [example
rating] Please generate another example for a similar product. Make sure the
rating for the review is [example rating]. Content:
Review | few-shot (3 or 5) |

#### Step 1

The Amazon customer review has a rating ranges from 1 to 5, 1 being the lowest
and 5 being the highest. Here are some examples Content: [examples: Content,
Rating] Please generate a customer review with a rating [first rating from
examples]. Content:
Task | Prompt Type |  Prompt
---|---|---
RTE | zero-shot |

#### Step 1

Given a premise and a hypothesis, a model needs to predict whether the
hypothesis can be logically inferred from the premise. The response should be
either True if the hypothesis can be inferred from the premise, or False if it
cannot be inferred. Here is the output format: Premise: Hypothesis: Label:
True or False Please generate an example where the Label is [random(label)].
Premise:
RTE | zero-shot topic |

#### Step 1

Given a premise and a hypothesis, a model needs to predict whether the
hypothesis can be logically inferred from the premise. The response should be
either True if the hypothesis can be inferred from the premise, or False if it
cannot be inferred. Here is the output format: Premise: Hypothesis: Label:
True or False Please generate an example about [premise] where the Label is
[random(label)]. Premise:
RTE | one-shot |

#### Step 1

Given a premise and a hypothesis, a model needs to predict whether the
hypothesis can be logically inferred from the premise. The response should be
either True if the hypothesis can be inferred from the premise, or False if it
cannot be inferred. Here is an example: Premise: [example premise] Hypothesis:
[example hypothesis] Label: [example label] Please generate another similar
example where the Label is [example label]. Premise:
RTE | few-shot (3 or 5) |

#### Step 1

Given a premise and a hypothesis, a model needs to predict whether the
hypothesis can be logically inferred from the premise. The response should be
either True if the hypothesis can be inferred from the premise, or False if it
cannot be inferred. Here are some examples: [examples: Premise, Hypothesis,
Label] Please generate a similar example. Make sure the label is [first label
from examples]. Premise:
SST-2 | zero-shot |

#### Step 1

Please generate a sentence that contains a [random(label)] sentiment.
Sentence:
SST-2 | zero-shot topic |

#### Step 1

Please consider this topic for generation: [topic from the topic list]. Please
generate a sentence that contains a [random(label)] sentiment. Sentence:
SST-2 | one-shot |

#### Step 1

The task is to predict whether the following sentence is positive or negative
sentiment. Sentence: [example sentence] Label:[example label] Please generate
a similar example on the same topic, including a Sentence and a Label.
Sentence:
SST-2 | few-shot (3 or 5) |

#### Step 1

The task is to predict whether the following sentence is positive or negative
sentiment. [examples: Sentence, Label] Please generate a similar example,
including a Sentence and a Label. Sentence:

##  Appendix D Appendix

Prompt used to evaluate LLM performance on each task.

Task | Prompt Type |  Prompt
---|---|---
RTE | zero-shot |

#### Step 1

Given a premise and a hypothesis, a model needs to predict whether the
hypothesis can be logically inferred from the premise. The response should be
either True if the hypothesis can be inferred from the premise, or False if it
cannot be inferred. Premise: [premise], Hypothesis: [hypothesis], Label:
RTE | 0/1/3/5-shot |

#### Step 1

Given a premise and a hypothesis, a model needs to predict whether the
hypothesis can be logically inferred from the premise. The response should be
either True if the hypothesis can be inferred from the premise, or False if it
cannot be inferred. Here are some examples: [example premise, hypothesis,
label] Premise: [premise], Hypothesis: [hypothesis], Label:
BoolQ | zero-shot |

#### Step 1

The task is to answer a question which is solely based on the content
provided. Passage: [passage] , Question: [question], Label:
BoolQ | 0/1/3/5-shot |

#### Step 1

The task is to answer a question which is solely based on the content
provided. Here are some examples: [example passage, question, label] Passage:
[passage], Question: [question], Label:
Review | zero-shot |

#### Step 1

The task is to predict the rating of an Amazon customer review based on the
content. The rating ranges from 1 to 5, with 1 being the lowest and 5 being
the highest. Text: [text] , Label:
Review | 0/1/3/5-shot |

#### Step 1

The task is to predict the rating of an Amazon customer review based on the
content. The rating ranges from 1 to 5, with 1 being the lowest and 5 being
the highest. Here are some examples: [example text, label] Text: [text],
Label:
NYT | zero-shot |

#### Step 1

The task is to predict the topic of a news headline. The topics include:
’sports’, ’arts, culture and entertainment’, ’business and finance’, ’health
and wellness’, ’lifestyle and fashion’, ’science and technology’, ’politics’,
’crime’. Text:[text], Label:
NYT | 0/1/3/5-shot |

#### Step 1

The task is to predict the topic of a news headline. The topics include:
’sports’, ’arts, culture and entertainment’, ’business and finance’, ’health
and wellness’, ’lifestyle and fashion’, ’science and technology’, ’politics’,
’crime’. Here are some examples: [example text, label] Text: [text], Label:
EMO | zero-shot |

#### Step 1

The task is to predict the emotion of a Twitter text. The emotions include six
categories: sadness, joy, love, anger, fear, surprise. Text: [text], Label:
EMO | 0/1/3/5-shot |

#### Step 1

The task is to predict the emotion of a Twitter text. The emotions include six
categories: sadness, joy, love, anger, fear, surprise. Here are some examples:
[example text, label] Text: [text], Label:
SST-2 | zero-shot |

#### Step 1

The task is to predict whether the given sentence has a positive or negative
sentiment. Sentence: [sentence], Label:
SST-2 | 0/1/3/5-shot |

#### Step 1

The task is to predict whether the given sentence has a positive or negative
sentiment. Here are some examples: [example sentence, label], Sentence:
[sentence], Label: