# Fighting Randomness with Randomness: Mitigating Optimisation Instability of
Fine-Tuning using Delayed Ensemble and Noisy Interpolation

Branislav Pecher♠†, Jan Cegin♠†, Robert Belanec♠†, Jakub Simko†, Ivan Srba†,
Maria Bielikova†
♠ Faculty of Information Technology, Brno University of Technology, Brno,
Czechia
† Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia
<EMAIL_ADDRESS>
###### Abstract

While fine-tuning of pre-trained language models generally helps to overcome
the lack of labelled training samples, it also displays model performance
instability. This instability mainly originates from randomness in
initialisation or data shuffling. To address this, researchers either modify
the training process or augment the available samples, which typically results
in increased computational costs. We propose a new mitigation strategy, called
Delayed Ensemble with Noisy Interpolation (DENI), that leverages the strengths
of ensembling, noise regularisation and model interpolation, while retaining
computational efficiency. We compare DENI with 9 representative mitigation
strategies across 3 models, 4 tuning strategies and 7 text classification
datasets. We show that: 1) DENI outperforms the best performing mitigation
strategy (Ensemble), while using only a fraction of its cost; 2) the
mitigation strategies are beneficial for parameter-efficient fine-tuning
(PEFT) methods, outperforming full fine-tuning in specific cases; and 3)
combining DENI with data augmentation often leads to even more effective
instability mitigation.

Fighting Randomness with Randomness: Mitigating Optimisation Instability of
Fine-Tuning using Delayed Ensemble and Noisy Interpolation

Branislav Pecher♠†, Jan Cegin♠†, Robert Belanec♠†, Jakub Simko†, Ivan Srba†,
Maria Bielikova† ♠ Faculty of Information Technology, Brno University of
Technology, Brno, Czechia † Kempelen Institute of Intelligent Technologies,
Bratislava, Slovakia<EMAIL_ADDRESS>

##  1 Introduction

Tuning of pre-trained language models such as BERT or RoBERTa using either
full fine-tuning or parameter efficient fine-tuning (PEFT) has achieved
significant success across a wide range of natural language processing tasks.
They are especially useful when faced with limited labelled data for quickly
adapting to the specific task. Despite the success, previous works observed
that fine-tuning still remains unstable (Dodge et al., 2020; Mosbach et al.,
2021; Chen et al., 2022), especially with limited data. Fine-tuning is
sensitive to the effects of randomness originating from random initialisation,
data shuffling or model randomness (e.g., use of non-deterministic layers such
as dropout in the model). As illustrated in Figure 1, repeating the fine-
tuning process multiple times, without mitigating the randomness, leads to
large performance variance in the results, both for full fine-tuning and for
PEFT methods.

Figure 1: Repeating BERT fine-tuning multiple times without any mitigation
leads to significant performance variance. Using DENI for randomness
mitigation, the variance is reduced and performance increased.

To deal with the fine-tuning instability, researchers propose various
strategies to mitigate the effects of randomness. In most cases, the
mitigation strategies focus on modifying the training process (Lee et al.,
2020; Dodge et al., 2020), such as adding noise (Hua et al., 2021; Wu et al.,
2022; Hua et al., 2023), ensembling multiple models (Hidey et al., 2022;
Khurana et al., 2021; Wang et al., 2023; Summers and Dinneen, 2021), or
improving the experimental setup that potentially leads to the instability
(Zhang et al., 2021; Mosbach et al., 2021), such as using bias correction or
training for longer. As many of these strategies are designed and evaluated on
high-resource datasets, almost no focus is dedicated to evaluating the
mitigation benefit of data augmentation (Zhang et al., 2018; Meng et al.,
2023a). At the same time, the instability of the whole training process is
addressed only when considering full fine-tuning. When using PEFT methods, the
focus is on the initialisation of soft prompts, even though the factors such
as data shuffling still lead to variance in results (Chen et al., 2022).
Overall, the best performing mitigation strategies are ensembles and model
interpolation methods (Gueta et al., 2023; Hidey et al., 2022; Wang et al.,
2023), which significantly reduce the deviation in results, but also
significantly increase the computation costs. The methods that add noise to
the model parameters (Hua et al., 2021; Wu et al., 2022) also perform well,
improving generalisability and overall performance, but not necessarily
reducing the instability.

Inspired by the success and mutual complementarity between ensemble methods,
model interpolation and noise regularisation, we propose a novel mitigation
method Delayed Ensemble with Noisy Interpolation (DENI). The DENI method
leverages the benefits of ensembling while reducing its computation costs. To
achieve this, the ensemble is created at the end of training from a single
model by perturbing its parameters using random noise (i.e., fighting
randomness with randomness). In addition, the method creates the ensemble by
adding noise, trains it for a few steps and then aggregates it into a single
model multiple times during training (see Figure 2), which leads to more
effective mitigation of the randomness. Using the DENI method leads to lower
variance in results and higher performance (as illustrated in Figure 1).

To evaluate the benefit of the DENI method, we compare it with other
representative mitigation strategies that modify the optimisation process. In
the comparison, we also include an augmentation strategy that uses large
language models to paraphrase samples, as such paraphrasing was observed to
improve robustness and stability by Cegin et al. (2023, 2024), especially in
limited data settings. Besides full fine-tuning, we also explore the benefit
of the proposed mitigation strategy and other baselines for representative
parameter-efficient fine-tuning (PEFT) methods, namely LoRA, IA3 and UniPELT.

Our main contributions and findings are111To support replicability and
extension of our results, we openly publish the source code of our proposed
method and experiments at https://anonymous.4open.science/r/DENI:

  * •

We propose DENI - a novel strategy for mitigating the randomness sensitivity
of fine-tuning (originating from initialisation, data shuffling and model
randomness). The proposed method leverages the benefit of ensembling and model
interpolation, while reducing their computation costs using noise.

  * •

We compare DENI with 9 representative mitigation strategies, which either
modify the training process or augment the data, across 7 text classification
datasets. The results show that, in comparison with the best performing
baselines, the DENI method improves the overall performance and reduces the
deviation in results, while introducing lower computation costs, leading to an
efficient mitigation.

  * •

We explore the benefit of mitigation strategies across 3 representative PEFT
methods. We find they often benefit more from the mitigation strategies,
especially data augmentation, showing a larger decrease in deviation and
increase in overall performance, even outperforming full fine-tuning in
specific cases.

##  2 Related Work: Mitigating Instability of Fine-Tuning

The majority of strategies for mitigating the instability of fine-tuning
modify the training process or the models themselves, addressing different
sources of instability. One set of strategies addresses the suboptimal setup
choices, suggesting that training for longer, using an optimiser with bias
correction and lower learning rate with warm-up and scheduling can reduce the
instability (Mosbach et al., 2021; Zhang et al., 2021). Another set of
approaches modifies the initialisation, such as using meta-learning (Dauphin
and Schoenholz, 2019), re-initialising top layers of the pretrained models
(Zhang et al., 2021), or initialising multiple models and stopping the ones
that show bad performance early in training (Dodge et al., 2020). Other
strategies optimise only parts of the network (Xu et al., 2021; Zhang et al.,
2022) or run supplementary pre-training on data rich tasks (Phang et al.,
2018). Mixout (Lee et al., 2020) randomly replaces parts of the parameters
with the original weights. Further strategies add noise to the input or model
parameters either before (Wu et al., 2022) or during (Hua et al., 2021, 2023;
Chen et al., 2023) training. The ensembling strategies provide the best
mitigation effectiveness, i.e., highest increase in performance and reduction
in results variance, but at the cost of significant increase in computation
costs (Hidey et al., 2022; Wang et al., 2023; Gueta et al., 2023). To reduce
the cost, the ensemble can be created from a single model, such using
Stochastic Weight Averaging (SWA) that performs an equal average of the
weights traversed by the optimiser (Izmailov et al., 2018; Khurana et al.,
2021; Lu et al., 2022), or Accelerated Ensembling that uses snapshots from
different parts of training (Summers and Dinneen, 2021; Huang et al., 2017).
However, such aggregation also reduces the mitigation effectiveness of the
ensemble.

Although many augmentation strategies exist, they are mainly used to improve
the overall performance in low-resource settings, without any consideration
for the instability (Feng et al., 2021; Obadinma et al., 2023). Only a few
papers consider using data augmentation for addressing the optimisation
instability by utilising pretrained language models as data generators (Cegin
et al., 2024; Meng et al., 2023b) or using common augmentation methods such as
Mixup (Guo et al., 2019; Guo, 2020). In specific cases, the augmentation is
applied to test data instead (Summers and Dinneen, 2021)

For the parameter-efficient fine-tuning methods, the focus is mostly on
initialising prompts and mitigating randomness in prompt-tuning (Chen et al.,
2023; Razdaibiedina et al., 2023; Pan et al., 2023; Köksal et al., 2023),
while the focus on mitigating optimisation instability for the remaining
methods is limited (Chen et al., 2022).

##  3 Mitigation Method: Delayed Ensemble With Noisy Interpolation

Figure 2: An illustrative example of the Delayed Ensemble with Noisy
Interpolation (DENI) method that mitigates model performance instability of
model fine-tuning with limited data. DENI alters regular fine-tuning, using
noise-adding, model aggregation, and ensembling to steer model(s) towards
optimal parameter setup in the parameter space. In comparison to simple
ensembling, the method requires only a fraction of computational resources.

The goal of our proposed method, the Delayed Ensemble with Noisy Interpolation
(DENI), is to mitigate the randomness in fine-tuning optimisation, reducing
the observed variability in results, while keeping or increasing the average
model performance and minimising additional computational costs. When
designing our method, we drew inspiration from findings and approaches from
previous works on ensembling (Summers and Dinneen, 2021), model interpolation
(Gueta et al., 2023) and noise regularisation (Hua et al., 2021; Wu et al.,
2022; Chen et al., 2023), that were shown to be effective techniques capable
of improving overall performance and mitigating randomness. Our method, which
is illustrated in Figure 2 and in Algorithm 1, comprises two main components:
1) Delayed Ensemble; and 2) Noisy Interpolation.

Algorithm 1 Delayed Ensemble with Noisy Interpolation

0: $var_{noise}$: noise variance $noise_{start}$: steps before adding noise
$noise_{end}$: steps after which no noise is added $ensemble_{start}$: steps
before creating final ensemble $steps_{noisy}$: number of steps with noisy
models $steps_{regular}$: number of steps with single aggregated model
$\lambda$: scaling factor for noise $M$: pretrained model $N$: number of
models in ensemble

1: Train model $M$ for $noise_{start}$ steps

2: Set $steps=noise_{start}$

3: while $steps<noise_{end}$ do

4: for all $n$ in $1,2,...,N$ do

5: $M_{n}=M+Noise(0,var_{noise})*\lambda_{steps}$

6: end for

7: Train $M,M_{1},...,M_{N}$ for $steps_{noisy}$

8: $M=average(M,M_{1},...,M_{N})$

9: Train $M$ for $steps_{regular}$

10: $steps=steps+steps_{noisy}+steps_{regular}$

11: end while

12: Train $M$ for $ensemble_{start}-noise_{end}$

13: for all $n$ in $1,2,...,N$ do

14: $M_{n}=M+Noise(0,var_{noise})*\lambda_{ensemble}$

15: end for

16: Train $M,M_{1},...,M_{N}$ for the remaining steps

17: Use hard voting of the final ensemble $M,M_{1},...,M_{N}$

#### Delayed Ensemble (DE).

The main idea is to exploit the benefit of ensembling multiple models, while
reducing the computation cost of obtaining such an ensemble. It was previously
shown, that independently training multiple models (initialised with different
random seeds) and aggregating their predictions represents an effective
regularisation technique capable of reducing the deviation in the results and
significantly improving performance. In addition, Gueta et al. (2023) show
that linear interpolation of multiple models in the weight space leads to a
better performing model than any of the individual ones used for its
aggregation. However, training multiple models with different set of weights
introduces a significant increase in computation costs. To reduce the
computation cost, while keeping as many of the benefits as possible, we
propose the Delayed Ensemble that creates the ensemble by adding noise to a
single trained model. Instead of initialising multiple models and training
them to full, we initialise and train only a single model $M$. The model is
trained for $ensemble_{start}$ steps (E in Figure 2), and afterwards is used
to create $N$ new models ($M_{1},M_{2},...,M_{N}$) by perturbing its
parameters by adding noise as follows:

$M_{n}=M+Noise(0,var_{noise})*\lambda$ (1)

where $Noise(0,var)$ represents Gaussian noise with $0$ mean and variance of
size $var$, and $\lambda$ represents a noise scaling factor. The models are
trained for the remainder of the steps, and the prediction is obtained using
hard voting (point F in Figure 2).

#### Noisy Interpolation (NI).

Besides allowing us to create the ensemble from a single trained model, adding
noise to the model before, during or even after training was found to be an
effective regularisation by itself that can improve the overall performance,
generalisability and mitigate the effects of randomness in training (Hua et
al., 2021; Wu et al., 2022; Hua et al., 2023). At the same time, training the
model that is a result of the linear interpolation of multiple models is more
effective and leads to better performance than the further training of the
individual models used for aggregation, especially when the interpolated model
is in close proximity to the optimal set of parameters (Gueta et al., 2023).
Based on these findings, we propose a repeated Noisy Interpolation, which
creates and interpolates the ensemble multiple times during training. After
initialising the model $M$ (A in Figure 2), we train it for $noise_{start}$
steps (B in Figure 2). Afterwards, we create $N$ new models
($M_{1},M_{2},...,M_{N}$) by perturbing the parameters of model $M$ by adding
noise (C1 in Figure 2), similarly to Delayed Ensemble. However, we use a
different scaling factor, $\lambda_{steps}$, which performs inverse scaling
based on the number of steps (i.e., in later stages of the training, the
introduced noise is smaller). These noisy models are then trained for
$steps_{noisy}$ steps (ending in C2 in Figure 2) and are aggregated together
using uniform interpolation (C3 in Figure 2). The aggregated model is then
trained for $steps_{regular}$ steps (ending in C4 in Figure 2). The whole
process of creating and interpolating ensembles is repeated until
$noise_{end}$ steps of training are finished (D in Figure 2). Afterwards, the
aggregated model is trained until the end.

#### Full Method (DENI)

is the combination of the two components, i.e., running Noisy Interpolation
for large part of the training before creating final Delayed Ensemble near the
end.

#### Hyperparameters.

As we define multiple hyperparameters for the proposed method that can affect
its mitigation effectiveness and efficiency, we provide recommendations on how
to set them up for optimal performance. For the most important parameters
(that represent the trade-off between mitigation and computations costs) we
run a thorough hyperparameter sensitivity analysis in Appendix B. For the
remaining hyperparameters we provide suggestions based on heuristics and our
preliminary experiments. Although there are multiple possibilities for the
$Noise()$ function, we have observed that using the Gaussian noise provides
the best results. In addition, we suggest to add noise only to the newly
initialised parameters. Adding noise to all layers makes the method
significantly more sensitive to the other hyperparameters (especially
$var_{noise}$ and $steps_{noisy}$) – we observed that finding the optimal
setup in such case becomes complicated, as adding slightly larger noise
completely breaks the model (i.e., performing worse than random), while adding
slightly smaller noise leads to negligible mitigation effects. Finally,
similar to Wu et al. (2022), we found that scaling the noise based on the
standard deviation of the model parameters provides the most benefit (again
reducing hyperparameter sensitivity). At the same time, scaling the noise
based on the number of steps is beneficial for the Noisy Interpolation, but
not Delayed Ensemble. As such, we define the scaling factors in following way:
$\lambda_{steps}=std(W_{i})*\frac{1}{steps}$ and
$\lambda_{ensemble}=std(W_{i})$, where $std(Wi)$ represents the standard
deviation of the parameter to which we are adding the noise. Overall, the DENI
method is not as sensitive to the hyperparameter setup, even though the number
of parameters is larger. In majority of the cases, the optimal parameters can
be determined based on heuristics, while slight change leads to only a small
difference in mitigation effectiveness, but only if using scaled noise and
only for the newly initialised model parameters.

##  4 Experimental Results and Findings

#### Baselines.

First, we use two basic baselines without any mitigation: 1) Default, where
the model is fine-tuned only on the limited data (representing the lower bound
of performance); and 2) All Data, where the model is fine-tuned on all the
samples available in the dataset (representing the upper bound of
performance). Second, we compare with other representative approaches that
introduce a change into the optimisation process, specifically: 1) Best
Practices as reported by Mosbach et al. (2021) and Zhang et al. (2021), which
includes using scheduling, warm-up, optimiser with bias correction and
training for longer; 2) Ensemble, where we concurrently train 10 models with
different initialisation and use hard voting to aggregate their predictions;
3) NoiseInput, where we periodically add noise to the input embeddings during
training; 4) NoiseWeights, where we periodically add noise to the parameters
during training; 5) Stochastic Weight Averaging (SWA) (Izmailov et al., 2018;
Khurana et al., 2021; Lu et al., 2022) that performs an equal average of the
weights traversed by the optimiser with modified learning rate schedule; and
6) Mixout (Lee et al., 2020) that stochastically mixes the parameters of two
models instead of dropout. In addition, we also compare with Augment N (Cegin
et al., 2023, 2024), which uses a large language model to paraphrase each
sample $N$ times to extend the training data (we report results only for the
best performing number of paraphrases, Augment $1$ and $2$). Finally, we
compare our proposed method (OurDENI), its individual components (OurDE and
OurNI) and its extended version Delayed Ensemble with Noisy Interpolation and
Augmented Labelled Samples (OurDENIALS) that combines DENI with Augment 1.
More details for each of the strategies are reported in Appendix C.2.

#### Datasets.

The experiments are conducted on 7 text classification datasets composed of
different tasks with different number of classes. We focus on 3 binary
classification datasets from the GLUE benchmark Wang et al. (2018): SST2
Socher et al. (2013) for sentiment classification, CoLA Warstadt et al. (2019)
for determining the grammatical acceptability of a sentence, and MRPC Dolan
and Brockett (2005) for determining the semantic equivalence relationship
between two sentences. In addition, we use 4 multi-class text datasets: AG
News Zhang et al. (2015) for news classification, TREC Voorhees and Tice
(2000) for question classification, DBPedia Lehmann et al. (2015) for topic
classification and SNIPS Coucke et al. (2018) for intent classification.

#### Experimental Setup.

As there is no consensus what constitutes a low-resource setting (e.g., for
some it is having less than 1000 labelled samples (Chen et al., 2022; Zhang et
al., 2021; Gueta et al., 2023; Chen et al., 2023) and for others less than 10
000 (Hua et al., 2021)), we opted to use only 1000 labelled samples from each
dataset for training for the main set of experiments and explore different
sizes of available training data in an experiment presented in Section 4.3.
Each experiment is repeated 20 times with different random seed that affects
the initialisation, order of samples and model randomness. For each
experiment, we report a mean F1 macro score and standard deviation over these
repeated runs. Further experimental setup details are reported in Appendix
C.1.

#### Models.

Each experiment is run using the BERT (Devlin et al., 2019), RoBERTa (Liu et
al., 2019) and ALBERT (Lan et al., 2020) base models. Besides regular fine-
tuning, we also report results for representative parameter-efficient fine-
tuning methods, specifically LoRA (Hu et al., 2022), IA3 (Liu et al., 2022)
and UniPELT (Mao et al., 2022) (which combines LoRA, Pfeiffer Adapter
(Pfeiffer et al., 2020) and Prefix-Tuning (Li and Liang, 2021)). Further
details regarding hyperparameter setup are reported in Appendix C.1.

###  4.1 DENI Method on Full Fine-Tuning

BERT | AG News | TREC | SNIPS | DBPedia | SST2 | MRPC | CoLA | Cost
---|---|---|---|---|---|---|---|---
Full FineTuning
Default |  84.950.482 |  90.000.682 |  97.990.109 |  98.750.047 |  88.270.230 |  62.731.497 |  75.110.662 | $1$
All Data |  88.650.343 |  95.660.579 |  98.860.066 |  99.210.030 |  95.150.078 |  68.860.893 |  79.560.440 | $5-55$
Best Practices |  84.960.444 |  90.400.585 |  98.050.137 |  98.760.049 |  88.320.313 |  63.071.593 |  75.300.718 | $2$
Ensemble |  85.560.304 |  90.670.400 |  98.220.076 |  98.820.021 | 89.050.079 |  65.690.674 |  76.540.325 | $10$
NoiseInput |  85.070.444 |  89.430.775 |  98.090.114 |  98.770.043 |  88.250.285 |  62.441.499 |  75.050.817 | $1$
NoiseWeights |  85.250.671 |  90.200.860 |  98.030.146 |  98.750.061 |  88.040.266 |  62.681.765 |  75.060.699 | $1$
SWA |  85.400.364 |  90.240.632 |  98.060.122 |  98.760.060 |  88.540.211 |  63.501.751 |  75.260.572 | $1.75$
Mixout |  84.960.538 |  90.200.603 |  98.030.111 |  98.730.040 |  88.390.280 |  63.231.137 |  75.330.572 | $1$
Augment 1 |  85.250.542 |  90.360.945 |  98.190.142 |  98.820.082 |  88.660.286 |  63.011.544 |  75.560.443 | $2.4$
Augment 2 |  85.280.433 |  90.450.676 |  98.100.143 |  98.720.076 |  88.630.327 |  62.401.514 |  74.580.426 | $3.6$
OurDE |  85.170.291 |  90.540.278 |  98.100.086 | 98.760.019 |  88.430.107 |  63.680.594 |  75.090.281 | $1.9$
OurNI |  86.810.760 |  91.420.901 |  98.420.131 |  98.990.044 |  90.080.284 |  65.381.571 |  77.160.586 | $2.8$
OurDENI |  87.670.261 |  92.040.377 |  98.650.099 | 99.070.023 |  90.820.129 |  66.660.613 |  77.740.263 | $3.7$
OurDENIALS | 88.180.196 | 92.270.236 | 98.730.075 |  99.030.040 | 91.220.132 | 67.040.544 | 78.290.153 | $7.4$
Table 1: Comparison of the DENI method with existing mitigation strategies and
baselines on full fine-tuning using BERT. The comparison is done in terms of
overall performance, deviation and cost (normalised training steps). The
highest performance is in bold and lowest deviation is underlined (not
considering All Data baseline).

In this section, our goal is to answer the following research question: RQ1:
How does the DENI method perform in comparison to other mitigation strategies?
We compare the proposed method with existing mitigation strategies in terms of
the overall performance, standard deviation and their computation cost. The
computation cost is calculated using the number of training steps normalised
by the steps of the Default strategy (i.e.,
$cost=\frac{n\\_steps_{strategy}}{n\\_steps_{Default}}$). In addition, we
explore what benefit the different components of the DENI method bring. The
results from the comparison on full fine-tuning are presented for the BERT
model in Table 1, with the full results in Appendix F and visualisation for
the relation between performance, deviation and the cost in Appendix D.

Strategies that significantly increase number of steps provide the most
benefit. The Ensemble represents the most effective mitigation strategy,
showing the highest increase in the overall performance (e.g., from $75.11$ to
$76.54$ on CoLA dataset) and reduction in standard deviation (e.g., from
$0.662$ to $0.325$ on CoLA dataset), with the deviation being often lower than
the All Data baseline. However, it also represent the most computationally
expensive strategy (when not considering training on all data where the cost
is dataset dependent). The Ensemble strategy is closely followed by SWA, Best
Practices and Augment N that are less consistent and show lower benefit, but
also require lower number of steps. The Augment strategy is beneficial only
when using at most $2$ paraphrases per samples, with higher number of
paraphrases leading to significant decrease in performance (see Appendix E for
results for higher number of $N$). The remaining mitigation strategies show
only negligible and dataset-dependent benefit.

The DENI method leads to higher performance and lower standard deviation while
requiring lower number of steps than the Ensemble strategy. In comparison to
the best performing mitigation strategy (Ensemble), we observe a statistically
significant increase of $0.25-2.11$ percentage points in performance (p-value
of 7e-5 using Mann-Whitney U test), with the deviation staying similar or even
lower in specific cases. At the same time, the DENI method requires only 37%
of the training steps. As such, it represents an effective and efficient
mitigation strategy.

BERT | AG News | TREC | SNIPS | DBPedia | SST2 | MRPC | CoLA
---|---|---|---|---|---|---|---
LoRA
Default |  85.100.488 |  89.680.770 |  98.060.142 |  98.720.077 |  87.950.315 |  61.706.377 |  74.800.907
Ensemble |  86.320.266 |  91.580.371 |  98.320.125 |  98.860.030 |  89.170.109 |  66.590.753 |  77.020.533
OurDENI |  87.860.305 |  92.390.377 |  98.670.084 |  99.050.041 |  90.740.122 |  67.490.554 |  77.660.425
OurDENIALS |  87.790.244 |  92.240.204 |  98.650.065 |  99.040.038 |  91.030.206 |  66.790.671 |  78.240.041
IA3
Default |  83.830.829 |  88.421.197 |  97.590.259 |  98.560.105 |  87.600.216 |  62.720.979 |  73.560.963
Ensemble |  85.390.327 |  90.540.304 |  98.160.082 |  98.780.034 |  88.550.092 |  64.540.767 |  75.990.440
OurDENI |  86.740.279 |  90.840.238 |  98.400.117 |  99.030.041 |  90.150.121 |  65.000.473 |  76.700.295
OurDENIALS |  87.080.574 |  92.260.127 |  98.420.128 |  99.040.042 |  90.350.224 |  67.020.378 |  78.300.037
UniPELT
Default |  84.970.582 |  87.131.316 |  97.630.194 |  98.560.262 |  87.820.233 |  63.770.798 |  74.910.403
Ensemble |  86.020.291 |  91.080.559 |  98.310.104 |  98.890.025 |  88.630.141 |  64.070.610 |  76.000.458
OurDENI |  87.540.301 |  91.430.678 |  98.560.084 |  99.060.045 |  90.570.078 |  66.720.275 |  77.680.409
OurDENIALS |  87.440.054 |  92.250.088 |  98.560.068 |  99.040.045 |  91.010.157 |  66.590.323 |  78.240.033
Table 2: Comparison of the best performing mitigation strategy, DENI method
and the Default baseline for different parameter-efficient fine-tuning
methods. The comparison is done in terms of overall performance and deviation.
Figure 3: Benefit of mitigation strategies for the different fine-tuning
methods using BERT on SNIPS dataset. The benefit is calculated as difference
to the mean performance of the Default baseline. The different mitigation
strategies are beneficial for all fine-tuning methods, but with different
overall benefit (e.g., Augment on IA3). In addition, the DENI method
outperforms all mitigation strategies, leading to higher performance and lower
deviation.

The DENI method components provide different benefits and complement each
other. The Delayed Ensemble component leads to reduction of standard
deviation, achieving deviation on par or lower than Ensemble, but may not
necessarily lead to increase in performance. For example on the MRPC dataset,
even though the deviation is lower ($0.594$ as opposed to $0.679$), the
performance is also significantly lower ($63.68$ as opposed to $65.69$).
However, the performance is higher than the one from Default baseline in all
cases. On the other hand, the Noisy Interpolation leads to significant
increase in performance, at the cost of higher deviation in results. For
example on the AG News dataset, the performance is increased to $87.67$ as
opposed to $85.56$ from Ensemble strategy, with the deviation also increasing
to $0.760$ as opposed to $0.304$. In the full method, the components
complement each other, suppressing the weaknesses and amplifying the strengths
(e.g., reducing the deviation of NI also leads to higher performance).
Finally, combining our proposed method with data augmentation may lead to
further benefits, although the overall benefit is lower ($0.2-0.6$) and not
statistically significant (p-value of 0.14 using Mann-Whitney U test) with a
significant cost increase (double the training steps). As such, combining
mitigation strategies that target optimisation with augmentation strategies
may lead to further increase for all strategies.

The effectiveness of the DENI method and the different mitigation strategies
is consistent across different models. On the RoBERTa and ALBERT model, we
still observe similar findings for the DENI method and the majority of
mitigation strategies, even though the effect of mitigation in absolute number
may be different. In addition, the benefit of specific mitigation strategies
grows on different models (SWA or Augment for ALBERT).

###  4.2 Mitigation Strategies for PEFT Methods

In this section, we aim to answer the following research question: RQ2: What
is the benefit of mitigation strategies for parameter-efficient fine-tuning
(PEFT) methods? We compare the mitigation strategies and the DENI method
across different PEFT methods and observe how the increase in overall
performance and the reduction in standard deviation changes. The results for
the BERT model on the best performing mitigation strategies is in Table 2, for
all strategies on the SNIPS dataset in Figure 3, with the full results in
Appendix F.

Combination of the PEFT methods with the mitigation strategies can often lead
to more effective mitigation. Compared to full fine-tuning combination of
Ensemble with LoRA or UniPELT leads to higher increase in performance ($86.32$
for LoRA and $86.02$ for UniPELT compared to $85.56$ for full fine-tuning on
TREC) and reduction in standard deviation ($0.266$ and $0.291$ compared to
$0.304$). Similar behaviour can be observed on the Default baseline as well,
where with lower number of trained parameters, the performance or the standard
deviation is on par, or even better, than when training all parameters. For
example, we observe deviation of $0.403$ on CoLA dataset with UniPELT as
compared to $0.662$ for full fine-tuning. As such, when faced with limited
labelled data the instability of training can be addressed by reducing the
number of trainable parameters.

The Augment mitigation strategy provides more benefit when reducing the number
of trainable parameters. The IA3 method, which trains approximately 0.5%
parameters, benefits significantly more from using augmented data. For
example, we observe an increase in performance of up to $2$ percentage points
on the MRPC dataset for the DENIALS method over the DENI method. At the same
time, the Augment N often outperforms the Ensemble in terms of performance
when using the IA3 method, but not necessarily in terms of deviation. The
performance benefit of data augmentation is lower for other PEFT method,
specifically up to $0.6$ percentage points for LoRA (CoLA dataset) and up to
$0.8$ percentage points for UniPELT (SST2 dataset). Similarly, using augmented
data leads to a significantly larger performance increase for ALBERT and lower
increase for RoBERTa. Finally, using augmented data in combination with the
DENI method often leads to lower deviation in results across all tuning
methods, especially for PEFT. For example, using UniPELT the deviation drops
from $0.301$ to $0.054$ on AG News dataset, or from $0.678$ to $0.088$ on TREC
dataset. Similar results can be observed across different models for all of
the datasets and tuning methods.

###  4.3 Behaviour on Different Sizes of Dataset

Figure 4: Mitigation effectiveness across different dataset sizes for the BERT
model on TREC dataset. Benefit of mitigation strategies is higher on lower
number of shots and gradually decrease with more shots.

Our goal in this section is to answer the following research question: RQ3:
How does the number of available labelled samples (shots) affect the benefit
and effectiveness of mitigation strategies? We vary the number of available
samples from 1 per class up to 1000 per class and compare the Ensemble and
DENI method with the Default and All Data baselines. The results of this
ablation study for the BERT model on the TREC dataset presented in Figure 4
(the number of samples ranging from 6 in 1-shot setting up to 6000 in
1000-shot setting).

Mitigation strategies provide larger benefit on low number of samples. The
difference in performance between the mitigation strategies and the Default
baseline is as high as $20$ percentage points ($80\%$ compared to $60\%$ when
using $30$ shots), with significantly lower standard deviation ($0.453$
compared to $4.34$). As we increase the number of shots, the benefit gradually
decreases. However, the mitigation strategies provide their benefit even when
using all data for training, increasing the performance by up to $1$
percentage point and reducing deviation by up to $75\%$ (from $0.448$ to
$0.116$).

The DENI method outperforms Ensemble across all dataset sizes. The performance
difference oscillates between $0.4-3$ percentage points across the different
shots. Similarly, the deviation stays the same, or even lower for the DENI
method, especially on lower number of shots (e.g., $2.095$ compared to $3.418$
when using 5 shots).

##  5 Conclusions

In this work, we have proposed a novel method for effective and efficient
mitigation of the instability in fine-tuning of pre-trained language models
based on ensembling, model interpolation and noise regularisation. We compare
the method with prior mitigation strategies, showing it leads to stronger
mitigation, with lower computation cost. In addition, we show that the
mitigation strategies provide more benefit to the PEFT methods, such as LoRA,
IA3 or UniPELT. Combining mitigation strategies with PEFT can often lead to
higher performance and lower deviation than their combination with full fine-
tuning. Moreover, we show that data augmentation provides more benefit when
limiting the number of trainable parameters, such as using IA3 or training
smaller models.

## Acknowledgements

This work was partially supported by VIGILANT, a project funded by the
European Union under the Horizon Europe under the Contract No. 101073921, the
Central European Digital Media Observatory 2.0 (CEDMO 2.0), a project funded
by the European Union under the Contract No. 101158609, and the MODERMED, a
project funded by the Slovak Research and Development Agency under GA No.
APVV-22-0414.

This work was supported by the Ministry of Education, Youth and Sports of the
Czech Republic through the e-INFRA CZ (ID:90254).

## Limitations

The setting we focus on in the main experiments may obscure the overall
benefit and effectiveness of different mitigation strategies. First, it was
previously observed that smaller models with lower number of parameters show
lower sensitivity to the effects of randomness when working with limited
labelled data. As we use smaller models, namely base versions of BERT, RoBERTa
and ALBERT, the benefit of our proposed method as well as other mitigation
strategies may be lower than if we used larger models. In addition, we are
quite conservative with what we consider as low resource setting and treat
availability of 1000 labelled samples to already represent such. However, as
we observed in the experiments, the performance is already quite high and the
deviation quite low on some datasets even with this number of labelled
samples. In addition, in the experiments from Section 4.3, we observe that the
benefit of mitigation strategies on significantly lower number of samples per
class is significantly higher. As such, this may obscure the real benefit of
different mitigation strategies. However, our choices in both cases are based
on previous works. As discussed in the Experimental Setup, many previous works
consider 1000 labelled samples to already be low resource setting. At the same
time, many of the mitigation strategies were designed and evaluated on high-
resource setting with full datasets. Finally, we choose the smaller models in
order to provide more extensive analysis and comparison, while limiting the
impact (in terms of generated CO2) as much as possible.

Even though a large number of mitigation strategies exist, we focus on a
smaller set of representative strategies. This is especially evident with
augmentation strategies, where we focus only on augmentation using the
paraphrasing from pretrained large language models. However, when choosing the
strategies for comparison, we specifically focused on selecting the most
representative ones, i.e., those that achieved the best mitigation (in terms
of performance and standard deviation) in related work (and in some cases in
our pilot experiments) from different groups of mitigation strategies. As
such, even though we could increase the number strategies we compare to, it
would not provide any additional findings. At the same time, our goal is to
limit the potential negative impact of our work as much as possible and
therefore have focused only on this set of samples.

Finally, we do not consider prompt-based parameter-efficient fine-tuning
approaches (P-tuning (Liu et al., 2023), Prefix-tuning (Li and Liang, 2021) or
Prompt-tuning (Lester et al., 2021)), even though they are one of the most
popular PEFT methods that are currently used for many pretrained large
language models. However, we specifically focus on fine-tuning with typical
classification models (that use classification head at the end and do not
perform classification through prompting), and as such the prompt-based PEFT
methods would not provide any significant benefit as they were specifically
designed for training models for prompting and in-context learning. In
addition, the UniPELT method already uses Prefix-tuning (which may be the most
relevant for typical classification models) as one of the aggregated methods –
however, we have observed that this PEFT method performed the worst in our
experiments and was especially sensitive to the hyperparameter setup (both for
the models and for the mitigation strategies).

## References

  * Cegin et al. (2024) Jan Cegin, Branislav Pecher, Jakub Simko, Ivan Srba, Maria Bielikova, and Peter Brusilovsky. 2024.  Effects of diversity incentives on sample diversity and downstream model performance in llm-based text augmentation.  _Preprint_ , arXiv:2401.06643.
  * Cegin et al. (2023) Jan Cegin, Jakub Simko, and Peter Brusilovsky. 2023.  ChatGPT to replace crowdsourcing of paraphrases for intent classification: Higher diversity and comparable model robustness.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 1889–1905, Singapore. Association for Computational Linguistics.
  * Chen et al. (2022) Guanzheng Chen, Fangyu Liu, Zaiqiao Meng, and Shangsong Liang. 2022.  Revisiting parameter-efficient tuning: Are we really there yet? In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 2612–2626, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Chen et al. (2023) Lichang Chen, Jiuhai Chen, Heng Huang, and Minhao Cheng. 2023.  PTP: Boosting stability and performance of prompt tuning with perturbation-based regularizer.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 13512–13525, Singapore. Association for Computational Linguistics.
  * Coucke et al. (2018) Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, et al. 2018.  Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces.  _arXiv preprint arXiv:1805.10190_.
  * Dauphin and Schoenholz (2019) Yann N Dauphin and Samuel Schoenholz. 2019.  Metainit: Initializing learning by learning to initialize.  _Advances in Neural Information Processing Systems_ , 32.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dodge et al. (2020) Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. 2020.  Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping.  _arXiv preprint arXiv:2002.06305_.
  * Dolan and Brockett (2005) William B. Dolan and Chris Brockett. 2005.  Automatically constructing a corpus of sentential paraphrases.  In _Proceedings of the Third International Workshop on Paraphrasing_.
  * Feng et al. (2021) Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy. 2021.  A survey of data augmentation approaches for NLP.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 968–988, Online. Association for Computational Linguistics.
  * Gueta et al. (2023) Almog Gueta, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, and Leshem Choshen. 2023.  Knowledge is a region in weight space for fine-tuned language models.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 1350–1370, Singapore. Association for Computational Linguistics.
  * Guo (2020) Hongyu Guo. 2020.  Nonlinear mixup: Out-of-manifold data augmentation for text classification.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 4044–4051.
  * Guo et al. (2019) Hongyu Guo, Yongyi Mao, and Richong Zhang. 2019.  Augmenting data with mixup for sentence classification: An empirical study.  _arXiv preprint arXiv:1905.08941_.
  * Hidey et al. (2022) Christopher Hidey, Fei Liu, and Rahul Goel. 2022.  Reducing model churn: Stable re-training of conversational agents.  In _Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , pages 14–25, Edinburgh, UK. Association for Computational Linguistics.
  * Hu et al. (2022) Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022.  LoRA: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_.
  * Hua et al. (2023) Hang Hua, Xingjian Li, Dejing Dou, Cheng-Zhong Xu, and Jiebo Luo. 2023.  Improving pretrained language model fine-tuning with noise stability regularization.  _IEEE Transactions on Neural Networks and Learning Systems_.
  * Hua et al. (2021) Hang Hua, Xingjian Li, Dejing Dou, Chengzhong Xu, and Jiebo Luo. 2021.  Noise stability regularization for improving BERT fine-tuning.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3229–3241, Online. Association for Computational Linguistics.
  * Huang et al. (2017) Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E. Hopcroft, and Kilian Q. Weinberger. 2017.  Snapshot ensembles: Train 1, get m for free.  In _International Conference on Learning Representations_.
  * Izmailov et al. (2018) P Izmailov, AG Wilson, D Podoprikhin, D Vetrov, and T Garipov. 2018.  Averaging weights leads to wider optima and better generalization.  In _34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018_ , pages 876–885.
  * Kalajdzievski (2023) Damjan Kalajdzievski. 2023.  A rank stabilization scaling factor for fine-tuning with lora.  _arXiv preprint arXiv:2312.03732_.
  * Khurana et al. (2021) Urja Khurana, Eric Nalisnick, and Antske Fokkens. 2021.  How emotionally stable is ALBERT? testing robustness with stochastic weight averaging on a sentiment analysis task.  In _Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems_ , pages 16–31, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Köksal et al. (2023) Abdullatif Köksal, Timo Schick, and Hinrich Schuetze. 2023.  MEAL: Stable and active learning for few-shot prompting.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 506–517, Singapore. Association for Computational Linguistics.
  * Lacoste et al. (2019) Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. 2019.  Quantifying the carbon emissions of machine learning.  _arXiv preprint arXiv:1910.09700_.
  * Lan et al. (2020) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020.  Albert: A lite bert for self-supervised learning of language representations.  In _International Conference on Learning Representations_.
  * Lee et al. (2020) Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang. 2020.  Mixout: Effective regularization to finetune large-scale pretrained language models.  In _International Conference on Learning Representations_.
  * Lehmann et al. (2015) Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick van Kleef, Sören Auer, and Christian Bizer. 2015.  DBpedia – a large-scale, multilingual knowledge base extracted from wikipedia.  _Semant. Web_ , 6(2):167–195.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.  The power of scale for parameter-efficient prompt tuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3045–3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021.  Prefix-tuning: Optimizing continuous prompts for generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4582–4597, Online. Association for Computational Linguistics.
  * Liu et al. (2022) Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. 2022.  Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning.  In _Advances in Neural Information Processing Systems_ , volume 35, pages 1950–1965. Curran Associates, Inc.
  * Liu et al. (2023) Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2023.  Gpt understands, too.  _AI Open_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Lu et al. (2022) Peng Lu, Ivan Kobyzev, Mehdi Rezagholizadeh, Ahmad Rashid, Ali Ghodsi, and Phillippe Langlais. 2022.  Improving generalization of pre-trained language models via stochastic weight averaging.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 4948–4954, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Mao et al. (2022) Yuning Mao, Lambert Mathias, Rui Hou, Amjad Almahairi, Hao Ma, Jiawei Han, Scott Yih, and Madian Khabsa. 2022.  UniPELT: A unified framework for parameter-efficient language model tuning.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 6253–6264, Dublin, Ireland. Association for Computational Linguistics.
  * McCoy et al. (2020) R. Thomas McCoy, Junghyun Min, and Tal Linzen. 2020.  BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performance.  In _Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_ , pages 217–227, Online. Association for Computational Linguistics.
  * Meng et al. (2023a) Yu Meng, Martin Michalski, Jiaxin Huang, Yu Zhang, Tarek Abdelzaher, and Jiawei Han. 2023a.  Tuning language models as training data generators for augmentation-enhanced few-shot learning.  In _International Conference on Machine Learning_ , pages 24457–24477. PMLR.
  * Meng et al. (2023b) Yu Meng, Martin Michalski, Jiaxin Huang, Yu Zhang, Tarek Abdelzaher, and Jiawei Han. 2023b.  Tuning language models as training data generators for augmentation-enhanced few-shot learning.  In _Proceedings of the 40th International Conference on Machine Learning_ , ICML’23. JMLR.org.
  * Mosbach et al. (2021) Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. 2021.  On the stability of fine-tuning {bert}: Misconceptions, explanations, and strong baselines.  In _International Conference on Learning Representations_.
  * Obadinma et al. (2023) Stephen Obadinma, Hongyu Guo, and Xiaodan Zhu. 2023.  Effectiveness of data augmentation for parameter efficient tuning with limited data.  _arXiv preprint arXiv:2303.02577_.
  * Pan et al. (2023) Kaihang Pan, Juncheng Li, Hongye Song, Jun Lin, Xiaozhong Liu, and Siliang Tang. 2023.  Self-supervised meta-prompt learning with meta-gradient regularization for few-shot generalization.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 1059–1077, Singapore. Association for Computational Linguistics.
  * Pfeiffer et al. (2020) Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020.  MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7654–7673, Online. Association for Computational Linguistics.
  * Phang et al. (2018) Jason Phang, Thibault Févry, and Samuel R Bowman. 2018.  Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks.  _arXiv preprint arXiv:1811.01088_.
  * Razdaibiedina et al. (2023) Anastasiia Razdaibiedina, Yuning Mao, Madian Khabsa, Mike Lewis, Rui Hou, Jimmy Ba, and Amjad Almahairi. 2023.  Residual prompt tuning: improving prompt tuning with residual reparameterization.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 6740–6757, Toronto, Canada. Association for Computational Linguistics.
  * Sellam et al. (2022) Thibault Sellam, Steve Yadlowsky, Ian Tenney, Jason Wei, Naomi Saphra, Alexander D’Amour, Tal Linzen, Jasmijn Bastings, Iulia Turc, Jacob Eisenstein, Dipanjan Das, and Ellie Pavlick. 2022.  The MultiBERTs: BERT Reproductions for Robustness Analysis.  In _International Conference on Learning Representations_ , page 30.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics.
  * Summers and Dinneen (2021) Cecilia Summers and Michael J Dinneen. 2021.  Nondeterminism and instability in neural network optimization.  In _International Conference on Machine Learning_ , pages 9913–9922. PMLR.
  * Voorhees and Tice (2000) Ellen M. Voorhees and Dawn M. Tice. 2000.  Building a question answering test collection.  In _Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’00, page 200–207, New York, NY, USA. Association for Computing Machinery.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355, Brussels, Belgium. Association for Computational Linguistics.
  * Wang et al. (2023) Lijing Wang, Yingya Li, Timothy Miller, Steven Bethard, and Guergana Savova. 2023.  Two-stage fine-tuning for improved bias and variance for large pretrained language models.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 15746–15761, Toronto, Canada. Association for Computational Linguistics.
  * Warstadt et al. (2019) Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019.  Neural network acceptability judgments.  _Transactions of the Association for Computational Linguistics_ , 7:625–641.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_.
  * Wu et al. (2022) Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. 2022.  NoisyTune: A little noise can help you finetune pretrained language models better.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 680–685, Dublin, Ireland. Association for Computational Linguistics.
  * Xu et al. (2021) Runxin Xu, Fuli Luo, Zhiyuan Zhang, Chuanqi Tan, Baobao Chang, Songfang Huang, and Fei Huang. 2021.  Raise a child in large language model: Towards effective and generalizable fine-tuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 9514–9528, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Zhang et al. (2022) Haojie Zhang, Ge Li, Jia Li, Zhongjin Zhang, Yuqi Zhu, and Zhi Jin. 2022.  Fine-tuning pre-trained language models effectively by optimizing subnetworks adaptively.  _Advances in Neural Information Processing Systems_ , 35:21442–21454.
  * Zhang et al. (2018) Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. 2018.  mixup: Beyond empirical risk minimization.  In _International Conference on Learning Representations_.
  * Zhang et al. (2021) Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Weinberger, and Yoav Artzi. 2021.  Revisiting few-sample {bert} fine-tuning.  In _International Conference on Learning Representations_.
  * Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015.  Character-level convolutional networks for text classification.  In _Advances in Neural Information Processing Systems_ , volume 28. Curran Associates, Inc.

##  Appendix A Ethical Considerations and Impact Statement

The experiments in this paper work with publicly available benchmark dataset
GLUE (SST2, MRPC and CoLA from the benchmark), and publicly available datasets
AG News, TREC, SNIPS and DB-Pedia, citing the original authors. As we were not
able to determine the license for all of the tasks and datasets used, we opted
to use them in as limited form as possible, adhering to the terms of use (no
annotation of the test set) for the GLUE benchmark dataset and applying it to
other datasets as well. As the datasets are commonly used, we do not check
them for any identifiable information or offensive content, assuming the
publicly available classification benchmark datasets do not contain such
content (as it was already removed by the authors of the datasets). We do not
work with any personally identifiable information or offensive content and
perform no crowdsourcing for further data annotation. In addition, we are not
aware of any potential ethical harms or negative societal impacts of our work,
apart from the ones related to the advancement of the field of Machine
Learning and Learning with Limited Labelled Data (mainly the use of
computation resources, consuming energy and generating CO2). Finally, we
follow the license terms for all the models we use – all models and datasets
allow their use as part of research. As we perform only classification and do
not release the predictions, we generate no potentially biased or offensive
content.

#### Impact Statement: CO2 Emissions Related to Experiments

The experiments presented in this paper used significant compute resources as
we train multiple models (3) over multiple random seeds (20), for different
mitigation strategies and baselines (17), fine-tuning methods (4) and datasets
(7). Overall, all the experiments (including preliminary experiments for which
we do not report results in this paper) were conducted using a private
infrastructure, which has a carbon efficiency of 0.432 kgCO2eq/kWh. A
cumulative of approximately 1000 hours of computation was performed on
hardware of type A100 PCIe 40GB (TDP of 250W). Total emissions are estimated
to be 108 kgCO2eq of which 0 percent were directly offset. These estimations
were conducted using the MachineLearning Impact calculator presented in
Lacoste et al. (2019). Whenever possible, we tried to reduce the compute
resources used as much as possible. We evaluate on smaller models (base
versions instead of large). In addition, we evaluate and compare our method
only to a set of representative mitigation strategies, determined based on
prior works and our preliminary experiments. Finally, the estimate does not
include the cost of generating the paraphrases as the information about the
efficiency for the large language model behind API is not available. To limit
the impact of the augmentation, we decide to run it only a single time on a
set of labelled samples that are used throughout the training.

##  Appendix B Hyperparameter Sensitivity Analysis

Figure 5: The effect of hyperparameter setup on the mitigation effectiveness
of the DENI method, based on hyperparameter search for the BERT model on TREC
dataset. The hyperparameters that affect each other are grouped together.

In this section, we explore the sensitivity of the DENI method to the
hyperparameter setup. We focus on the hyperparameters that represents the
trade-off between mitigation effectiveness and computation costs. The
hypeparameters that directly affect each other are grouped together as
follows: 1) the size of noise we are adding ($var_{noise}$) together with how
often the noise is added ($steps_{noisy}$); 2) number of steps after which we
start ($noise_{start}$) and stop ($noise_{end}$) adding noise; 3) number of
steps after which the final ensemble is created ($ensemble_{start}$); and 4)
number of models in the ensemble ($N$). The results of this analysis are
presented in Figure 5 for BERT on TREC dataset.

The number of models in ensemble shows highest influence. Increasing the
number of models that are used in the ensemble leads to significant increase
in mitigation effectiveness (increasing performance and reducing deviation),
but also a significant increase in the cost. However, after a certain number
($10$), the improvement of mitigation strength is negligible. As such, we
recommend using $8-10$ models in the ensemble, as it represents the optimal
trade-off between computation and mitigation.

When increasing the size of noise, we also need to add it less often (and vice
versa) to get optimal performance, in terms of both performance and deviation.
Small noise should be added often, while large noise should be added only
occasionally. On the other hand, adding large noise often leads to large
decrease in overall performance, while adding small noise only sporadically
leads to no mitigation. We find that the optimal value for both factors is at
their midpoint – adding noise of medium size ($0.15$) every so often ($125$
steps). However, this only holds when using the scaling factor. When foregoing
the scaling factor, we observe a significant sensitivity to the size of noise.

Noise-adding noise should be started later in the training. At the start of
optimisation, we observe a higher sensitivity of the model to the added noise,
negatively affecting the mitigation effectiveness (i.e., leading to smaller
performance increase and smaller decrease in deviation, often even being
counterproductive). The sensitivity to the added noise slowly decreases as the
training progresses (also due to the learning rate and scaling factor). As
such, we observe that the optimal value for the $noise_{start}$ is after 30%
of the training. At the same time, adding noise for a longer period of time
increases the overall deviation in results, while increasing the overall
performance only until a certain point. For example, starting the noise early
and adding it for 60% of training leads to a large increase in deviation (and
decrease in performance). Overall, optimal value for $noise_{end}$ is 30% of
training steps after the $noise_{start}$ value, regardless of its value.
Finally, we observe negligible impact of the $ensemble_{start}$
hyperparameter. The only requirement is to leave enough steps of optimisation
for the model to deal with the added noise that can break the learned
knowledge (i.e., leaving enough steps for the different models to converge to
their optimal parameters). As such, it is enough to create the ensemble after
90% of training steps are already done, as creating the ensemble after this
point leads to significant decrease mitigation (although if using larger
number of steps, the start of ensemble may be delayed even further).

Overall, the hyperparameter sensitivity of the DENI method is not as
significant. In majority of the cases, the optimal parameters can be
determined based on heuristics (e.g., adding large noise often breaks the
models), while slight deviation from the optimal values does not leave to a
significant changes in the mitigation effectiveness. However, this holds true
only when using the recommended hyperparameter setup introduced in Section 3
(using the scaling factor based on standard deviation of the individual
parameters in the model and adding noise only to the newly initialised
parameters).

##  Appendix C Experimental Setup: Further Details

###  C.1 General Experimental Setup

For the experiments, we are using English only datasets from the GLUE (Wang et
al., 2018) benchmark suite and other publicly available datasets. The datasets
we use from GLUE benchmark are: 1) SST2 (Socher et al., 2013), binary dataset
with up to 68 000 labelled samples; 2) CoLA (Warstadt et al., 2019), binary
dataset with up to 10 000 labelled samples; and 3) MRPC (Dolan and Brockett,
2005), binary dataset with up to 6 000 labelled samples. The remaining
datasets are all multi-class datasets, specifically: 1) Ag News Zhang et al.
(2015) with 4 classes and up to 120 000 labelled samples; 2) TREC Voorhees and
Tice (2000) with 6 classes and up to 6 000 labelled samples; 3) SNIPS (Coucke
et al., 2018) with 7 classes and up to 15 000 labelled samples; and 4) DBPedia
(Lehmann et al., 2015) with 14 classes and up to 630 000 labelled samples. All
datasets are split into training and testing using 80-20 split. In addition,
the training dataset is split into training and validation set, with the
validation set used for hyperparameter tuning. When selecting the 1 000
labelled samples for training for the main experiments (or smaller/larger
number of samples for the size change experiments), we use uniform split,
i.e., we select the same number of labelled samples for each class.

Each experiment is repeated 20 times, each time with different random seeds
that influences the randomness originating from model initialisation, data
shuffling and non-deterministic operations in the model (e.g., dropout). Each
such repeat uses the same set of training and testing data, mainly in order to
make the data augmentation easier (i.e., we can use the augmentation only
single time on the set of labelled samples without worrying about introducing
an information leak, such as using for training samples that are paraphrases
of the ones in the test or validation set).

For the experiments, we use the base versions of 3 representative pretrained
language models (obtained from HuggingFace (Wolf et al., 2019)), specifically:
1) BERT (Devlin et al., 2019)222https://huggingface.co/bert-base-uncased with
110M parameters; 2) RoBERTa (Liu et al.,
2019)333https://huggingface.co/roberta-base with 130M parameters; and 3)
ALBERT (Lan et al., 2020)444https://huggingface.co/albert/albert-base-v2 with
12M parameters. We add a Dropout layer with a drop rate of 0.3 followed by
fully-connected classification layer on top of each of these models. Each
model is trained for 10 epochs using Adam optimiser, learning rate of 1e-5 and
batch size of 8 (with further modifications to these hyperparameters based on
the mitigation strategy used, more information in Appendix C.2).

Besides full fine-tuning, where we train all parameters of the models, we also
perform the experiments with parameter-efficient fine-tuning (PEFT) methods,
specifically: 1) LoRA (Hu et al., 2022), applied to all attention layers, with
rank $64$, alpha $64$, dropout $0.1$, with bias for all layers and using rank
stabilisation (Kalajdzievski, 2023), that trains approximately $2\%$ of the
parameters; 2) IA3 (Liu et al., 2022), applied to all atention and linear
layers, that trains approximately $0.5\%$ of parameters; and 3) UniPELT (Mao
et al., 2022), which combines LoRA (same as above), Pfeiffer Adapter (Pfeiffer
et al., 2020), with reduction factor of $16$, and Prefix-Tuning (Li and Liang,
2021)), with prefix-length of $25$, that trains approximately $10\%$ of the
parameters. As specified in the Limitations section, we do not focus on any
prompt-based PEFT methods (P-tuning (Liu et al., 2023), Prefix-tuning (Li and
Liang, 2021) or Prompt-tuning (Lester et al., 2021)), as these were not
designed for optimising models with classification heads (and in our
preliminary experiments, the prompt-based PEFT methods achieved significantly
lower performance). Each of the PEFT methods has slightly different
hyperparameters for training, as we have observed a significant sensitivity of
the individual PEFT methods to the hyperparameter setup (with LoRA showing
lowest sensitivity and UniPELT showing highest sensitivity). Specifically, we
use following setup: 1) for LoRA we use learning rate of 1e-4 for BERT and
ALBERT models and 1e-5 for RoBERTa model; 2) for IA3 we use learning rate of
75e-4; and 3) for UniPELT we use learning rate of 1e-4 for BERT, 25e-5 for
RoBERTa and 6e-5 for ALBERT.

All of the hyperparameters, for all the combination of model and fine-tuning
approach, are set using a separate hyperparameter optimisation using the
validation data. This hyperparameter optimisation is done in a two-level
fashion. First, the optimisation is run using large differences in the
hyperparameter values, to find the approximate set of hyperparameters that
should provide good performance on the given dataset. In the second step, we
explore the hyperparameter space around these approximate hyperparameters, to
find the optimal set of parameters. In addition, the performance in the
hyperparameter search is evaluated from 5 repeated runs (changing the model
initialisation, data order and model non-determinism, but not the labelled
training or validation data), taking their average into consideration. When
choosing the hyperparameter values in the first level, we draw inspiration
from related work, using the optimal parameters reported in papers that
propose, or use these approaches (such as (Dodge et al., 2020; McCoy et al.,
2020; Mosbach et al., 2021; Sellam et al., 2022). However, we also search
through additional hyperparameter values besides those reported in related
works to better explore the parameter space and obtain as precise results from
the investigation as possible.

For the size change experiments (Section 4.3) the experimental setup is
slightly different. Instead of choosing the 1 000 labelled samples, we instead
choose $N$ samples per class (called $N$-shots) and change the value of $N$,
going from a low value up to the large part of the dataset. Specifically, we
cover following $N$ values: 1, 5, 10, 15, 20, 25, 30, 40, 50, 100, 150, 200,
300, 400, 500, 600, 700, 800, 900 and 1000. Due to the large computation cost
of the size change experiment, the experiment is done only for the full fine-
tuning of the BERT model on the TREC dataset. The remaining experimental setup
is kept the same as for the main experiments.

Similarly, the experimental setup for the hyperparameter sensitivity analysis
(Appendix B) is slightly different. Due to the large number of combinations,
we limit the number of repeated runs only to 5. In addition, we run the
sensitivity analysis only for the full fine-tuning of the BERT model on the
TREC dataset. The hyperparameters and their values we search through are: 1)
$var_{noise}$ with values of 0.05, 0.1, 0.15, 0.2, 0.25 and 0.3; 2)
$step_{noisy}$ with values of 5, 10, 15, 20, 25, 50, 75, 100, 125, 150, 200,
250 and 300; 3) $noise_{start}$ with values of 0.1, 0.2 and 0.3; 4)
$noise_{end}$ with values of 0.4, 0.5, 0.6 and 0.7; 5) $ensemle_{start}$ with
values of 0.7, 0.75, 0.8, 0.85, 0.9 and 0.95; and 6) ensemble size with values
of 1, 2, 3, 5, 7, 9, 10 and 12. The remaining experimental setup is kept the
same as for the main experiments.

###  C.2 Mitigation Strategies Setup

Dataset |  Prompt
---|---
AG News, TREC, SNIPS, DBPedia, SST2 |  Rephrase an original question or statement 10 times. Format your responses like: 1. rephrase 1, 2. rephrase 2, … , 10. rephrase 10. Original phrase: "{}".
MRPC |  Paraphrase text consisting of two sentences in the format "Sentence 1: … ; Sentence 2:…" Paraphrases both sentences 10 times. Format your responses in a JSON format. The text: "{}"
CoLA |  Rephrase an original grammatically correct/incorrect text 10 times. Ensure that the rephrases are grammatically correct/incorrect. Format your responses like: 1. rephrase 1, 2. rephrase 2, … , 10. rephrase 10. Original text: "{}".
Table 3: Prompts used for generating the paraphrases for the different
datasets. The MRPC and CoLA dataset use a modified prompt to guarantee that
the generated paraphrases are valid. For the CoLA dataset, the prompt
guarantees that the paraphrase has the same class for the sample (either
grammatically acceptable or not). For the MRPC dataset, the prompt guarantees
that the two sentences are paraphrased at once, reducing the number of queries
needed otherwise.

In this section, we provide further details regarding the hyperparameter setup
for the individual mitigation strategies and how this affects their cost
(represented as the number of steps). The hyperparameter setup for each of the
methods, except for the baselines (Default and Full) and our method is
determined using a hyperparameter search (with the same setup as for
individual models, i.e., using average over 5 repeated runs on validation
data).

The Default baseline uses the setup as specified in Appendix C.1, i.e.,
training for 10 epochs on 1 000 samples, with batch size of 8 and Adam
optimiser. As such, the number of steps for each dataset is 1250, which we use
as the normalisation number for the remaining mitigation strategies (i.e.,
having normalised cost value of $1$).

The All Data baseline uses all the labelled samples available in the dataset,
with the same setup as Default baseline. The exception are the AG News and
DBPedia dataset, which are downsampled to use up to 55 000 samples. As such,
the number of steps (and the cost of this baseline) is dataset dependent. For
the TREC and MRPC datasets, the cost is $5$ times the one of Default, for the
CoLA the cost is $10$, for SNIPS $11.5$ and for the AG News, SST2 and DBPedia
the cost is $55$ times the Default.

The Best Practices mitigation strategy uses the AdamW optimiser with bias
correction, warmup for 10% of training step followed by linear scheduler and
increases the number of training epoch to 20. As such, the cost is $2$ times
the one in Default baseline.

The Ensemble mitigation strategy uses the same setup as the Default baselines,
but trains 10 models with different initialisation and data order. As such,
the cost is 10 times the one in Default baseline.

The NoiseInput mitigation strategy introduces a Gaussian noise with the
variance of 0.15 every 25 steps into the input embeddings of the model.
Similarly, the NoiseWeights mitigation strategy introduces a Gaussian noise
with the variance of 0.15 (using scaling based on parameter standard deviation
and number of steps) every 25 steps to all the randomly initialised parameters
of the model (i.e., only the classification head or also the parameters added
by PEFT method). As this does not modify the number of steps, the cost is the
same as Default baseline.

For the Stochastic Weight Averaging (SWA) mitigation strategy we start the
averaging after 25% of training steps using the same small learning rate as
used for the model (as recommended by Lu et al. (2022)). As such, for the 75%
of training, we perform optimisation on 2 models and so the cost is $1.75$
time the one in Default baseline.

The Mixout mitigation strategy replaces the Dropout layers with mix
probability of $0.9$ as recommended by Lee et al. (2020). As it does not
introduce any increase in the number of steps, the cost is the same as the one
in Default baseline.

For the Augment N mitigation strategy, we follow the experimental setup from
Cegin et al. (2023, 2024). A pre-trained large language model is used to
paraphrase all the training samples 10 times and the generated paraphrases are
then use as additional samples for training. For paraphrasing we use the
ChatGPT 3.5 (version gpt-3.5-turbo-0125) model, with temperature value of 1
and other parameters set to default values. The prompts used to generate the
paraphrases are in Table 3. For MRPC dataset, we formatted the augmented
phrases in JSON for easier parsing. For CoLA dataset, we noticed that the
resulting paraphrases for the grammatically incorrect sentences were in some
cases ( 10% of the time) grammatically correct, meaning the label was
incorrect. We did not do any additional manual filtering though (as it would
be costly) and used the method as is with the all of the collected data. To
reduce the required computation (and other) costs of the augmentation, we run
it only a single time, asking the large language model to generate 10
paraphrases at once. When selecting what paraphrases to use for different $N$,
we always select from top ones, i.e., when selecting $1$ paraphrase, we always
take the first generated by the model, or when selecting $3$, we always take
top three paraphrases. In addition, we increase the number of epochs to 12, as
we have observed this significantly improves the performance. As the
mitigation strategy introduces new samples and we increase the number of
steps, the resulting cost of the strategy can be calculated as $1.2*(N+1)$.

Finally, for our method, we set the hyperparameters based on the
hyperparameter search (Appendix B), superficially, we use $var_{noise}$ set to
0.15, $steps_{noisy}$ set to 125, $noise_{start}$ set to 30% of training
steps; $noise_{end}$ set to 60% of the training steps; $ensemble_{start}$ set
to 90% of the training steps; ensemble size of 10; and the remaining
parameters as recommended in Section 3 (i.e., using Gaussian noise introduced
only to newly initialised parameters, same number of steps for $steps_{noisy}$
and $steps_{regular}$, using scaled noise based on the parameters and also on
number of already done steps for the Noisy Interpolation part). In addition,
for the DENIALS configuration, we use the Augment 1 to generate the new
samples. We calculate the cost for the DENI method in following way. For the
Delayed Ensemble (DE), a single model is trained for 90% of the training steps
(cost of $0.9$) and 10 models are trained for the remaining training steps
(cost of $10*0.1$) resulting in a cost of $1.9$. For the Noisy Interpolation
part a single model is trained for 70% of the training steps (cost of $0.7$),
and the noise is added for 30% of training steps, from which we train the
ensemble for 10% of training (cost of $1$), single model for additional 10% of
training (cost of $0.1$) and another ensemble for additional 10% of training
(cost of $1$), resulting in cost of $2.8$. The cost of the Delayed Ensemble
with Noisy Interpolation (DENI) is the combination of the Noisy Interpolation
($2.8$) and the cost for the final ensemble ($0.9$) resulting in cost of
$3.7$. Finally for the Delayed Ensemble with Noisy Interpolation and Augmented
Labelled Samples (DENIALS) we double the number of steps and so the cost is
calculated as $(N+1)*3.7$, resulting in cost of $7.4$.

##  Appendix D Additional Results: Relationship Between Performance, Deviation
and Normalised Cost

Figure 6: The relationship between performance, standard deviation and the
normalised cost for the different mitigation strategies. The results are
presented for the BERT model and all fine-tuning methods on all datasets. For
the sake of better readability, we highlight only the most efficient
mitigation strategies and baselines (highest performance and lowest standard
deviation), greying out all the remaining ones. We can observe that the DENI
method provides the most effective mitigation, with the performance and
standard deviation being on par or better than other strategies, while
requiring only a fraction of their computation costs.

In this Appendix, we provide a visualisation of the relationship between the
performance, standard deviation and the normalised cost for the different
mitigation strategies. The visualisation serves to allow for better evaluation
and comparison for the benefit of the mitigation strategies, which also takes
cost into consideration. The relationship is visualised in Figure 6 for the
BERT model for all fine-tuning methods across all datasets.

We observe that the mitigation strategies that provide the highest benefit,
increasing performance and reducing standard deviation, also introduce
increase in the computation cost. When taking the cost into consideration, the
DENI method (without the augmented labelled samples) appears the most
efficient, as it provides similar (and often higher) performance, similar (and
often lower) standard deviation, while requiring only half of the computation
costs. Similarly, the DENI method provides higher performance and similar (and
often lower) standard deviation than the Ensemble mitigation strategy, while
requiring only a 1/3 of the computation cost. An additional interesting
comparison is with the All Data baseline, where the DENI method, but often
also Ensemble mitigation strategy, appear close to the baseline, often
outperforming it in terms standard deviation.

##  Appendix E Additional Results: Augment N With Higher Number of Paraphrases

Figure 7: The performance of the Augment mitigation strategy for the different
values of $N$ for BERT using full fine-tuning across all datasets. The
reported performance is normalised using the mean value of the Default
baselines for each of the given datasets (i.e., subtracting the mean from all
values). Figure 8: The performance of the Augment mitigation strategy for the
different values of $N$ for BERT using LoRA across all datasets. The reported
performance is normalised using the mean value of the Default baselines for
each of the given datasets (i.e., subtracting the mean from all values).
Figure 9: The performance of the Augment mitigation strategy for the different
values of $N$ for BERT using IA3 across all datasets. The reported performance
is normalised using the mean value of the Default baselines for each of the
given datasets (i.e., subtracting the mean from all values). Figure 10: The
performance of the Augment mitigation strategy for the different values of $N$
for BERT using UniPELT across all datasets. The reported performance is
normalised using the mean value of the Default baselines for each of the given
datasets (i.e., subtracting the mean from all values). Figure 11: The
performance of the Augment mitigation strategy for the different values of $N$
for RoBERTa using full fine-tuning across all datasets. The reported
performance is normalised using the mean value of the Default baselines for
each of the given datasets (i.e., subtracting the mean from all values).
Figure 12: The performance of the Augment mitigation strategy for the
different values of $N$ for RoBERTa using LoRA across all datasets. The
reported performance is normalised using the mean value of the Default
baselines for each of the given datasets (i.e., subtracting the mean from all
values). Figure 13: The performance of the Augment mitigation strategy for the
different values of $N$ for RoBERTa using IA3 across all datasets. The
reported performance is normalised using the mean value of the Default
baselines for each of the given datasets (i.e., subtracting the mean from all
values). Figure 14: The performance of the Augment mitigation strategy for the
different values of $N$ for RoBERTa using UniPELT across all datasets. The
reported performance is normalised using the mean value of the Default
baselines for each of the given datasets (i.e., subtracting the mean from all
values). Figure 15: The performance of the Augment mitigation strategy for the
different values of $N$ for ALBERT using full fine-tuning across all datasets.
The reported performance is normalised using the mean value of the Default
baselines for each of the given datasets (i.e., subtracting the mean from all
values). Figure 16: The performance of the Augment mitigation strategy for the
different values of $N$ for ALBERT using LoRA across all datasets. The
reported performance is normalised using the mean value of the Default
baselines for each of the given datasets (i.e., subtracting the mean from all
values). Figure 17: The performance of the Augment mitigation strategy for the
different values of $N$ for ALBERT using IA3 across all datasets. The reported
performance is normalised using the mean value of the Default baselines for
each of the given datasets (i.e., subtracting the mean from all values).
Figure 18: The performance of the Augment mitigation strategy for the
different values of $N$ for ALBERT using UniPELT across all datasets. The
reported performance is normalised using the mean value of the Default
baselines for each of the given datasets (i.e., subtracting the mean from all
values).

In this Appendix, we provide the results for the Augment N mitigation strategy
with additional values for $N$, specifically $3$, $5$ and $10$. The results
are reported for all the models and datasets and are normalised by the mean
value of Default baseline for each of the given datasets. For the BERT model,
the results are reported in Figure 7 for full fine-tuning, Figure 8 for LoRA,
Figure 9 for IA3 and Figure 10 for UniPELT. For the RoBERTa model, the results
are reported in Figure 11 for full fine-tuning, Figure 12 for LoRA, Figure 13
for IA3 and Figure 14 for UniPELT. For the ALBERT model, the results are
reported in Figure 15 for full fine-tuning, Figure 16 for LoRA, Figure 17 for
IA3 and Figure 18 for UniPELT.

For the majority of the cases, the highest performance increase is observed
when using only 1 or 2 paraphrases for each of the labelled samples (the
performance reported in the main content of the paper). Increasing the number
of paraphrases, the performance gradually decreases, achieving lowest
performance with 10 paraphrased samples. The reason for this decrease can be
twofold. First, we run the paraphrasing only a single time, generating 10
paraphrases at once and selecting from the top. As the model tries to generate
10 paraphrases at once, the later ones may be significantly different, using
more obscure words. As such, adding such paraphrases into the training may
confuse the model as the distribution of words is significantly different (and
may even represent noise for the model). Second, with the higher number of
paraphrases, the model may start to lose its generalisability as it is trained
on very similar samples (i.e., paraphrases) and thus start overfitting the
data distribution which is not representative of the test data. However, we
observe some exceptions, where using full set of paraphrases does not lower
the performance.

In addition, we observe that the benefit of data augmentation is dependent on
the model and the fine-tuning approach used. When limiting the number of
trainable parameters, either by using the IA3 PEFT method (which trains
approximately 0.5% of the parameters) or using the ALBERT model (which has
only 12M parameters), we observe a significantly higher benefit of the data
augmentation. In specific cases, using the augmented data can lead to a
performance higher than from any other mitigation strategy, such as using
ALBERT model trained with IA3 method on CoLA dataset (see Table 6).

Finally, we can observe that in specific cases, the augmented data cause
significant problems to the models. Specifically, we observe a significant
decrease in performance for the UniPELT method on the MRPC dataset across all
models. In addition, we observe that the augmented data on MRPC dataset show
higher standard deviation across all models and fine-tuning approaches. The
reason for both may be due to the characteristics of the MRPC dataset – the
task of determining the semantic similarity between 2 sentences. Although we
specifically design the prompt to paraphrase both sentences at once, which
should keep their semantic similarity, the large language model may not
produce correct paraphrases in this case (e.g., introducing a significant
changes, making the sentences too dissimilar). Such augmented data may cause
problems to the models and especially to UniPELT, which we observed to be the
most brittle fine-tuning approach.

##  Appendix F Additional Results: All Datasets, Models and Fine-Tuning
Approaches

BERT | AG News | TREC | SNIPS | DBPedia | SST2 | MRPC | CoLA
---|---|---|---|---|---|---|---
Full FineTuning
Default |  84.950.482 |  90.000.682 |  97.990.109 |  98.750.047 |  88.270.230 |  62.731.497 |  75.110.662
All Data |  88.650.343 |  95.660.579 |  98.860.066 |  99.210.030 |  95.150.078 |  68.860.893 |  79.560.440
Best Practices |  84.960.444 |  90.400.585 |  98.050.137 |  98.760.049 |  88.320.313 |  63.071.593 |  75.300.718
Ensemble |  85.560.304 |  90.670.400 |  98.220.076 | 98.820.021 |  89.050.079 |  65.690.674 |  76.540.325
NoiseInput |  85.070.444 |  89.430.775 |  98.090.114 |  98.770.043 |  88.250.285 |  62.441.499 |  75.050.817
NoiseWeights |  85.250.671 |  90.200.860 |  98.030.146 |  98.750.061 |  88.040.266 |  62.681.765 |  75.060.699
SWA |  85.400.364 |  90.240.632 |  98.060.122 |  98.760.060 |  88.540.211 |  63.501.751 |  75.260.572
Mixout |  84.960.538 |  90.200.603 |  98.030.111 |  98.730.040 |  88.390.280 |  63.231.137 |  75.330.572
Augment 1 |  85.250.542 |  90.360.945 |  98.190.142 |  98.820.082 |  88.660.286 |  63.011.544 |  75.560.443
Augment 2 |  85.280.433 |  90.450.676 |  98.100.143 |  98.720.076 |  88.630.327 |  62.401.514 |  74.580.426
OurDE |  85.170.291 |  90.540.278 |  98.100.086 | 98.760.019 |  88.430.107 |  63.680.594 |  75.090.281
OurNI |  86.810.760 |  91.420.901 |  98.420.131 |  98.990.044 |  90.080.284 |  65.381.571 |  77.160.586
OurDENI |  87.670.261 |  92.040.377 |  98.650.099 | 99.070.023 |  90.820.129 |  66.660.613 |  77.740.263
OurDENIALS | 88.180.196 | 92.270.236 | 98.730.075 |  99.030.040 | 91.220.132 | 67.040.544 | 78.290.153
LoRA
Default |  85.100.488 |  89.680.770 |  98.060.142 |  98.720.077 |  87.950.315 |  61.706.377 |  74.800.907
All Data |  88.450.485 |  95.290.393 |  98.880.082 |  99.210.029 |  95.160.086 |  70.641.156 |  77.278.283
Best Practices |  85.460.533 |  90.511.135 |  98.100.167 |  98.740.058 |  88.130.263 |  65.360.925 |  74.970.662
Ensemble |  86.320.266 |  91.580.371 |  98.320.125 | 98.860.030 | 89.170.109 |  66.590.753 |  77.020.533
NoiseInput |  85.420.482 |  90.420.986 |  98.070.169 |  98.760.037 |  88.160.285 |  65.021.277 |  75.480.545
NoiseWeights |  85.090.532 |  90.740.840 |  98.020.141 |  98.730.061 |  88.140.280 |  64.253.900 |  75.250.851
SWA |  85.670.446 |  90.330.948 |  98.130.145 |  98.750.058 |  86.787.381 |  63.268.973 |  74.650.739
Mixout |  85.470.509 |  90.910.864 |  98.140.128 |  98.680.071 |  88.210.306 |  64.311.418 |  75.150.860
Augment 1 |  85.650.542 |  90.840.998 |  98.180.163 |  98.850.079 |  88.870.339 |  64.235.017 |  76.440.662
Augment 2 |  85.220.524 |  90.150.814 |  98.210.164 |  98.780.068 |  88.820.149 |  64.701.366 |  76.240.808
OurDE | 85.920.187 |  91.230.359 |  98.210.087 |  98.750.035 |  88.300.111 |  65.000.644 |  75.650.439
OurNI |  87.360.688 |  91.391.065 |  98.440.167 |  98.930.072 |  90.100.332 |  65.294.747 |  77.280.680
OurDENI | 87.860.305 | 92.390.377 | 98.670.084 | 99.050.041 |  90.740.122 | 67.490.554 |  77.660.425
OurDENIALS |  87.790.244 | 92.240.204 | 98.650.065 |  99.040.038 | 91.030.206 |  66.790.671 | 78.240.041
IA3
Default |  83.830.829 |  88.421.197 |  97.590.259 |  98.560.105 |  87.600.216 |  62.720.979 |  73.560.963
All Data |  87.720.538 |  94.930.449 |  98.740.065 |  99.170.025 |  94.830.069 |  67.511.139 |  78.280.806
Best Practices |  84.340.501 |  89.470.923 |  97.850.188 |  98.670.065 |  87.580.225 |  62.920.994 |  74.530.646
Ensemble |  85.390.327 |  90.540.304 | 98.160.082 | 98.780.034 | 88.550.092 |  64.540.767 |  75.990.440
NoiseInput |  84.180.609 |  89.300.919 |  97.830.207 |  98.610.068 |  87.510.223 |  62.051.070 |  74.520.812
NoiseWeights |  84.350.488 |  89.200.805 |  97.820.182 |  98.670.089 |  87.630.277 |  63.021.426 |  74.260.739
SWA |  84.790.479 |  90.150.618 |  97.920.099 |  98.780.060 |  88.190.196 |  62.541.127 |  73.870.663
Mixout |  84.010.714 |  88.661.660 |  97.630.203 |  98.700.072 |  87.560.253 |  62.712.212 |  74.050.738
Augment 1 |  85.230.590 |  90.030.861 |  98.180.180 |  98.880.077 |  88.650.229 |  64.251.079 |  75.960.710
Augment 2 |  85.030.587 |  90.100.928 |  98.250.149 |  98.880.089 |  88.680.234 |  63.611.078 |  75.580.672
OurDE |  84.200.309 |  89.330.371 |  97.820.128 |  98.700.040 |  87.740.107 | 63.010.335 |  74.670.381
OurNI |  85.740.641 |  90.531.158 |  98.200.161 |  98.920.094 |  89.620.233 |  64.221.152 |  75.670.749
OurDENI | 86.740.279 |  90.840.238 |  98.400.117 |  99.030.041 |  90.150.121 |  65.000.473 |  76.700.295
OurDENIALS | 87.080.574 | 92.260.127 | 98.420.128 | 99.040.042 | 90.350.224 | 67.020.378 | 78.300.037
UniPELT
Default |  84.970.582 |  87.131.316 |  97.630.194 |  98.560.262 |  87.820.233 |  63.770.798 |  74.910.403
All Data |  88.580.532 |  94.420.782 |  98.770.105 |  99.230.029 |  95.260.095 |  69.771.139 |  79.370.590
Best Practices |  84.750.649 |  89.610.897 |  97.910.177 |  98.760.073 |  88.010.285 |  63.061.003 |  75.320.642
Ensemble |  86.020.291 |  91.080.559 |  98.310.104 | 98.890.025 |  88.630.141 |  64.070.610 |  76.000.458
NoiseInput |  84.670.629 |  88.931.068 |  97.860.190 |  98.750.067 |  87.820.247 |  62.941.128 |  75.160.652
NoiseWeights |  84.870.464 |  89.721.120 |  98.000.237 |  98.730.073 |  86.327.537 |  63.271.278 |  75.090.687
SWA |  85.600.589 |  89.711.038 |  98.000.128 |  98.760.080 |  88.370.163 |  64.260.919 |  75.690.685
Mixout |  84.810.502 |  89.820.887 |  97.960.173 |  98.690.069 |  87.950.217 |  62.841.371 |  75.040.770
Augment 1 |  85.060.366 |  88.651.101 |  97.870.118 |  98.720.077 |  88.210.200 |  61.001.055 |  74.730.564
Augment 2 |  84.690.584 |  88.520.860 |  97.890.193 |  98.640.076 |  88.180.192 |  60.040.990 |  74.220.702
OurDE |  85.110.244 |  90.280.461 |  97.890.119 |  98.780.047 |  88.130.128 |  64.070.412 |  75.460.250
OurNI |  86.760.588 |  90.811.272 |  98.350.161 |  99.020.074 |  90.080.309 |  65.041.179 |  77.050.631
OurDENI | 87.540.301 |  91.430.678 | 98.560.084 | 99.060.045 | 90.570.078 | 66.720.275 |  77.680.409
OurDENIALS | 87.440.054 | 92.250.088 | 98.560.068 |  99.040.045 | 91.010.157 |  66.590.323 | 78.240.033
Table 4: Comparison of the DENI method with existing mitigation strategies and
baselines on full fine-tuning, LoRA, IA3 and UniPELT using BERT model. The
comparison is done in terms of overall performance and deviation. The highest
performance for each fine-tuning method is in bold and lowest deviation is
underlined (not considering All Data baseline). Figure 19: Benefit of
mitigation strategies for the different fine-tuning methods using BERT on AG
News dataset. The benefit is calculated as difference to the mean performance
of the Default baseline when using full fine-tuning. The different mitigation
strategies are beneficial for all fine-tuning methods, but with different
overall benefit (e.g., Augment on IA3).

In this Appendix we provide the results for each combination of dataset,
model, mitigation strategy and fine-tuning approach. All the results are
provided in Table 4 for the BERT model, in Table 5 for the RoBERTa model and
in Table 6 for the ALBERT model. In addition, we provide results in a form of
figures (boxplots) separately for each dataset and model in a following way:

  * •

For BERT model in Figure 19 for AG News dataset, Figure 20 for TREC dataset,
Figure 21 for SNIPS dataset, Figure 22 for DBPedia dataset, Figure 23 for SST2
dataset, Figure 24 for MRPC dataset and Figure 25 for CoLA dataset.

  * •

For RoBERTa model in Figure 26 for AG News dataset, Figure 27 for TREC
dataset, Figure 28 for SNIPS dataset, Figure 29 for DBPedia dataset, Figure 30
for SST2 dataset, Figure 31 for MRPC dataset and Figure 32 for CoLA dataset.

  * •

For ALBERT model in Figure 33 for AG News dataset, Figure 34 for TREC dataset,
Figure 35 for SNIPS dataset, Figure 36 for DBPedia dataset, Figure 37 for SST2
dataset, Figure 38 for MRPC dataset and Figure 39 for CoLA dataset.

For the figures, we omit the failed runs in the visualisations (i.e., runs
that show significantly lower performance for the given strategy or than the
average of all the other strategies, for example achieving F1 score of 1%
instead of 80% on TREC dataset). As such, specific mitigation strategies may
be missing in the figures in some cases – in such case, all the runs using the
mitigation strategies resulted in failed runs, such as Mixout on MRPC or CoLA
dataset with RoBERTa model trained using IA3, or the Default baseline on TREC
dataset with ALBERT model trained using UniPELT.

When it comes to the further models and dataset, we observe only minor
differences in the results compared to the ones reported in the main body of
the paper. We summarise the main findings and potential differences below.

On the different datasets, we observe different benefit for the individual
mitigation strategies. For example, for the BERT model trained using UniPELT,
the SWA mitigation strategy achieves similar performance to the Ensemble
mitigation strategies on the AG News, MRPC or CoLA datasets, while on SNIPS,
DBPedia or TREC it leads to lower benefit. Similar behaviour can be observed
for other methods as well. The data augmentation provides significantly larger
benefit when the number of trainable parameters is small, such as when using
IA3 or the ALBERT model. For example, the Augment N strategy outperforms the
Ensemble strategy in these cases. In addition, this has also the effect on our
proposed method. The DENI method (or DENIALS method) achieves the highest
performance increase and standard deviation reduction across all datasets,
models and fine-tuning approaches, with the exception of RoBERTa and ALBERT
trained using IA3 on CoLA dataset, where the Augment 2 strategy provides the
most benefit. This can be explained by: 1) the benefit of augmented samples
when the number of trainable parameters is small; and 2) we observe that the
augmented samples for CoLA dataset appear to provide larger benefit across all
models and fine-tuning approaches. As such, in these cases the benefit of
adding further samples through augmentation outweighs benefit of regular
mitigation strategies.

In addition, we observe different level of standard deviation for the
different models, especially for the ALBERT model. As such, we also observe a
different benefit of the individual mitigation strategies on these models as
compared to BERT. For example, we observe higher benefit of the Stochastic
Weight Averaging, even outperforming the Ensemble mitigation strategy and all
of our proposed methods when using ALBERT model trained using IA3 on the CoLA
dataset. At the same time, the Ensemble strategy and the DENI method provide
more consistent and more significant reduction in the standard deviation on
the ALBERT and RoBERTa models across the different datasets and fine-tuning
approaches, while the majority of the remaining mitigation strategies often
show significantly lower consistency, often not leading to any mitigation
benefit over the Default baseline. As such, we can better observe the
efficiency and effectiveness of the different mitigation strategies on these
cases – with Ensemble and our DENI method being significantly more effective
and efficient.

The different components of the DENI method provide the same benefit across
all the models, datasets and fine-tuning methods. The Delayed Ensemble (DE)
reduces the standard deviation, achieving deviation on par or often even lower
than the one from Ensemble. However, it also leads to lower performance than
the Ensemble method. Only in specific cases, such as BERT on the MRPC dataset
trained using UniPELT or ALBERT on the CoLA dataset trained using IA3, we
observe similar performance of the Delayed Ensemble to the Ensemble or other
components of the DENI method. The Noisy Interpolation (NI) leads to an
increase in the performance, but often at the cost of higher standard
deviation. When combined in the DENI method, the complementarity of these two
components leads to a significant benefit, increasing the overall performance
while reducing the standard deviation. Finally, combining the DENI method with
augmented labelled samples (to obtain the DENIALS method) provides different
benefit based on the dataset and the fine-tuning method used. In about half
the cases (mainly when using full fine-tuning or UniPELT) the benefit of the
DENI method is higher, while in other (mainly when using LoRA or IA3) the
DENIALS method leads to higher performance and lower standard deviation.
However, taking the cost of the method into consideration, the DENI method can
be viewed as more efficient, as it requires half of the computation cost,
while producing similar increase in performance and reduction in standard
deviation – the difference in the performance and deviation in results between
the DENI and DENIALS methods is not statistically significant (p-value of 0.14
using the Mann-Whitney U test; p-value of 0.82 using the Levene test).

To summarise, the DENI method provides consistent benefit across different
models, datasets and fine-tuning strategies, even in cases when the deviation
in results is significantly higher or even negligible. At the same time, the
other mitigation strategies we compare against (with the exception of
Ensemble) show lower consistency in their benefit. In some cases, using the
mitigation strategies, especially DENIALS or even DENI leads to a performance
that is approaching the one when using all data in the dataset, while also
reducing the standard deviation. Finally, increasing the number of available
labelled samples through augmentation has significantly higher benefit when
the number of trainable parameters is low (using IA3 fine-tuning or ALBERT
model), outperforming all the regular mitigation strategies that modify the
optimisation process in specific cases.

Figure 20: Benefit of mitigation strategies for the different fine-tuning methods using BERT on TREC dataset. The benefit is calculated as difference to the mean performance of the Default baseline when using full fine-tuning. The different mitigation strategies are beneficial for all fine-tuning methods, but with different overall benefit (e.g., Augment on IA3). Figure 21: Benefit of mitigation strategies for the different fine-tuning methods using BERT on SNIPS dataset. The benefit is calculated as difference to the mean performance of the Default baseline when using full fine-tuning. The different mitigation strategies are beneficial for all fine-tuning methods, but with different overall benefit (e.g., Augment on IA3). Figure 22: Benefit of mitigation strategies for the different fine-tuning methods using BERT on DBPedia dataset. The benefit is calculated as difference to the mean performance of the Default baseline when using full fine-tuning. The different mitigation strategies are beneficial for all fine-tuning methods, but with different overall benefit (e.g., Augment on IA3). Figure 23: Benefit of mitigation strategies for the different fine-tuning methods using BERT on SST2 dataset. The benefit is calculated as difference to the mean performance of the Default baseline when using full fine-tuning. The different mitigation strategies are beneficial for all fine-tuning methods, but with different overall benefit (e.g., Augment on IA3). Figure 24: Benefit of mitigation strategies for the different fine-tuning methods using BERT on MRPC dataset. The benefit is calculated as difference to the mean performance of the Default baseline when using full fine-tuning. The different mitigation strategies are beneficial for all fine-tuning methods, but with different overall benefit (e.g., Augment on IA3). Figure 25: Benefit of mitigation strategies for the different fine-tuning methods using BERT on CoLA dataset. The benefit is calculated as difference to the mean performance of the Default baseline when using full fine-tuning. The different mitigation strategies are beneficial for all fine-tuning methods, but with different overall benefit (e.g., Augment on IA3). RoBERTa | AG News | TREC | SNIPS | DBPedia | SST2 | MRPC | CoLA
---|---|---|---|---|---|---|---
Full FineTuning
Default |  85.770.433 |  90.980.979 |  98.420.092 |  98.360.082 |  90.140.270 |  65.551.654 |  77.160.934
All Data |  89.270.391 |  94.810.691 |  99.070.086 |  99.060.028 |  95.150.084 |  73.451.031 |  81.560.632
Best Practices |  86.120.418 |  92.250.733 |  98.360.124 |  98.380.059 |  89.990.288 |  66.431.276 |  77.270.870
Ensemble |  86.560.236 |  92.800.378 |  98.490.073 |  98.530.023 |  90.690.119 |  67.510.703 |  78.730.431
NoiseInput |  86.040.485 |  92.120.645 |  98.370.144 |  98.400.073 |  89.930.297 |  66.131.933 |  77.040.853
NoiseWeights |  86.230.547 |  91.900.954 |  98.290.111 |  98.290.083 |  89.940.280 |  66.561.326 |  76.920.678
SWA |  86.440.450 |  91.811.037 |  98.460.108 |  98.400.044 |  90.150.257 |  66.351.373 |  76.701.357
Mixout |  86.020.471 |  91.870.724 |  98.370.126 |  98.350.059 |  90.130.251 |  65.941.641 |  77.200.760
Augment 1 |  85.910.524 |  91.090.708 |  98.400.104 |  98.520.067 |  90.300.226 |  65.541.723 |  77.250.656
Augment 2 |  85.960.413 |  90.170.770 |  98.420.103 |  98.430.068 |  90.430.246 |  63.491.031 |  76.610.631
OurDE |  86.240.192 |  91.760.335 |  98.420.031 |  98.370.024 |  89.890.138 |  65.820.693 |  77.400.350
OurNI |  88.100.530 |  92.270.972 |  98.860.105 |  98.560.061 |  91.850.375 |  67.881.692 |  79.430.882
OurDENI | 88.670.133 | 93.150.287 | 98.920.027 | 98.660.018 |  92.250.085 | 68.230.483 | 79.690.365
OurDENIALS |  88.260.150 | 93.110.217 | 98.850.027 |  98.530.025 | 92.430.067 |  67.830.530 | 79.310.235
LoRA
Default |  85.730.448 |  89.550.907 |  98.340.112 |  98.160.133 |  89.950.359 |  64.191.592 |  76.711.040
All Data |  89.450.374 |  94.320.686 |  98.880.102 |  99.080.027 |  95.120.075 |  72.950.984 |  81.070.632
Best Practices |  85.870.477 |  91.120.648 |  98.250.146 |  98.250.087 |  90.130.285 |  64.211.711 |  77.250.912
Ensemble |  86.560.273 |  92.500.422 |  98.390.054 |  98.440.037 |  90.770.122 |  65.490.807 |  78.110.636
NoiseInput |  86.040.515 |  91.061.140 |  98.310.151 |  98.300.086 |  90.020.340 |  63.382.061 |  77.000.762
NoiseWeights |  85.820.471 |  91.200.885 |  98.280.103 |  98.230.107 |  90.090.250 |  63.652.281 |  77.010.715
SWA |  86.280.436 |  91.500.624 |  98.370.125 |  98.310.087 |  90.210.498 |  65.242.331 |  77.110.871
Mixout |  85.830.551 |  91.490.865 |  98.310.118 |  98.200.131 |  90.110.285 |  64.602.130 |  77.200.855
Augment 1 |  86.360.506 |  91.190.625 |  98.490.143 |  98.320.115 |  90.600.189 |  64.388.130 |  78.110.921
Augment 2 |  86.130.423 |  90.450.641 |  98.470.157 |  98.260.123 |  90.460.224 |  64.881.702 |  77.900.886
OurDE |  85.610.243 |  91.100.402 |  98.290.045 |  98.210.029 |  90.100.105 |  64.520.804 |  77.260.586
OurNI |  87.550.575 |  92.730.759 |  98.700.139 |  98.510.110 |  91.980.279 |  66.511.856 |  79.170.851
OurDENI |  87.980.219 |  93.010.226 | 98.840.029 | 98.580.027 |  92.440.072 | 66.950.469 | 79.740.397
OurDENIALS | 88.140.175 | 93.250.205 | 98.960.038 | 98.670.034 | 92.700.070 | 67.060.490 | 79.950.882
IA3
Default |  84.980.557 |  90.590.887 |  98.070.156 |  97.890.128 |  89.000.230 |  58.584.528 |  73.882.586
All Data |  88.830.562 |  94.660.560 |  98.930.094 |  98.850.059 |  94.510.122 |  64.838.397 |  73.0313.417
Best Practices |  85.030.503 |  86.2018.439 |  93.3220.601 |  97.690.192 |  88.510.374 |  55.348.258 |  73.047.342
Ensemble |  86.390.296 |  91.490.441 |  98.650.072 |  98.290.063 |  90.110.158 |  57.731.880 |  76.630.542
NoiseInput |  84.810.557 |  90.351.104 |  98.130.199 |  97.630.192 |  88.460.395 |  56.395.678 |  71.4110.636
NoiseWeights |  80.8815.803 |  90.291.211 |  98.160.239 |  97.610.179 |  87.970.409 |  49.5213.034 |  71.597.086
SWA |  86.130.474 |  89.660.951 |  98.290.175 |  95.4710.860 |  88.020.426 |  52.7412.001 |  68.296.409
Mixout |  84.810.507 |  90.261.052 |  98.020.215 |  97.970.119 |  85.2211.347 |  33.227.739 |  33.019.143
Augment 1 |  86.300.660 |  91.480.846 |  98.390.192 |  98.210.146 |  86.8112.490 |  56.494.422 |  75.566.946
Augment 2 |  86.410.543 |  91.190.796 |  98.640.176 |  98.260.129 |  90.210.261 |  57.951.282 | 78.700.790
OurDE |  84.930.275 |  89.830.495 |  98.190.053 |  97.730.058 | 88.550.107 | 58.960.407 | 75.090.396
OurNI |  87.490.673 |  91.650.952 |  98.640.163 |  98.430.196 |  90.330.313 |  58.4310.157 |  74.3111.236
OurDENI |  87.700.225 |  91.850.457 | 98.790.035 | 98.560.037 |  90.820.108 |  62.400.517 |  77.250.559
OurDENIALS | 87.750.207 | 92.000.337 | 98.880.042 | 98.610.042 | 91.020.116 | 62.810.554 |  77.410.546
UniPELT
Default |  85.980.359 |  91.840.870 |  98.360.108 |  98.420.066 |  90.340.304 |  62.437.489 |  76.308.074
All Data |  89.730.353 |  94.310.526 |  99.020.114 |  99.080.039 |  95.070.093 |  73.330.873 |  82.030.407
Best Practices |  86.070.446 |  92.620.777 |  98.350.121 |  98.380.102 |  90.260.252 |  64.921.433 |  78.450.754
Ensemble |  87.180.240 |  93.250.575 |  98.590.076 |  98.550.024 | 91.000.067 |  66.040.656 |  80.260.372
NoiseInput |  86.160.452 |  92.180.809 |  98.420.137 |  98.380.098 |  90.250.206 |  59.9010.928 |  76.0410.489
NoiseWeights |  85.960.623 |  92.410.745 |  98.320.123 |  98.250.114 |  90.310.198 |  65.501.254 |  78.070.758
SWA |  86.640.402 |  92.280.822 |  98.400.133 |  98.420.088 |  88.628.224 |  62.667.641 |  70.0619.817
Mixout |  86.270.550 |  92.231.027 |  98.360.163 |  98.420.079 |  90.220.257 |  64.661.695 |  66.1521.219
Augment 1 |  86.020.432 |  91.881.072 |  98.350.136 |  98.550.073 |  90.420.252 |  60.401.399 |  75.9711.790
Augment 2 |  85.890.459 |  90.880.564 |  98.330.153 |  98.530.089 |  90.180.230 |  59.231.251 |  78.480.800
OurDE |  86.090.235 |  92.390.417 |  98.360.056 |  98.420.026 |  90.270.106 |  65.420.512 |  79.770.445
OurNI |  87.950.493 |  92.991.076 |  98.760.144 |  98.560.110 |  92.170.262 |  65.975.566 |  72.0419.807
OurDENI | 88.560.189 | 93.680.284 |  98.910.043 | 98.680.018 |  92.670.072 | 67.560.275 | 80.680.343
OurDENIALS | 88.560.189 | 93.680.298 | 98.940.037 |  98.670.030 | 92.700.084 |  67.460.396 | 80.550.241
Table 5: Comparison of the DENI method with existing mitigation strategies and baselines on full fine-tuning, LoRA, IA3 and UniPELT using RoBERTa model. The comparison is done in terms of overall performance and deviation. The highest performance for each fine-tuning method is in bold and lowest deviation is underlined (not considering All Data baseline). Figure 26: Benefit of mitigation strategies for the different fine-tuning methods using RoBERTa on AG News dataset. The benefit is calculated as difference to the mean performance of the Default baseline when using full fine-tuning. The different mitigation strategies are beneficial for all fine-tuning methods, but with different overall benefit (e.g., Augment on IA3). Figure 27: Benefit of mitigation strategies for the different fine-tuning methods using RoBERTa on TREC dataset. The benefit is calculated as difference to the mean performance of the Default baseline when using full fine-tuning. The different mitigation strategies are beneficial for all fine-tuning methods, but with different overall benefit (e.g., Augment on IA3). Figure 28: Benefit of mitigation strategies for the different fine-tuning methods using RoBERTa on SNIPS dataset. The benefit is calculated as difference to the mean performance of the Default baseline when using full fine-tuning. The different mitigation strategies are beneficial for all fine-tuning methods, but with different overall benefit (e.g., Augment on IA3). Figure 29: Benefit of mitigation strategies for the different fine-tuning methods using RoBERTa on DBPedia dataset. The benefit is calculated as difference to the mean performance of the Default baseline when using full fine-tuning. The different mitigation strategies are beneficial for all fine-tuning methods, but with different overall benefit (e.g., Augment on IA3). Figure 30: Benefit of mitigation strategies for the different fine-tuning methods using RoBERTa on SST2 dataset. The benefit is calculated as difference to the mean performance of the Default baseline when using full fine-tuning. The different mitigation strategies are beneficial for all fine-tuning methods, but with different overall benefit (e.g., Augment on IA3). Figure 31: Benefit of mitigation strategies for the different fine-tuning methods using RoBERTa on MRPC dataset. The benefit is calculated as difference to the mean performance of the Default baseline when using full fine-tuning. The different mitigation strategies are beneficial for all fine-tuning methods, but with different overall benefit (e.g., Augment on IA3). Figure 32: Benefit of mitigation strategies for the different fine-tuning methods using RoBERTa on CoLA dataset. The benefit is calculated as difference to the mean performance of the Default baseline when using full fine-tuning. The different mitigation strategies are beneficial for all fine-tuning methods, but with different overall benefit (e.g., Augment on IA3). ALBERT | AG News | TREC | SNIPS | DBPedia | SST2 | MRPC | CoLA
---|---|---|---|---|---|---|---
Full FineTuning
Default |  83.110.716 |  88.451.794 |  97.230.245 |  98.230.127 |  85.493.983 |  60.122.438 |  65.419.853
All Data |  87.460.680 |  93.980.912 |  98.620.126 |  99.050.039 |  94.580.189 |  67.077.899 |  78.941.257
Best Practices |  82.950.679 |  89.661.033 |  97.300.222 |  98.210.143 |  86.031.169 |  60.203.691 |  70.221.473
Ensemble |  83.760.347 |  90.670.376 |  97.830.122 | 98.440.036 |  88.200.151 |  66.411.571 |  72.130.664
NoiseInput |  83.070.489 |  88.333.974 |  97.270.269 |  98.130.320 |  84.647.497 |  60.634.068 |  68.932.667
NoiseWeights |  83.090.683 |  89.241.021 |  97.190.349 |  97.820.312 |  85.402.145 |  60.462.962 |  69.541.370
SWA |  83.770.766 |  89.181.447 |  97.490.264 |  98.070.367 |  86.062.042 |  61.053.537 |  67.068.933
Mixout |  82.640.688 |  89.431.237 |  97.370.321 |  98.170.177 |  86.090.901 |  59.624.338 |  67.567.110
Augment 1 |  83.090.522 |  89.661.202 |  97.280.329 |  98.230.205 |  86.461.782 |  60.804.930 |  69.963.846
Augment 2 |  82.470.820 |  88.691.292 |  97.320.283 |  98.180.118 |  87.050.823 |  60.552.952 |  70.363.699
OurDE |  83.020.209 |  89.210.346 |  97.340.079 |  98.090.044 |  86.260.225 |  60.260.919 |  70.170.568
OurNI |  85.500.652 |  90.760.907 |  97.950.667 |  98.460.245 |  88.890.603 |  66.702.416 |  72.841.095
OurDENI | 85.780.156 | 91.000.241 | 98.210.166 | 98.680.055 | 89.490.176 | 67.510.628 | 73.090.441
OurDENIALS |  85.580.169 |  90.860.331 | 97.950.067 |  98.390.052 | 89.370.102 | 67.400.577 | 72.610.437
LoRA
Default |  79.7710.530 |  76.5723.043 |  87.1227.864 |  92.6221.163 |  72.7321.716 |  48.4514.781 |  58.7214.973
All Data |  87.170.451 |  81.7122.575 |  91.1422.510 |  98.930.506 |  87.5617.387 |  62.2812.957 |  59.8618.586
Best Practices |  78.7716.671 |  82.9018.145 |  96.153.883 |  97.890.206 |  75.0218.841 |  61.435.798 |  55.8718.034
Ensemble |  84.360.452 |  90.110.699 |  97.860.133 |  98.440.052 |  86.672.238 |  64.674.205 |  69.3310.165
NoiseInput |  82.630.866 |  76.3625.050 |  87.6628.042 |  97.920.188 |  74.2721.892 |  55.9414.109 |  61.6218.108
NoiseWeights |  81.902.025 |  83.509.570 |  88.9822.796 |  86.6327.099 |  74.2021.399 |  57.7611.426 |  57.3817.570
SWA |  74.0623.224 |  69.6632.682 |  83.4233.489 |  87.8626.928 |  54.7023.038 |  48.2814.489 |  48.0117.929
Mixout |  82.450.905 |  83.4018.324 |  97.000.983 |  96.834.359 |  74.7821.506 |  56.8712.139 |  58.3619.808
Augment 1 |  82.331.315 |  76.7524.220 |  87.8027.988 |  97.951.020 |  72.6822.844 |  55.5911.736 |  58.1517.500
Augment 2 |  82.371.293 |  78.1324.119 |  92.7020.126 |  98.090.163 |  76.1320.011 |  58.949.478 |  64.6213.715
OurDE |  80.470.316 |  88.090.689 |  97.830.123 |  98.240.074 | 86.180.475 |  64.840.956 | 71.671.595
OurNI |  84.780.901 |  88.205.218 |  97.371.502 |  93.4121.096 |  84.3511.181 |  60.7711.306 |  59.1921.022
OurDENI | 85.210.111 | 91.820.560 | 98.450.079 |  98.790.047 |  87.920.774 | 68.870.943 |  72.951.791
OurDENIALS |  84.730.149 |  91.760.598 |  98.020.504 | 98.810.032 | 87.931.055 | 68.760.721 | 73.361.730
IA3
Default |  80.901.296 |  77.8511.902 |  96.340.702 |  95.995.347 |  73.6317.822 |  51.3710.640 |  54.8913.130
All Data |  85.570.631 |  87.2317.164 |  97.971.760 |  98.510.818 |  90.372.450 |  65.533.442 |  63.6013.290
Best Practices |  78.847.546 |  84.225.928 |  96.720.469 |  95.049.257 |  77.7712.933 |  55.464.093 |  50.1411.646
Ensemble |  84.580.441 |  90.110.830 |  97.730.118 |  98.190.064 |  84.991.262 |  56.576.732 |  54.8411.430
NoiseInput |  74.2721.131 |  76.7324.718 |  92.8513.980 |  93.1818.298 |  75.1414.414 |  51.7711.849 |  47.7715.243
NoiseWeights |  81.052.024 |  77.9120.630 |  95.055.857 |  95.664.713 |  75.6011.375 |  54.777.076 |  48.419.886
SWA |  81.376.187 |  85.4411.288 |  93.3315.927 |  96.126.814 |  73.0418.684 |  51.8312.638 |  54.2115.483
Mixout |  78.545.491 |  56.3428.193 |  84.6723.558 |  86.3922.527 |  46.2717.586 |  42.6213.977 |  36.269.978
Augment 1 |  82.761.926 |  82.2513.731 |  95.114.686 |  97.303.050 |  80.7611.785 |  54.169.843 |  64.787.860
Augment 2 |  82.393.087 |  79.2818.981 |  93.0619.200 |  97.930.854 |  81.6910.545 |  58.184.052 | 67.196.209
OurDE |  82.870.209 |  87.880.407 |  97.660.075 |  97.940.097 |  84.840.698 |  59.990.408 | 61.991.657
OurNI |  83.821.657 |  88.992.079 |  97.640.225 |  98.180.366 |  81.5012.484 |  60.404.143 |  57.0815.157
OurDENI | 84.960.143 | 91.020.320 | 97.940.066 | 98.350.051 |  86.880.644 | 62.550.468 |  63.352.045
OurDENIALS | 85.240.163 |  90.990.716 | 98.080.123 | 98.530.275 | 87.270.441 |  62.510.751 |  64.122.093
UniPELT
Default |  74.932.312 |  51.526.672 |  83.827.394 |  83.0516.371 |  80.738.096 |  58.911.770 |  60.376.526
All Data |  84.213.139 |  78.144.317 |  97.820.441 |  99.000.034 |  93.102.212 |  65.221.804 |  74.237.717
Best Practices |  79.660.960 |  69.898.061 |  93.482.481 |  95.521.320 |  84.041.959 |  57.312.411 |  63.677.417
Ensemble |  83.150.387 |  84.491.287 |  96.920.308 |  97.610.161 |  87.170.487 |  60.630.955 |  68.461.455
NoiseInput |  76.928.502 |  66.417.107 |  92.942.395 |  89.5620.237 |  81.4711.838 |  57.852.175 |  65.194.367
NoiseWeights |  80.160.903 |  73.467.650 |  94.691.304 |  92.6414.449 |  82.378.680 |  59.222.550 |  61.5611.011
SWA |  79.991.582 |  74.046.722 |  94.511.184 |  96.051.194 |  81.979.083 |  58.292.459 |  60.2214.255
Mixout |  79.631.743 |  72.865.823 |  94.551.471 |  94.384.587 |  79.0512.674 |  57.851.957 |  63.496.818
Augment 1 |  77.6713.450 |  82.672.436 |  95.550.665 |  97.260.547 |  82.559.967 |  57.881.454 |  68.523.480
Augment 2 |  80.471.248 |  81.602.290 |  95.360.597 |  97.080.884 |  83.696.801 |  57.471.482 |  68.551.417
OurDE |  81.150.217 |  80.471.109 |  96.250.325 |  97.230.096 |  85.810.349 |  59.880.489 |  68.230.794
OurNI |  81.013.406 |  76.865.754 |  95.471.276 |  97.400.670 |  85.383.221 |  59.731.891 |  64.916.295
OurDENI | 83.910.160 |  85.480.914 | 97.460.221 |  97.890.095 | 87.890.335 |  62.320.309 |  72.070.552
OurDENIALS |  83.780.182 | 86.200.529 | 96.740.090 | 98.960.047 |  87.640.564 | 62.790.232 | 72.600.231
Table 6: Comparison of the DENI method with existing mitigation strategies and
baselines on full fine-tuning, LoRA, IA3 and UniPELT using ALBERT model. The
comparison is done in terms of overall performance and deviation. The highest
performance for each fine-tuning method is in bold and lowest deviation is
underlined (not considering All Data baseline). Figure 33: Benefit of
mitigation strategies for the different fine-tuning methods using ALBERT on AG
News dataset. The benefit is calculated as difference to the mean performance
of the Default baseline when using full fine-tuning. The different mitigation
strategies are beneficial for all fine-tuning methods, but with different
overall benefit (e.g., Augment on IA3). Figure 34: Benefit of mitigation
strategies for the different fine-tuning methods using ALBERT on TREC dataset.
The benefit is calculated as difference to the mean performance of the Default
baseline when using full fine-tuning. The different mitigation strategies are
beneficial for all fine-tuning methods, but with different overall benefit
(e.g., Augment on IA3). Figure 35: Benefit of mitigation strategies for the
different fine-tuning methods using ALBERT on SNIPS dataset. The benefit is
calculated as difference to the mean performance of the Default baseline when
using full fine-tuning. The different mitigation strategies are beneficial for
all fine-tuning methods, but with different overall benefit (e.g., Augment on
IA3). Figure 36: Benefit of mitigation strategies for the different fine-
tuning methods using ALBERT on DBPedia dataset. The benefit is calculated as
difference to the mean performance of the Default baseline when using full
fine-tuning. The different mitigation strategies are beneficial for all fine-
tuning methods, but with different overall benefit (e.g., Augment on IA3).
Figure 37: Benefit of mitigation strategies for the different fine-tuning
methods using ALBERT on SST2 dataset. The benefit is calculated as difference
to the mean performance of the Default baseline when using full fine-tuning.
The different mitigation strategies are beneficial for all fine-tuning
methods, but with different overall benefit (e.g., Augment on IA3). Figure 38:
Benefit of mitigation strategies for the different fine-tuning methods using
ALBERT on MRPC dataset. The benefit is calculated as difference to the mean
performance of the Default baseline when using full fine-tuning. The different
mitigation strategies are beneficial for all fine-tuning methods, but with
different overall benefit (e.g., Augment on IA3). Figure 39: Benefit of
mitigation strategies for the different fine-tuning methods using ALBERT on
CoLA dataset. The benefit is calculated as difference to the mean performance
of the Default baseline when using full fine-tuning. The different mitigation
strategies are beneficial for all fine-tuning methods, but with different
overall benefit (e.g., Augment on IA3).