# Parameter-Efficient Language Model Tuning with Active Learning in Low-
Resource Settings

Josip Jukić Jan Šnajder
TakeLab
University of Zagreb, Faculty of Electrical Engineering and Computing
{josip.jukic<EMAIL_ADDRESS>
###### Abstract

Pre-trained language models (PLMs) have ignited a surge in demand for
effective fine-tuning techniques, particularly in low-resource domains and
languages. Active learning (AL), a set of algorithms designed to decrease
labeling costs by minimizing label complexity, has shown promise in
confronting the labeling bottleneck. Concurrently, adapter modules, designed
for parameter-efficient fine-tuning (PEFT), have showcased notable potential
in low-resource settings. However, the interplay between AL and adapter-based
PEFT remains unexplored. In our study, we empirically investigate PEFT
behavior with AL in low-resource settings for text classification tasks. Our
findings affirm the superiority of PEFT over full-fine tuning (FFT) in low-
resource settings and demonstrate that this advantage persists in AL setups.
Finally, we delve into the properties of PEFT and FFT through the lens of
forgetting dynamics and instance-level representations, linking them to AL
instance selection behavior and the stability of PEFT. Our research
underscores the synergistic potential of AL, PEFT, and TAPT in low-resource
settings, paving the way for advancements in efficient and effective fine-
tuning.

##  1 Introduction

Pre-trained language models (PLMs) have rapidly become a staple in the field
of natural language processing. With the growing demand for data for training
these models, developing efficient fine-tuning methods has become critical.
This is particularly relevant for many domains and languages where it is
difficult or downright impossible to obtain large amounts of labeled training
data. In such low-resource settings, it becomes essential to effectively
leverage and adapt PLMs while minimizing the need for extensive labeled data.

Data labeling is notoriously time-consuming and expensive, often hindering the
development of sizable labeled datasets required for training high-performance
models. Active learning (AL) Cohn et al. (1996); Settles (2009) has emerged as
a potential solution to this challenge. In contrast to passive learning, in
which the training set is sampled at random, AL encompasses a unique family of
machine learning algorithms specifically designed to reduce labeling costs by
reducing label complexity, i.e., the number of labels required by an
acquisition model to achieve a certain level of performance Dasgupta (2011).
With the advent of PLMs, AL research has pivoted towards investigating
training regimes for PLMs, such as task-adaptive pre-training (TAPT;
Gururangan et al., 2020), that could be combined with AL and further reduce
label complexity.

While AL explicitly aims to minimize the label complexity of learning, it is
also important to consider reducing the parameter complexity of the
acquisition model. Effectively lowering parameter complexity can lead to a
corresponding reduction in label complexity, further improving training
efficiency. As PLMs grow larger, fine-tuning becomes increasingly challenging
due to the sheer number of parameters involved. To address this issue,
adapters Houlsby et al. (2019) have been recently introduced as compact
modules that can be incorporated between the layers of PLMs. These adapters
enable a considerable degree of parameter-sharing, thereby fostering the
practice of parameter-efficient fine-tuning (PEFT) through modular learning
Pfeiffer et al. (2023). During the tuning process for a downstream task, only
the parameters of the adapters are updated, resulting in a significantly
reduced number of parameters. Recent research He et al. (2021); Li and Liang
(2021); Karimi Mahabadi et al. (2021) has revealed that some PEFT methods
outperform full fine-tuning (FFT) in low-resource settings, potentially due to
better stability and a decreased risk of overfitting. In contrast, FFT has
been shown to exhibit instability in scenarios with limited data, whereas PEFT
methods have proven to be well-suited for such circumstances.

Despite the promising results demonstrated by PEFT methods in low-resource
settings, a notable gap remains in the research landscape. Given that the
majority of real-world AL scenarios involve a restricted amount of data, PEFT
methods emerge as strong candidates as acquisition models for AL. However,
there has been no exploration of AL in conjunction with adapters.
Investigating this uncharted territory can further advance our understanding
of AL and reveal novel strategies for optimizing performance in low-resource
settings.

In this paper, we present an empirical study on the behavior of PEFT in low-
resource settings for text classification tasks. We analyze PEFT with and
without AL and compare it against FFT. While our results confirm that PEFT
exhibits superior performance in low-resource setups compared to FFT, we show
that the improved performance with PEFT extends to AL scenarios in terms of
performance gains over passive learning. Furthermore, we analyze the efficacy
of TAPT in conjunction with AL and PEFT. We find that TAPT is beneficial in AL
scenarios for both PEFT and fully fine-tuned models, thus representing a
viable technique for improving performance in low-resource scenarios. Finally,
aiming to illuminate why PEFT and TAPT improve AL performance in low-resource
settings, we analyze the properties of PEFT and FFT via forgetting dynamics
Toneva et al. (2019) and PLMs’ instance-level representations. We find that AL
methods choose fewer unforgettable and more moderately forgettable examples
when combined with PEFT and TAPT, where forgetfulness indicates the model’s
tendency to learn and forget the gold label of a particular instance. Compared
to FFT, we observe that PEFT yields early and middle layer representations
more similar to the base model, which might suggest increased stability and
targeted regularization during training in the low-resource setup. We
hypothesize that the stability of early and middle layers mitigates the issue
of forgetting the knowledge obtained during pre-training when fine-tuning for
downstream tasks.

In summary, we showed that (1) PEFT yields greater improvements in performance
and increased stability compared to FFT in AL low-resource scenarios for text
classification and (2) TAPT enhances the overall text classification
performance of adapters and is well-suited for AL scenarios. We also found
that (3) AL methods choose fewer unforgettable and more moderately forgettable
examples with PEFT and TAPT and (4) PEFT produces instance-level
representations of early and middle layers that are more similar to the base
PLM than FFT. Our results uncover the intricacies of positive interactions
between AL, PEFT, and TAPT, providing empirical justification for their
combined use in low-resource settings.

##  2 Related Work

Our study encompasses research efforts that marry AL with PLMs and explore the
application of PEFT techniques in low-resource contexts.

#### AL with PLMs.

Until recently, the conventional approach for integrating PLMs with AL
involved performing full fine-tuning with a fixed number of training epochs
and training the model from scratch in each AL step Ein-Dor et al. (2020);
Margatina et al. (2021); Shelmanov et al. (2021); Karamcheti et al. (2021);
Schröder et al. (2022). However, studies by Mosbach et al. (2021) and Zhang et
al. (2021) revealed that fine-tuning in low-resource setups is prone to
instability, particularly when training for only a few epochs. This
instability, often sensitive to weight initialization and data ordering Dodge
et al. (2020), presents a significant challenge for AL, which frequently
operates in low-resource settings. Recent research has looked into the impact
of PLM training regimes on AL performance Grießhaber et al. (2020); Yuan et
al. (2020); Yu et al. (2022), suggesting that the choice of training regime is
more critical than the choice of the AL method. Notably, TAPT has proven
particularly effective in enhancing AL performance Margatina et al. (2022);
Jukić and Šnajder (2022).

#### Adapters in low-resource settings.

Research on adapters in low-resource settings has primarily focused on areas
such as cross-lingual transfer for low-resource languages Ansell et al.
(2021); Lee et al. (2022); Parović et al. (2022), where the emphasis lies on
exploring diverse methods of fusing adapters. In monolingual settings with
scarce data, adapters have been found to outperform full fine-tuning Li and
Liang (2021); Mao et al. (2022). A study by He et al. (2021) demonstrated that
adapter-based tuning exhibits enhanced stability and generalization
capabilities by virtue of being less sensitive to learning rates than
traditional fine-tuning methods. Furthermore, incorporating task adaptation
techniques, such as TAPT, has been shown to improve performance in limited-
data scenarios. However, it appears that TAPT yields diminishing returns when
sufficient data is available Kim et al. (2021).

####

While adapters have shown promise in low-resource settings, their application
in the context of AL, where such conditions are common, remains unexplored. We
investigate the performance of adapters in AL scenarios, aiming to expand our
understanding of why they exhibit favorable properties in low-resource
settings.

##  3 Preliminaries

In this section, we outline our experimental setup, providing details on the
datasets we employed, as well as the PEFT and AL methods used in our study.

###  3.1 Datasets

We employ four single-text classification tasks commonly used for AL
evaluation. These include:

  * •

The Subjectivity dataset (subj; Pang and Lee, 2004), designed to assess the
subjectivity of a given text;

  * •

The Question Type classification dataset (trec; Li and Roth, 2002), designed
for categorizing questions according to their types;

  * •

The Stanford Sentiment Treebank (sst; Socher et al., 2013), which focuses on
sentiment analysis;

  * •

AG’s News classification dataset (agn; Zhang et al., 2015), which classifies
news articles into different categories.

We provide the dataset statistics in the appendix for further reference (cf.
Appendix Table 3).

###  3.2 PEFT methods

In our experiments, we explore four distinct PEFT techniques that introduce
additional trainable parameters, covering the main approaches to PEFT.

#### Adapter

incorporates trainable bottleneck layers after both the multi-head attention
and feed-forward block in each Transformer layer Houlsby et al. (2019).

#### Prefix-tuning

adds new parameters in the multi-head attention blocks within each Transformer
layer Li and Liang (2021).

#### LoRA

(Low-rank adaptation) represents an additive method that incorporates
trainable low-rank decomposition matrices into the layers of a pre-trained
model Hu et al. (2022).

#### UniPELT

combines multiple PEFT approaches, namely LoRA, Prefix-tuning, and Adapter, in
a single unified setup Mao et al. (2022). Each constituent is a submodule, and
UniPELT employs gating mechanisms to activate them effectively.

All the mentioned PEFT methods fall under the category of lightweight fine-
tuning. While prefix-tuning does not technically qualify as an adapter, (He et
al., 2022) demonstrated that it shares formal similarities with adapters, with
prefix-tuning performing weighted addition and an adapter employing unweighted
addition. We refer to all four considered PEFT methods as adapters for
terminological simplicity. We use BERT Devlin et al. (2018) as the base model
for every adapter.

###  3.3 Active learning methods

Our study employs five sampling strategies, including random selection as a
passive learning baseline. The other four strategies are active learning
methods originating from different families. We have deliberately chosen these
methods for their robustness (ability to perform well across various tasks)
and widespread usage in the field.

#### Random selection

(rnd) uniformly selects instances from the unlabeled pool.

#### Maximum entropy

(ent; Lewis and Gale, 1994) comes from the family of uncertainty strategies.
The method queries instances where the model is least certain based on the
maximum entropy criterion of the prediction output.

#### Monte Carlo dropout

(mc; Gal and Ghahramani, 2016) resembles ent but utilizes the stochasticity of
forward passes with dropout layers Srivastava et al. (2014) to estimate the
entropy for a given instance. In experiments, we use ten inference cycles to
approximate the entropy of the output via Monte-Carlo dropout sampling.

#### Core-set

(cs; Sener and Savarese, 2018) encourages instance diversity by using the
learned representations of the acquisition model. This method aims to minimize
the distance between an example in the unlabeled set and its closest
counterpart in the labeled subset. We use the [cls] token representation from
the Transformer’s penultimate layer. We follow the greedy method described in
the original work.

#### Discriminative active learning

(dal; Gissin and Shalev-Shwartz, 2019) frames active learning as a
classification of whether a particular instance is labeled or not to make the
labeled and unlabeled sets indistinguishable. Specifically, dal queries
instances that a trained classifier is most likely to consider as part of the
unlabeled subset.

###  3.4 Experimental setup

In AL runs, we selected $50$ new examples in each step of each AL experiment,
using $100$ examples for the warm start (randomly sampled labeled data to
initiate the model). To probe different PEFT approaches with and without AL in
low-resource settings, we established a labeling budget limit of $1,000$
instances.

To circumvent the need for a validation set in our experiments, which is
frequently unavailable in real-world AL scenarios, we adopted the Besov early
stopping Jukić and Šnajder (2022). This approach leverages the smoothness of
Transformer layers to determine the appropriate epoch at which to halt
training.

In the case of TAPT, we pre-trained the base model on a masked language
modeling task using unlabeled training data. For adapters, we only updated the
injected parameters while keeping the remaining parameters of the base model
frozen. This approach aligns with the primary function of adapters, which is
to utilize a common base model across diverse tasks. In the implementation of
TAPT, we randomly masked $15\%$ of tokens for both FFT models and adapters and
trained the model via self-supervision for $50$ epochs with the learning rate
set to $10^{-5}$. Other details of training and hyperparameters details are
given in Section A.3.

For every setting, we performed five runs with different random seeds. We
report the average $F_{1}$ score at each sampling step (with and without AL
for FFT and PEFT) to show the corresponding learning curve averaged over five
runs with standard error denoting the confidence intervals.

###  3.5 Evaluation

In order to gauge the overall impact of an AL method, we resort to the area
under the performance curve (AUC). In each individual AL step with a specific
quantity of labeled examples, we measure the classification performance in
terms of the $F_{1}$ score, which is subsequently used in computing AUC. We
advocate for using AUC alongside the AL curves, as it serves as a suitable
approximation of AL feasibility through a summary numeric score, which is in
line with recent AL literature Schröder et al. (2022); Jukić and Šnajder
(2022). As our experiments involve different training regimes, we compare each
AL sampling strategy $S_{\mathrm{AL}}$ to passive learning $S_{\mathrm{PL}}$
within the same training regime to isolate the effects of AL. The primary
objective of AL is to improve the label efficiency of passive learning. To
test whether AL is successful, we compare it against random sampling as the
baseline, where we compute the difference between the AUC of a specific AL
method and the corresponding passive learning curve. We normalize this
difference with respect to AUC$(S_{\mathrm{PL}})$ in order to evaluate the
relative improvement over passive learning (RIPL) and make the metric more
comparable across different datasets. We define this metric as follows:

$\text{RIPL}(S_{\mathrm{AL}},S_{\mathrm{PL}})=\frac{\text{AUC}(S_{\mathrm{AL}})-\text{AUC}(S_{\mathrm{PL}})}{1-\text{AUC}(S_{\mathrm{PL}})}$

Intuitively, the RIPL metric provides an estimate of the proportion of maximum
possible improvement achievable by a given AL method in comparison to the
passive learning baseline. A score of $1$ signifies the maximum theoretical
improvement, which would be tantamount to attaining an $F_{1}$ score of $1$ in
the initial sampling step and sustaining that score through the final step.
Conversely, a negative score indicates that the AL method performs worse than
passive learning.

##  4 Experiments

In this section, we first examine the performance of PEFT methods in
comparison to FFT within passive learning scenarios. Subsequently, we conduct
a battery of experiments in AL settings to explore their efficacy further.

###  4.1 PEFT vs. FFT

Previous research Li and Liang (2021); Mao et al. (2022); He et al. (2021),
that investigated the use of adapters in low-resource settings demonstrated
that these adapters perform at least as well as, and sometimes even better
than FFT. However, these conclusions were primarily derived from comparing FFT
to a single adapter variant on a full dataset or from evaluating the
performance of adapters at only a few discrete points.

In the first part of our experiments, we build upon these findings by
conducting a more nuanced analysis. We generate detailed learning curves that
facilitate the comparison of multiple adapters with FFT across a range of
conditions. Our comparison, summarized by the AUC metric in Table 1, reveals
that UniPELT and Prefix-tuning consistently outshine FFT across all datasets
used in our study. Conversely, the performance of Adapter and LoRA is mostly
comparable to FFT, although there are instances where they either outperform
or underperform FFT. Figure 1 shows the corresponding learning curves under
the passive learning setup, where it is possible to dissect the performance
dynamics as the training set increases. The performance disparities between
adapters and FFT become more noticeable under conditions of extreme data
scarcity, particularly when dealing with only $100$–$300$ labeled instances.
Notably, the greatest differences in performance appear at the initial step,
with only $100$ labels, which is consistent across all datasets. These
properties highlight the potential of adapter-based methods in effectively
handling low-resource scenarios, with Prefix-tuning and UniPELT emerging as
the more successful approaches compared to Adapter and LoRA.

|  | subj | trec | sst | agn
---|---|---|---|---|---
adapters | Adapter | $.926$ | $.804$ | $.800$ | $.871$
LoRA | $.929$ | $.750$ | $.798$ | $.860$
Prefix-tuning | .936 | $.847$ | $.847$ | $.875$
UniPELT | $.934$ | .877 | .836 | .875
| FFT | $.928$ | $.810$ | $.787$ | $.860$
Table 1: Comparison of adapters with FFT in passive learning setup. We report
the AUC metric (based on $F_{1}$ score) averaged over five runs. Numbers in
bold denote the best-performing approach for a particular dataset.

(a) subj

(b) trec

(c) sst

(d) agn

Figure 1: Learning curves for passive learning with different PEFT methods and
FFT. The results are averaged over five runs. The confidence intervals denote
the standard deviation. Best viewed on a computer screen.

###  4.2 AL with PEFT

|  | regular | TAPT
---|---|---|---
|  | ent | mc | cs | dal | ent | mc | cs | dal
subj | Adapter | $.112$ | $.102$ | $.100$ | $.092$ | $.137$ | $.151$ | $.111$ | $.067$
LoRA | $.127$ | $.115$ | $.091$ | $.081$ | $.165$ | $.160$ | $.122$ | $.100$
Prefix-tuning | $.095$ | $.110$ | $.106$ | $.111$ | $.186$ | $.181$ | $.170$ | $.151$
UniPELT | $.129$ | $.153$ | $.131$ | $.128$ | $.167$ | $.159$ | $.163$ | $.157$
FFT | $.050$ | $.059$ | $.061$ | $.077$ | $.140$ | $.140$ | $.142$ | $.126$
trec | Adapter | $.027$ | $.069$ | $0.137$ | $.084$ | $.124$ | $.146$ | $.079$ | $.154$
LoRA | $.098$ | $.065$ | $.408$ | $.007$ | $.254$ | $.237$ | $.343$ | $.074$
Prefix-tuning | $.093$ | $.105$ | $.068$ | $.093$ | $.246$ | $.227$ | $.205$ | $.241$
UniPELT | $.138$ | $.165$ | $.082$ | $.200$ | $.302$ | $.236$ | $.276$ | $.334$
fft | $.011$ | $.022$ | $0.038$ | $.034$ | $.180$ | $.162$ | $.131$ | $.121$
sst | Adapter | $.015$ | $.048$ | $.025$ | $.002$ | $.005$ | $.035$ | $.034$ | $.028$
LoRA | $.001$ | $.007$ | $.064$ | $.031$ | $.036$ | $.022$ | $.328$ | $.014$
Prefix-tuning | $.049$ | $.060$ | $.114$ | $.031$ | $.152$ | $.143$ | $.005$ | $.126$
UniPELT | $.037$ | $.043$ | $.040$ | $.008$ | $.082$ | $.101$ | $.083$ | $.080$
fft | $.002$ | $.011$ | $-.039$ | $.004$ | $.080$ | $.079$ | $-.012$ | $.070$
agn-4 | Adapter | $.074$ | $.046$ | $.015$ | $.062$ | $.115$ | $.089$ | $.077$ | $.080$
LoRA | $.020$ | $0.025$ | $0.067$ | $.016$ | $.028$ | $.102$ | $.071$ | $.023$
Prefix-tuning | $.054$ | $.023$ | $.040$ | $.033$ | $.035$ | $.143$ | $.098$ | $.092$
UniPELT | $.074$ | $.096$ | $.089$ | $.095$ | $.203$ | $.185$ | $.112$ | $.081$
fft | $.014$ | $.032$ | $.007$ | $.092$ | $.114$ | $.021$ | $.089$ | $.017$
Table 2: Improvement over passive learning in terms of the RIPL metric. We
present the results for all combinations of the adapters and datasets utilized
in our study. In addition, we compare the outcomes of the regular approach
with those achieved when we apply TAPT to the pre-trained base model. Positive
values indicate improvement over passive learning, while negative values
represent situations where performance drops compared to passive learning.

Our initial findings using PEFT in low-resource settings further support the
notion of employing PEFT in AL scenarios. We conducted evaluations of
individual PEFT methods in AL scenarios with and without the application of
TAPT. As shown in Table 2, the performance of PEFT methods is presented in
terms of gains over random sampling (passive learning). We observe that
without applying TAPT, PEFT methods demonstrate greater gains over FFT. When
TAPT is implemented, FFT becomes comparable to PEFT, although Prefix-tuning
and UniPELT still yield the most significant improvements, depending on the
specific dataset and AL method employed.

We further investigate the behavior of AL methods and their performance
throughout individual steps, where we compare adapters against FFT models.
Figure 2 shows the learning curves for corresponding adapter models with and
without applying TAPT. We show the learning curves for the subj dataset to
avoid clutter, as we noticed similar trends on other datasets we used. Prefix-
tuning and UniPELT seem to be invariant of a particular AL method, showing
better overall performance compared to Adapter and LoRA. These latter
adapters, however, show superior label efficiency with ent and mc, whereas cs
and dal yield modest improvements but still outdo passive learning. These
results are particularly pronounced when TAPT is applied to Adapter and LoRA.
TAPT typically boosts performance throughout the AL steps, though it appears
that cs and dal glean less benefit from it than the entropy-based methods we
used.

Figure 2: Learning curves for AL compared with random sampling. We report the
results for subj. The first row features learning curves for adapters without
the application of TAPT, while the second row highlights their corresponding
curves with TAPT implemented. The final row compares FFT, both with and
without TAPT. The results are averaged over five iterations, and the
confidence intervals represent the standard deviation. For the best viewing
experience, we recommend observing these figures on a computer screen.

##  5 Analysis

(a) subj; rnd

(b) subj; mc

(c) subj; mc \+ TAPT

(d) trec; rnd

(e) trec; mc

(f) trec; mc \+ TAPT

Figure 3: Forget profiles for random sampling and the mc without and with
TAPT. The x-axis shows the number of instances in each of the forgetting
categories: the never category (corresponds to unforgettable instances),
moderately forgettable instances, and highly forgettable instances. We chose
subj and trec as the representatives of a simple binary task and a harder
multi-class classification, respectively.

(a) ent

(b) mc

(c) cs

(d) dal

(e) rnd

Figure 4: Layerwise difference in representation similarity for the UniPELT
adapter and the FFT model on subj. We observed similar trends in other
adapters and datasets we used. The differences are computed as
$CKA(\textit{adapter},\textit{base})-CKA(\textit{FFT},\textit{base})$, where
base is the corresponding pre-trained bert model. Warm colors (positive
values) illustrate layer pairs that demonstrate higher similarity to the base
model with the adapter than with FFT. Conversely, cool colors (negative
values) represent layer pairs that are more similar to the base model when
using the FFT model. Best viewed on a computer screen.

We now look into the properties of adapters and FFT models to examine their
behavior with AL. First, we analyze the influence of TAPT on the forgetting
dynamics during training. We continue with example-level representation
analysis, where we investigate the representation similarity of PEFT and FFT
to their respective base models.

###  5.1 Forgetting dynamics

To understand forgetting dynamics, we draw upon the study by Toneva et al.
(2019), focusing on the occurrence of forgetting events \- cases where a
specific training example transitions from correct to incorrect classification
over the course of multiple learning epochs. We delve into forgetting dynamics
by dividing instances into three categories: (1) unforgettable instances,
i.e., the ones that have never experienced a forgetting event during training,
(2) instances that have witnessed one or two forgetting events, labeled as
moderately forgettable, and (3) instances subjected to three or more
forgetting events, referred to as highly forgettable instances.

Figure 3 shows the frequency of instances across the three forgetting
categories. Particularly, we contrast rnd with mc—an AL method which
consistently enhances performance across all datasets. We find that FFT
selects a larger number of unforgettable instances compared to adapters, while
it picks fewer moderately forgettable instances. Intriguingly, when TAPT is
introduced, these disparities in forgetting profiles between FFT and the other
two adapters, Prefix-tuning and UniPELT, seem to vanish. Conversely, TAPT
appears to intensify the differences between FFT and the remaining two
adapters, LoRA and Adapter, which generally exhibit smaller improvements than
Prefix-tuning and UniPELT. We speculate that LoRA and Adapter might not reap
as much benefit from TAPT, thus they fail to achieve an improved forgetting
profile that, ideally, consists of fewer unforgettable instances and a greater
number of moderately forgettable ones, as compared to the profiles achieved
with FFT without TAPT.

###  5.2 Representation analysis

To analyze the effect of PEFT and FFT on AL selection, we utilized centered
kernel alignment (CKA) as a similarity measure between two sets of
representations Kornblith et al. (2019). It has been shown previously that
PEFT methods result in representations closer to the base model at the token-
level He et al. (2021). We extend the analysis to example-level representation
to explore the behavior of models with AL. CKA is designed to be invariant
invertible linear transformation, which is important for measuring meaningful
similarities between representations of higher dimensions than the number of
data points.

In Figure 4, we compare the similarity between the base model and the adapter
model and between the base model and the fully fine-tuned model. We find that
PEFT representations are more similar to the base model in the early and
middle layers when using a specific AL method. Specifically, up to the eighth
layer, representations are greatly more similar in adapters than in FFT
models. In the final four layers, the difference in CKA scores between the
adapter and FFT model is close to zero. Interestingly, the penultimate,
$11$-th layer is more similar in the FFT model with respect to the base model.

##  6 Conclusion

Our study has shed light on the advantages of parameter-efficient fine-tuning
(PEFT) in low-resource settings, confirming its superiority over full fine-
tuning (FFT) methods. Importantly, we have demonstrated that the integration
of PEFT with active learning (AL) can offer substantial performance gains
compared to passive learning strategies, even in settings where labeled data
is scarce. Furthermore, we highlighted the potential of task-adaptive pre-
training (TAPT) to improve model performance further when used in conjunction
with both PEFT and AL. We found that AL methods in combination with PEFT and
TAPT tend to select fewer unforgettable instances and more moderately
forgettable examples. We further found that PEFT maintains the integrity of
early and middle layer representations similar to the base model, thereby
preventing forgetting during downstream task fine-tuning. These insights
inform us of a possible underpinning mechanism that contributes to PEFT’s
superior performance and stability in low-resource settings. Overall, our work
highlights the potential of PEFT and AL and establishes a foundation for
developing increasingly efficient and cost-effective approaches for training
models in low-resource settings.

## References

  * Ansell et al. (2021) Alan Ansell, Edoardo Maria Ponti, Jonas Pfeiffer, Sebastian Ruder, Goran Glavaš, Ivan Vulić, and Anna Korhonen. 2021.  MAD-G: Multilingual adapter generation for efficient cross-lingual transfer.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 4762–4781, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Cohn et al. (1996) David A Cohn, Zoubin Ghahramani, and Michael I Jordan. 1996.  Active learning with statistical models.  _Journal of artificial intelligence research_ , 4:129–145.
  * Dasgupta (2011) Sanjoy Dasgupta. 2011.  Two faces of active learning.  _Theoretical Computer Science_ , 412(19):1767–1781.  Algorithmic Learning Theory (ALT 2009).
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  BERT: pre-training of deep bidirectional transformers for language understanding.  _CoRR_ , abs/1810.04805.
  * Dodge et al. (2020) Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah A. Smith. 2020.  Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping.  _CoRR_ , abs/2002.06305.
  * Ein-Dor et al. (2020) Liat Ein-Dor, Alon Halfon, Ariel Gera, Eyal Shnarch, Lena Dankin, Leshem Choshen, Marina Danilevsky, Ranit Aharonov, Yoav Katz, and Noam Slonim. 2020.  Active Learning for BERT: An Empirical Study.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7949–7962, Online. Association for Computational Linguistics.
  * Gal and Ghahramani (2016) Yarin Gal and Zoubin Ghahramani. 2016.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In _Proceedings of The 33rd International Conference on Machine Learning_ , volume 48 of _Proceedings of Machine Learning Research_ , pages 1050–1059, New York, New York, USA. PMLR.
  * Gissin and Shalev-Shwartz (2019) Daniel Gissin and Shai Shalev-Shwartz. 2019.  Discriminative active learning.  _CoRR_ , abs/1907.06347.
  * Grießhaber et al. (2020) Daniel Grießhaber, Johannes Maucher, and Ngoc Thang Vu. 2020.  Fine-tuning BERT for low-resource natural language understanding via active learning.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 1158–1171, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Gururangan et al. (2020) Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020.  Don’t stop pretraining: Adapt language models to domains and tasks.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8342–8360, Online. Association for Computational Linguistics.
  * He et al. (2022) Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2022.  Towards a unified view of parameter-efficient transfer learning.  In _International Conference on Learning Representations_.
  * He et al. (2021) Ruidan He, Linlin Liu, Hai Ye, Qingyu Tan, Bosheng Ding, Liying Cheng, Jiawei Low, Lidong Bing, and Luo Si. 2021.  On the effectiveness of adapter-based tuning for pretrained language model adaptation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 2208–2222, Online. Association for Computational Linguistics.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.  Parameter-efficient transfer learning for NLP.  In _International Conference on Machine Learning_ , pages 2790–2799. PMLR.
  * Hu et al. (2022) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022.  LoRA: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_.
  * Jukić and Šnajder (2022) Josip Jukić and Jan Šnajder. 2022.  Smooth sailing: Improving active learning for pre-trained language models with representation smoothness analysis.
  * Karamcheti et al. (2021) Siddharth Karamcheti, Ranjay Krishna, Li Fei-Fei, and Christopher Manning. 2021\.  Mind your outliers! investigating the negative impact of outliers on active learning for visual question answering.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 7265–7281, Online. Association for Computational Linguistics.
  * Karimi Mahabadi et al. (2021) Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. 2021\.  Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 565–576, Online. Association for Computational Linguistics.
  * Kim et al. (2021) Seungwon Kim, Alex Shum, Nathan Susanj, and Jonathan Hilgart. 2021.  Revisiting pretraining with adapters.  In _Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021)_ , pages 90–99, Online. Association for Computational Linguistics.
  * Kornblith et al. (2019) Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. 2019.  Similarity of neural network representations revisited.  In _International Conference on Machine Learning_ , pages 3519–3529. PMLR.
  * Lee et al. (2022) Jaeseong Lee, Seung-won Hwang, and Taesup Kim. 2022.  FAD-X: Fusing adapters for cross-lingual transfer to low-resource languages.  In _Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 57–64, Online only. Association for Computational Linguistics.
  * Lewis and Gale (1994) David D Lewis and William A Gale. 1994.  A sequential algorithm for training text classifiers.  In _SIGIR’94_ , pages 3–12. Springer.
  * Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021.  Prefix-tuning: Optimizing continuous prompts for generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4582–4597, Online. Association for Computational Linguistics.
  * Li and Roth (2002) Xin Li and Dan Roth. 2002.  Learning question classifiers.  In _COLING 2002: The 19th International Conference on Computational Linguistics_.
  * Mao et al. (2022) Yuning Mao, Lambert Mathias, Rui Hou, Amjad Almahairi, Hao Ma, Jiawei Han, Scott Yih, and Madian Khabsa. 2022.  UniPELT: A unified framework for parameter-efficient language model tuning.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 6253–6264, Dublin, Ireland. Association for Computational Linguistics.
  * Margatina et al. (2022) Katerina Margatina, Loic Barrault, and Nikolaos Aletras. 2022.  On the importance of effectively adapting pretrained language models for active learning.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 825–836, Dublin, Ireland. Association for Computational Linguistics.
  * Margatina et al. (2021) Katerina Margatina, Giorgos Vernikos, Loïc Barrault, and Nikolaos Aletras. 2021\.  Active learning by acquiring contrastive examples.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 650–663, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Mosbach et al. (2021) Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. 2021.  On the stability of fine-tuning BERT: Misconceptions, explanations, and strong baselines.  In _International Conference on Learning Representations_.
  * Pang and Lee (2004) Bo Pang and Lillian Lee. 2004.  A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts.  In _Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04)_ , pages 271–278, Barcelona, Spain.
  * Parović et al. (2022) Marinela Parović, Goran Glavaš, Ivan Vulić, and Anna Korhonen. 2022\.  BAD-X: Bilingual adapters improve zero-shot cross-lingual transfer.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1791–1799, Seattle, United States. Association for Computational Linguistics.
  * Pfeiffer et al. (2023) Jonas Pfeiffer, Sebastian Ruder, Ivan Vulić, and Edoardo Maria Ponti. 2023.  Modular deep learning.
  * Schröder et al. (2022) Christopher Schröder, Andreas Niekler, and Martin Potthast. 2022.  Revisiting uncertainty-based query strategies for active learning with transformers.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 2194–2203, Dublin, Ireland. Association for Computational Linguistics.
  * Sener and Savarese (2018) Ozan Sener and Silvio Savarese. 2018.  Active learning for convolutional neural networks: A core-set approach.  In _International Conference on Learning Representations_.
  * Settles (2009) Burr Settles. 2009.  Active learning literature survey.  Computer sciences technical report.
  * Shelmanov et al. (2021) Artem Shelmanov, Dmitri Puzyrev, Lyubov Kupriyanova, Denis Belyakov, Daniil Larionov, Nikita Khromov, Olga Kozlova, Ekaterina Artemova, Dmitry V. Dylov, and Alexander Panchenko. 2021.  Active learning for sequence tagging with deep pre-trained models and Bayesian uncertainty estimates.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 1698–1712, Online. Association for Computational Linguistics.
  * Socher et al. (2013) Richard Socher, John Bauer, Christopher D. Manning, and Andrew Y. Ng. 2013.  Parsing with compositional vector grammars.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 455–465, Sofia, Bulgaria. Association for Computational Linguistics.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014.  Dropout: A simple way to prevent neural networks from overfitting.  _Journal of Machine Learning Research_ , 15(56):1929–1958.
  * Toneva et al. (2019) Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J. Gordon. 2019.  An empirical study of example forgetting during deep neural network learning.  In _International Conference on Learning Representations_.
  * Yu et al. (2022) Yue Yu, Lingkai Kong, Jieyu Zhang, Rongzhi Zhang, and Chao Zhang. 2022.  AcTune: Uncertainty-based active self-training for active fine-tuning of pretrained language models.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1422–1436, Seattle, United States. Association for Computational Linguistics.
  * Yuan et al. (2020) Michelle Yuan, Hsuan-Tien Lin, and Jordan Boyd-Graber. 2020.  Cold-start active learning through self-supervised language modeling.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7935–7948, Online. Association for Computational Linguistics.
  * Zhang et al. (2021) Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Weinberger, and Yoav Artzi. 2021\.  Revisiting few-sample {bert} fine-tuning.  In _International Conference on Learning Representations_.
  * Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015.  Character-level convolutional networks for text classification.  _Advances in neural information processing systems_ , 28.

##  Appendix A Reproducibility

###  A.1 Dataset statistics

The sizes of the datasets per split are provided in Table 3. Predominantly,
the datasets encompass texts in English.

| train | val | test | total
---|---|---|---|---
subj | $7,000$ | $1,000$ | $2,000$ | $10,000$
sst | $6,647$ | $868$ | $1,425$ | $8,940$
trec | $4,881$ | $452$ | $500$ | $5,833$
agn | $20,000$ | $7,600$ | $7,600$ | $35,200$
Table 3: Dataset sizes by splits. Although we do not use a validation set
(val) in our experiments, we report its size for completeness. For the agn
dataset, we performed uniform subsampling to ensure computational feasibility
of the experiments.

###  A.2 Models

We used base and uncased variants of Transformer models in our experiments.
Specifically, we used the “bert-base-uncased” version of bert, which has
$109,514,298$ trainable parameters.

###  A.3 Hyperparameters

We used a fixed learning rate of $2\times 10^{-5}$ for FFT and $10^{-4}$ for
adapters. Additionally, we set the gradient clipping to $1$ during training.

###  A.4 Computing infrastructure

We conducted our experiments on $4\times$ AMD Ryzen Threadripper 3970X 32-Core
Processors and $4\times$ NVIDIA GeForce RTX 3090 GPUs with $24$GB of RAM. We
used PyTorch version $1.9.0$ and CUDA $11.4$.