# Arbitrary Few Parameters are Good Enough for Adapting Large-scale Pre-
trained Language Models

Yusheng Su1 , Chi-Min Chan1∗, Yujia Qin1, Yankai Lin2,
Shengding Hu1, Zonghan Yang1, Ning Ding1, Zhiyuan Liu1, Maosong Sun1
1Department of Computer Science and Technology, Tsinghua University
2Gaoling School of Artificial Intelligence, Renmin University
<EMAIL_ADDRESS>
The first two authors contributed equally.

###### Abstract

Parameter-efficient tuning (PET) methods can effectively drive extremely large
pre-trained language models (PLMs) by only training minimal parameters.
Different PET methods utilize different manually designed modules. In a small
PLM, there are usually noticeable performance differences among PET methods.
Nevertheless, when a PLM’s scale grows up to tens of billions of parameters,
all PET methods achieve almost the same performance and even perform on par
with the full-parameter fine-tuning method. Hence, we hypothesize that model
scaling can mitigate the design differences (the module structures and the
number of trainable parameters) among PET methods. To study this hypothesis,
we introduce a more flexible PET method – arbitrary PET (APET) method – to be
compatible with arbitrary module structures and any number of trainable
parameters. Then, we experiment on $11$ NLP tasks of $5$ types and $2$
representative PLMs. From our investigations, we find that the model scaling
(1) mitigates the effects of the arbitrary module structure on the performance
of tuning methods, and (2) enables the tuning methods to optimize fewer
parameters to achieve the full-parameter fine-tuning performance.
Intriguingly, we also observe that all tuning methods require almost the same
number of trainable parameters to drive PLMs. We discuss this phenomenon and
the above two findings collectively from optimization perspectives to fathom
the mechanisms behind them. These conclusions not only demonstrate the
positive impact of model scaling on tuning methods but disclose its
mechanisms, which help us design more effective and efficient tuning methods
on larger-scale PLMs.

Figure 1: When the scale of a PLM contains up to tens of billions of
parameters, such as T5${}_{\textsc{xxl}}$, all PET methods perform comparably
and almost achieve the full-parameter fine-tuning performance.

##  1 Introduction

Pre-trained language models (PLMs), such as GPT Radford et al. (2018), BERT
Devlin et al. (2019), and T5 Raffel et al. (2020), have achieved great success
on various natural language processing (NLP) tasks. Despite their
effectiveness, fine-tuning (FT) these large-scale PLMs with full parameters
incurs both unaffordable computational and storage costs. To solve this
problem, researchers have proposed a series of parameter-efficient tuning
(PET) methods Houlsby et al. (2019a); Li and Liang (2021); Karimi Mahabadi et
al. (2021a); Lester et al. (2021); Mahabadi et al. (2021); Hu et al. (2022a);
Ben Zaken et al. (2022); He et al. (2022b) which only update an assigned
trainable module consisting of minimal parameters while freezing the rest
parameters in a PLM during model adaptation.

Although these existing representative PET methods can reduce computational
and storage costs, there are usually noticeable performance gaps among these
representative PET methods on downstream tasks. Intriguingly, we find that
when a PLM’s scale grows up to tens of billions of parameters, the performance
gap among different PET methods shrinks and all PET methods can almost achieve
full-parameter performance, as shown in Figure 1. These findings are
interesting and worth exploring because the existing representative PET
methods are designed with disparate philosophies, e.g., trainable modules that
are composed of different module structures and numbers of trainable
parameters. Hence, we hypothesize that model scaling mitigates the effects of
the design differences among the PET methods on the performance. To validate
this hypothesis, we further conduct two lines of ablation analyses:

  1. (A1)

The impact of model scaling on the module structure.

  2. (A2)

The impact of model scaling on the number of trainable parameters.

However, only investigating the four representative PET methods (Figure 1)
might be insufficient to cover enough variations of module structure for
ablation analyses (A1). Besides, the trainable modules of these four PET
methods are limited to being composed of layer-level tensors or matrices,
which is hard to precisely control the number of trainable parameters at the
fine-grained (parameter) level in ablation analyses (A2). To facilitate the
ablation analyses, we develop a more flexible PET method - Arbitrary
Parameter-Efficient Tuning (APET) method (§ 5.1) - to be compatible with
arbitrary modules and any number of trainable parameters.

In analysis (A1), we compare the performance of APET methods with different
module structures under the same number of trainable parameters. From the
experimental results, we find although APET methods with diverse structures
require different training steps to reach convergence under the same training
configurations, they can eventually achieve almost comparable performance on
larger-scale models. This indicates that the model scaling can mitigate the
effects caused by the structure difference on the performance but not on the
convergence speed.

In analysis (A2), we compare the performance of the same APET methods under
different numbers of trainable parameters. From the experimental results, we
find that the model scaling cannot mitigate the effect of the number of
trainable parameters on tuning methods. Besides, we find two interesting
phenomena when the number of trainable parameters reaches two thresholds: high
parameter threshold and low parameter threshold. (1) When the number of
trainable parameters equals the high threshold, all APET methods can achieve
the full-parameter fine-tuning performance, and the high parameter threshold
tends to be smaller on the larger models. Namely, we can optimize fewer
necessary trained parameters for tuning methods to achieve full-parameter
fine-tuning performance on the larger models. (2) On the other hand, when the
number of trainable parameters exceeds the low parameter threshold, all APET
methods outperform random guess performance. Furthermore, we find that the low
parameter thresholds of all APET methods on the same models are almost the
same, even for different tasks. This suggests that tuning methods require the
same number of trainable parameters to drive the same PLMs.

In summary, we design a more flexible PET methods - APET methods - to conduct
the extensive ablation analyses and reveal the impact of model scaling on the
designing differences among the tuning methods, e.g., (1) module structures (§
5.2) and (2) number of trainable parameters (§ 5.3). (3) Furthermore, we
discuss the further findings in ablation analyses from the perspective of
optimization (§ 6). We hope these results not only provide guidance for
designing tuning methods on the larger models but also encourage more
researchers to explore the impact of model scaling on tuning methods from the
theoretical perspective.

##  2 Related Work

#### Parameter-Efficient Tuning (PET) Methods

With larger PLMs continuously being developed, fine-tuning all of the
parameters and storing the adapted weights become increasingly cumbersome. To
address the issue, researchers propose PET methods which keep most of the
parameters of PLMs frozen and optimize only a trainable module consisting of a
few parameters during downstream adaptation. Over the recent years, many
different designs of PET methods have emerged. For instance, some PET methods
insert the external trainable modules after the feed-forward and attention
layers in a PLM (Houlsby et al., 2019a; Pfeiffer et al., 2021; Karimi Mahabadi
et al., 2021b); others prepend the trainable modules into attention layers (Li
and Liang, 2021; Hu et al., 2022a) or the embedding layer (Lester et al.,
2021). Another line of PET method selects the existing parameters in a PLM
(Ben Zaken et al., 2022; Guo et al., 2021) as the trainable module to
optimize.

PET Methods Unified View of PET Methods PET Trainable Modules $\theta$
Transformations Modification $\Delta\mathbf{h}$ Trainable Weight Positions
Prompt Lester et al. (2021)
$\mathbf{h}^{out}=f(\mathbf{h}^{in})+\Delta\mathbf{h}$
$f(\mathbf{W}_{prompt})$ $\mathbf{W}_{prompt}$ Input hidden states Adapter
Houlsby et al. (2019a)
$\sigma(f(\mathbf{h}^{in})\mathbf{W}_{\textit{down}})\mathbf{W}_{\textit{up}}$
$\mathbf{W}_{\textit{down}},\mathbf{W}_{\textit{up}}$ SelfAttn./FFN. LoRA Hu
et al. (2022a) $\alpha\ast x\mathbf{W}_{down}\mathbf{W}_{up}$
$\mathbf{W}_{\textit{down}},\mathbf{W}_{\textit{up}}$ SelfAttn BitFit Ben
Zaken et al. (2022) $\mathbf{W}_{b}$ $\mathbf{W}_{b}$ Bias terms

Table 1: We uniformly re-frame the transformations of PET methods as
modifications $\Delta\mathbf{h}$ of specific hidden states in the
corresponding PLM layer ($f$) where $\mathbf{W}$ is introduced in computing
$\Delta\mathbf{h}$. Each PET method is composed of $p$ trainable weights
$\mathbf{W}$ in different positions. Hence, we represent the PET module as
$\theta=\\{\mathbf{W}_{1},\mathbf{W}_{2},...,\mathbf{W}_{p}\\}$.

To fathom the mechanisms behind the PET methods, He et al. (2022a) formalize
PET methods as a unified framework to study the connections among PET methods.
Yi et al. (2022) also conduct the same study and further indicate that the
optimization of different PET methods can be unified in a similar subspace. In
this paper, we further explicitly demonstrate that the optimization of
different PET methods requires a close number of trainable parameters (§ 5.3).

#### The Power of Model Scaling

With model size scaling, PLMs emerge many abilities, such as reasoning ability
Wei et al. (2022b, a), human-like behavior Nakano et al. (2021); Ouyang et al.
(2022), and can achieve state-of-the-art results on many understanding and
generation tasks Du et al. (2021); Chowdhery et al. (2022).

Besides, some researchers find that performing some PET methods Lester et al.
(2021); Ding et al. (2022) on large-scale models can almost achieve the full-
parameter fine-tuning performance. In this paper, we verify this phenomenon is
general in the existing PET methods (§ 4) and further study the impact of
model scaling on the design differences among parameter-efficient methods (§
5). Furthermore, we will explain the mechanism behind our findings from the
optimization perspective (§ 6).

##  3 Preliminary

In this section, we first introduce the Transformer framework (§ 3.1) and the
most representative parameter-efficient tuning methods (§ 3.2).

###  3.1 Transformer Framework

The Transformer model Vaswani et al. (2017) is the mainstream architecture for
most powerful PLMs. The model is stacked of $L$ blocks, each of which consists
of a sequence of layers, including self-attention and feed-forward network.
During the forward pass through each block, the input hidden state is applied
with the sequence of layers. For simplicity, we formalize the transformation
of each layer as

Figure 2: PET methods only optimize the trainable modules $\theta$, while
freezing the rest of the parameters $\Phi$ in a model during downstream
adaptation.

$\mathbf{h}^{out}=f(\mathbf{h}^{in}).$ (1)

Under the layer as the operator $f$, the input hidden state
$\mathbf{h}^{in}\in\mathbb{R}^{s\times d_{in}}$ is transformed into the output
hidden state $\mathbf{h}^{out}\in\mathbb{R}^{s\times d_{out}}$, where $s$ is
the input length and $d_{in}$, $d_{out}$ are dimensions.

Figure 3: We investigate the performance of the PET methods, including Prompt,
BitFit, LoRA, and Adapter, on two series of models (T5 and Bert). The figure
shows the average performance of PET on all tasks. As the model scaling
increases, the performance gaps among PET methods become smaller, and all PET
methods gradually achieve full-parameter fine-tuning (FT) performance.

###  3.2 Parameter Efficient Tuning (PET)

Different PET methods111More implementation details are left in appendix B are
equipped with diverse modules $\theta$. These modules are composed of
trainable parameters $\mathbf{W}$ that modify the original layers and the
corresponding transformations in PLMs. To make comparisons, we follow the
unified view He et al. (2022a); Hu et al. (2022b) to re-frame the
transformations of all PET methods as the modifications $\Delta\mathbf{h}$ of
specific hidden states in the corresponding PLM’s layers as shown in Table 1,

$\mathbf{h}^{out}=f(\mathbf{h}^{in})+\Delta\mathbf{h}.$ (2)

In the training process, given a downstream task $\mathcal{D}=\\{X,Y\\}$, we
only optimize all trainable parameters of the module $\theta$ for each PET
method to generate desired outputs $Y$ of a downstream task while freezing the
rest of the parameters $\Phi$ in a PLM $\mathcal{M}$, as shown in Figure 2.
Formally, the training objective is to minimize $\mathcal{L}$ as follows:

$\min\nolimits_{\theta}\mathcal{L}(\mathcal{M}_{(\Phi,\theta)}(X),Y).$ (3)

##  4 Experiment and Analysis

To explore the impact of model scaling on these PET methods, we first
introduce the investigated tasks, PLMs, and settings of the existing
representative PET methods in the experiments (§ 4.1), and then report the
main experimental results (§ 4.2).

###  4.1 Experimental Settings

#### Investigated NLP Tasks

We investigate $11$ tasks, which can be divided into $5$ categories: (1)
Sentiment Analysis (SA), including SST-2 Socher et al. (2013), IMDB Maas et
al. (2011), and Rotten Tomatoes Pang and Lee (2005); (2) Natural Language
Inference (NLI), including MNLI Williams et al. (2018), QNLI Wang et al.
(2019), and RTE Bos and Markert (2005); (3) Paraphrase Identification (PI),
including MRPC Dolan and Brockett (2005) and QQP Sharma et al. (2019); (4)
Question Answering (QA), including NQ-Open Lee et al. (2019); (5)
Summarization (SUM), including SAMSum Gliwa et al. (2019) and Multi-News
Fabbri et al. (2019). More details are in appendix A.

#### Investigated PLMs

We will experiment on two series of PLM backbones: BERT Devlin et al. (2019)
and T5 Raffel et al. (2020) representing masked language models and sequence-
to-sequence models, respectively. Since masked language models are typically
applied on discriminative tasks, we only investigate SA, PI, and NLI
categories of tasks on the BERT backbone. Differently, sequence-to-sequence
models have no fixed-length output limitation; thus, we investigate all tasks
on the T5 backbone.

#### Training Details of PET Methods

We choose four representative PET methods, Prompt Lester et al. (2021), BitFit
Ben Zaken et al. (2022), Adapter Houlsby et al. (2019a), and LoRA Hu et al.
(2022a), to conduct analysis experiments. To guarantee the performance of PET
methods, we keep the same design of each method, including the module
structure and the number of trainable parameters, as reported in the original
paper. Besides, we train each PET method on $11$ tasks with $3$ different
random seeds and report their average performance. Details for the training
configurations are in appendix B.

###  4.2 Preliminary Experiments

To observe the impact of the model scaling on PET methods, we range PLMs in
ascending order of the model scale and report the performance of PET methods
on each PLM.

#### Model scaling impact on PET methods

Results are reported in Figure 3. First, we can observe that the PET methods
have noticeable performance gaps between each other on the general scale of
models (T5${}_{\textsc{small}}$ and T5${}_{\textsc{base}}$ in the sub-figure
[a]; BERT${}_{\textsc{small}}$ and BERT${}_{\textsc{base}}$ in the sub-figure
[b]). This phenomenon is intuitive and shows the critical impact of design
differences (the module structure and the number of trainable parameters) on
the performance of PET methods, as it has been consistently found in many
prior works Ding et al. (2022); Hu et al. (2022b).

However, we find that (1) as the model scaling increases (from
T5${}_{\textsc{small}}$ to T5${}_{\textsc{xxl}}$ in the sub-figure [a]; from
BERT${}_{\textsc{small}}$ to BERT${}_{\textsc{large}}$ in the sub-figure [b]),
the largest and smallest performance gaps among PET methods all become minor
(from 6%~4% to 4%~0% on T5 [a]; from 11%~2% to 6%~1% on BERT [b]); (2) all PET
methods can even achieve the full-parameter fine-tuning performance when the
model scale grows up to tens of billions parameters, i.e.,
T5${}_{\textsc{xxl}}$. These two findings imply that the model scaling can
mitigate the impact of the design differences (module structure and number of
trainable parameters) on the performance of all tuning PET methods.

##  5 Main Experiments

To further verify whether the model scaling will respectively remove the
effects of the above differences on PET methods, we conduct two ablations: the
model scaling impact on the (1) module structure and (2) number of trainable
parameters. However, only investigating the above four respective PET methods
is insufficient to cover enough variations of module structure for ablation
study (1). Besides, their trainable modules are composed of the layer-level
weights as shown in Figure 2.

This limitation makes us hard to preciously control the number of trainable
parameters at the fine-grained (parameter level) in (2). Hence, we develop a
more flexible PET method, Arbitrary Parameter-Efficient Tuning (APET) method.
Its trainable module can be arbitrary structure (§ 5.1) that facilitates us to
explore various module structures in the ablation study (§ 5.2) and easier
control the number of trainable parameters in the ablation study of trainable
parameters (§ 5.3).

###  5.1 Arbitrarily Few Parameter Tuning (APET)

Similar to PET methods, the APET method can be re-framed in the unified form
as shown in Table 1. Its trainable module $\theta$ is also composed of $L$
trainable weights $\mathbf{\widehat{W}}$, which can be expressed as
$\theta=\\{\mathbf{\widehat{W}}_{1},\mathbf{\widehat{W}}_{2},...,\mathbf{\widehat{W}}_{L}\\}$.
Differently, each trainable weight $\mathbf{\widehat{W}}$ of the APET method
is generated by $\mathbf{\widehat{W}}=\mathbf{W}\odot m$, where
$\mathbf{W}\in\mathbb{R}^{i\times j}$ is a weight, $m\in\\{0,1\\}^{i\times j}$
is a given binary pruning mask, and $\odot$ is a Hadamard product. The $i$ and
$j$ are dimensions. The trainable weight $\mathbf{\widehat{W}}$ will be added
or plugged into the PLM to modify the original PLM layers and the
corresponding transformations222More implementation details are left in
appendix C. Furthermore, $\mathbf{\widehat{W}}$ can be at any position of the
PLM, and we can control its parameter number and distribution. Based on the
parameter distribution as shown in Figure 4, we have two APET methods:

#### APET${}_{\textsc{Adjacent}}$

The trainable parameters in $\mathbf{\widehat{W}}$ are adjacent in rows or
columns.

#### APET${}_{\textsc{Discrete}}$

The trainable parameters in $\mathbf{\widehat{W}}$ are discrete.

In the training process, we also follow Equation (3) only to optimize
$\theta=\\{\mathbf{\widehat{W}}_{1},\mathbf{\widehat{W}}_{2},...,\mathbf{\widehat{W}}_{L}\\}$
of the APET method while freezing the rest of the parameters in a PLM.

It is worth noting that the trainable modules of APET methods can be any
arbitrary structure that is composed of the corresponding number of trainable
parameters. In this sense, we can see the previously introduced PET methods as
special cases of APET.

Figure 4: The trainable modules of APET methods are composed $L$ trainable
weights $\mathbf{\widehat{W}}$, which can be expressed as
$\theta=\\{\mathbf{\widehat{W}}_{1},\mathbf{\widehat{W}}_{2},...,\mathbf{\widehat{W}}_{L}\\}$.
Based on the parameter distribution in $\mathbf{\widehat{W}}$, we have two
APET methods: APET${}_{\textsc{Adjacent}}$ and APET${}_{\textsc{Discrete}}$.
Figure 5: Since we are hard to control the number of trainable parameters of
four PET methods, we set the number of trainable parameters of APET
(APET${}_{\textsc{Adjacent}}$, APET${}_{\textsc{Discrete}}$) methods equal to
four PET methods’ and then compare their performance. In each group (bar: h),
all these tuning methods have disparate module structures but have the same
number of trainable parameters (Parm). We find that as the model scaling
increases, the impact of module structures on the performance becomes minor,
especially on T5${}_{\textsc{xxl}}$.

###  5.2 The Impact of Model Scaling on The Module Structure

To conduct the ablation study of module structure, we first freeze the other
substantial factors (the number of trainable parameters) that might affect the
performance of tuning methods in the experiments. Since we are hard to control
the number of trainable parameters of the four representative PET methods, we
set the number of trainable parameters of APET methods equal to four PET
methods’ and then compare their performance in the same group (bar: h) as
shown in each sub-graph of the Figure 5. In the same group, although these
tuning methods have disparate module structures, they have the same number of
trainable parameters (Parm).

#### Performance Comparison

As shown in Figure 5, there are four groups of comparisons in each sub-graph.
We can observe that as a PLM size scales (T5: from [a.1] to [a.2]; BERT: from
[b.1] to [b.2]), the performance gaps between all tuning methods (PET, APET)
in the same group shrink. Although this phenomenon is also in both series of
PLMs (T5 and BERT), the performance gaps become much smaller on
T5${}_{\textsc{xxl}}$. We argue that this is because the larger model has more
powerful effectiveness in mitigating the impact of the module structure on the
performance.

In addition, we also find that even though tuning methods have different
numbers of trainable parameters in the four groups (bar: h), they all almost
achieve the same performance as well on the large-scale model, i.e.,
T5${}_{\textsc{xxl}}$, as shown in [a.2]. We will provide more discussions
about this finding and explain the reasons behind this phenomenon.

#### Convergence Comparison

Although two APET methods can achieve the same final performance on the larger
models, we find that APET${}_{\textsc{Adjacent}}$ and
APET${}_{\textsc{Discrete}}$ methods still require different training steps to
achieve convergence. This phenomenon is not only on the smaller models
(T5${}_{\textsc{small}}$ and BERT${}_{\textsc{small}}$) but also on the larger
models as shown in Figure 6. Hence, we can infer that (1) the model scaling
can only mitigate the effects of the structure on the performance, but not on
the convergence speed; (2) the module structure with adjacent trainable
parameters might be a better choice. We will discuss these inferences in § 6.

Figure 6: Under the same setting, APET${}_{\textsc{Adjacent}}$ method requires
fewer training steps than APET${}_{\textsc{Discrete}}$ method to achieve
convergence on T5${}_{\textsc{xxl}}$. Figure 7: Given the same tuning methods
under the different numbers of trainable parameters, we observe their
performance on two series of models and tasks. We find that (1) the model
scaling can make tuning methods optimize fewer necessary trained parameters
(smaller high parameter thresholds) to reach full-parameter fine-tuning
performance (\- - - and —–); (2) all tuning methods require almost the same
number of trainable parameters (same low parameter thresholds) to exceed
random guessing performance on the same models.

###  5.3 The Impact of Model Scaling on The Number of Trainable Parameters

In this section, given the same tuning methods under different numbers of
trainable parameters, we observe their performance to study the ablation.

From the reported results in Figure 7, we can find that (1) on the smaller
models, e.g., T5${}_{\textsc{small}}$ (\- - - , \- - -),
BERT${}_{\textsc{small}}$ (\- - - , \- - -), when the trainable parameters of
tuning methods are fewer than a certain number, the performance will drop to
randomly guess performance; (2) similarly, this phenomenon still holds on the
larger models, T5${}_{\textsc{xxl}}$ (—– , —–), BERT${}_{\textsc{large}}$ (—–
, —–). Strictly speaking, these findings demonstrate that the model scaling
cannot sufficiently remove the effect of the number of trainable parameters on
the performance of tuning methods.

Interestingly, we find two parameter thresholds of trainable parameters on all
models and name them as low parameter threshold of necessary trained
parameters and high parameter threshold of necessary trained parameters,
respectively. For a tuning method, when its trainable parameters are more than
low parameter threshold, the tuning method can exceed random performance
(e.g., 0% on T5 and $\frac{1\times 100}{\text{Number of label types}}\%$ on
BERT); when the trainable parameters are more than high parameter threshold,
the tuning method can almost achieve the full-parameter fine-tuning (FT)
performance. Furthermore, we also find that the model scaling affects the two
parameter thresholds. Hence, we explore this phenomenon in the following
paragraphs.

#### High threshold of necessary trained parameters

We observe the high threshold of each tuning method in the sub-graph [a.1]
(SST2) of Figure 7. From the experimental results, we find that the high
threshold of the larger model, i.e., T5${}_{\textsc{xxl}}$ is always lower
than the high threshold of the smaller model, i.e., T5${}_{\textsc{small}}$.
Except in the sub-graph [a.1] (SST2), we observe the same phenomenon over all
tasks (SST2, RTE, MRPC), and on two series of models (T5 and BERT) as shown in
sub-graphs ([a.2], [a.3], [b.1], [b.2], [b.3]). Hence, we can conclude that
the model scaling enables the tuning methods to train fewer necessary
parameters to achieve full-parameter performance.

This conclusion can intuitively disclose the reason why tuning methods (PET
and APET methods) perform equally and can almost achieve the full-parameter
performance on larger models (T5${}_{\textsc{xxl}}$) in Figure 3 and Figure 5.
That is because the numbers of trainable parameters of overall tuning methods
in Figure 3 and Figure 5 exceed the high parameter thresholds on
T5${}_{\textsc{xxl}}$; hence, they all can achieve the full-parameter fine-
tuning performance.

#### Low threshold of necessary trained parameters

From the results, as shown in the sub-graph [a.1] SST2 of Figure 7, we find
that tuning methods will exceed the random performance (0% on T5; 50% on BERT)
and immediately reach the 80~90% full-parameter fine-tuning performance when
the trainable parameters are more than low thresholds. However, the low
thresholds are relatively higher on T5${}_{\textsc{xxl}}$. Namely, tuning
methods require more trained parameters to exceed the random performance. This
phenomenon is consistent over all tasks on two series of models. Hence, we can
infer that the model scaling cannot reduce the number of necessary trained
parameters to drive PLMs to perform downstream tasks.

Furthermore, it is worth noting that the low parameter thresholds of all
tuning methods almost lie in the same range on the same models. Specifically,
the range of low thresholds are in [7.9e+3, 8.5e+3] on T5${}_{\textsc{xxl}}$,
[8.0e+2, 7.9e+3] on T5${}_{\textsc{small}}$, [8.0e+2, 3.2e+3] on
BERT${}_{\textsc{large}}$, and [4.0e+2, 1.6e+3] on BERT${}_{\textsc{small}}$.
We will explain this phenomenon from the optimization perspective in § 6.

##  6 Discussion & Conclusion

The objectives of all tuning methods (PET, APET) can be expressed as
$\min\nolimits_{\theta}\mathcal{L}(\mathcal{M}_{(\Phi,\theta)}(X),Y)$ as
introduced in Equation (3), where $\theta$ is a trainable module. The training
module $\theta$ of different tuning methods is composed of different
structures and the numbers of trainable parameters. In this paper, we explore
the impact of model scaling on structures and the trainable parameters of
training module $\theta$ in different tuning methods. We find that the model
scaling can (1) mitigate the effects of module structures on the performance
(§ 5.2) and (2) make tuning methods optimize fewer trainable parameters to
achieve full-parameter fine-tuning performance (§ 5.3). To further fathom the
reasons for these phenomena, we will explain them from the optimization
perspective. (3) Besides, we also observe that all tuning methods can optimize
almost the same number of minimal trained parameters to exceed random guessing
performance on the same models (§ 5.3). Although phenomenon (3) is not caused
by model scaling, we can also explain it from the optimization perspective.
Hence, we together discuss it and the above two findings (1) and (2) in the
following paragraphs.

#### Why can model scaling mitigate the effects of the module structure of
tuning methods?

From the optimal control perspective, a trainable module ($\theta$) of a
tuning method can be seen as a controller Yang and Liu (2022); Ding et al.
(2022) to drive PLMs towards downstream tasks. As the model scale increases,
the larger model has higher parameter redundancy Aghajanyan et al. (2021),
allowing arbitrary selection of trainable parameters for tuning without
greatly degrading performance Desai et al. (2019); Chen et al. (2020);
Prasanna et al. (2020); Evci et al. (2020); thus, controllers (modules) might
have higher degrees of freedom. This might explain why the differences among
structures of the trainable module ($\theta$) have less impact such that all
tuning methods can achieve the same performance on the larger models. It is
worth noting that even though tuning methods can achieve the same performance,
we found the module structures will affect converge speeds. Thus, finding a
better module structure to improve the converge speeds for tuning methods is a
direction worthy of exploring.

#### Why can model scaling leverage the fewer trainable parameters to achieve
fine-tuning performance?

Training $\theta$ to steer a PLM towards downstream NLP tasks can be seen as
adaptations. From the perspective of representation space, the adaptations of
the tuning methods (PET, APET, and FT methods) can be re-parameterized into a
unified low dimensional subspace Qin et al. (2021); Aghajanyan et al. (2021);
Yi et al. (2022). Aghajanyan et al. (2021) further demonstrate that adaptation
on a larger PLM can be re-parameterized into the lower dimensional space; this
implicitly explains why tuning methods can optimize fewer parameters on
larger-scale models, e.g., T5${}_{\textsc{xxl}}$, to meet the full-parameter
fine-tuning performance on tasks.

#### Why can tuning methods optimize the near numbers of trainable parameters
to exceed random guessing?

As stated above, the adaptations of the tuning methods can be re-parameterized
into a unified subspace. Qin et al. (2021) shows that this low dimensional
subspace is shared among all NLP tasks for the same tuning methods. Yi et al.
(2022) further suggests that this subspace is also shared among various tuning
methods. This might implicitly explain why all tuning methods can train the
near numbers of necessary trained parameters to exceed the random guessing
performance on the same models, even for the different tasks (§ 5.3).

We hope that these discussions disclose the mechanisms behind the model
scaling impact on tuning methods and can inspire more research toward
exploring the advantages of the model scaling.

## Limitations

This paper might have some possible limitations as follows: (1) we only
explore the effects of the scaling law on performance. There might be other
research points worth exploring, such as the power of model scale to
convergence speed; (2) we study the power of model scale with comprehensive
empirical experiments and explain the findings from the optimization
perspective. There might be more theoretical proofs to explain these exciting
findings.

## References

  * Aghajanyan et al. (2021) Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. 2021.  Intrinsic dimensionality explains the effectiveness of language model fine-tuning.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 7319–7328, Online. Association for Computational Linguistics.
  * Ben Zaken et al. (2022) Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022.  BitFit: Simple parameter-efficient fine-tuning for transformer-based masked language-models.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 1–9, Dublin, Ireland. Association for Computational Linguistics.
  * Bos and Markert (2005) Johan Bos and Katja Markert. 2005.  Recognising textual entailment with logical inference.  In _Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing_ , pages 628–635, Vancouver, British Columbia, Canada. Association for Computational Linguistics.
  * Chen et al. (2020) Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. 2020.  The lottery ticket hypothesis for pre-trained BERT networks.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022\.  Palm: Scaling language modeling with pathways.
  * Desai et al. (2019) Shrey Desai, Hongyuan Zhan, and Ahmed Aly. 2019.  Evaluating lottery tickets under distributional shifts.  In _Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019)_ , pages 153–162, Hong Kong, China. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ding et al. (2022) Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozhi Wang, Zhiyuan Liu, Hai-Tao Zheng, Jianfei Chen, Yang Liu, Jie Tang, Juanzi Li, and Maosong Sun. 2022.  Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models.  volume abs/2203.06904.
  * Dolan and Brockett (2005) William B. Dolan and Chris Brockett. 2005.  Automatically constructing a corpus of sentential paraphrases.  In _Proceedings of the Third International Workshop on Paraphrasing (IWP2005)_.
  * Du et al. (2021) Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathy Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. 2021.  Glam: Efficient scaling of language models with mixture-of-experts.
  * Evci et al. (2020) Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. 2020\.  Rigging the lottery: Making all tickets winners.  In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 2943–2952. PMLR.
  * Fabbri et al. (2019) Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019.  Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1074–1084, Florence, Italy. Association for Computational Linguistics.
  * Gliwa et al. (2019) Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019.  SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization.  In _Proceedings of the 2nd Workshop on New Frontiers in Summarization_ , pages 70–79, Hong Kong, China. Association for Computational Linguistics.
  * Guo et al. (2021) Demi Guo, Alexander Rush, and Yoon Kim. 2021.  Parameter-efficient transfer learning with diff pruning.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4884–4896, Online. Association for Computational Linguistics.
  * He et al. (2022a) Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2022a.  Towards a unified view of parameter-efficient transfer learning.  In _International Conference on Learning Representations_.
  * He et al. (2022b) Shwai He, Liang Ding, Daize Dong, Miao Zhang, and Dacheng Tao. 2022b.  Sparseadapter: An easy approach for improving the parameter-efficiency of adapters.
  * Houlsby et al. (2019a) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019a.  Parameter-efficient transfer learning for NLP.  In _Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 2790–2799. PMLR.
  * Houlsby et al. (2019b) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019b.  Parameter-efficient transfer learning for NLP.  In _Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 2790–2799. PMLR.
  * Hu et al. (2021) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021.  Lora: Low-rank adaptation of large language models.  _ArXiv preprint_ , abs/2106.09685.
  * Hu et al. (2022a) Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022a.  LoRA: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_.
  * Hu et al. (2022b) Shengding Hu, Zhen Zhang, Ning Ding, Yadao Wang, Yasheng Wang, Zhiyuan Liu, and Maosong Sun. 2022b.  Sparse structure search for delta tuning.  In _Advances in Neural Information Processing Systems_.
  * Karimi Mahabadi et al. (2021a) Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. 2021a.  Compacter: Efficient low-rank hypercomplex adapter layers.  In _Advances in Neural Information Processing Systems_ , volume 34, pages 1022–1035. Curran Associates, Inc.
  * Karimi Mahabadi et al. (2021b) Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. 2021b.  Compacter: Efficient low-rank hypercomplex adapter layers.  _Advances in Neural Information Processing Systems_ , 34.
  * Lee et al. (2019) Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019.  Latent retrieval for weakly supervised open domain question answering.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6086–6096, Florence, Italy. Association for Computational Linguistics.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.  The power of scale for parameter-efficient prompt tuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3045–3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021.  Prefix-tuning: Optimizing continuous prompts for generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4582–4597, Online. Association for Computational Linguistics.
  * Maas et al. (2011) Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011.  Learning word vectors for sentiment analysis.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 142–150, Portland, Oregon, USA. Association for Computational Linguistics.
  * Mahabadi et al. (2021) Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. 2021.  Compacter: Efficient low-rank hypercomplex adapter layers.
  * Nakano et al. (2021) Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2021.  Webgpt: Browser-assisted question-answering with human feedback.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.
  * Pang and Lee (2005) Bo Pang and Lillian Lee. 2005.  Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales.  In _Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05)_ , pages 115–124, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Pfeiffer et al. (2021) Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2021.  AdapterFusion: Non-destructive task composition for transfer learning.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 487–503, Online. Association for Computational Linguistics.
  * Prasanna et al. (2020) Sai Prasanna, Anna Rogers, and Anna Rumshisky. 2020.  When BERT Plays the Lottery, All Tickets Are Winning.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 3208–3229, Online. Association for Computational Linguistics.
  * Qin et al. (2021) Yujia Qin, Xiaozhi Wang, Yusheng Su, Yankai Lin, Ning Ding, Zhiyuan Liu, Juanzi Li, Lei Hou, Peng Li, Maosong Sun, et al. 2021.  Exploring low-dimensional intrinsic task subspace via prompt tuning.  _ArXiv preprint_ , abs/2110.07867.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.  Improving language understanding by generative pre-training.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  In _Proceedings of JMLR_ , volume 21, pages 1–67.
  * Sharma et al. (2019) Lakshay Sharma, Laura Graesser, Nikita Nangia, and Utku Evci. 2019.  Natural language understanding with the quora question pairs dataset.  _ArXiv preprint_ , abs/1907.01041.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics.
  * Su et al. (2022) Yusheng Su, Xiaozhi Wang, Yujia Qin, Chi-Min Chan, Yankai Lin, Huadong Wang, Kaiyue Wen, Zhiyuan Liu, Peng Li, Juanzi Li, Lei Hou, Maosong Sun, and Jie Zhou. 2022.  On transferability of prompt tuning for natural language processing.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3949–3969, Seattle, United States. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pages 5998–6008.
  * Vu et al. (2022) Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou’, and Daniel Cer. 2022.  SPoT: Better frozen model adaptation through soft prompt transfer.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5039–5059, Dublin, Ireland. Association for Computational Linguistics.
  * Wang et al. (2019) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
  * Wei et al. (2022a) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022a.  Emergent abilities of large language models.  _Transactions on Machine Learning Research_.
  * Wei et al. (2022b) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022b.  Chain of thought prompting elicits reasoning in large language models.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.
  * Yang and Liu (2022) Zonghan Yang and Yang Liu. 2022.  On robust prefix-tuning for text classification.  In _International Conference on Learning Representations_.
  * Yi et al. (2022) Jing Yi, Weize Chen, Yujia Qin, Yankai Lin, Ning Ding, Xu Han, Zhiyuan Liu, Maosong Sun, and Jie Zhou. 2022.  Different tunes played with equal skill: Exploring a unified optimization subspace for delta tuning.

##  Appendix A Task and Dataset

We use various NLP tasks to evaluate the APET methods, which can be divided
into the following $5$ categories:

#### Sentiment Analysis (SA)

SA tasks evaluate if a model can correctly predict the sentiment labels of an
input sentence. In this paper, we choose SST-2 Socher et al. (2013), IMDB Maas
et al. (2011), and Rotten Tomatoes Pang and Lee (2005).

#### Natural Language Inference (NLI)

NLI tasks evaluate a model’s ability to correctly classify if a hypothesis can
be entailed or not given a premise. In this paper, we choose MNLI Williams et
al. (2018), QNLI Wang et al. (2019), and RTE Bos and Markert (2005).

#### Paraphrase Identification (PI)

PI tasks evaluate if a model can correctly identify paraphrases, which means
two sentences are identical in semantic meaning. In this paper, we choose MRPC
Dolan and Brockett (2005), and QQP Sharma et al. (2019).

#### Question Answering (QA)

QA tasks evaluate a model’s ability to answer questions. Context may be
present. In this paper, we choose NQ-Open Lee et al. (2019), an open-world QA
dataset without context.

#### Summarization (SUM)

SUM tasks evaluate a model’s ability to summarize a long paragraph into a
shorter abstract without loosing the semantics of the original text. In this
paper, we choose SamSUM Gliwa et al. (2019), and Multi-News Fabbri et al.
(2019) in our experiments.

##  Appendix B Parameter-efficient Tuning (PET) Methods

Here, we first recap the PLM (transformer) layer. Then, we describe the detail
and training configurations of all the PET methods mentioned in Table 1.

###  B.1 Transformer Architecture

A PLM is generally a stack of multiple Transformer layers, each composed of a
multi-headed attention and a feed-forward network. The multi-headed attention
contains $h$ attention heads working in parallel. Specifically, given an input
$\mathbf{X}\in\mathbb{R}^{n\times d}$, the $i$-th attention head works as
follows:

$\mathbf{h}_{i}=\texttt{softmax}(\frac{(\mathbf{XW}^{i}_{q})(\mathbf{XW}^{i}_{k})^{T}}{\sqrt{d/h}}(\mathbf{XW}^{i}_{v})),$
(4)

where $n$ is sequence length, $d$ is the hidden dimension,
$\mathbf{W}^{i}_{q}\in\mathbb{R}^{n\times d}$ is query,
$\mathbf{W}^{i}_{k}\in\mathbb{R}^{n\times d}$ is key, and
$\mathbf{W}^{i}_{v}\in\mathbb{R}^{n\times d}$ is value. The output from each
attention head will be concatenated and further transformed by
$\mathbf{W}_{o}\in\mathbb{R}^{d\times d}$ and be denoted as:

$\mathbf{h}_{\texttt{MHA}}=\texttt{concat}(\mathbf{h}_{1},\mathbf{h}_{2},...,\mathbf{h}_{h})\mathbf{W}_{o},$
(5)

where $\mathbf{h}_{\texttt{MHA}}\in\mathbb{R}^{n\times d}$ is the output
hidden state of multi-headed attention layer. After that, $\mathbf{h}$ will be
fed into a two-layer feed-forward network

$\mathbf{h}_{\texttt{FFN}}=\sigma(\mathbf{hW}_{1}+\mathbf{b}_{1})\mathbf{W}_{2}+\mathbf{b}_{2},$
(6)

where $\mathbf{W}_{1}\in\mathbb{R}^{d\times d_{m}}$,
$\mathbf{W}_{2}\in\mathbb{R}^{d_{m}\times d}$,
$\mathbf{b}_{1}\in\mathbb{R}^{d_{m}}$, $\mathbf{b}_{2}\in\mathbb{R}^{d}$, and
$d_{m}>d$ is an integer.

During the forward pass through each (transformer) block, the input hidden
state is applied with the sequence of layers. For simplicity, we formalize the
transformation of each layer as

$\mathbf{h}^{out}=f(\mathbf{h}^{in}).$ (7)

Under the layer as the operator $f$, the input hidden state
$\mathbf{h}^{in}\in\mathbb{R}^{n\times d}$ is transformed into the output
hidden state $\mathbf{h}^{out}\in\mathbb{R}^{n\times d}$, where $s$ is the
input length, and $d$ is the dimension.

###  B.2 Implementation Details of PET Methods

We follow the unified view He et al. (2022a); Hu et al. (2022b) to re-frame
the transformations of all PET methods as the modifications $\Delta\mathbf{h}$
of specific hidden states in the corresponding PLM’s layers as:

$\mathbf{h}^{out}=f(\mathbf{h}^{in})+\Delta\mathbf{h}.$ (8)

#### Prompt

Prompt-tuning Lester et al. (2021) prepends $N_{p}$ trainable soft tokens,
i.e. embeddings, to the input sentences and asks the model to predict the
probability of the next word. During training, only the newly added embeddings
are optimized and the backbone model is frozen. Given an input embedding
$\mathbf{X}\in\mathbb{R}^{n\times d}$, prompt-tuning can be seen as performing
the following operation:

$\mathbf{h}^{out}=f([\mathbf{X};\mathbf{W}_{prompt}]),$ (9)

where $\mathbf{W}_{prompt}\in\mathbb{R}^{N_{p}\times d}$,
$\mathbf{X}\in\mathbb{R}^{N\times d}$, [;] means "concatenate", $d$ is
dimension, and $N$, $N_{p}$ are the sequence lengths. The $N_{p}=100$ and
$N_{p}=256$. To re-frame Prompt into the form of Equation 8, we concatenate a
zero weight $\mathbf{W}_{\texttt{0}}\in\\{0\\}^{N_{p}\times d}$ and
$\mathbf{W}_{prompt}\in\mathbb{R}^{N_{p}\times d}$, which can be denoted as
$\mathbf{W}_{prompt^{{}^{\prime}}}\in\mathbb{R}^{(N+N_{p})\times d}$. Hence,
the Equation 9 becomes
$\mathbf{h}^{out}=f(\mathbf{X}+\mathbf{W}_{prompt^{{}^{\prime}}})$ and we can
further reform it as the unified form:

$\mathbf{h}^{out}=f(\mathbf{X})+f(\mathbf{W}_{prompt^{{}^{\prime}}}).$ (10)

#### BitFit

BitFit Ben Zaken et al. (2022) is a method that only tunes all the bias terms
$\mathbf{W}_{b}\in\mathbb{R}^{d}$ in the PLM, which lie in the self-attention
and layer norm layers. For a Transformer layer $f$, BitFit performs the same
operation as the normal Transformer model with no modification:

$\mathbf{h}^{out}=f(\mathbf{h}^{in})+\mathbf{W}_{b}.$ (11)

#### LoRA

LoRA Hu et al. (2021) is a method that adapts a PLM in a low-rank space. It
down-projects the attention weights into a lower dimension and up-projects it
back to the original dimension. Only these projection weights are optimized.
For a Transformer layer $f$, Lora computes the output hidden states as
follows:

$\mathbf{h}^{out}=f(\mathbf{h}^{in})+\alpha\mathbf{h}^{in}\mathbf{W}_{down}\mathbf{W}_{up},$
(12)

where $\mathbf{W}_{down}\in\mathbb{R}^{d\times r}$ denotes the down projection
matrix, $\mathbf{W}_{up}\in\mathbb{R}^{r\times d}$ denotes the up projection
matrix, $r$ denotes the rank and is set to 8, and $\alpha=16$.

#### Adapter

Adapter Houlsby et al. (2019b) is a method that only tunes the inserted
adapter modules, which consist of down projection, non-linear transformation,
up projection, and a skip-connection. For each existing Transformer layer in a
PLM, the adapter modules are inserted at two locations: (1) after the first
feed-forward layer, and (2) after the two consecutive feed-forward layers.
During training, only the adapter modules are optimized and the rest of the
PLM is frozen. For a Transformer layer $f$, Adapter can be seen as performing
the following operation:

$\mathbf{h}^{out}=f(\mathbf{h}^{in})+\sigma(f(\mathbf{h}^{in})\mathbf{W}_{down})\mathbf{W}_{up},$
(13)

where $\sigma$ is a non-linear activation function,
$\mathbf{W}_{down}\in\mathbb{R}^{d\times r_{\texttt{A}}}$ denotes the down
projection matrix, $\mathbf{W}_{down}\in\mathbb{R}^{d\times r_{\texttt{A}}}$
denotes the down projection matrix, $r_{\texttt{A}}=24$ denotes the bottleneck
dimension.

###  B.3 Training Configurations of PET Methods

The trainable module of a PET method $\theta$ is composed of $L$ trainable
weights $\mathbf{W}$ (all trainable weights) of the specific PET method, which
can be expressed as
$\theta=\\{\mathbf{{W}}_{1},\mathbf{{W}}_{2},...,\mathbf{{W}}_{L}\\}$. We also
follow Equation (3) to train the PET method. During training, we only optimize
$\theta$ while freezing the rest of the parameters in the PLM. We adopt a
batch size of $32$ and a learning rate of 3e-4 with no warm-up for most of the
PET models and tasks. However, here are two special cases soft-prompts and RTE
datasets. Since soft-prompts are hard to optimize, we adopt a learning rate of
3e-2. Besides, due to the small size of RTE dataset, we set the learning rate
for all PET methods as 5e-5 on RTE dataset. The maximum input length is $128$
for single sentence tasks (SA) and $256$ for multi-sentence tasks (NLI, PI,
QA, SUM). The maximum generation length is $1$ for classification tasks (SA,
NLI, PI), $64$ for Multi-News, and $128$ for SAMSum.

Figure 8: The trainable modules of APET methods are composed $p$ trainable
weights $\mathbf{\widehat{W}}$, which can be expressed as
$\theta=\\{\mathbf{\widehat{W}}_{1},\mathbf{\widehat{W}}_{2},...,\mathbf{\widehat{W}}_{L}\\}$.
We introduce three operations to insert the trainable weight
$\mathbf{\widehat{W}}$ into the PLM and the corresponding transformation.

Task Category \- - SA (3 tasks) NLI (3 tasks) PI (2 tasks) QA (1 task) SUM (2
tasks) \- - SA (3 tasks) NLI (3 tasks) PI (2 tasks) Task \- - SST2 IMDB Rotten
Tomatoes MNLI QNLI RTE MRPC QQP NQ-Open SAMSum Multi-News \- - SST2 IMDB
Rotten Tomatoes MNLI QNLI RTE MRPC QQP Metric \- - Acc. Acc. Acc. F1. ROUGE-L
\- - Acc. Acc. Acc. Backbone \- - T5 \- - BERT Scale \- -
T5${}_{\textsc{small}}$ \- - BERT${}_{\textsc{small}}$ Methods Ratio Ratio
Fine-tune 1 91.8 89.2 82.6 82.4 90.3 69.9 89.7 88.0 7.0 42.8 25.1 1 88.1 86.0
81.6 72.9 87.1 63.8 80.6 85.8 APET${}_{\textsc{Adjacent}}$ 0.07% 75.1 83.6
68.5 74.9 80.6 58.0 79.0 79.9 2.8 30.2 21.6 0.09% 77.7 74.0 74.5 63.7 79.6
62.4 77.9 76.2 APET${}_{\textsc{Adjacent}}$ 0.16% 81.4 81.4 78.0 72.7 82.0
59.6 78.2 82.4 2.7 34.9 23.5 0.05% 80.7 85.7 79.1 74.7 82.9 59.9 83.3 70.0
APET${}_{\textsc{Adjacent}}$ 0.30% 85.7 87.4 82.7 65.4 81.9 62.4 80.9 72.7 3.4
33.9 21.7 0.23% 86.1 84.4 80.8 69.3 84.7 65.6 81.0 82.4
APET${}_{\textsc{Adjacent}}$ 1.04% 87.9 84.4 80.7 80.4 86.8 62.6 83.3 86.6 5.5
41.0 24.5 0.70% 85.7 92.8 83.1 79.8 86.8 64.9 85.1 76.0 Scale \- -
T5${}_{\textsc{base}}$ \- - BERT${}_{\textsc{base}}$ Methods Ratio Ratio Fine-
tune 1 95.2 90.3 86.6 87.1 93.7 80.1 88.0 89.4 13.1 45.6 26.1 1 92.3 89.0 86.7
81.7 91.1 67.5 86.5 88.3 APET${}_{\textsc{Adjacent}}$ 0.03% 80.8 82.1 76.9
83.9 69.0 61.2 79.7 74.1 3.4 42.7 22.8 0.07% 81.7 84.0 78.8 76.8 78.2 68.7
81.6 69.8 APET${}_{\textsc{Adjacent}}$ 0.11% 85.8 84.6 80.5 85.0 75.1 64.8
81.1 77.2 6.0 44.0 25.3 0.06% 82.9 83.8 80.0 79.2 80.6 68.6 83.1 59.7
APET${}_{\textsc{Adjacent}}$ 0.24% 81.5 71.3 77.0 78.2 73.4 56.0 76.2 73.0 5.9
43.6 25.2 0.27% 91.5 88.1 86.2 78.7 89.6 70.4 87.0 86.2
APET${}_{\textsc{Adjacent}}$ 0.73% 94.0 84.4 81.6 83.2 86.7 65.8 80.4 86.5 7.9
43.0 25.0 0.83% 87.2 90.1 84.6 84.3 86.2 70.0 87.1 64.9 Scale \- -
T5${}_{\textsc{xxl}}$ \- - BERT${}_{\textsc{large}}$ Methods Ratio Ratio Fine-
tune 1 97.5 94.3 93.5 92.2 96.9 89.9 90.4 90.6 28.0 47.6 26.5 1 91.1 89.7 87.5
84.3 92.0 70.1 85.5 89.9 APET${}_{\textsc{Adjacent}}$ 0.004% 96.7 92.8 94.0
92.1 95.6 88.5 92.2 88.4 20.4 49.1 24.0 0.03% 87.3 91.1 82.2 82.3 80.9 61.3
85.2 64.1 APET${}_{\textsc{Adjacent}}$ 0.02% 97.2 91.1 93.7 90.0 95.8 91.4
83.6 88.3 23.8 49.3 24.2 0.05% 89.4 92.9 85.3 84.3 82.1 65.6 86.3 66.0
APET${}_{\textsc{Adjacent}}$ 0.06% 97.3 94.0 94.0 91.7 95.7 92.3 91.8 89.2
22.6 50.3 25.5 0.23% 94.1 89.7 88.5 82.6 91.4 74.6 86.7 87.4
APET${}_{\textsc{Adjacent}}$ 0.17% 96.9 92.9 93.8 92.0 95.6 79.6 86.0 89.0
26.2 49.2 24.2 0.72% 92.5 95.3 86.6 87.2 88.9 68.7 86.7 68.8

Table 2: The performance of all APET methods on the all investigated tasks.

##  Appendix C Arbitrary Few Parameter (APET) Methods

We introduce a more flexible PET method, Arbitrary Parameter-Efficient Tuning
(APET) method. Its trainable module can be arbitrary structure that
facilitates us to explore various module structures and easier control the
number of trainable parameters.

###  C.1 Implementation Details of APET Methods

As we previously introduced in § 5.1, the trainable module of the APET method
is composed of trainable weights. Each trainable weight $\mathbf{\widehat{W}}$
of the APET method is generated by $\mathbf{\widehat{W}}=\mathbf{W}\odot m$,
where $\mathbf{W}\in\mathbb{R}^{i\times j}$ is a weight,
$m\in\\{0,1\\}^{i\times j}$ is a given binary pruning mask, and $\odot$ is a
Hadamard product. Here, we have three operations to insert the trainable
weight $\mathbf{\widehat{W}}$ into the PLM to modify the specific layers and
their corresponding transformations as follows:

#### Add

We will add the trainable weight $\mathbf{\widehat{W}}=\mathbf{W}\odot m$ into
the PLM layer. The corresponding transformation can be denoted as
$\mathbf{h}^{out}$:

$f(\mathbf{h}^{in})+\mathbf{W}_{\texttt{1}}\odot m_{\texttt{1}}.$ (14)

This form is similar to the operation of Bitfit (Equation 11).

#### Concatenate

We will concatenate the trainable weight $\mathbf{\widehat{W}}=\mathbf{W}\odot
m$ and the hidden state or the layer in the PLM. The corresponding
transformation can be denoted as $\mathbf{h}^{out}$:

$\displaystyle
f(\mathbf{h}^{in})+\left\\{\begin{matrix}f(\mathbf{W}_{\texttt{2}}\odot
m_{\texttt{2}})\\\ \alpha\mathbf{h}^{in}\mathbf{W}_{\texttt{3}}\odot
m_{\texttt{3}}\mathbf{W}_{\texttt{4}}\odot m_{\texttt{4}}\end{matrix}\right.$
(15)

This form is similar to the operation of Prompt (Equation 9) and LoRA
(Equation 12).

#### Plug in

We will plug the trainable weight $\mathbf{\widehat{W}}=\mathbf{W}\odot m$
between PLM layers. The corresponding transformation can be denoted as
$\mathbf{h}^{out}$:

$f(\mathbf{h}^{in})+\sigma(f(\mathbf{h}^{in})\mathbf{W}_{\texttt{5}}\odot
m_{\texttt{5}})\mathbf{W}_{\texttt{6}}\odot m_{\texttt{6}}.$ (16)

This form is similar to Adapter (Equation 13).

According to these operations and the corresponding transformations, we can
express the APET methods as $\mathbf{h}^{out}$:

$\displaystyle
f(\mathbf{h}^{in})+\left\\{\begin{matrix}\mathbf{W}_{\texttt{1}}\odot
m_{\texttt{1}}f(\mathbf{W}_{\texttt{2}}\odot m_{\texttt{2}})\\\
\alpha\mathbf{h}^{in}\mathbf{W}_{\texttt{3}}\odot
m_{\texttt{3}}\mathbf{W}_{\texttt{4}}\odot m_{\texttt{4}}\\\
\sigma(f(\mathbf{h}^{in})\mathbf{W}_{\texttt{5}}\odot
m_{\texttt{5}})\mathbf{W}_{\texttt{6}}\odot m_{\texttt{6}}\\\
\vdots\end{matrix}\right.$ (17)

By comparing the Equation 17 with the equations of the previously introduced
PET methods, we can clearly find that the PET methods are special cases of
APET methods. Besides, based on the distribution of the masked parameters in
the $\mathbf{\widehat{W}}$, we have two APET methods as shown in Figure 8:
APET${}_{\textsc{Adjacent}}$ and APET${}_{\textsc{Discrete}}$.

(a) BERT${}_{\textsc{small}}$ and T5${}_{\textsc{small}}$ (b)
BERT${}_{\textsc{large}}$ and T5${}_{\textsc{xxl}}$

Figure 9: Relative performance (zero-shot transfer performance / original
performance) (%) on the target tasks (columns) of the APET methods trained on
the source tasks (rows). Colors of the task names indicate task types.

###  C.2 Training Configurations of APET methods

The trainable module of a APET method $\theta$ is composed of $L$ trainable
weights $\mathbf{\widehat{W}}$, which can be expressed as
$\theta=\\{\mathbf{\widehat{W}}_{1},\mathbf{\widehat{W}}_{2},...,\mathbf{\widehat{W}}_{L}\\}$.
We also follow Equation (3) to train the APET method. During training, we only
optimize $\theta$ while freezing the rest of the parameters in the PLM.

Besides, we adopt a batch size of $32$ and a learning rate of 3e-4 with no
warm-up for most of the APET models and tasks. In addition, The maximum input
length is $128$ for single sentence tasks (SA) and $256$ for multi-sentence
tasks (NLI, PI, QA, SUM). The maximum generation length is $1$ for
classification tasks (SA, NLI, PI), $64$ for Multi-News, and $128$ for SAMSum.

##  Appendix D Main Experiment Results

Due to the limitation space, we only report the average performance in Figure
2. Here, we show all results of APET methods on all investigated tasks.

##  Appendix E Power of Model Scale to Transferability

Furthermore, to explore whether the power of model scale can also facilitate
generalization ability of tuning methods, we explore the transferability
between the NLP tasks in the zero-shot setting Vu et al. (2022); Su et al.
(2022); Ding et al. (2022). In the experiments, we first train the parameters
of APET methods on the source tasks and directly reuse them on the target
tasks in zero-shot setting. We will investigate two series of PLMs T5
(T5${}_{\textsc{small}}$ and T5${}_{\textsc{xxl}}$) and BERT
(BERT${}_{\textsc{small}}$ and BERT${}_{\textsc{large}}$) and report the
relative performance.

Note that for different types of tasks, they are expected to share different
groups of label sets (e.g. for task like SA, the labels are usually
positive/negative, whereas, for tasks like NLI, the labels are usually
entailment/not entailment). Reusing the parameters trained on the source task
to test on the target task will naturally fail since the model is not able to
generate the labels they have never seen in the training stage. To this end,
we generally map the original label sets to a unified label set (e.g.
negative/not entailment/false –> 0, positive/entailment/true –> 1). Utilizing
a unified label set makes it feasible to evaluate the transferability of the
AFP method among different types of tasks regardless of the divergence of
original labels.

The results are shown in Figure 9, from which we can find that the APET
(APET${}_{\textsc{Discrete}}$ and APET${}_{\textsc{Adjacent}}$) methods can
transfer to the same type of tasks demonstrated by the darker color alongside
the diagonal of the matrix and generally perform well both on small-scale PLMs
(Figure 9 (a): BERT${}_{\textsc{small}}$ and T5${}_{\textsc{small}}$) and
large-scale PLMs (Figure 9 (b): BERT${}_{\textsc{large}}$ and
T5${}_{\textsc{xxl}}$). However, the lighter color indicates that APET methods
have difficulty performing different types of tasks overall, and both small-
scale and large-scale PLMs share this phenomenon. This finding indicates that
the power of scale does not necessarily facilitate the generalization ability
of AFP methods which is in line with the prevalent assumption that fewer
parameters often cause underfitting, whereas more parameters tend to cause
overfitting. Nevertheless, the mechanism behind this phenomenon still arouses
our deep concern and is worth expanding that we will systematically analyze it
in our future work.