# Efficient Prompt Tuning by Multi-Space Projection and Prompt Fusion

Pengxiang Lan  Enneng Yang  Yuting Liu  Guibing Guo Corresponding Author.
Linying Jiang  Jianzhe Zhao  Xingwei Wang  Northeastern University, China
<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Prompt tuning is a promising method to fine-tune a pre-trained language model
without retraining its large-scale parameters. Instead, it attaches a soft
prompt to the input text, whereby downstream tasks can be well adapted by
merely learning the embeddings of prompt tokens. Nevertheless, existing
methods still suffer from two challenges: (i) they are hard to balance
accuracy and efficiency. A longer (shorter) soft prompt generally leads to a
better (worse) accuracy but at the cost of more (less) training time. (ii) The
performance may not be consistent when adapting to different downstream tasks.
We attribute it to the same embedding space but responsible for different
requirements of downstream tasks. To address these issues, we propose an
Efficient Prompt Tuning method (EPT) by multi-space projection and prompt
fusion. Specifically, it decomposes a given soft prompt into a shorter prompt
and two low-rank matrices, whereby the number of parameters is greatly reduced
as well as the training time. The accuracy is also enhanced by leveraging low-
rank matrices and the short prompt as additional knowledge sources to enrich
the semantics of the original short prompt. In addition, we project the soft
prompt into multiple subspaces to improve the performance consistency, and
then adaptively learn the combination weights of different spaces through a
gating network. Experimental experiments on 13 natural language processing
downstream tasks show that our method significantly and consistently
outperforms 11 comparison methods with the relative percentage of improvements
up to 28.8%, and training time decreased by 14%.

123

##  1 Introduction

Fine-tuning methods have become a growing focus to adapt a pre-trained
language model (PLM) to a variety of downstream tasks [7, 23]. However, the
continuous expansion of the PLMs scale has led to a significant increase in
the number of parameters [42], such as the T5 model [24] containing hundreds
of millions of parameters. Therefore, full fine-tuning PLMs on all parameters
is unrealistic in practical applications. The discrete phrase-based tuning
provides task descriptions in the form of input text [2], guiding PLMs to
perform corresponding downstream tasks effectively, avoiding full-parameter
fine-tuning. Unfortunately, manually designing an effective set of task prompt
phrases heavily relies on experts’ domain knowledge, which is still
challenging in the face of a wide variety of tasks.

Figure 1: Average performance ($y$-axis) against the number of trainable
parameters ($x$-axis) on the GLUE and SuperGLUE benchmarks. We utilize the
T5-Base for all of the models. $*$ indicates that some tasks utilize multi-
task learning.

Recently, prompt tuning (PT) [19] based method has become an effective
alternative to convert discrete phrases into a set of learnable parameters. PT
freezes the parameters of PLMs and only trains the attached soft (continuous)
prompt vectors to the input text. Therefore, its parameters do not
dramatically scale up with the expansion of the model size, making PT stand
out in the parameter-efficient fine-tuning (PEFT) approaches [28]. Recent
studies have leveraged some successful approaches to reduce training
parameters in PT, such as parameter-efficient transfer learning (PETL) [33,
1], multi-task learning [36], and decomposing soft prompts [28, 39]. Despite
these PT variants effectively improving soft prompt performance in downstream
tasks, PT still faces several limitations that cannot be ignored. First,
existing PT-based methods encounter the challenge of balancing accuracy and
efficiency [28, 39, 19]. Attaching the soft prompt to the input extends the
overall length of the input sequence. Due to the quadratic complexity of the
Transformer [32], lengthening the soft prompt introduces additional training
time. PT requires training a substantial number of prompt tokens to achieve
competitive performance [19]; directly shortening the soft prompt to reduce
training time may result in sub-optimal performance for PT. Second, existing
PT-based variants are not well adapted to various downstream tasks and are
causing inconsistent performance. This is because they attempt to handle the
different needs of various downstream tasks with the same embedding space [28,
36, 1]. However, text information in natural language processing tasks
involves different types [35] and degrees of difficulty, and models pay
limited attention to semantics in the short prompt. For example, on the
SuperGLUE [35] benchmark, which is more complex than the GLUE [34] benchmark,
the performance of PT’s variants is not very satisfactory.

To tackle the aforementioned knotty issues, we propose a novel efficient
prompt tuning (EPT) that consists of two core modules: prompt fusion and
multi-space projection. EPT initially decomposes a whole soft prompt into two
independent parts: a short prompt and two low-rank matrices. Only the short
prompt is attached to the front of the input, to reduce the training time.
Low-rank matrices are utilized to update the frozen input text embedding.
Next, to offset the semantic loss of the short prompt compared with long ones,
we design a prompt fusion module, which utilizes the attention network to
capture the knowledge difference between low-rank matrices and the short
prompt, and instills this difference into the short prompt to improve the
semantic richness of the short prompt. Then, to adapt PT to different
downstream tasks more consistently, we leverage a multi-space projection
module to project a single soft prompt into multiple subspaces and reweight
the soft prompt in these subspaces according to the task through the gating
network. Finally, a joint representation of the prompt (obtained from the
prompt fusion and multi-space modules) replaces the vanilla prompt.

In summary, the main contributions of this paper are as follows:

  * •

We point out that PT-based methods suffer from the trade-off dilemma of
“accuracy and efficiency” as well as performance inconsistency. To address
these issues, we propose a novel efficient prompt tuning (EPT) method.

  * •

We design two effective modules in EPT, prompt fusion and prompt projection.
The former helps to maintain the efficiency of the short prompt and compensate
for the semantic missing of the short prompt to enhance performance, and the
latter reweights prompts in multiple subspaces to adapt to downstream tasks.

  * •

We comprehensively evaluate EPT on the GLUE and SuperGLUE benchmarks, where
EPT outperformed other PEFT methods, including LoRA and multi-task transfer
learning-based PT variants (see Fig.1). In particular, EPT achieves a 14%
reduction in training time compared to vanilla PT on the GLUE benchmark.

##  2 The Proposed Method

In this section, we first introduce the background of the prompt tuning and
then elaborate our proposed EPT method as shown in Fig. 2. It consists of four
main modules: (1) Prompt Decomposition, (2) Prompt Fusion, (3) Multi-Space
Projection, and (4) Reconstructed Prompt.

###  2.1 Background: Prompt Tuning

We first introduce the training method of PT. PT has gained widespread
adoption in downstream tasks due to its advantage of the parameters not
increasing sharply with the expansion of the model [28]. Let labeled training
data
$(\boldsymbol{X},\boldsymbol{Y})=\left\\{\boldsymbol{x}_{i},\boldsymbol{y}_{i}\right\\}_{i=1}^{N}$
for one target task $\mathcal{T}$, where $N$ is the number of training data.
Given a PLM with parameters $\Theta$ and each input text $x_{i}$. The
embedding of $x_{i}\in\boldsymbol{X}$ is represented as
$\mathbf{E}_{i}\in\mathbb{R}^{{m\times d}}$, where $m$ is maximum sequence
length and $d$ is the dimension of input text embedding.
$\mathbf{P}\in\mathbb{R}^{{l\times d}}$ is initialized to form a target
prompt, $l$ is a hyper-parameter for the length of the soft prompt. It is
concatenated with $\mathbf{E}_{i}\in\mathbb{R}^{{m\times d}}$, which does not
involve gradient updates during training, to form a new input embedding
$\left[\mathbf{P};\mathbf{E}_{i}\right]\in\mathbb{R}^{(l+m)\times d}$. The
target task is formulated as follows:

$\displaystyle\mathcal{L}_{PT}=-\sum_{i}\log
P\left(\mathbf{y}_{i}|\left[\mathbf{P};\mathbf{E}_{i}\right];\Theta\right)$
(1)

where $\mathcal{L}_{PT}$ is a loss function only optimized with the prompt
$\mathbf{P}$. However, vanilla PT requires training a large number of prompt
tokens (i.e., a larger value of $l$ in $\mathbf{P}$) to achieve the expected
performance [19].

Figure 2:  The overview of the EPT model. Firstly, we decompose the whole soft
prompt into a short prompt and two low-rank matrices. Low-rank matrices are
multiplied and added element-wise to the frozen input text embedding.
Secondly, the Multi-Space Projection Module maps the short prompt to multiple
subspaces to account for the different requirements of downstream tasks. Then,
the Prompt Fusion module enhances the semantic knowledge of the short prompt.
Finally, EPT learns a joint representation of prompts (from the Multi-Space
Projection and Prompt Fusion module) to supersede the original prompt. The new
prompt and the updated input text embedding are concatenated to input into the
PLM.

###  2.2 Prompt Decomposition

Most studies have shown that the performance of PT is comparable to full fine-
tuning [26, 36]. However, a challenging issue persists: PT requires training a
substantial number of prompt tokens to achieve competitive performance,
resulting in an increased length of the entire input sequence [19]. It causes
greater resource consumption in the training/inference phase [28]. We
decompose $\mathbf{P}\in\mathbb{R}^{l\times d}$ into a short trainable prompt
$\mathbf{P}_{s}\in\mathbb{R}^{s\times d}$ and two trainable low-rank matrices
($\mathbf{A}\in\mathbb{R}^{m\times r}$ and $\mathbf{B}\in\mathbb{R}^{r\times
d}$) to reduce computing costs. Among them, $r$ is the rank in low-rank
matrices and $r\ll\min(m,d)$, $s$ ($s<l$) is the length of the short prompt,
$d$ is the dimension of input text embedding, $m$ is the maximum sequence
length.

In order to keep the same amount of trainable parameters, the selection of $s$
and $r$ satisfies the equation $l\times d=s\times d+(m+d)\times r$, where $s$
and $r$ are hyper-parameters and $s<l$ when $r>0$. For the decomposition of
the vanilla PT, the specific values of $s$ and $r$ affect each other. For
example, in the T5-base, $d$ (dimension) is 768. If $l$ is 100 and $m$ is 256,
when the length of $\textbf{P}_{s}$ is 60, $r$ is 30 ($60\times
768+(256+768)\times 30$). When the length of $\textbf{P}_{s}$ is 40, $r$ is 45
($40\times 768+(256+768)\times 45$). When $r=0$, $s=l$, the decomposed PT
proposed in this paper degenerates to vanilla PT. The purpose of the low-rank
matrices is to update the frozen input word embedding. When $s=0$, only low-
rank matrices are used to update the frozen input word embedding:

$\displaystyle\mathbf{I}^{up}_{i}=\mathbf{E}_{i}+\mathbf{A}\otimes\mathbf{B}$
(2)

where $\mathbf{A}\otimes\mathbf{B}$ represents the multiplication operation of
$\mathbf{A}$ and $\mathbf{B}$, $\mathbf{I}^{up}_{i}\in\mathbb{R}^{m\times d}$
represents the result of adding $\mathbf{A}\otimes\mathbf{B}$ to the frozen
input text embedding $\mathbf{E}_{i}$.

###  2.3 Prompt Fusion

In this section, we design a novel prompt fusion module to keep the short
prompt efficiency and further compensate for the semantic loss of the
decomposition of the long prompt into a short prompt and two low-rank matrices
in the previous section. Specifically, supposing the short prompt
$\mathbf{P}_{s}$ is directly injected into PLMs (the vanilla prompt has the
same operation). In that case, although shortening the length of the soft
prompt reduces the training time, this still will lead to poor performance of
PT because of the lack of knowledge of the original prompt $\mathbf{P}$. PT
requires a substantial number of prompt tokens (exceeding 100) to achieve
optimal performance [19]. Therefore, enriching the knowledge of the decomposed
soft prompt becomes exceptionally crucial while reducing the training time.

Building upon this foundation, we first leverage an attention network to
consider the difference in knowledge richness between low-rank matrices and
the short prompt. Then, we add the short prompt with the output of the
attention network to enhance the knowledge of the original short prompt:
beginequation

$\displaystyle\textbf{W}_{attn}=\mbox{softmax}(\frac{1}{\sqrt{d}}\mathbf{P}_{s}\cdot(\mathbf{A}\otimes\mathbf{B})^{\top})$
(3)
$\displaystyle\mathbf{P}_{f}=\mathbf{P}_{s}+Ein(\textbf{W}_{attn}\cdot\mathbf{P}_{s})$
(4)

where $(\mathbf{A}\otimes\mathbf{B})^{\top}$ is the transpose of
$\mathbf{A}\otimes\mathbf{B}$, $\textbf{W}_{attn}$ is the weighted vector
representation, and $Ein(\cdot)$ is the Einstein Summation (the way the
dimensions change is ${}^{\prime}bpl,bpd\rightarrow bpd^{\prime}$). The
attention mechanism $\textbf{W}_{attn}\cdot\mathbf{P}_{s}$ considers the
knowledge association between low-rank matrices and $\mathbf{P}_{s}$.
$\mathbf{P}_{f}\in\mathbb{R}^{m\times d}$ enhances the knowledge within the
original short prompt based on reducing the consumption of computing
resources.

###  2.4 Multi-Space Projection

In this section, we propose the multi-space projection module to project a
single prompt into multiple subspaces to solve the performance inconsistency
problem of the original PT only fine-tuning in a single space, which reweights
the prompt representations in different spaces through a gating network at
each downstream task. Text information in text classification tasks usually
involves different types and degrees of difficulty (such as Natural Language
Inference, Question Answering, etc.). However, PT is inputted into PLMs in the
same embedding space to adapt to downstream tasks, and a single space does not
consider the different requirements in downstream tasks. This results in
potentially inconsistent performance of PT - as it performs well on some tasks
and poorly on others. The Mixture-of-Experts [16] provides an excellent idea
to solve the aforementioned problem. Motivated by this, we map
$\mathbf{P}_{s}$ to distinct spaces and utilize a gating network to control
each space’s weight distribution. Prompt tokens are assigned different degree
weights by achieving the parameter selection:

$\displaystyle
E_{i}(\mathbf{P}_{s})=\mbox{linear}_{1}\left(\sigma\left(\mbox{linear}_{2}\left(\mathbf{P}_{s}\right)\right)\right),i\in[1,...,N_{e}]$
(5)

where $E_{i}(\mathbf{P}_{s})\in\mathbb{R}^{s\times d}$ is the $i$-th space,
$\mbox{linear}_{1}\in\mathbb{R}^{m\times d}$,
$\mbox{linear}_{2}\in\mathbb{R}^{d\times m}$, $N_{e}$ is the maximum number of
spaces, the activation function $\sigma(\cdot)$ is a ReLU [18] function. The
gate network is formulated as follows:

$\displaystyle\begin{split}&f_{i}(\mathbf{P}_{s})=\mbox{linear}(\mathbf{P}_{s}),i\in[1,...,N_{e}]\\\
&G_{i}(\mathbf{P}_{s})=\frac{\mbox{exp}^{f_{i}(\mathbf{P}_{s})}}{\sum^{N_{e}}_{i=1}\;\mbox{exp}^{f_{i}(\mathbf{P}_{s})}}\end{split}$
(6)

where $G_{i}(\mathbf{P}_{s})\in\mathbb{R}^{s\times 1}$ is used to control the
importance of each space, $\mbox{linear}\in\mathbb{R}^{d\times 1}$.
Reweighting each space by leveraging a gating mechanism:

$\displaystyle\mathbf{P}_{amend}=\sum_{i=1}^{N_{e}}\;G_{i}(\mathbf{P}_{s})\cdot
E_{i}(\mathbf{P}_{s})$ (7)

where $\mathbf{P}_{amend}\in\mathbb{R}^{s\times d}$ is the result of
reweighting $\mathbf{P}_{s}$. $G_{i}(\mathbf{P}_{s})$ makes one or more spaces
in an active state better for different parameter selections.

###  2.5 Reconstructed Prompt

In this section, our EPT method integrates prompt representations of the
fusion module in Sec. 2.3 and the multi-space module in Sec. 2.4 to obtain a
joint representation to have both advantages. To be specific, we learn the
joint representation $\mathbf{P}_{new}$ of $\mathbf{P}_{amend}$ and
$\mathbf{P}_{f}$. Weights of $\mathbf{P}_{amend}$ are allocated in different
spaces, and the soft prompt $\mathbf{P}_{f}$ in the prompt fusion module. The
purpose of learning a joint representation of soft prompts is to replace the
original prompt $\mathbf{P}$ with $\mathbf{P}_{new}$:

$\displaystyle\mathbf{P}_{new}=\mathbf{P}_{amend}+\mathbf{P}_{f}$ (8)

when the initialized $\mathbf{P}_{s}$ performs poorly on specific tasks,
$\mathbf{P}_{amend}$ and $\mathbf{P}_{f}$ redistribute the importance of
$\mathbf{P}_{s}$. After learning $\mathbf{P}_{new}$, the constructed network
is discarded, and $\mathbf{P}_{new}$ is utilized for training in the PLM. By
$\mathbf{P}_{new}$ and $\mathbf{I}^{up}_{i}$, Eq.(1) is displaced by:

$\displaystyle\mathcal{L}_{PT}=-\sum_{i}\;\log
P(\mathbf{y}_{i}|[\mathbf{P}_{new};\mathbf{I}^{up}_{i}];\mathbf{P}_{new})$ (9)

where $[\mathbf{P}_{new};\mathbf{I}^{up}_{i}]$ is a input embedding of PLMs
through the connection of $\mathbf{P}_{new}$ and $\mathbf{I}^{up}_{i}$.

##  3 Experiments

We conduct extensive experiments to answer these key research questions: RQ1:
How does EPT compare with state-of-the-art baselines across different
datasets? RQ2: How do we understand the impact of the critical components of
EPT and model scaling on the performance of EPT? RQ3: How do the few-shot
adaptability and hyper-parameter tuning affect the performance of EPT?

Table 1: The details of the 15 datasets used in our experiment. NLI stands for
natural language inference. GLUE Benchmark
---
Dataset | #Train | #Valid | #Test | Type | Domain | #Metric
MNLI | 392,702 | 9,832 | 9,815 | NLI | various | accuracy
QQP | 362,846 | 1,000 | 40,431 | Paraphrase | social QA questions (Quora) | accuracy
QNLI | 103,743 | 1,000 | 5,463 | NLI | Wikipedia | accuracy
MRPC | 3,668 | 204 | 204 | Paraphrase | news | accuracy
STS-B | 5,749 | 750 | 750 | Sent. Similarity | various | Pearson corr.
SST-2 | 66,349 | 1,000 | 872 | Sentiment | Movie Reviews | accuracy
CoLA | 8,551 | 521 | 522 | Acceptability | various | Matthews corr.
RTE | 2,490 | 138 | 139 | NLI | News, Wikipedia | accuracy
SuperGLUE Benchmark
Dataset | #Train | #Valid | #Test | Type | Domain | #Metric
MulticRC | 27,243 | 2,424 | 2,424 | Question Answering | various | F1
Wic | 5,428 | 319 | 319 | Word Sense Disambiguation | lexical databases | accuracy
WSC | 554 | 52 | 52 | Common Sense Reasoning | fiction books | accuracy
BoolQ | 9,427 | 1,635 | 1,635 | Question Answering | Wikipedia | accuracy
CB | 250 | 28 | 28 | NLI | various | accuracy
ReCoRD | 137,484 | 1,370 | 15,176 | Common Sense Reasoning | news (CNN, Daily Mail) | F1
MRQA 2019 Shared Task
SQuAD | 87,599 | 10,570 | - | Question Answering | Wikipedia | F1

###  3.1 Evaluation Datasets and Source Tasks

We conducted multi-angle experiments on the EPT method to demonstrate its
outstanding applicability to 13 publicly available NLP tasks (8 from the GLUE
benchmark 111https://huggingface.co/datasets/glue and 5 from the SuperGLUE
benchmark 222https://huggingface.co/datasets/super_glue). Specifically, (1)
GLUE [34] is a benchmark for evaluating natural language understanding
performance. It consists of diverse tasks that test the model’s ability to
understand language in different contexts. To fully prove the performance
effect of EPT, we maintain consistency with previous work, and the NLP
datasets are MNLI [38], QQP [34], QNLI [25], SST-2 [29], STS-B [3], MRPC [8],
RTE [10] and CoLA [37] from GLUE. (2) SuperGLUE [35] is an extension of GLUE,
that includes more complex and challenging tasks. This paper uses five tasks
from SuperGLUE: MultiRC [17], BoolQ [5], WiC [22], WSC [20] and CB [6]. We
follow the previous working setup [30, 1, 28], which only utilizes ReCoRD [41]
and SQuAD [25] in the few-shot experiment. Tab. 1 shows the complete
statistics of all experimental datasets, including the size of training,
validation, and test sets, the task type, the domain to which each dataset
belongs, and the evaluation metrics. There are many datasets in the GLUE and
SuperGLUE benchmarks, and we use average performance to compare the
improvement rate.

###  3.2 Baselines

We focus on exploring a high-performance and less training parameter method of
PEFT, so the number of training parameters is also an essential factor.
Methods such as KronA [9], S4 [4], etc. have more training parameters, for
example, the training parameter of PT is 0.1% of full fine-tuning, while the
training parameter of MAM adapter [12] is 6.7% of full fine-tuning. Therefore,
we focus more on the latest methods of PT-type in the baseline selection.

The baselines for comparison with EPT are: (1) Full Fine-tuning (FT), which
updates all parameters of PLMs. (2) PEFT approaches, including Adapter [13],
AdapterDrop [27], BitFit [40], LoRA [14], LST [31], HyperFormer [21], and
HyperDecoder [15]. Notably, multi-task learning variants include FT, Adapter,
HyperFormer, and HyperDecoder. (3) PT-based method, where the vanilla PT [19]
updates parameters with prompt prefix to accommodate various downstream tasks,
and its variants include SPoT [33], ATTEMPT [1], MPT [36], and their transfer
and multi-task learning variants. SPoT and ATTEMPT find optimal prompt
initializations by pre-training prompts on informative source tasks. (4)
Prompt decomposition, DEPT [28], is a parameter-efficient method that
decomposes the soft prompt.

###  3.3 Training Detail Settings

Implementation details: The EPT is implemented by the Python library of
PyTorch 1.13.1333https://pytorch.org/, Huggingface Transformers
444https://github.com/huggingface/transformers, and Huggingface PEFT
555https://github.com/huggingface/peft. The main experiments of EPT and
baseline are performed using the T5-Base model [28], which has a parameter
size of 220M and the hidden size $d$ is 768. Consistent with the experimental
setup of DEPT, we decompose the vanilla prompt (parameter size is 76,800) with
the length of prompt tokens of 100. We train for 30,000 epochs on small
datasets with less than 100k training examples and 300,000 epochs on large-
size data with more than 100k examples. We conduct a grid search for learning
rates. For the soft prompt, we search the learning rate ($\alpha_{1}$) within
the set {3e-1, 4e-1, 5e-1}. For the low-rank matrices, we search the learning
rate ($\alpha_{2}$) within the set {1e-04, 5e-4, 5e-03}. Following the DEPT,
we utilize five source tasks - MNLI, QQP, SST-2, SQuAD, and ReCoRD - for the
few-shot experiments. We derive our soft prompt from one of these selected
source tasks to initialize our soft prompt and low-rank matrices. In addition,
we search for the number of spaces and the length of soft prompts, and analyze
their respective effects on EPT in detail in Section 3.6. Finally, the maximum
sequence length of the model is usually set to 256 (the length on SuperGLUE-
MultiRC is set to 348), and we evaluate performance every 1000 steps.

Table 2: Performance comparison on GLUE benchmark, all experimental results are based on the T5-Base model. The evaluation metrics are Pearson correlation for STS-B and accuracy for other tasks. “Param" represents the amount of trainable parameters for each task. Where $\bigstar$ indicates that some tasks utilize the PETL method, $\lozenge$ indicates that some tasks utilize multi-task learning (resulting in the reduction of trainable parameters). The best result is marked in bold. The second-best result is marked with an underline. The numbers under datasets refer to the number of training examples in each dataset. |  | GLUE
---|---|---
|  | MNLI | QQP | QNLI | MRPC | STS-B | SST-2 | CoLA | RTE | Mean
Model | Param | (393K) | (364K) | (105K) | (3.7K) | (7K) | (67K) | (8.5K) | (2.5K) | (%)
Fine-tuning | 220M | 86.8 | 91.6 | 93.0 | 90.2 | 89.7 | 94.6 | 61.8 | 71.9 | 84.9
LoRA | 3.8M | 86.3 | 89.0 | 93.2 | 90.1 | 90.9 | 94.3 | 63.3 | 75.5 | 85.3
LST | 3.8M | 85.6 | 88.8 | 93.3 | 90.4 | 90.7 | 94.1 | 58.1 | 71.9 | 84.1
Adapter | 1.9M | 86.5 | 90.2 | 93.2 | 85.3 | 90.7 | 93.8 | 64.0 | 71.9 | 84.5
AdapterDrop | 1.1M | 86.3 | 90.2 | 93.2 | 86.3 | 91.4 | 93.6 | 62.7 | 71.2 | 84.4
BitFit | 280K | 85.3 | 90.1 | 93.0 | 86.8 | 90.9 | 94.2 | 58.2 | 67.6 | 83.3
ATTEMPT★ | 232K | 84.3 | 90.3 | 93.0 | 85.7 | 89.7 | 93.2 | 57.4 | 73.4 | 83.4
MPT★ | 77.6K | 85.9 | 90.3 | 93.1 | 89.1 | 90.4 | 93.8 | 62.4 | 79.4 | 85.6
PT | 76.8K | 83.4 | 90.2 | 93.1 | 90.1 | 90.2 | 91.9 | 60.7 | 78.8 | 84.8
SPoT★ | 76.8K | 85.4 | 90.1 | 93.0 | 79.7 | 90.0 | 93.4 | 57.1 | 69.8 | 82.3
DEPT | 76.8K | 85.0 | 90.4 | 93.2 | 90.7 | 90.8 | 94.2 | 63.8 | 79.1 | 85.9
EPT (ours) | 76.8K | 85.8 | 90.3 | 93.3 | 90.2 | 91.1 | 94.5 | 66.9 | 82.0 | 86.8
Fine-tuning◆ | 28M | 85.7 | 91.1 | 92.0 | 90.2 | 88.8 | 92.5 | 54.9 | 75.4 | 83.8
Adapter◆ | 1.8M | 86.3 | 90.5 | 93.2 | 90.2 | 89.9 | 93.0 | 61.5 | 70.3 | 84.4
HyperDecoder◆ | 1.8M | 86.0 | 90.5 | 93.4 | 87.7 | 90.5 | 94.0 | 55.9 | 71.7 | 83.7
HyperFormer◆ | 638K | 85.7 | 90.0 | 93.0 | 87.2 | 89.7 | 94.0 | 63.7 | 75.4 | 84.8
ATTEMPT◆★ | 96K | 83.7 | 90.1 | 93.2 | 87.3 | 90.8 | 94.3 | 64.3 | 82.7 | 85.8
MPT◆★ | 10.5K | 84.3 | 90.0 | 93.0 | 89.2 | 90.4 | 93.3 | 63.5 | 82.7 | 85.8
Table 3: Performance comparison on SuperGLUE. All experimental results are based on the T5-Base model. The evaluation metrics are F1 for MultiRC (Multi) and accuracy for other tasks. The second-best result is marked with an underline. The numbers under datasets refer to the number of training examples in each dataset. |  | SuperGLUE
---|---|---
|  | Multi | Wic | WSC | Bool | CB | Mean
Model | Param | (5.1K) | (6K) | (554) | (9.4K) | (250) | (%)
Fine-tuning | 220M | 72.8 | 70.2 | 59.6 | 81.1 | 85.7 | 73.9
Adapter | 1.9M | 75.9 | 67.1 | 67.3 | 82.5 | 85.7 | 75.7
AdapterDrop | 1.1M | 72.9 | 68.3 | 67.3 | 82.3 | 85.7 | 75.3
BitFit | 280K | 74.5 | 70.0 | 59.6 | 79.6 | 78.6 | 72.5
ATTEMPT★ | 232K | 74.4 | 66.8 | 53.8 | 78.8 | 78.6 | 70.5
PT | 76.8K | 65.7 | 50.8 | 51.9 | 63.7 | 67.9 | 60.0
SPoT★ | 76.8K | 74.0 | 67.0 | 50.0 | 77.2 | 46.4 | 62.9
MPT★ | 77.6K | 74.8 | 69.0 | 67.3 | 79.6 | 79.8 | 74.1
DEPT | 76.8K | 74.3 | 68.7 | 67.3 | 79.3 | 92.9 | 76.5
EPT (ours) | 76.8K | 74.8 | 69.0 | 69.2 | 80.8 | 92.9 | 77.3
ATTEMPT◆★ | 96K | 74.4 | 66.5 | 69.2 | 78.5 | 82.1 | 74.1
MPT◆★ | 10.5K | 74.8 | 70.2 | 67.3 | 79.2 | 89.3 | 76.1

Models: Previous studies have demonstrated that prompt tuning (PT) is close to
full model tuning when model size increases, closing the performance gap when
the model contains over 11B parameters [19]. However, in smaller models, PT
still performs poorly, and its performance may vary significantly depending on
the selection of hyperparameters [33], such as prompt length. Therefore, our
models for evaluating EPT performance are T5-Small (60M), T5-Base (220M), and
T5-Large (770M).

###  3.4  Overall Performance Comparison (RQ1)

Overall, Tab.2 and Tab.3 show the results of EPT and other baselines on the
GLUE and SuperGLUE benchmarks, respectively. Overall, EPT only utilizes a
small number of trainable parameters, and its average performance on the two
benchmarks outperforms 11 PEFT methods, including PT variants based on multi-
task and transfer learning, the visualized results are shown in Fig.1. In
particular, EPT has the best results for CoLA in the GLUE benchmark and WSC
and CB in the SuperGLUE benchmark.

Among all baselines, although the full fine-tuning performs best in some
datasets (MNLI, QQP, SST-2, and SuperGLUE_Wic), the number of parameters
required for training is 2,904 times that EPT, making full fine-tuning
undoubtedly very computationally resource intensive. SPoT, DEPT, and EPT
perform better while keeping the same training parameters as the original PT.
This proves that randomly sampled tokens from the vocabulary for
initialization and then directly injecting them into PLMs cannot make PT well
adaptable to different downstream tasks. EPT and DEPT also utilize decomposing
the soft prompt to reduce computing resources. It is worth noting that the
performance of our proposed EPT surpasses DEPT by 1% on the GLUE benchmark and
by 1.1% on the SuperGLUE benchmark. This affirms the necessity of considering
projecting the short prompt into multiple subspaces improves adaptability to
different downstream tasks with different requirements and utilizing the
prompt fusion module to enrich semantic knowledge of the short prompt.
Additionally, compared to the baseline MPT and ATTEMPT, which are the best-
performing transfer learning methods, EPT performs better. EPT not only does
not require additional pre-training source tasks but also trains fewer
parameters.

Figure 3: Performance of different baselines varies with model scale (from
T5-Small, T5-Base, T5-Large).

(a)  (b)

Figure 4: (a) The performance changes of EPT(Ours), MPT, and PT at different
K-shot on the GLUE benchmark. (b) Comparison of training time consumption and
the performance changes (EPT, DEPT, and PT) according to different lengths of
the short prompt in EPT and DEPT on the GLUE benchmark. (c) and (d) are the
performance comparisons of PT and EPT as the length changes on the CoLA and
BoolQ, respectively.

Unlike DEPT and ATTEMPT, EPT has consistent performance in downstream tasks
with different requirements, whereas they all utilize the attention mechanism.
Additionally, SPoT and ATTEMPT only consider the relationship between source
prompts of different tasks. EPT enhances the semantic knowledge of the short
prompt through the prompt fusion module and improves the adaptability to
downstream tasks with different requirements by reweighting the short prompt
in the multi-space projection module, which is why EPT performs the best. Full
fine-tuning performs best in some datasets, such as MNLI and QQP. We analyze
that EPT is more efficient in datasets with fewer training samples. Overall,
in the GLUE benchmark, our optimal baseline DEPT is only 0.3% higher than MPT
in single-task setting. DEPT is only 0.5% better than MPT on multi-task
setting on the SuperGLUE benchmark. On the contrary, on the GLUE benchmark,
our proposed EPT outperforms DEPT by 1% and vanilla PT by 2.4%. On the
SuperGLUE benchmark, EPT outperforms DEPT by a relative 1.1% and vanilla PT by
a relative 28.8%. Therefore, while training time decreased by 14%, the degree
of performance improvement is already very noticeable.

###  3.5 Ablation Experiment Analysis (RQ2)

Analysis the Critical Components of EPT.

To verify the contribution of each critical component (Prompt Decomposition,
Prompt Fusion, and Multi-Space Projection) in EPT. We divided EPT into five
different variants for ablation experiments, as shown in Tab.4. Overall, the
result of EPT considering all critical components (i.e., the last line) is the
most outstanding. The lack of any critical component in EPT significantly
reduces performance, proving that each critical component positively impacts
EPT. Compared to the vanilla PT, the performance improvement of EPT is 2.4% on
the GLUE benchmark and 28.8% on the SuperGLUE benchmark. When not considering
all critical components (i.e., the first line), EPT is a vanilla PT. When
using the prompt fusion or multi-space projection module, EPT is superior to
only performing the prompt decomposition. This again proves the effectiveness
of the prompt fusion and multi-space projection module.

Table 4: Performance comparison on the critical components of EPT on GLUE and SuperGLUE benchmarks. Prompt | Prompt | Multi-Space | GLUE | SuperGLUE
---|---|---|---|---
Decomposition | Fusion | Projection | (%) | (%)
✗ | ✗ | ✗ | 84.8 | 60.0
✓ | ✗ | ✗ | 85.9 | 76.5
✓ | ✓ | ✗ | 86.5 | 76.7
✓ | ✗ | ✓ | 86.4 | 77.1
✓ | ✓ | ✓ | 86.8 | 77.3

Power of Model Scale. We empirically analyze the impact of the model scale
(T5-Small, T5-Base, T5-Large) on the performance of different baselines (PT,
ATTEMPT, Adapter, Fine-tuning, MPT, and EPT) on BoolQ and MultiRC in the
SuperGLUE benchmark. As shown in Fig.3, the performance get improvements with
the increase of the model scale. This is aligned with the findings of [19].
When the model specification is T5-Large, the performance of EPT on MultiRC
also is optimal. Furthermore, the performance of EPT outperforms the other
baselines on the SuperGLUE benchmark with T5-Large, and the parameters
required to train EPT are far less than full fine-tuning and Adapter.

###  3.6 Few-shot Adaptation and Sensitivity of the Hyper-parameter (RQ3)

Few-shot adaptation. Following previous work [1, 36, 28], we pre-trained the
soft prompt and the low-rank matrices on source tasks. We evaluate the
performance of EPT, vanilla PT, and MPT in $k$-shot ($k=4,16,32$) on the GLUE
benchmark. As shown in Fig.4(a), the performance improvement of EPT is mainly
due to using the PETL framework for pre-training source prompts. EPT
outperforms other variants of PT under few-shot learning tasks, which proves
its effectiveness.

The Length of Soft Prompt. For our EPT method, we keep the same number of
trainable parameters (76,800) as vanilla PT (length is 100), so when comparing
the time cost on EPT and PT, we only compare the decomposed EPT with the
original PT (length is 100). Fig.4(b) shows that EPT takes more training time
as the length of the short prompt increases. When the length of the short
prompt is 60, EPT has the best performance on the GLUE benchmark, and the
training time of EPT is 14% lower than that of the original PT.

On the GLUE benchmark, EPT significantly outperforms DEPT and PT at different
prompt’s lengths (except for length 0). When the prompt’s length is 0, the
soft prompt is only decomposed into two low-rank matrices and the prompt
fusion and multi-space projection module in EPT cannot work. Therefore, EPT
and DEPT have the same performance. When the length of the soft prompt is 0,
the parameters are frozen and not updated, so the vanilla PT is no performance
result. When the length of soft prompt is 100, EPT is superior to DEPT because
the short prompt is mapped to different subspaces to reweight prompt tokens,
which positively impacts EPT. This also proves that the vanilla PT is
difficult to adapt downstream tasks with different requirements by utilizing
the same embedding space to fine-tune.

Due to the GLUE and SuperGLUE benchmarks, which include many datasets, using
average performance to compare improvement rates may create an illusion of
non-significant parameter influence. As shown in Fig.4(c) and Fig.4(d), To
display the performance of EPT more intuitively, we randomly compared the
performance of vanilla PT and EPT on the CoLA and BoolQ datasets. EPT performs
better than PT under different prompt lengths.

The Number of Spaces in the Multi-Space Projection Module. To eliminate the
noise generated by the prompt fusion module, when analyzing the impact of
changes in the number of spaces on performance, we only leverage a multi-space
projection module that learns the reweighted short prompt. As shown in Fig.5,
we dynamically alter the number of spaces $N$ from 2 to 8 with a step size of
1 during training. Overall, there are many datasets in both the GLUE and
SuperGLUE benchmarks, so the fluctuations in EPT on the two benchmarks are
small. The increase in the number of spaces requires more computing resources.
Hence, the number of spaces we comprehensively selected is 4.

Figure 5: Performance of the number of spaces in the Multi-Space Projection
module on the GLUE and SuperGLUE benchmarks.

###  3.7 Interpretability

Accuracy vs. efficiency Fig.1 visualizes the relationship between all baseline
performance and the amount of training parameters on the GLUE and SuperGLUE
benchmarks. EPT utilize the efficient joint representation of prompts to
displace the original prompt, and the trainable parameters input to PLMs
remain consistent with ordinary prompts. We experimentally demonstrate that
EPT training with fewer parameters, outstanding performance, and strong
adaptability to different downstream tasks.

Interpretability of Weights in Spaces. As shown in Fig.6, after mapping the
short prompt into distinct spaces, we visualized the weight allocation of the
gating network to different prompt tokens. We are comparing the weight
distribution of the original prompt tokens initialized in the existing
vocabulary on four datasets (i.e., CoLA, BoolQ, CB, and RTE). It is evident
that different downstream tasks exhibit distinct attention levels towards
tokens in prompts, which proves that reweighting short prompt tokens in the
multi-space projection module is indispensable for improving the stability of
PT.

##  4 Related Works

Parameter-efficient Fine-tuning. Parameter-efficient fine-tuning approaches
can adapt well to various downstream tasks by updating a limited number of
training parameters compared to full fine-tuning. AdapterDrop [27] dynamically
dropping the Adapter reduces the number of model parameters as much as
possible and improves the efficiency of model training/inference. BitFit [40]
only updates the mask layer parameters of PLMs. Diff pruning [11] learns a
task-specific “diff” vector that extends the original pre-trained parameters.
LoRA [14] only updates the parameters of low-rank matrix pairs. HyperDecoder
[15] efficient adaptation of parameters for decoder generation using a hyper-
network conditioned on encoder output in multi-task. LST [31] aims to reduce
the training memory by a ladder-side network for transformers. Prompt tuning
(PT) is a promising parameter-efficient fine-tuning (PEFT) approach, as its
parameters do not exhibit dramatic growth even when the model size expands
significantly.

PT-based methods. The expansion in PLMs size does not lead to a surge in the
training parameters of PT. The recent research aims to improve the performance
of PT through various approaches. SPoT [33] learns one or more source prompts,
constructing the interaction with the target task to initialize the target
prompt. ATTEMPT [1] considers the impact of knowledge in the source prompts on
the input sequence to generate different attention weights, achieving
weighting target prompts. MPT [36] decomposes each source prompt into a one-
rank matrix, performs Hadamard product with shared prompts to construct
student prompts, and then improves the performance of PT through knowledge
distillation. DPT [39] initializes a soft prompt to reduce the number of
trainable parameters by utilizing singular value decomposition (SVD). Most
relevant to this paper, DEPT [28] decomposes the soft prompt into a shorter
prompt and low-rank matrix pairs and utilizes low-rank matrix pairs to update
the input embedding. These variants, which are built upon soft prompts, have
exhibited remarkable performance. However, these PT-based methods still
struggle to balance efficiency and accuracy. Moreover, they typically work in
a single space, thus resulting in performance inconsistencies across different
downstream tasks.

(a)

(b)

(c)

(d)

(e)

Figure 6: The short prompt (length is 20) maps into four different subspaces
and the attention level of each short prompt token is compared with
initialization (derived from an existing vocabulary) on four datasets (i.e.,
CoLA, BoolQ, CB, and RTE).

##  5 Conclusions and Future Work

In this work, we propose an efficient soft prompt tuning (EPT) method by
prompt fusion and multi-space projection. Specifically, the prompt fusion
module can help enhance the semantic of the soft prompt, leading to a balance
between accuracy and efficiency. The multi-space module projects a single soft
prompt into multiple subspaces with reweighted prompt tokens, improving the
performance consistency. Experimental results across three model architectures
(T5-Small, T5-Base, T5-Large) demonstrate that EPT reduces training time,
achieves optimal and consistent performance using the shorter soft prompt, and
validates the effectiveness of critical components in EPT.

For future work, we will address the computational overhead introduced by
using two learning rates in EPT for parameter search. Furthermore, we intend
to explore the integration of EPT with soft prompt methods based on multi-task
transfer learning, aiming to reduce training parameters further while
maintaining optimal performance.

## References

  * Asai et al. [2022] A. Asai, M. Salehi, M. E. Peters, and H. Hajishirzi.  Attempt: Parameter-efficient multi-task tuning via attentional mixtures of soft prompts.  In _EMNLP 2022_ , pages 6655–6672, 2022.
  * Brown et al. [2020] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Cer et al. [2017] D. Cer, M. Diab, E. E. Agirre, I. Lopez-Gazpio, and L. Specia.  Semeval-2017 task 1: Semantic textual similarity multilingual and cross-lingual focused evaluation.  In _SemEval 2017_ , pages 1–14, 2017.
  * Chen et al. [2022] J. Chen, A. Zhang, X. Shi, M. Li, A. Smola, and D. Yang.  Parameter-efficient fine-tuning design spaces.  In _ICLR 2022_ , 2022.
  * Clark et al. [2019] C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova.  Boolq: Exploring the surprising difficulty of natural yes/no questions.  In _NAACL-HLT 2019_ , pages 2924–2936, 2019.
  * De Marneffe et al. [2019] M.-C. De Marneffe, M. Simons, and J. Tonhauser.  The commitmentbank: Investigating projection in naturally occurring discourse.  In _SuB_ , volume 23, pages 107–124, 2019.
  * Devlin et al. [2019] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _NAACL-HLT 2019_ , pages 4171–4186, 2019.
  * Dolan and Brockett [2005] B. Dolan and C. Brockett.  Automatically constructing a corpus of sentential paraphrases.  In _IWP 2005_ , 2005.
  * Edalati et al. [2022] A. Edalati, M. Tahaei, I. Kobyzev, V. P. Nia, J. J. Clark, and M. Rezagholizadeh.  Krona: Parameter efficient tuning with kronecker adapter.  _arXiv preprint arXiv:2212.10650_ , 2022.
  * Giampiccolo et al. [2007] D. Giampiccolo, B. Magnini, I. Dagan, and W. B. Dolan.  The third pascal recognizing textual entailment challenge.  In _Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing_ , pages 1–9, 2007.
  * Guo et al. [2021] D. Guo, A. M. Rush, and Y. Kim.  Parameter-efficient transfer learning with diff pruning.  In _ACL-IJCNLP 2021_ , pages 4884–4896, 2021.
  * He et al. [2021] J. He, C. Zhou, X. Ma, T. Berg-Kirkpatrick, and G. Neubig.  Towards a unified view of parameter-efficient transfer learning.  _arXiv preprint arXiv:2110.04366_ , 2021.
  * Houlsby et al. [2019] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly.  Parameter-efficient transfer learning for nlp.  In _ICML 2019_ , pages 2790–2799, 2019.
  * Hu et al. [2021] E. J. Hu, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al.  Lora: Low-rank adaptation of large language models.  In _ICLR 2021_ , 2021.
  * Ivison and Peters [2022] H. Ivison and M. E. Peters.  Hyperdecoders: Instance-specific decoders for multi-task nlp.  In _EMNLP 2022_ , pages 1715–1730, 2022.
  * Jacobs et al. [1991] R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton.  Adaptive mixtures of local experts.  _Neural computation_ , 3(1):79–87, 1991.
  * Khashabi et al. [2018] D. Khashabi, S. Chaturvedi, M. Roth, S. Upadhyay, and D. Roth.  Looking beyond the surface: A challenge set for reading comprehension over multiple sentences.  In _NAACL-HLT 2018_ , pages 252–262, 2018.
  * Krizhevsky et al. [2012] A. Krizhevsky, I. Sutskever, and G. E. Hinton.  Imagenet classification with deep convolutional neural networks.  _Advances in neural information processing systems_ , 25, 2012.
  * Lester et al. [2021] B. Lester, R. Al-Rfou, and N. Constant.  The power of scale for parameter-efficient prompt tuning.  In _EMNLP 2021_ , pages 3045–3059, 2021.
  * Levesque et al. [2012] H. Levesque, E. Davis, and L. Morgenstern.  The winograd schema challenge.  In _KR 2012_ , 2012.
  * Mahabadi et al. [2021] R. K. Mahabadi, S. Ruder, M. Dehghani, and J. Henderson.  Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks.  In _ACL-IJCNLP 2021_ , pages 565–576, 2021.
  * Pilehvar and Camacho-Collados [2019] M. T. Pilehvar and J. Camacho-Collados.  Wic: the word-in-context dataset for evaluating context-sensitive meaning representations.  In _NAACL-HLT 2019_ , pages 1267–1273, 2019.
  * Radford et al. [2019] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Raffel et al. [2020] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67, 2020.
  * Rajpurkar et al. [2016] P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang.  Squad: 100,000+ questions for machine comprehension of text.  In _EMNLP 2016_ , pages 2383–2392, 2016.
  * Razdaibiedina et al. [2023] A. Razdaibiedina, Y. Mao, M. Khabsa, M. Lewis, R. Hou, J. Ba, and A. Almahairi.  Residual prompt tuning: improving prompt tuning with residual reparameterization.  In _ACL 2023_ , pages 6740–6757, 2023.
  * Rücklé et al. [2021] A. Rücklé, G. Geigle, M. Glockner, T. Beck, J. Pfeiffer, N. Reimers, and I. Gurevych.  Adapterdrop: On the efficiency of adapters in transformers.  In _EMNLP 2021_ , pages 7930–7946, 2021.
  * Shi and Lipani [2023] Z. Shi and A. Lipani.  Dept: Decomposed prompt tuning for parameter-efficient fine-tuning.  _arXiv preprint arXiv:2309.05173_ , 2023.
  * Socher et al. [2013] R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y. Ng, and C. Potts.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _EMNLP 2013_ , pages 1631–1642, 2013.
  * Su et al. [2022] Y. Su, X. Wang, Y. Qin, C.-M. Chan, Y. Lin, H. Wang, K. Wen, Z. Liu, P. Li, J. Li, et al.  On transferability of prompt tuning for natural language processing.  In _NAACL-HLT 2022_ , pages 3949–3969, 2022.
  * Sung et al. [2022] Y.-L. Sung, J. Cho, and M. Bansal.  Lst: Ladder side-tuning for parameter and memory efficient transfer learning.  _Advances in Neural Information Processing Systems_ , 35:12991–13005, 2022.
  * Vaswani et al. [2017] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin.  Attention is all you need.  In _NeurIPS 2017_ , pages 6000–6010, 2017.
  * Vu et al. [2022] T. Vu, B. Lester, N. Constant, R. Al-Rfou, and D. Cer.  Spot: Better frozen model adaptation through soft prompt transfer.  In _ACL 2022_ , pages 5039–5059, 2022.
  * Wang et al. [2018] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  In _EMNLP 2018_ , pages 353–355, 2018.
  * Wang et al. [2019] A. Wang, Y. Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman.  Superglue: a stickier benchmark for general-purpose language understanding systems.  In _NeurIPS 2019_ , pages 3266–3280, 2019.
  * Wang et al. [2022] Z. Wang, R. Panda, L. Karlinsky, R. Feris, H. Sun, and Y. Kim.  Multitask prompt tuning enables parameter-efficient transfer learning.  In _ICLR 2022_ , 2022.
  * Warstadt et al. [2019] A. Warstadt, A. Singh, and S. R. Bowman.  Neural network acceptability judgments.  _Transactions of the Association for Computational Linguistics_ , 7:625–641, 2019.
  * Williams et al. [2018] A. Williams, N. Nangia, and S. Bowman.  A broad-coverage challenge corpus for sentence understanding through inference.  In _NAACL-HLT 2018_ , pages 1112–1122, 2018.
  * Xiao et al. [2023] Y. Xiao, L. Xu, J. Li, W. Lu, and X. Li.  Decomposed prompt tuning via low-rank reparameterization.  In _EMNLP 2023_ , 2023.
  * Zaken et al. [2022] E. B. Zaken, Y. Goldberg, and S. Ravfogel.  Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 1–9, 2022.
  * Zhang et al. [2018] S. Zhang, X. Liu, J. Liu, J. Gao, K. Duh, and B. Van Durme.  Record: Bridging the gap between human and machine commonsense reading comprehension.  _arXiv preprint arXiv:1810.12885_ , 2018.
  * Zhang et al. [2022] S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al.  Opt: Open pre-trained transformer language models.  _arXiv preprint arXiv:2205.01068_ , 2022.