# Tied-LoRA : Enhancing parameter efficiency of LoRA with Weight Tying

Adithya Renduchintala
&Tugrul Konuk

NVIDIA
<EMAIL_ADDRESS>
&Oleksii Kuchaiev

###### Abstract

We propose Tied-LoRA , a simple paradigm utilizes weight tying and selective
training to further increase parameter efficiency of the Low-rank adaptation
(LoRA) method. Our investigations include all feasible combinations parameter
training/freezing in conjunction with weight tying to identify the optimal
balance between performance and the number of trainable parameters. Through
experiments covering a variety of tasks and two base language models, we
provide analysis revealing trade-offs between efficiency and performance. Our
experiments uncovered a particular Tied-LoRA configuration that stands out by
demonstrating comparable performance across several tasks while employing only
13 % percent of parameters utilized by the standard LoRA method.

##  1 Introduction

Large language models (LLMs) are increasingly employed as the fundamental
component in a wide array of applications that require proficiency in Natural
Language Processing (NLP) tasks. A key contributor to this widespread adoption
is the capacity to fine-tune pretrained LLMs for downstream tasks in a
parameter-efficient manner. This fine-tuning procedure enables the development
of specialized language models that excel on particular domains and tasks.
Despite smaller training data sizes compared to pretraining, finetuning still
demands considerable computational resources, particularly for large models
containing billions of parameters. Furthermore, it is often desirable to
maintain several different customization for each combination of user $\times$
task of the service, many of which are often served simultaneously. As the
number of users and tasks per user grow, so do the costs associated with
customization. Thus, making efficient use of customizable parameters
paramount.

$W\in\mathbb{R}^{d\times 3d}$$\mathbf{A}\in\mathbb{R}^{d\times
r}$$\mathbf{v}\in\mathbb{R}^{3d\times 1}$$\mathbf{B}\in\mathbb{R}^{r\times
3d}$$x$$z$$\mathbf{u}\in\mathbb{R}^{r\times 1}$ Figure 1: Schematic of our
Tied-Lora paradigm, the main low-rank matrices $A$ and $B$ are tied across
(indicated by the symbol) all the layers of the base language model. We use
the gradient shading to indicate that these parameters can either be trained
or frozen.

Low-rank Adaptation (LoRA) (Hu et al., 2021) has emerged as a popular
parameter-efficient finetuning (PEFT) method because of its straightforward
implementation and the ability to merge LoRA weights into the base model.
However, despite its advantages, LoRA training can still be expensive,
especially as the base models become increasingly larger. While prior work has
attempted to make LoRA more parameter efficient, they concentrated on
appropriate low-rank selection. However, we introduce a novel approach,
Instead of controlling the number of parameters by the rank, we employ simple
weight tying coupled with selective training. By integrating these two core
ideas, we propose a range of Tied-LoRA configurations and study the
performance of each configuration on five diverse customization tasks.

Our contributions are threefold:

  1. 1.

We propose a range of Tied-LoRA configurations that use simple weight tying in
LoRA along with selective training to boost the parameter efficiency of LoRA.

  2. 2.

We study this spectrum of possible Tied-LoRA configurations on diverse tasks
that resemble real-world customization problems.

  3. 3.

Based on the results of our study, we propose the specific
$\mathbf{v}\mathbf{B}_{{}_{\text{\faChain}}}\mathbf{u}\mathbf{A}_{{}_{\text{\faChain}}}$
configuration as the best option for maintaining performance while reducing
parameters by$~{}87\%$.

##  2 Related Work

#### Parameter-efficient fine-tuning (PEFT).

Recent work on PEFT of pretrained language models has shown competitive
capabilities, often matching full fine-tuning performance for task-specific
model customization while utilizing significantly fewer trainable parameters
(Houlsby et al., 2019; Lin et al., 2020; Pfeiffer et al., 2021; Rücklé et al.,
2021; Liu et al., 2022).

#### Low-Rank adaptation (LoRA).

One of the most popular PEFT techniques is LoRA, introduced by Hu et al.
(2021). LoRA employs low-rank matrix approximations of full weights’ gradient-
descent (GD) update to significantly reduce the number of trainable
parameters. Importantly, LoRA can incorporate the low-rank updates into the
frozen base weights after the fine-tuning process, avoiding any inference
speed penalties or model architecture changes. In summary, LoRA paves the way
for efficient fine-tuning for task-specific customization of large models with
minimal computational overhead and no changes to the model’s architecture.

#### Extensions to LoRA.

Since its arrival, there have been several efforts to improve the LoRA method.
These methods mostly concentrated around reducing the trainable parameters and
memory footprint while increasing the performance of the method on downstream
tasks. AdaLoRA (Zhang et al., 2023) introduces dynamic rank adjustment for the
low-rank matrices during the fine-tuning process. The fundamental premise of
this extension is to optimally distribute the parameter budget over model
layers. Chavan et al. (2023) combined the adapter tuning with LoRA to derive a
generalized framework that utilized both methods for increased flexibility and
capability across a wide variety of tasks and datasets. Kopiczko et al. (2023)
proposes the VeRA method the freezes randomly initialized projection matrices
and introduces trainable scaling vectors that vary across layers. This method
shows similar performance to the
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{u}\mathbf{A}$(LoRA)
method while dramatically reducing the number of trainable parameters. Our
work draws significant inspiration from the principles of the VeRA method.

Tangential to the efforts that aim to reduce trainable parameters, QLoRA
(Dettmers et al., 2023), significantly reduces the memory usage of LoRA using
a 4-bit or 8-bit quantized base language model during training. The method
provides algorithms and custom kernels to backpropagate gradients through the
frozen, quantized base model to update low-rank matrices during training,
resulting in considerable reduction in memory usage. Combining quantization
and reduction in the number of trainable parameters is a direction of future
work.

#### Weight tying.

Weight tying (Press and Wolf, 2017) is a common approach that reduces the
number of parameters by using the same set of weights for both the input word
embedding layer and the output word embedding layer (sometimes referred to as
the language model head). In this study, we apply weight tying to the low-rank
weight matrices used in LoRA, and share them across the layers of the base
language model. This simple procedure leads to efficient training methods
where the number of trainable parameters are either unaffected by, or only
increases marginally with the number of hidden layers. As models get deeper
this approach naturally provides greater parameter reduction over original
LoRA method.

##  3 Method

In this section, we introduce tied
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{u}\mathbf{A}$(LoRA)
, a generalized paradigm for parameter-efficient fine-tuning of large language
models through low-rank weight-update approximations. Our framework offers a
range of training strategies through a series of design choices over selective
parameter training and weight tying, including some of the existing PEFT
methodologies available in the literature. Specifically, we use weight tying
alongside pairs of projection matrices and scaling vectors that can be
selectively either trained or frozen. As the low-rank computation path does
not introduce any non-linearity, all Tied-LoRA configurations can be merged
into the base model weights to preventing additional latency during inference.
Table 1 provides an overview of the scenarios we study.

###  3.1 Formulation

The overall structure of the tied LoRA framework can be seen in Figure 1. Note
that the original LoRA (Hu et al., 2021) uses a dedicated pair of low-rank
projections for each of the $Q,K,V$ matrices. However, in our formulation, $W$
is a $d\times 3d$ matrix that jointly projects $Q,K$, and $V$ attention
matrices, where $d$ is the hidden size of the base language model. Therefore,
our down projection $A$ is a $d\times r$ shaped matrix and up projection
matrix $B$ has shape $r\times 3d$, where $r$ is the low-rank bottleneck
dimension. Essentially, the down projection $A$ is _shared_ by $Q,K$, and $V$,
leading to fewer trainable parameters ($4dr$) than the original LoRA ($6dr$).

For a linear layer with a frozen pretrained weight matrix $W$, we define the
layer output as

$z=Wx+\Delta Wx\approx Wx+\frac{\alpha}{r}\Lambda_{v}B\Lambda_{u}Ax,$ (1)

where $\Delta W$ is the full-rank update matrix, $\alpha$ is a scaling factor,
$A$ and $B$ are low-rank projection matrices, and $\Lambda_{u}$ and
$\Lambda_{v}$ are diagonal matrices with diagonal elements given by $u$ and
$v$, respectively. Herein, $\Lambda_{v}B\Lambda_{u}Ax$ is the low-rank
approximation to the parameter update matrix $\Delta W$. Unlike the original
LoRA, where $\alpha$ is a hyper-parameter that can be manually set, we simply
set $\alpha=r$, effectively removing its scaling effect.

Equation 1 is a generalized formulation for methods that utilize low-rank
approximations to estimate parameter updates. Particular settings of parameter
updates and weight tying reduces this equation to some of the existing
formulations in the literature. Setting and freezing
$\Lambda_{u}=\Lambda_{v}=I$ and untying $A$ and $B$ results in LoRA:

$z=Wx+BAx.$ (2)

Similarly, randomly initializing $A$ and $B$ matrices and tying them across
all layer leads the the VeRA formulation (Kopiczko et al., 2023):

$z=Wx+\Lambda_{v}B\Lambda_{u}Ax,$ (3)

Method | Parameters | Initialization
---|---|---
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{u}\mathbf{A}$(LoRA) | $4Ldr$ | $A\sim\mathcal{N},B=0,u,v=1$
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}_{{}_{\text{\faChain}}}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{u}\mathbf{A}_{{}_{\text{\faChain}}}$ | $4dr$ | $A\sim\mathcal{N},B=0,u,v=1$
$\mathbf{v}\mathbf{B}_{{}_{\text{\faChain}}}\mathbf{u}\mathbf{A}_{{}_{\text{\faChain}}}$ | $4dr+L(r+3d)$ | $A,B\sim\mathcal{N},u=1,v=0$
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}_{{}_{\text{\faChain}}}\mathbf{u}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{A}_{{}_{\text{\faChain}}}$ | $(L+3d)r$ | $A,B\sim\mathcal{N},v,u=1$
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}_{{}_{\text{\faChain}}}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{u}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{A}_{{}_{\text{\faChain}}}$ | $3dr$ | $A,B\sim\mathcal{N},u,v=1$
$\mathbf{v}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{B}_{{}_{\text{\faChain}}}\mathbf{u}\mathbf{A}_{{}_{\text{\faChain}}}$ | $dr+L(r+3d)$ | $A,B\sim\mathcal{N},u=1,v=0$
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{B}_{{}_{\text{\faChain}}}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{u}\mathbf{A}_{{}_{\text{\faChain}}}$ | $dr$ | $A,B\sim\mathcal{N},u,v=1$
$\mathbf{v}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{B}_{{}_{\text{\faChain}}}\mathbf{u}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{A}_{{}_{\text{\faChain}}}$(Vera) | $L(r+3d)$ | $A,B\sim\mathcal{N},u=1,v=0$

Table 1: Tied-LoRA configurations included in our study. The first column shows acronyms used to identify each Tied-LoRA configuration (i.e., method). Symbols with subscript ${}_{{}_{\text{\faChain}}}$ indicate that it is shared across all layers and the color blue indicates that the parameter is frozen. Formulas for the number of trainable parameters in each configuration as a function of number of layers $L$, hidden size $d$, and low-rank $r$ are also provided. Base Model | Method | Avg. Score over tasks @ Rank $r$
---|---|---
2 | 8 | 32 | 128
2B |  $\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{u}\mathbf{A}$(LoRA) | 50.29 (100)$\CIRCLE$ | 51.41 (100)$\CIRCLE$ | 51.16 (100)$\CIRCLE$ | 51.22 (100)$\CIRCLE$
$\mathbf{v}\mathbf{B}_{{}_{\text{\faChain}}}\mathbf{u}\mathbf{A}_{{}_{\text{\faChain}}}$ | 50.25 (41.7)$\CIRCLE$ | 50.77 (13.6)$\CIRCLE$ | 50.73 (6.5)$\CIRCLE$ | 50.95 (4.8)$\CIRCLE$
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}_{{}_{\text{\faChain}}}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{u}\mathbf{A}_{{}_{\text{\faChain}}}$ | 47.80 (4.2) | 49.59 (4.2) | 49.86 (4.2) | 44.24 (4.2)
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}_{{}_{\text{\faChain}}}\mathbf{u}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{A}_{{}_{\text{\faChain}}}$ | 47.35 (3.1) | 48.32 (3.1) | 46.68 (3.1) | 31.94 (3.1)
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}_{{}_{\text{\faChain}}}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{u}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{A}_{{}_{\text{\faChain}}}$ | 46.55 (3.1) | 47.52 (3.1) | 46.86 (3.1) | 31.41 (3.1)
$\mathbf{v}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{B}_{{}_{\text{\faChain}}}\mathbf{u}\mathbf{A}_{{}_{\text{\faChain}}}$ | 49.99 (38.6)$\CIRCLE$ | 49.77 (10.4)$\CIRCLE$ | 48.19 (3.4) | 46.00 (1.6)$\CIRCLE$
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{B}_{{}_{\text{\faChain}}}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{u}\mathbf{A}_{{}_{\text{\faChain}}}$ | 49.79 (1.0) | 49.51 (1.0) | 48.26 (1.0)$\CIRCLE$ | 45.98 (1.0)
$\mathbf{v}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{B}_{{}_{\text{\faChain}}}\mathbf{u}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{A}_{{}_{\text{\faChain}}}$(Vera) | 49.92 (37.5) | 49.23 (9.4) | 47.99 (2.4) | 45.60 (0.6)
7B |  $\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{u}\mathbf{A}$(LoRA) | 58.05 (100)$\CIRCLE$ | 58.64 (100)$\CIRCLE$ | 57.98 (100)$\CIRCLE$ | 58.43 (100)$\CIRCLE$
$\mathbf{v}\mathbf{B}_{{}_{\text{\faChain}}}\mathbf{u}\mathbf{A}_{{}_{\text{\faChain}}}$ | 57.23 (40.6)$\CIRCLE$ | 57.80 (12.5)$\CIRCLE$ | 57.93 (5.5)$\CIRCLE$ | 57.86 (3.7)$\CIRCLE$
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}_{{}_{\text{\faChain}}}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{u}\mathbf{A}_{{}_{\text{\faChain}}}$ | 55.59 (3.1) | 57.53 (3.1)$\CIRCLE$ | 57.47 (3.1)$\CIRCLE$ | 55.53 (3.1)
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}_{{}_{\text{\faChain}}}\mathbf{u}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{A}_{{}_{\text{\faChain}}}$ | 54.79 (2.3) | 56.48 (2.3) | 54.90 (2.3) | 29.00 (3.3)
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}_{{}_{\text{\faChain}}}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{u}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{A}_{{}_{\text{\faChain}}}$ | 54.93 (2.3) | 55.85 (2.3) | 55.09 (2.3) | 28.02 (2.3)
$\mathbf{v}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{B}_{{}_{\text{\faChain}}}\mathbf{u}\mathbf{A}_{{}_{\text{\faChain}}}$ | 57.02 (38.3)$\CIRCLE$ | 56.79 (10.2) | 56.60 (3.1) | 55.66 (1.4)$\CIRCLE$
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{B}_{{}_{\text{\faChain}}}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{u}\mathbf{A}_{{}_{\text{\faChain}}}$ | 57.01 (0.8) | 56.63 (0.8) | 56.45 (0.8) | 55.52 (0.8)
$\mathbf{v}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{B}_{{}_{\text{\faChain}}}\mathbf{u}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{A}_{{}_{\text{\faChain}}}$(Vera) | 56.75 (37.5) | 56.44 (9.4) | 56.65 (2.3) | 55.38 (0.6)
Table 2: Average scores across all tasks at each low rank value for the Tied-
LoRA methods in comparison with LoRA. The fraction of trainable parameters
used by each method compared to
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{u}\mathbf{A}$(LoRA)
is show in brackets. We indicate the first, second and third highest scoring
method for each low-rank setting with $\CIRCLE$, $\CIRCLE$and $\CIRCLE$,
respectively.

###  3.2 Weight Tying

The third column of Table 1 presents representations for number of trainable
parameters each Tied-Lora configuration requires. As is apparent from the
table, weight tying is a critical ingredient of our proposed approach which
drastically reduces the number of trainable parameters. For example,
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{u}\mathbf{A}$(LoRA)
training using the 7B LLaMA-2 (Touvron et al., 2023) language model with a
typical low rank setting of $8$ requires $\sim 4.2$M trainable parameters. By
merely introducing weight tying across the $32$ layers of this model reduces
the trainable parameters to $\sim 131$K, which is a $96.875\%$ reduction. In
comparison, the Vera method results in a reduction of $90.6\%$.

###  3.3 Selective Training

Through the flexible framework that equation 1 offers, we are given the
opportunity to investigate a range training configurations. By selectively
updating the components $A,B,u$, and $v$ during the training process, we can
generate a variety of methodological variations. These variations not only
exhibit differences in parameter count, but they also demonstrate distinct
capabilities across a variety of tasks. This exploration allows us to
investigate the intriguing regime of extremely low-parameter and low-rank PEFT
models. This is a key step towards the customization of models, enabling them
to excel at specific tasks while maintaining a minimal parameter count. Our
ultimate goal here is to harness the power of this methodology to create
highly efficient, task-specific models that achieve high performance with
reduced complexity.

##  4 Experiments

We now turn to evaluating the different configurations possible within our
Tied-LoRA paradigm. While
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{u}\mathbf{A}$(LoRA)
and PEFT methods can be used to train models for general instruction following
(Sun et al., 2023; Lermen et al., 2023; Sun et al., 2023), we focus our
evaluations in a “task customization” perspective, where each model is trained
on a specific task and is evaluated on a test set from the same task.

###  4.1 Tasks & Datasets

To evaluate the performance of each Tied-LoRA configuration across diverse
data settings, we utilized the following types of tasks:

#### Extractive QA

is a common task where the model is expected to “read” some relevant text (the
context) and answer questions. The answers are usually exact sub-strings from
the provided context. We use SQuADv1 dataset (Rajpurkar et al., 2016) in our
experiments. Since the official test split of this dataset does not contain
ground-truth answers, we use the validation set as our test set. We create a
validation set comprising of a random sample of $4800$ examples extracted from
the training set.

#### Summarization

is a central problem in NLP and several variations of summarization datasets
have been proposed. We employ the DialogSum dataset (Chen et al., 2021) to
study our models’ performance on this task. DialogSum includes summaries of
real-word conversations on a diverse set of topics and scenarios. This dataset
was an attractive option as the length of the conversations and summarizes are
within the context lengths ($4096$ tokens) of the base language models.

#### Commonsense Natural Language Inference (NLI)

is a task designed to probe the ability of language models to apply
“commonsense reasoning” to choose a possible ending for a given situation
described in natural language. These tasks are typically trivial for humans
but language models can still struggle. We use the HellaSwag dataset (Zellers
et al., 2019) to study the performance of our proposed models on this type of
task. As HellaSwag contains multiple-choice questions, it can be viewed as a
classification problem.

#### Translation

Machine translation is a natural language generation task which is widely used
in research and industry. Translation is inherently multilingual and thus
offers a challenging domain to study our Tied-LoRA paradigm. There are several
large scale translation datasets but we focus on a moderately sized IWSLT 2017
German-to-English translation dataset (Cettolo et al., 2017). The dataset
contains translation of spoken language into various other natural languages.
With over $206k$ training examples this is the largest dataset that we study.

#### Mathematical Reasoning

is a challenging domain where large language models still lag behind human
performance. Using PEFT methods on such tasks further amplifies these
challenges as there are very few trainable parameters. In our experiments, we
use the GSM8K benchmark (Cobbe et al., 2021) which contains $8.5$K high-
quality, grade-school level math word problems. Each example in the GSM8K
benchmark contains a question and an answer. The answers are provided with
natural language solutions which contain explanations of each step used to
obtain the final answer. The final numerical answer is demarcated from the
rest of the natural language solution. We evaluate our models by comparing
these final numerical answers.

$2^{1}$$2^{3}$$2^{5}$$2^{7}$$78$$80$$82$$84$Acc. (a) squad,2B

$2^{1}$$2^{3}$$2^{5}$$2^{7}$$84$$85$$86$$87$$88$Acc. (b) squad,7B

$2^{1}$$2^{3}$$2^{5}$$2^{7}$$35.5$$36$$36.5$$37$$37.5$$38$RougeL (c)
dialogsum,2B

$2^{1}$$2^{3}$$2^{5}$$2^{7}$$38$$39$$40$$41$RougeL (d) dialogsum,7B

$2^{1}$$2^{3}$$2^{5}$$2^{7}$$60$$65$$70$$75$$80$$85$Acc. (e) hellaswag,2B

$2^{1}$$2^{3}$$2^{5}$$2^{7}$$84$$86$$88$$90$$92$Acc. (f) hellaswag,7B

$2^{1}$$2^{3}$$2^{5}$$2^{7}$$38$$39$$40$BLEU (g) iwslt2017,2B

$2^{1}$$2^{3}$$2^{5}$$2^{7}$$39$$39.5$$40$$40.5$$41$$41.5$BLEU (h)
iwslt2017,7B

$2^{1}$$2^{3}$$2^{5}$$2^{7}$$4$$6$$8$$10$$12$Acc. (i) gsm8k,2B

$2^{1}$$2^{3}$$2^{5}$$2^{7}$$22$$24$$26$$28$$30$$32$Acc. (j) gsm8k,7B

(k) *

[]
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{u}\mathbf{A}$(LoRA)
$\mathbf{v}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{B}_{{}_{\text{\faChain}}}\mathbf{u}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{A}_{{}_{\text{\faChain}}}$(Vera)
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{B}_{{}_{\text{\faChain}}}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{u}\mathbf{A}_{{}_{\text{\faChain}}}$
$\mathbf{v}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{B}_{{}_{\text{\faChain}}}\mathbf{u}\mathbf{A}_{{}_{\text{\faChain}}}$
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}_{{}_{\text{\faChain}}}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{u}\mathbf{A}_{{}_{\text{\faChain}}}$
$\mathbf{v}\mathbf{B}_{{}_{\text{\faChain}}}\mathbf{u}\mathbf{A}_{{}_{\text{\faChain}}}$
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}_{{}_{\text{\faChain}}}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{u}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{A}_{{}_{\text{\faChain}}}$
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}_{{}_{\text{\faChain}}}\mathbf{u}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{A}_{{}_{\text{\faChain}}}$

Figure 2: Plots showing the performance of the tied-Lora configurations along
with the baseline
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{u}\mathbf{A}$(LoRA)
for $5$ diverse tasks at $4$ different values for low-rank dimension setting.
Note that we let the plot for
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}_{{}_{\text{\faChain}}}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{u}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{A}_{{}_{\text{\faChain}}}$
and
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}_{{}_{\text{\faChain}}}\mathbf{u}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{A}_{{}_{\text{\faChain}}}$
go out of bounds to show details for the other curves.

###  4.2 Base Language Models

Although PEFT enables the base language model to perform new tasks, the final
performance heavily depends on the inherent abilities learned during
pretraining. This necessitates investigating the performance of Tied-LoRA on
multiple base models with different inherent capabilities. Therefore, we use a
relatively small two billion parameter, GPT-2B-001
model111https://huggingface.co/nvidia/GPT-2B-001 released by NVIDIA and the
moderately large $7$B LLaMA 2 model (Touvron et al., 2023) released by Meta.

In addition to the size differences, these models also differ in the amount of
pretraining data used. The GPT-2B-001 model was trained on $1.1$ trillion
tokens of text from publicly available multilingual text spaning $53$
languages. The LLaMA2 $7$B model was trained on $2$ trillion tokens of
predominately English text. Both models are auto-regressive language models
with a context size of $4096$ tokens.

###  4.3 Implementation Details

We use the open-source NeMo Framework to implement all the algorithms
presented in this paper. Our implementation is publicly available through the
NeMo GitHub repository.222https://github.com/NVIDIA/NeMo/tree/adithyare/vera
All training routines were run for $2k$ max steps, but training was terminated
sooner using early stopping with a patience of $10$ to prevent overfitting. We
trained all configurations using AdamW optimizer (Loshchilov and Hutter, 2017)
with a weight decay of $0.01$ and a cosine learning rate schedule with $50$
warm-up steps. For each Tied-Lora method we tried two learning rates, a high
rate of $1e-4$ and a low learning rate of $1e-5$. While the “typical” range of
the low-rank dimension $r$ is $4-16$ we find that some complex tasks benefit
from higher $r$ so we trained all our models with a wide range of
$r\in\\{2,8,32,128\\}$. Each task was trained with a global batch size of
$256$ and a validation check interval of $30$ steps. The only exception was
the IWSLT translation dataset for which we set global batch size and
validation check interval of $1024$ and $60$ respectively. No extensive hyper-
parameter search was conducted.

During inference, we used greedy-decoding to generate the models’ predictions
with a limit of $500$ tokens.

##  5 Results

Table 2 shows average scores attained by each Tied-Lora configuration over the
$5$ tasks, per low-rank value. We can immediately see that
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{u}\mathbf{A}$(LoRA)
is the best performing model for both the 2B and 7B base language models. This
is hardly surprising as
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{u}\mathbf{A}$(LoRA)
is the most expensive method which does not use tied weights. With this in
mind we see that
$\mathbf{v}\mathbf{B}_{{}_{\text{\faChain}}}\mathbf{u}\mathbf{A}_{{}_{\text{\faChain}}}$
is a consistently the next best performing method with average scores
comparable to
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{u}\mathbf{A}$(LoRA)
, demonstrating the efficacy of weight tying.
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}_{{}_{\text{\faChain}}}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{u}\mathbf{A}_{{}_{\text{\faChain}}}$
however does not perform as well suggesting that the scaling vectors
$\mathbf{u}$ and $\mathbf{v}$ provide an additional boost in performance
especially as the rank $r$ is increased to $128$ (at the cost of more
trainable parameters).

Next best Tied-Lora configuration is
$\mathbf{v}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{B}_{{}_{\text{\faChain}}}\mathbf{u}\mathbf{A}_{{}_{\text{\faChain}}}$
which obtains third place for $6$ out of the $8$ settings shown in Table 2.
Note that
$\mathbf{v}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{B}_{{}_{\text{\faChain}}}\mathbf{u}\mathbf{A}_{{}_{\text{\faChain}}}$
beats other Tied-LoRA methods which use more parameters. Interestingly,
$\mathbf{v}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{B}_{{}_{\text{\faChain}}}\mathbf{u}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{A}_{{}_{\text{\faChain}}}$(Vera)
which uses fewer parameters than
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}_{{}_{\text{\faChain}}}\mathbf{u}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{A}_{{}_{\text{\faChain}}}$
and
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}_{{}_{\text{\faChain}}}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{u}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{A}_{{}_{\text{\faChain}}}$
has better performance.
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}_{{}_{\text{\faChain}}}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{u}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{A}_{{}_{\text{\faChain}}}$
and
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}_{{}_{\text{\faChain}}}\mathbf{u}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{A}_{{}_{\text{\faChain}}}$
does the worst in most cases, especially when $r$ is increased.

Figure 2 shows the performance for each task individually. We see that for
tasks like HellaSwag and SQuAD Tied-LoRA methods
($\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}_{{}_{\text{\faChain}}}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{u}\mathbf{A}_{{}_{\text{\faChain}}}$
and
$\mathbf{v}\mathbf{B}_{{}_{\text{\faChain}}}\mathbf{u}\mathbf{A}_{{}_{\text{\faChain}}}$
specifically) are virtually the same as
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{u}\mathbf{A}$(LoRA)
over the entire range of ranks, fewer parameters.
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}_{{}_{\text{\faChain}}}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{u}\mathbf{A}_{{}_{\text{\faChain}}}$
for example, only uses $4.2\%$ and $3.1\%$ of parameters that
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{u}\mathbf{A}$(LoRA)
uses for the 2B and 7B models, respectively.

On the flip side tasks like GSM8K seem to benefit from the additional
parameters provided by
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{u}\mathbf{A}$(LoRA)
. A similar gap between
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{u}\mathbf{A}$(LoRA)
and Tied-LoRA methods can be seen for the translation task as well especially
in the 2B model. We hypothesize that tasks in which the base language model
already performs well can be easily enhanced by Tied-Lora, while tasks that
are not “natutal” to the base model (like mathematical reasoning) requires
more parameters.

Again, we can see that in Tied-LoRA methods the addition of untied parameters
$u$ and $v$ are most helpful as the $r$ is increased. This suggests that the
untied parameters act as a per layer “adjustment” in the Tied-LoRA paradigm.
We also see that it is best to either train both $A$ and $B$ or just freeze
$B$ and train $A$ (with untied weights $u$ and $v$ when applicable). Lastly,
we see that in the specific cases of
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}_{{}_{\text{\faChain}}}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{u}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{A}_{{}_{\text{\faChain}}}$
and
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}_{{}_{\text{\faChain}}}\mathbf{u}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{A}_{{}_{\text{\faChain}}}$
there is extreme instability when $r$ is increased. This pattern is consistent
across all the tasks we studied.

##  6 Conclusion & Future Work

We have presented our Tied-Lora paradigm of extending the parameter efficiency
of Lora by using simple technique of weight-tying and selective training of
low-rank matrices. Our study suggests that for several tasks
$\mathbf{v}\mathbf{B}_{{}_{\text{\faChain}}}\mathbf{u}\mathbf{A}_{{}_{\text{\faChain}}}$
configuration can perform as well as Lora (over a range of low-rank
dimensions) with just $13\%$ of the parameters of Lora when $r$ is within the
typical setting of $8$. Increasing to larger $r$ result is more aggressive
reduction of trainable parameters compared to
$\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{v}\mathbf{B}\color[rgb]{0.26171875,0.578125,0.765625}\definecolor[named]{pgfstrokecolor}{rgb}{0.26171875,0.578125,0.765625}\mathbf{u}\mathbf{A}$(LoRA)
. This is especially true for tasks which the base language model already has
some abilities, such as commonsense NLI, extractive QA and summarization.
Given that the baseline abilities of LLMs are consistently improving with each
iteration of LLMs, we hope that our best Tied-LoRA configuration can be used
as a replacement for LoRA for more tasks in the future.

## References

  * Cettolo et al. (2017) Mauro Cettolo, Marcello Federico, Luisa Bentivogli, Jan Niehues, Sebastian Stüker, Katsuhito Sudoh, Koichiro Yoshino, and Christian Federmann. 2017.  Overview of the IWSLT 2017 evaluation campaign.  In _Proceedings of the 14th International Conference on Spoken Language Translation_ , pages 2–14, Tokyo, Japan. International Workshop on Spoken Language Translation.
  * Chavan et al. (2023) Arnav Chavan, Zhuang Liu, Deepak Gupta, Eric Xing, and Zhiqiang Shen. 2023.  One-for-all: Generalized lora for parameter-efficient fine-tuning.  _arXiv preprint arXiv:2306.07967_.
  * Chen et al. (2021) Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. 2021.  DialogSum: A real-life scenario dialogue summarization dataset.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 5062–5074, Online. Association for Computational Linguistics.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021.  Training verifiers to solve math word problems.  _arXiv preprint arXiv:2110.14168_.
  * Dettmers et al. (2023) Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023.  Qlora: Efficient finetuning of quantized llms.  _arXiv preprint arXiv:2305.14314_.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.  Parameter-efficient transfer learning for nlp.  In _International Conference on Machine Learning_ , pages 2790–2799. PMLR.
  * Hu et al. (2021) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_.
  * Kopiczko et al. (2023) Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki Markus Asano. 2023.  Vera: Vector-based random matrix adaptation.  _arXiv preprint arXiv:2310.11454_.
  * Lermen et al. (2023) Simon Lermen, Charlie Rogers-Smith, and Jeffrey Ladish. 2023.  Lora fine-tuning efficiently undoes safety training in llama 2-chat 70b.  _arXiv preprint arXiv:2310.20624_.
  * Lin et al. (2020) Zhaojiang Lin, Andrea Madotto, and Pascale Fung. 2020.  Exploring versatile generative language model via parameter-efficient transfer learning.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 441–459, Online. Association for Computational Linguistics.
  * Liu et al. (2022) Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. 2022.  Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning.  _Advances in Neural Information Processing Systems_ , 35:1950–1965.
  * Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter. 2017.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_.
  * Pfeiffer et al. (2021) Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2021.  AdapterFusion: Non-destructive task composition for transfer learning.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 487–503, Online. Association for Computational Linguistics.
  * Press and Wolf (2017) Ofir Press and Lior Wolf. 2017.  Using the output embedding to improve language models.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_ , pages 157–163, Valencia, Spain. Association for Computational Linguistics.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Rücklé et al. (2021) Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. 2021.  AdapterDrop: On the efficiency of adapters in transformers.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7930–7946, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Sun et al. (2023) Xianghui Sun, Yunjie Ji, Baochang Ma, and Xiangang Li. 2023.  A comparative study between full-parameter and lora-based fine-tuning on chinese instruction data for instruction following large language model.  _arXiv preprint arXiv:2304.08109_.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019.  HellaSwag: Can a machine really finish your sentence? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4791–4800, Florence, Italy. Association for Computational Linguistics.
  * Zhang et al. (2023) Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023.  Adaptive budget allocation for parameter-efficient fine-tuning.  _arXiv preprint arXiv:2303.10512_.