# VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Banks

Yang Li
Dept. of Computer Science
Georgia State University
Atlanta, GA 30303
<EMAIL_ADDRESS>
&Shaobo Han
NEC Laboratories America
Princeton, NJ 08540
<EMAIL_ADDRESS>
&Shihao Ji
Dept. of Computer Science
Georgia State University
Atlanta, GA 30303
<EMAIL_ADDRESS>

###### Abstract

As the adoption of large language models increases and the need for per-user
or per-task model customization grows, the parameter-efficient fine-tuning
(PEFT) methods, such as low-rank adaptation (LoRA) and its variants, incur
substantial storage and transmission costs. To further reduce stored
parameters, we introduce a "divide-and-share" paradigm that breaks the
barriers of low-rank decomposition across matrix dimensions, modules and
layers by sharing parameters globally via a vector bank. As an instantiation
of the paradigm to LoRA, our proposed VB-LoRA composites all the low-rank
matrices of LoRA from a shared vector bank with a differentiable top-$k$
admixture module. VB-LoRA achieves extreme parameter efficiency while
maintaining comparable or better performance compared to state-of-the-art PEFT
methods. Extensive experiments demonstrate the effectiveness of VB-LoRA on
natural language understanding, natural language generation, and instruction
tuning tasks. When fine-tuning the Llama2-13B model, VB-LoRA only uses 0.4% of
LoRA’s stored parameters, yet achieves superior results. Our source code is
available at https://github.com/leo-yangli/VB-LoRA.

##  1 Introduction

Figure 1: Comparison of the PEFT methods on RoBERTa-Large. Our VB-LoRA
achieves higher scores with significantly smaller number of stored parameters.

Parameter-efficient fine-tuning (PEFT) casts a new paradigm that leverages
strong prior knowledge built in foundation models and adapts them to a wide
range of downstream tasks by updating a small amount of trainable parameters
(He et al., 2021). Compared to prefix/prompt tuning (Li and Liang, 2021;
Lester et al., 2021) or in-context learning (Brown et al., 2020), fine-tuning
a large-scale pre-trained model yields better domain specialization dictated
by high-quality datasets (Brown et al., 2020; Liu et al., 2022; Zhao et al.,
2023). This process can be repeated to suit the needs of ever-changing
deployment scenarios and personalizations. However, the sheer volume of
parameter space across a multitude of instantiations (Sheng et al., 2023)
poses challenges for storage, transmission, and computation, especially for
low-resource hardware and consumer-grade networks (Borzunov et al., 2024).

To mitigate these challenges, various PEFT methods have been proposed by
adding or adapting a small amount of trainable parameters per task without
sacrificing performance (Houlsby et al., 2019; Karimi Mahabadi et al., 2021;
Ding et al., 2023). These methods exploit the dependencies among model
parameters to reduce the redundancy. For example, Hu et al. (2021) propose the
low-rank adaptation (LoRA) to approximate the accumulated gradient update for
self-attention modules, and induces the intra-matrix parameter coupling.
Renduchintala et al. (2023) further study the options of allowing the inter-
matrix parameter sharing via weight tying across all the layers. In both
cases, the number of trainable parameters is reduced significantly. These two
methods stand at the two extremes of spectrum in deciding the range of model
components reuse (locally or across-layers) and designating which low-rank
matrices needs to be shared and updated. However, as the model size increases
and the demand for user-customized models across various services rises, the
expense of storing and transmitting the customizations for each combination
escalates and emerges as a critical issue. Hence, investigating PEFT methods
with significantly smaller number of trainable parameters has attracted a
flurry of research interests (Kopiczko et al., 2024; Renduchintala et al.,
2023).

Figure 2: Left: The model parameters can be represented as a composition of
vectors from a vector bank, which is shared across sub-vectors, modules and
layers. Right: Architecture of VB-LoRA. We use a top-$k$ softmax function to
select $k$ vectors from the vector bank. The selected vectors are then pooled
into a sub-vector, which is arranged at a desired position, forming the
parameters of LoRA.

This paper introduces VB-LoRA, extreme parameter-efficient fine-tuning with
_vector banks_ based on a simple yet effective "divide-and-share" paradigm. We
push the limits of LoRA parameter efficiency by breaking the two barriers of
low-rank decomposition: (1) locally within each module and each layer, and (2)
only across the two original matrix dimensions (without division; see Sec. 3.2
for details). We argue that the parameters across different modules and layers
can be shared, and thus the redundancy in parameters can be further reduced.
In addition, by partitioning rank-one component vectors into sub-vectors, we
introduce "virtrual" dimensions such that deep structure in the parameter
space can be represented by a highly compressed matrix factorization.

VB-LoRA draws inspirations from previous line of work on quantized tensor
networks (Oseledets, 2010; Cichocki, 2014) in breaking the constraint of
physical dimension for extreme parameter compression. Specifically, VB-LoRA
reparameterizes LoRA’s low-rank adaptation by a rank-one decomposition and
then divides the resulting vectors into sub-vectors of the same size. A
_global_ sharing mechanism is then learnt based on a sparse top-$k$ admixture
module. The same sized sub-vectors allows parameters to be shared across
modules and layers at the sub-vector level. Moreover, compared to the post-hoc
matrix compression methods (Oseledets, 2010; Khoromskij, 2011), VB-LoRA is
end-to-end differentiable, and therefore the fine-tuning process is aware of
the compressed form, enabling task-oriented compression. Figure 1 illustrates
the parameter efficiency of VB-LoRA as compared with state-of-the-art PEFT
methods. Our contributions are summarized as follows:

  1. 1.

We introduce a "divide-and-share" paradigm that breaks the barriers of low-
rank decomposition across matrix dimensions, modules, and layers by sharing
parameters globally via a vector bank.

  2. 2.

We reparameterize LoRA’s low-rank decomposition by a rank-one decomposition,
and divide the resulting vectors further into sub-vectors of the same size,
enabling extreme parameter efficiency at the sub-vector level.

  3. 3.

We propose a sparse top-$k$ module based on the admixture model to learn a
global sharing mechanism, making our framework end-to-end differentiable and
compression-aware.

  4. 4.

Our method achieves extreme parameter efficiency while maintaining comparable
or better empirical performance compared to the state-of-the-art PEFT methods
on natural language understanding, natural language generation, and
instruction tuning tasks.

##  2 Related Work

#### Exploit Global Redundancy for Enhanced Parameter Efficiency

The parameters of deep neural networks (DNNs) can be naturally divided by
layers, heads, or types (MHA or FFN). While LoRA (Hu et al., 2021) only
exploits the _intra-matrix_ dependency, Tied-LoRA (Renduchintala et al., 2023)
employs a simple weight tying scheme on the low-rank matrices $A$ and $B$
across layers to reduce the _inter-matrix_ redundancy. When $A$ and $B$ are
randomly initialized, frozen, and shared across all layers, Tied-LoRA
degenerates to VeRA (Kopiczko et al., 2024), which only requires two scaling
vectors to be updated, leading to impressive parameter efficiency. Our VB-LoRA
pushes the limits of LoRA parameter efficiency by sharing parameters globally
across modules and layers at the sub-vector level.

On the low-dimensional reparameterization, Aghajanyan et al. (2020)
empirically show that there exists a low-dimensional reparameterization that
is as effective for fine-tuning as the full parameter space. The actualization
of the random projection is achieved through the Fastfood transform (Le et
al., 2013) for large-scale pre-trained language models. To make it structure-
aware, a set of layer-wise scaling parameters are included as part of the
training parameters. Following this intuition, we study the lightweight fine-
tuning performance within LoRA based on the customized reparameterization that
arises from the rank-one matrix decomposition.

Moreover, tensor decomposition has been leveraged for PEFT in ViT models (Jie
and Deng, 2023) based on classical formats, such as tensor-train or Tucker
(Kolda and Bader, 2009). We find that forcing multilinear decomposition across
multiple modes results in a higher rank number, which is detrimental to the
objective of parameter compression. An indirect comparison of VB-LoRA to Jie
and Deng (2023) can be conducted by referring the compression rate to LoRA.
From this perspective, our VB-LoRA can be viewed as a customized tensor format
endowed by a convex geometry structure, which is enabled by a sparse top-$k$
admixture model we proposed.

#### Parameter Modeling based on Sparse Admixture Models

Admixture models have been widely used in population genetics (Pritchard et
al., 2000), topic modeling (Reisinger et al., 2010; Inouye et al., 2014), and
hyperspectral unmixing (Li and Bioucas-Dias, 2008; Fu et al., 2015) to extract
archetypal (or endmember) components from observed data. The archetypal
components can be relaxed to have mixed sign (Ding et al., 2008) with
identifiability guarantees (Lin et al., 2015). Conventionally, parameters
estimation are conducted based on linear programming (Chan et al., 2009) or
combinatorial algorithms (Arora et al., 2013). However, an involved integer
programming problem arises when incorporating an extra top-$k$ constraint into
the mixing weights that is especially challenging for the large-scale language
models. In this work, we propose learning archetypal vector banks not from
observed data but from model parameters of LLMs. By modifying the sparse
top-$k$ module (Shazeer et al., 2016) commonly used in Mixture-of-Expert
models (Jiang et al., 2024), the mixing weights and vector banks are optimized
by backpropgatation under the objective of downstream fine-tuning tasks. The
proposed top-$k$ admixture model is model-agnostic in the sense that it can be
readily integrated into any neural network parameters or accumulated gradient
updates.

##  3 Proposed Method

###  3.1 Preliminaries: Transformer Architecture and LoRA Adapters

The transformer architecture (Vaswani et al., 2017) consists of $L$ layers,
each containing two types of blocks: Multi-Head Attention (MHA) and Feed-
Forward Network (FFN). We denote the query, key, value, and output matrices of
MHA at layer $\ell$ as
$\bm{\mathcal{W}}_{t}^{\ell}=\\{\bm{W}_{t}^{i}\\}_{i=1}^{N_{h}}$,
$t\in\\{q,k,v,o\\}$, where $\bm{W}_{t}^{i}\in\mathbb{R}^{d\times d}$, and
$N_{h}$ is the number of heads. Given
$\textup{FFN}(\bm{x})=\bm{W}_{\textrm{down}}\textup{ReLU}(\bm{W}_{\textrm{up}}\bm{x})$
with $\bm{x}\in\mathbb{R}^{d}$, viewing FFN as a multi-head operation, we
further divide $\bm{W}_{\textrm{up}}\in\mathbb{R}^{cd\times d}$ and
$\bm{W}_{\textrm{down}}\in\mathbb{R}^{d\times cd}$ into $c$ matrices of size
$d\times d$, denoted by
$\bm{\mathcal{W}}_{\textrm{up}}^{\ell}=\\{\bm{W}_{\textrm{up}}^{\ell,i}\\}_{i=1}^{c}$
and
$\bm{\mathcal{W}}_{\textrm{down}}^{\ell}=\\{\bm{W}_{\textrm{down}}^{\ell,i}\\}_{i=1}^{c}$,
$c=4$.

Given a pre-trained matrix $\bm{W}_{0}\in\mathbb{R}^{m\times n}$, LoRA (Hu et
al., 2021) constrains the weight increments $\Delta\bm{W}$ as a low-rank
decomposition $\Delta\bm{W}=\bm{B}\bm{A}$, where $\bm{B}\in\mathbb{R}^{m\times
r}$, $\bm{A}\in\mathbb{R}^{r\times n}$ are trainable parameters, with
$r\ll\min(m,n)$. VeRA (Kopiczko et al., 2024) further limits the trainable
parameters to two scaling vectors $b$ and $d$, which form the diagonal
elements of two diagonal matrices $\Lambda_{b}$ and $\Lambda_{d}$. Hence, VeRA
can be expressed as $\Delta\bm{W}=\Lambda_{b}\bm{B}\Lambda_{d}\bm{A}$, where
$\bm{B}$ and $\bm{A}$ are randomly initialized, frozen and shared across
layers.

Collectively, we denote the model parameters of transformer as
$\bm{\Omega}\\!=\\!\\{\\{\bm{\mathcal{W}}_{q}^{\ell},\bm{\mathcal{W}}_{k}^{\ell},\bm{\mathcal{W}}_{v}^{\ell},\bm{\mathcal{W}}_{o}^{\ell}\\}\cup\\{\bm{\mathcal{W}}_{\textrm{up}}^{\ell},\bm{\mathcal{W}}_{\textrm{down}}^{\ell}\\}\\}_{\ell=1}^{L}\in\mathbb{R}^{12L\times
d\times d}$. In the sequel, we propose a _global_ reparameterization on the
weight increments of $\bm{W}\in\bm{\Omega}$ based on the LoRA decomposition
$\Delta\bm{W}=\bm{B}\bm{A}$. we will show how extreme parameter efficiency can
be achieved by (1) parameter sharing across matrix dimensions of $\bm{A}$ and
$\bm{B}$ based on a rank-one decomposition and sub-vector partitions (Sec.
3.2), and (2) across modules and layers regardless of the index or matrix type
(Sec. 3.3).

###  3.2 Divide-and-Share: a New Paradigm for Parameter Sharing

The low rank decomposition of LoRA can be equivalently expressed in a rank-one
form as follows:

$\displaystyle\Delta\bm{W}=\bm{B}\bm{A}=\displaystyle{\sum}_{k=1}^{r}\bm{b}_{k}\otimes\bm{a}_{k}=\displaystyle{\sum}_{k=1}^{r}\otimes_{i=1}^{2}\bm{v}_{k}^{(i)},\quad\bm{v}_{k}^{(1)}=\bm{b}_{k},\quad\bm{v}_{k}^{(2)}=\bm{a}_{k},$
(3.1)

where $\otimes$ denotes the outer product operator and $\bm{v}_{k}^{(i)}$ is a
vector of size $d_{i}$.

#### Divide

Based on the rank-one decomposition above, we further represent each component
vector $\bm{v}_{k}^{(i)}$ as a concatenation of a set of sub-vectors,

$\displaystyle\bm{v}_{k}^{(i)}=\text{concat}(\mathbf{u}_{k,1}^{(i)},\mathbf{u}_{k,2}^{(i)},\ldots,\mathbf{u}_{k,{d_{i}^{\prime}}}^{(i)}),\quad\bm{u}_{k,j}^{(i)}\in\mathbb{R}^{b},\quad
j\in\\{1,\ldots,d_{i}^{\prime}\\},$ (3.2)

where $\\{d_{i}\\}_{i=1,2}$ represents the size of the matrix dimension of
$\Delta\bm{W}$. In general, $\\{d_{i}\\}_{i=1,2}$ are not equal across
$\bm{A}$ and $\bm{B}$, and we choose $b$ as a common factor of $d_{i}$ such
that $d_{i}^{\prime}=d_{i}/b$ and $d_{i}^{\prime}\in\mathbb{Z}$.

Remarks The divide operator was first introduced in Quantized Tensor Train
(QTT) for super compression of large-scale matrices (Oseledets, 2010;
Cichocki, 2014). For example, dyadic division reshapes a vector of length
$L=2^{p}$ into a $p$-dimensional array which facilitates the efficient Tensor
Train decomposition to be used. Our divide operator instead applies to the
rank-one component vectors $\bm{v}_{k}^{(i)}$, and the resulting hierarchical
tensorial representation of $\Delta\bm{W}$ can be viewed as a Canonical
Polyadic Decomposition (CPD) (Kolda and Bader, 2009) with component vectors
$\bm{v}_{k}^{(i)}$ folded into $2$\- dimensional arrays with sub-vectors
$\bm{u}_{k,j}^{(i)}$ as columns.

#### Share

To facilitate parameter sharing across model dimensions, we assume each sub-
vector $\bm{u}_{k,j}^{(i)}$ as a top-$k$ admixture of basic elements from
vector bank $\mathcal{B}=\\{\bm{\alpha}_{1},\ldots,\bm{\alpha}_{h}\\}$, where
$\bm{\alpha}_{i}\in\mathbb{R}^{b}$ for $i\in\\{1,\ldots,h\\}$, and is defined
as follows (with the subscripts omitted for clarity):

$\displaystyle\bm{u}=\displaystyle{\sum}_{s=1}^{h}w_{s}(\bm{\sigma})\bm{\alpha}_{s},\quad\mathbf{w}(\bm{\sigma})=\mathrm{Softmax}(\mathrm{TopK}(\bm{\sigma},k)),$
(3.3)

where $\operatorname{TopK}\left(\bm{\sigma},k\right)_{i}=\sigma_{i}$ if
$\sigma_{i}$ is among the top-$k$ of $\bm{\sigma}$ and
$\operatorname{TopK}\left(\bm{\sigma},k\right)_{i}=-\infty$ otherwise. For
each sub-vector $\bm{u}$, we introduce logits $\bm{\sigma}\in\mathbb{R}^{h}$
as its learnable parameters. We call the model expressed in Eq. 3.3 as the
_top- $k$ admixture module_ (TKAM), which is differentiable. This design
enables the joint learning of vector bank $\mathcal{B}$ and logits
$\bm{\sigma}$ in an end-to-end manner, which is amenable for model fine-tuning
to the downstream tasks.

Remarks The TKAM module promotes sparsity by selecting $k$ vectors of the
largest logits from the vector bank. By setting $k\ll h$, we restrict the sub-
vector $\bm{u}$ to be sparse. That is, in each iteration, the updates to the
vector bank remain locally dominated – with at most $k$ basis vectors
$\bm{\alpha}\in\mathcal{B}$ affected by the backpropagation through $\bm{u}$ –
in the hope that the learnt vectors can be more specialized and the knowledge
encapsulated in the vector bank can be activated and updated sparsely.

The TKAM module can also be viewed as a factor model with simplex constraints
on the mixing weight (e.g., $k=2$, the sub-vector $\bm{u}$ lies on the edges
of the simplex) and common factors stored in $\mathcal{B}$. Let
$\bm{u}\in\mathbb{R}^{b}$ and $\bm{u}=\sum_{s=1}^{h}\bm{\alpha}_{s}w_{s}$,
where $\bm{\alpha}_{s}$ is the $s$-th factor, and $\bm{w}$ is the factor score
for the sub-vector $\bm{u}$. We consider the following options for $\bm{w}$:
(1) Admixture (convex combination): $\bm{w}\in[0,1]^{h}$ and
$\sum_{s=1}^{h}w_{s}=1$, which is commonly used in various communities. (2)
Sparse Admixture (TKAM): $\bm{w}\in[0,1]^{h}$ and $\sum_{s=1}^{h}w_{s}=1$ with
only $k\ll h$ non-zero elements allowed.

In addition, the Noisy Top-$k$ Gating module (Shazeer et al., 2016) has been
widely used to replace the fully connected layers with the Mixture of Experts
(MoE) layers in large language models (Jiang et al., 2024). In contrast, we
use Eq. 3.3 to learn the selective sharing scheme across the rank-one
component vectors without changing the original model. We find that the
decomposed cumulative gradient parameter are more sensitive than the original
model parameters during training process. Therefore, keeping zero noise in the
gating function can help make the learning more efficient and stable. An
ablation study of different vector selection methods is provided in Sec. 4.4.

###  3.3 Breaking Boundaries of LoRA for Global Parameter Sharing

While LoRA only applies the low rank decomposition to each individual weight
increment, the boundary can be broken by the _divide-and-share_ scheme we
proposed in Sec. 3.2. In our global parameter sharing scheme, each sub-vector
$\bm{u}_{\mathbf{i}}$ is composed from a globally shared vector bank
$\mathcal{B}$ via TKAM, where $\mathbf{i}=[\mathbf{j},\mathbf{v}]$ is a multi-
index including physical indices $\mathbf{j}$, such as module, layer, head,
and left/right decomposed matrix, and virtual indices $\mathbf{v}$ (created
from vector partition).

Specifically, LoRA provides a _local_ low-rank factorization for each
$d_{1}\times d_{2}$ matrix $\Delta\bm{W}$ independently. In contrast, our VB-
LoRA proposes a _global_ low-rank factorization on a
$b\times|\\{\textbf{i}\\}|$ matrix, where $|\\{\textbf{i}\\}|$ represents the
cardinality of the index set including both physical and virtual indices. As
we will see below, this differentiation can better leverage the redundancy in
the cumulative gradients, leading to extreme parameter efficiency. It’s worth
mentioning that adding the multi-index information to the vector selection
mechanism can make the TKAM model structure-aware, potentially yielding
additional benefits. But we will leave this as future work.

Figure 2 provides an overview of our method. The left section demonstrates the
high-level idea of VB-LoRA: the vector bank is shared across sub-vectors,
modules, and layers. The right section details the architecture of VB-LoRA. To
form each sub-vector, we use a top-$k$ softmax function to select $k$ vectors
from the vector bank, which are then pooled into a sub-vector. These sub-
vectors are arranged in the desired positions, forming the parameters for LoRA
with negligible computational overhead. Our method can be seamlessly
integrated into the PyTorch framework and the HuggingFace PEFT
library111https://github.com/huggingface/peft. Algorithm 1 provides the
PyTorch-like pseudocode for VB-LoRA.

Algorithm 1 Pseudocode of VB-LoRA in a PyTorch-like style

⬇

# d: hidden dimension; b: length of sub-vectors; r: rank; h: size of vector
bank

# k: number of selected vectors used in the top-k admixture module

# logits: Each linear layer has two trainable parameters: logits_A and
logits_B.

# Both parameters have a shape of (d/b)*r*h.

# vector_bank: The shared vector bank with a shape of h*b.

def get_low_rank_matrix(logits, vector_bank, k):

topk_logits, topk_indices = logits.topk(k, dim=-1)

topk_weights = torch.softmax(topk_logits, dim=-1)

matrix = (topk_weights * vector_bank[topk_indices]).sum(-2)

return matrix

def VBLoRA_forward(vector_bank, logits_A, logits_B, k):

r = logits_A.shape[1]

A = get_low_rank_matrix(logits_A, vector_bank, k).transpose(0, 1).reshape(r,
-1)

B = get_low_rank_matrix(logits_B, vector_bank, k).transpose(1, 2).reshape(-1,
r)

delta_W = B @ A

return delta_W

###  3.4 Parameter Count

In full fine-tuning, the number of trainable parameters is equal to the model
size, i.e., $LMd^{2}$, where $L$ is the numebr of layers, $M$ is the number of
fine-tuned modules, and $d$ is hidden dimension. LoRA reduces this number to
$2LMdr$, while VeRA further reduces it to $LM(d+r)$. The trainable parameters
of LoRA and VeRA are the same as the parameters they need to store.

In VB-LoRA, the trainable parameters consist of two parts: the parameters of
the vector bank $\mathcal{B}$ and the parameters of logits $\bm{\sigma}$.
However, at the end of training, the logit parameters can be discarded and
only the $k$ selected indices and the top-$k$ admixture weights need to be
stored. Therefore, the stored parameters can be represented by a triplet
$\Theta=\\{\mathcal{B},\mathcal{I},\mathcal{V}\\}$, where
$\mathcal{B}\in\mathbb{R}^{h\times b}$ is a vector bank containing $h$ vectors
of $b$-dimensional, $\mathcal{I}\in\mathbb{R}^{L\times M\times
r\times(d/b)\times k}$ is the top-$k$ indices of the vectors in $\mathcal{B}$
for all sub-vectors, and $\mathcal{V}\in\mathbb{R}^{L\times M\times
r\times(d/b)\times(k-1)}$ is the top-$k$ admixture weights used to composite
the sub-vectors from the bank. It is worth noting that the top-$k$ admixture
weights have only $k-1$ degrees of freedom since they must be summed to 1.
Additionally, depending on the size of the vector bank $h$, the indices
$\mathcal{I}$ can be efficiently stored as unsigned integers (e.g., uint8 when
$h\leq 256$), and hence, we count the number of parameters as the
float32-equivalent size for a fair comparison. When we use $k=2$ and uint8 for
indices, the number of stored parameters of VB-LoRA is $hb+1.5LMr(d/b)$.

Unlike LoRA and VeRA, the number of parameters in VB-LoRA does not increase
linearly with the model size (determined by $L$ and $d$) or the number of
fine-tuned modules, i.e., $M$. While the second term of VB-LoRA’s parameters
is a linear function of $LMd$, the coefficient is $1.5r/b$, which is typically
very small. For example, in our experiments, the typical values are $r=4$ and
$b=256$, leading to a coefficient of 0.02, whereas the coefficient is $2r$ for
LoRA and 1 for VeRA. Most of the parameters in VB-LoRA reside within the
shared vector bank, whose size does not increase linearly with the model size
or number of fine-tuned modules.

##  4 Experiments

In this section, we conduct a comprehensive evaluation of our method through a
series of experiments. We begin by comparing VB-LoRA to the state-of-the-art
PEFT methods: LoRA, VeRA, and Tied-LoRA on the GLUE benchmark. Next, we extend
our analysis to natural language generation tasks using GPT-2, as well as
instruction tuning tasks on the Llama2 models with 7B and 13B parameters. All
our experiments were conducted on a server equipped with 8 A100 GPUs. For
reproducibility of our results, we provide detailed hyperparameters and
specifications of computing resources for each experiment in the appendix. Our
source code is available at https://github.com/leo-yangli/VB-LoRA.

###  4.1 Natural Language Understanding

We adopt the General Language Understanding Evaluation (GLUE)
benchmark222https://gluebenchmark.com/ (Wang et al., 2018) to assess the
performance of VB-LoRA across various natural language understanding tasks,
including similarity, paraphrase, and inference tasks. Following Kopiczko et
al. (2024), we focus on six tasks from GLUE: CoLA (Warstadt et al., 2019)
(linguistic acceptability), SST-2 (Socher et al., 2013) (sentiment analysis),
MRPC (Dolan and Brockett, 2005) (paraphrase detection), STS-B (Cer et al.,
2017) (semantic textual similarity), QNLI (Rajpurkar et al., 2018)
(inference), and RTE (inference).

Our experiments are performed with RoBERTabase and RoBERTalarge (Liu et al.,
2019). While LoRA and VeRA only finetune the query and value modules, we
explore two fine-tuning strategies: query and value only (VB-LoRAqv), and all
linear modules (VB-LoRAall), including
$\bm{W}_{q},\bm{W}_{k},\bm{W}_{v},\bm{W}_{o}$, $\bm{W}_{\textrm{up}}$, and
$\bm{W}_{\textrm{down}}$. We create a vector bank of 90 vectors of a length of
256, initialized with a uniform distribution $\mathcal{U}(-0.02,0.02)$. The
logits are initialized with a normal distribution $\mathcal{N}(0,0.01)$. The
learning rates for the vector bank and logit parameters are set to 0.001 and
0.01, respectively. We set the rank to 4 and $k=2$ for all our experiments.

Table 1 reveals that VB-LoRA achieves competitive or superior performance
compared to VeRA and Tied-LoRA, while being more parameter efficient. For
example, when fine-tuning the query and value modules on the RoBERTalarge
model, our method reduces the stored parameters to less than 40% of those
required by VeRA or Tied-LoRA, while outperforming them across all tasks.

Moreover, the results consistently indicate that fine-tuning all modules,
beyond just the query and value modules, enhances performance for all the
methods. However, LoRA, VeRA and Tied-LoRA requires 2–4 times of the
parameters in this case because their parameter counts increase linearly with
the number of fine-tuned modules. In contrast, our method uses only 37.5%
additional parameters as we maintain the same vector bank size but add
additional parameters for indices and top-$k$ weights. Thus, with only 12.8%
of the parameters compared to VeRAall (4% compared to LoRAqv), our method
achieves the best average performance.

Table 1: Results with RoBERTabase and RoBERTalarge on the GLUE benchmark. The
best results in each group are shown in bold. We report Matthew’s correlation
for CoLA, Pearson correlation for STS-B, and accuracy for all other datasets.
Results for LoRAqv and VeRAqv are sourced from their respective original
papers, while the other results are based on our implementations. We report
the median performance from 5 runs using different random seeds.

| Method | # Params | SST-2 | MRPC | CoLA | QNLI | RTE | STS-B | Avg.
---|---|---|---|---|---|---|---|---|---
| FT | 125M | 94.8 | 90.2 | 63.6 | 92.8 | 78.7 | 91.2 | 85.2
|  LoRAqv | 0.295M |  95.1±0.2 |  89.7±0.7 |  63.4±1.2 |  93.3±0.3 |  86.6±0.7 |  91.5±0.2 | 86.6
|  VeRAqv | 0.043M | 94.6±0.1 | 89.5±0.5 | 65.6±0.8 |  91.8±0.2 |  78.7±0.7 |  90.7±0.2 | 85.2
|  Tied-LoRAqv | 0.043M |  94.4±0.5 |  88.5±1.0 |  61.9±1.6 |  92.0±0.1 |  76.2±1.0 |  89.8±0.3 | 83.8
| VB-LoRAqv (Ours) | 0.023M | 94.4±0.2 | 89.5±0.5 | 63.3±0.7 | 92.2±0.2 | 82.3±1.3 | 90.8±0.1 | 85.4
|  VeRAall | 0.157M | 95.1±0.4 |  88.7±0.5 |  64.5±1.0 |  92.3±0.2 |  81.9±1.4 |  90.2±0.3 | 85.5
|  Tied-LoRAall | 0.109M |  94.7±0.2 |  88.5±0.8 | 64.7±0.8 | 92.4±0.1 |  76.5±1.3 |  90.3±0.1 | 84.5
Base | VB-LoRAall (Ours) | 0.027M | 95.0±0.2 | 89.7±0.2 | 64.3±1.4 | 92.3±0.2 | 82.3±0.9 | 90.7±0.2 | 85.7
Large |  LoRAqv | 0.786M |  96.2±0.5 |  90.2±1.0 |  68.2±1.9 |  94.8±0.3 |  85.2±1.1 |  92.3±0.5 | 87.8
VeRAqv | 0.061M | 96.1±0.1 |  90.9±0.7 |  68.0±0.8 |  94.4±0.2 |  85.9±0.7 |  91.7±0.8 | 87.8
Tied-LoRAqv | 0.066M |  94.8±0.6 |  89.7±1.0 |  64.7±1.2 |  94.1±0.1 |  81.2 ±0.1 |  90.8 ±0.3 | 85.9
VB-LoRAqv (Ours) | 0.024M | 96.1±0.2 | 91.4±0.6 | 68.3±0.7 | 94.7±0.5 | 86.6±1.3 | 91.8±0.1 | 88.2
VeRAall | 0.258M | 96.6±0.5 |  90.9±0.8 |  68.5±1.4 | 94.4±0.4 |  85.9±1.2 | 92.2±0.2 | 88.1
Tied-LoRAall | 0.239M |  94.8±0.3 |  90.0±0.4 |  66.8±0.1 |  94.1±0.1 |  82.3±2.0 |  91.6±0.2 | 86.6
VB-LoRAall (Ours) | 0.033M | 96.3±0.2 | 91.9±0.9 | 69.3±1.5 | 94.4±0.2 | 87.4±0.7 | 91.8±0.2 | 88.5

###  4.2 Natural Language Generation

For natural language generation experiments, we fine-tune the GPT-2 Medium and
Large models Radford et al. (2019) on the E2E dataset333Licensed under CC BY-
SA 4.0. URL: https://github.com/tuetschek/e2e-dataset (Novikova et al., 2017),
which contains approximately 42,000 training examples, 4,600 validation
examples, and 4,600 test examples from the restaurant domain. We use a vector
bank of size 256 for GPT-2 Medium and 350 for GPT-2 Large. The vector length
is set to 256 and the rank is set to 4 for both models. To achieve the best
performance, we fine-tune all attention layers and FFN layers. As shown in
Table 2, our approach achieves competitive performance compared to VeRA, while
requiring about 20% less stored parameters for both models.

Table 2: Results with GPT-2 Medium and GPT-2 Large on the E2E benchmark. The results for FT and LoRA are taken from Hu et al. (2021), and the results for VeRA are taken from Kopiczko et al. (2024). We report the mean of 3 runs using different random seeds.  | Method | # Params | BLEU | NIST | METEOR | ROUGE-L | CIDEr
---|---|---|---|---|---|---|---
Medium | FT | 354.92M | 68.2 | 8.62 | 46.2 | 71.0 | 2.47
LoRA | 0.35M | 68.9 | 8.69 | 46.4 | 71.3 | 2.51
VeRA | 0.098M | 70.1 | 8.81 | 46.6 | 71.5 | 2.50
VB-LoRA (Ours) | 0.076M | 70.0 | 8.81 | 46.6 | 71.5 | 2.52
Large | FT | 774.03M | 68.5 | 8.78 | 46.0 | 69.9 | 2.45
LoRA | 0.77M | 70.1 | 8.80 | 46.7 | 71.9 | 2.52
VeRA | 0.17M | 70.3 | 8.85 | 46.9 | 71.6 | 2.54
VB-LoRA (Ours) | 0.13M | 70.3 | 8.86 | 46.7 | 72.2 | 2.54

###  4.3 Instruction Tuning

Instruction tuning is a process of fine-tuning model with a set of
instructions or prompts to enhance its performance on specific instructions
(Ouyang et al., 2022). Following Kopiczko et al. (2024), we fine-tune the
Llama2 model (Touvron et al., 2023) within the QLoRA (Dettmers et al., 2023)
framework444https://github.com/artidoro/qlora, which aims to reduce memory
usage when fine-tuning large language models on a single GPU. We utilize the
quantization strategy provided by QLoRA, including 4-bit NormalFloat for
storage data, BFloat16 for computation parameters, double quantization and
paged optimizers to train it on a single GPU. We use the Cleaned Alpaca
Dataset 555The original and cleaned Alpaca datasets are licensed under CC BY-
NC 4.0. URLs: https://huggingface.co/datasets/tatsu-lab/alpaca,
https://huggingface.co/datasets/yahma/alpaca-cleaned, which improves the data
quality of the original Alpaca dataset (Taori et al., 2023). We evaluate the
fine-tuned models on the MT-Bench666Licensed under CC BY 4.0. URL:
https://huggingface.co/datasets/lmsys/mt_bench_human_judgments (Zheng et al.,
2024), which contains 80 multi-turn questions. Our fine-tuned models generate
responses to these questions, and subsequently, GPT-4 is employed to review
and evaluate the generated answers, assigning a quantitative score on a scale
of 10. Note that aligning with VeRA, we report the score of the first turn of
the conversation. We apply VB-LoRA to all linear layers except the top one,
following Kopiczko et al. (2024). For Llama2 7B, we use a vector bank of 2,048
vectors, each with a length of 256, and the rank is set to 4, resulting in a
total of 0.8M stored parameters. For Llama2 13B, we use the same-sized vector
bank but increase the rank to 6, leading to 1.1M stored parameters. For all
the experiments, we train for one epoch.

The results are reported in Table 3. Notably, we report two sets of LoRA
results for each experiment: one from our implementation and the other from
Kopiczko et al. (2024), due to a noticeable discrepancy between the scores.
Since we closely follow the experimental settings of Kopiczko et al. (2024),
we speculate that the difference is due to changes in the GPT-4 model over
time. However, comparing the relative improvements of VeRA and VB-LoRA with
their respective implementations of LoRA remains fair. VB-LoRA achieves higher
scores than LoRA while using only 0.5% (Llama2 7B) and 0.4% (Llama2 13B) of
the stored parameters. While VeRA can reach similar scores with their
implementation of LoRA, it requires more than twice of parameters compared to
VB-LoRA.

Table 3: Results with Llama2 on the MT-Bench dataset. The scores are assigned by GPT-4 on a scale of 10. LoRA† and VeRA are sourced from Kopiczko et al. (2024). LoRA‡ and VB-LoRA are from our implementations. The discrepancy between LoRA† and LoRA‡ may be due to changes in the GPT-4 model over time.  Model | Method | # Parameters | Score
---|---|---|---
Llama2 7B | w/o FT | - | 4.79
LoRA† | 159.9M | 5.19
VeRA | 1.6M | 5.08
LoRA‡ | 159.9M | 5.63
VB-LoRA (Ours) | 0.8M | 5.71
Llama2 13B | w/o FT | - | 5.38
LoRA† | 250.3M | 5.77
VeRA | 2.4M | 5.93
LoRA‡ | 250.3M | 6.13
VB-LoRA (Ours) | 1.1M | 6.31

###  4.4 Ablation Study

We conduct an ablation study to examine the impact of each individual
component of VB-LoRA. The experiments are performed on RoBERTa-large, fine-
tuning only the query and value modules.

#### Vector Selection Methods

Besides the top-$k$ admixture module (abbreviated as Top-$k$ below), there
exist several commonly used discrete optimization methods for vector
selection, including Noisy Top-$k$ (Shazeer et al., 2016), Gumbel-Softmax
(GS), and Straight-Through Gumbel-Softmax (Jang et al., 2017; Maddison et al.,
2016). For Top-$k$ and Noisy Top-$k$, we evaluate the impact of different $k$
to the performances on the CoLA dataset. For GS and Straight-Through GS, we
set the temperature $\tau=1/3$ during training and use Top-1 and Top-2 Softmax
for inference. Additionally, we explore "Select All", a special case of
Top-$k$ with $k$ equals to the vector bank size $h$. As shown in Table 5,
Noisy Top-$k$, GS, and Straight-Through GS significantly underperform Top-$k$
and "Select All". We hypothesize that random noise injected by these methods
likely disrupts the parameters of vector bank, leading to instability in the
learning process.

We further investigate the impact of $k$ to the training dynamics and
performance of VB-LoRA. As discussed in Sec. 3.4, the choice of $k$ affects
not only the model’s performance but also the number of parameters to be
stored. Hence, a smaller $k$ is generally preferred for improved parameter
efficiency. Table 5 shows that $k=2$ yields the best result on CoLA, whereas
$k=1$ performs significantly worse. To explain this, we delve into the
training dynamics of VB-LoRA. As shown in Figure 3 (a), when $k=1$, the
selected vectors remain largely unchanged during training. In contrast, when
$k>1$, the model actively explore the vector bank as illustrated in Figure 3
(b) and (c), i.e., different vectors are selected and updated actively during
the training process.

#### Sub-vector Length $b$

VB-LoRA introduces a new virtual dimension $b$ that divides the original
dimensions of LoRA matrices into sub-vectors. As discussed in Sec. 3.2, using
finer granularity in sub-vectors promotes sharing across modules and layers,
leading to extreme parameter efficiency. Furthermore, such a division is also
necessary when the weight matrices have different shapes. As shown in Table 5,
we maintain the same number of parameters in the vector bank while varying the
sub-vector length $b$. The best performance is achieved with a sub-vector
length of 256.

Table 4: Ablation study of different vector selection methods. S.: Softmax,
GS: Gumbel-Softmax, ST-GS: Straight Through Gumbel-Softmax.

Method | Training | Inference | CoLA
---|---|---|---
Select All | S. | S. |  67.5±1.2
Top-$k$ | Top 1 S. | Top 1 S. |  66.9±0.5
Top 2 S. | Top 2 S. | 68.3±0.7
Top 3 S. | Top 3 S. |  68.1±1.3
Top 6 S. | Top 6 S. |  67.1±0.5
Noisy Top-$k$ | Noisy Top 1 S. | Top 1 S. |  45.3±2.2
Noisy Top 2 S. | Top 2 S. |  62.6±0.2
GS |  GS ($\tau$=1/3) | Top 1 S. |  57.1±0.6
GS ($\tau$=1/3) | Top 2 S. |  57.3±1.6
ST-GS |  ST-GS ($\tau$=1/3) | Top 1 S. |  55.6±1.6
ST-GS ($\tau$=1/3) | Top 2 S. |  54.7±1.2

Table 5: Ablation study of sub-vector length.

Length $b$ | Vector Bank Size | CoLA
---|---|---
128 | 240 |  67.0±0.8
256 | 120 | 68.7±0.7
512 | 60 |  67.8±0.8
1024 | 30 |  67.3±1.1

(a) When $k=1$

(b) When $k=2$

(c) When $k=3$

(d) Noisy Top-2

Figure 3: VB-LoRA’s vector selection footprints during training. The x-axis
represents the 96 sub-vectors formed by the vectors from a bank of 90 vectors,
while the y-axis represents the indices of selected vectors from the bank. The
blue blocks indicate the selection footprint during training.

##  5 Conclusion

This paper introduces a "divide-and-share" paradigm and a differentiable
top-$k$ admixture module for extreme parameter-efficient fine-tuning with
vector banks. Our proposed VB-LoRA achieves the competitive or higher accuracy
while using significantly smaller number of stored parameters compared to the
state-of-the-art PEFT methods, including LoRA, VeRA, and Tied-LoRA. In
addition, VB-LoRA is model-agnostic and applicable to other PEFT methods (Ding
et al., 2023). , including inserted adapters (Karimi Mahabadi et al., 2021),
prompt tuning (Qin et al., 2021), and BitFit (Ben Zaken et al., 2022).
Although VB-LoRA focuses on reducing the storage and transmission costs for
LLM fine-tuning, we believe the proposed scheme can be extended to memory-
efficient fine-tuning and parameter-efficient pre-training. But we leave these
for future exploration.

Fine-tuning a pre-trained model requires making design choices about which
layers of the model should be fronzen or updated. Multitask fine-tuning adds
extra complexity about which parameters should be shared or task-specific.
Along this line of work, Polytropon (Ponti et al., 2022) jointly learns a
small inventory of LoRA adapters and a routing function that selects a
variable-sized subset of adapters for few-shot adaptation. Caccia et al.
(2023) emphasize the importance of routing granularity and further propose a
finer-grained mixing across multiple heads. Following these works, it would be
interesting to explore a finer-grained parameter transfer across tasks, heads,
types, and layers at the sub-vector level for multitask fine-tuning.

Limitations and broader impacts VB-LoRA does not have any significant
limitations other than the method is not structure-aware in the sense that the
selection of vectors from vector bank is not informed by module, layer, and
matrix type, etc. But our framework can be easily extended to be structure-
aware. Regarding the broader impacts, VB-LoRA reduces the storage and
transmission costs of LLM adaptors and makes customized LLMs more accessible.
As far as we can foresee, there is no negative societal impact beyond what
LLMs may have.

## References

  * Aghajanyan et al. [2020] Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta.  Intrinsic dimensionality explains the effectiveness of language model fine-tuning.  _arXiv preprint arXiv:2012.13255_ , 2020.
  * Arora et al. [2013] Sanjeev Arora, Rong Ge, Yonatan Halpern, David Mimno, Ankur Moitra, David Sontag, Yichen Wu, and Michael Zhu.  A practical algorithm for topic modeling with provable guarantees.  In _International Conference on Machine Learning_ , pages 280–288. PMLR, 2013.
  * Ben Zaken et al. [2022] Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel.  BitFit: Simple parameter-efficient fine-tuning for transformer-based masked language-models.  In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 1–9, Dublin, Ireland, May 2022.
  * Borzunov et al. [2024] Alexander Borzunov, Max Ryabinin, Artem Chumachenko, Dmitry Baranchuk, Tim Dettmers, Younes Belkada, Pavel Samygin, and Colin A Raffel.  Distributed inference and fine-tuning of large language models over the internet.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in Neural Information Processing Systems_ , 33:1877–1901, 2020.
  * Caccia et al. [2023] Lucas Caccia, Edoardo Ponti, Zhan Su, Matheus Pereira, Nicolas Le Roux, and Alessandro Sordoni.  Multi-head adapter routing for cross-task generalization.  In _Advances in Neural Information Processing Systems_ , 2023.
  * Cer et al. [2017] Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia.  SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation.  In Steven Bethard, Marine Carpuat, Marianna Apidianaki, Saif M. Mohammad, Daniel Cer, and David Jurgens, editors, _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_ , pages 1–14, Vancouver, Canada, August 2017.
  * Chan et al. [2009] Tsung-Han Chan, Chong-Yung Chi, Yu-Min Huang, and Wing-Kin Ma.  A convex analysis-based minimum-volume enclosing simplex algorithm for hyperspectral unmixing.  _IEEE Transactions on Signal Processing_ , 57(11):4418–4432, 2009.
  * Cichocki [2014] Andrzej Cichocki.  Era of big data processing: A new approach via tensor networks and tensor decompositions.  _arXiv preprint arXiv:1403.2048_ , 2014.
  * Dettmers et al. [2023] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer.  QLoRA: Efficient finetuning of quantized LLMs.  In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, _Advances in Neural Information Processing Systems_ , volume 36, pages 10088–10115, 2023.
  * Ding et al. [2008] Chris HQ Ding, Tao Li, and Michael I Jordan.  Convex and semi-nonnegative matrix factorizations.  _IEEE transactions on pattern analysis and machine intelligence_ , 32(1):45–55, 2008.
  * Ding et al. [2023] Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al.  Parameter-efficient fine-tuning of large-scale pre-trained language models.  _Nature Machine Intelligence_ , 5(3):220–235, 2023.
  * Dolan and Brockett [2005] William B. Dolan and Chris Brockett.  Automatically constructing a corpus of sentential paraphrases.  In _Proceedings of the Third International Workshop on Paraphrasing (IWP2005)_ , 2005.
  * Fu et al. [2015] Xiao Fu, Wing-Kin Ma, Kejun Huang, and Nicholas D Sidiropoulos.  Blind separation of quasi-stationary sources: Exploiting convex geometry in covariance domain.  _IEEE Transactions on Signal Processing_ , 63(9):2306–2320, 2015.
  * He et al. [2021] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig.  Towards a unified view of parameter-efficient transfer learning.  In _International Conference on Learning Representations_ , 2021.
  * Houlsby et al. [2019] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.  Parameter-efficient transfer learning for NLP.  In _International Conference on Machine Learning_ , pages 2790–2799. PMLR, 2019.
  * Hu et al. [2021] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al.  LoRA: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_ , 2021.
  * Inouye et al. [2014] David Inouye, Pradeep Ravikumar, and Inderjit Dhillon.  Admixture of Poisson MRFs: A topic model with word dependencies.  In _International Conference on Machine Learning_ , pages 683–691. PMLR, 2014.
  * Jang et al. [2017] Eric Jang, Shixiang Gu, and Ben Poole.  Categorical reparametrization with Gumble-softmax.  In _International Conference on Learning Representations_ , 2017.
  * Jiang et al. [2024] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al.  Mixtral of experts.  _arXiv preprint arXiv:2401.04088_ , 2024.
  * Jie and Deng [2023] Shibo Jie and Zhi-Hong Deng.  Fact: Factor-tuning for lightweight adaptation on vision transformer.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 37, pages 1060–1068, 2023.
  * Karimi Mahabadi et al. [2021] Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder.  Compacter: Efficient low-rank hypercomplex adapter layers.  _Advances in Neural Information Processing Systems_ , 34:1022–1035, 2021.
  * Khoromskij [2011] Boris N Khoromskij.  O (d log n)-quantics approximation of n-d tensors in high-dimensional numerical modeling.  _Constructive Approximation_ , 34:257–280, 2011.
  * Kolda and Bader [2009] Tamara G Kolda and Brett W Bader.  Tensor decompositions and applications.  _SIAM review_ , 51(3):455–500, 2009.
  * Kopiczko et al. [2024] Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano.  VeRA: Vector-based random matrix adaptation.  In _International Conference on Learning Representations_ , 2024.
  * Le et al. [2013] Quoc Le, Tamás Sarlós, Alex Smola, et al.  Fastfood-approximating kernel expansions in loglinear time.  In _International Conference on Machine Learning_ , volume 85, 2013.
  * Lester et al. [2021] Brian Lester, Rami Al-Rfou, and Noah Constant.  The power of scale for parameter-efficient prompt tuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, 2021.
  * Li and Bioucas-Dias [2008] Jun Li and José M Bioucas-Dias.  Minimum volume simplex analysis: A fast algorithm to unmix hyperspectral data.  In _IGARSS 2008-2008 IEEE International Geoscience and Remote Sensing Symposium_ , volume 3, pages III–250. IEEE, 2008.
  * Li and Liang [2021] Xiang Lisa Li and Percy Liang.  Prefix-tuning: Optimizing continuous prompts for generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4582–4597, 2021.
  * Lin et al. [2015] Chia-Hsiang Lin, Wing-Kin Ma, Wei-Chiang Li, Chong-Yung Chi, and ArulMurugan Ambikapathi.  Identifiability of the simplex volume minimization criterion for blind hyperspectral unmixing: The no-pure-pixel case.  _IEEE Transactions on Geoscience and Remote Sensing_ , 53(10):5530–5546, 2015.
  * Liu et al. [2022] Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel.  Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning.  _Advances in Neural Information Processing Systems_ , 35:1950–1965, 2022.
  * Liu et al. [2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  RoBERTa: A robustly optimized BERT pretraining approach.  _arXiv preprint arXiv:1907.11692_ , 2019.
  * Maddison et al. [2016] Chris J Maddison, Andriy Mnih, and Yee Whye Teh.  The concrete distribution: A continuous relaxation of discrete random variables.  In _International Conference on Learning Representations_ , 2016.
  * Novikova et al. [2017] Jekaterina Novikova, Ondřej Dušek, and Verena Rieser.  The E2E dataset: New challenges for end-to-end generation.  In Kristiina Jokinen, Manfred Stede, David DeVault, and Annie Louis, editors, _Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue_ , pages 201–206, Saarbrücken, Germany, August 2017.
  * Oseledets [2010] Ivan V Oseledets.  Approximation of $2d\times 2d$ matrices using tensor decomposition.  _SIAM Journal on Matrix Analysis and Applications_ , 31(4):2130–2145, 2010.
  * Ouyang et al. [2022] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744, 2022.
  * Ponti et al. [2022] Edoardo M Ponti, Alessandro Sordoni, Yoshua Bengio, and Siva Reddy.  Combining modular skills in multitask learning.  _arXiv preprint arXiv:2202.13914_ , 2022.
  * Pritchard et al. [2000] Jonathan K Pritchard, Matthew Stephens, and Peter Donnelly.  Inference of population structure using multilocus genotype data.  _Genetics_ , 155(2):945–959, 2000.
  * Qin et al. [2021] Yujia Qin, Xiaozhi Wang, Yusheng Su, Yankai Lin, Ning Ding, Jing Yi, Weize Chen, Zhiyuan Liu, Juanzi Li, Lei Hou, et al.  Exploring universal intrinsic task subspace via prompt tuning.  _arXiv preprint arXiv:2110.07867_ , 2021.
  * Radford et al. [2019] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  2019\.
  * Rajpurkar et al. [2018] Pranav Rajpurkar, Robin Jia, and Percy Liang.  Know what you don’t know: Unanswerable questions for SQuAD.  In Iryna Gurevych and Yusuke Miyao, editors, _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 784–789, Melbourne, Australia, July 2018.
  * Reisinger et al. [2010] Joseph Reisinger, Austin Waters, Bryan Silverthorn, and Raymond J Mooney.  Spherical topic models.  In _International Conference on Machine Learning_ , pages 903–910. Citeseer, 2010.
  * Renduchintala et al. [2023] Adithya Renduchintala, Tugrul Konuk, and Oleksii Kuchaiev.  Tied-LoRA: Enhacing parameter efficiency of LoRA with weight tying.  _arXiv preprint arXiv:2311.09578_ , 2023.
  * Shazeer et al. [2016] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean.  Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.  In _International Conference on Learning Representations_ , 2016.
  * Sheng et al. [2023] Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, et al.  S-LoRA: Serving thousands of concurrent LoRA adapters.  _arXiv preprint arXiv:2311.03285_ , 2023.
  * Socher et al. [2013] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts.  Recursive deep models for semantic compositionality over a sentiment treebank.  In David Yarowsky, Timothy Baldwin, Anna Korhonen, Karen Livescu, and Steven Bethard, editors, _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1631–1642, Seattle, Washington, USA, October 2013.
  * Taori et al. [2023] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca, 2023.
  * Touvron et al. [2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023.
  * van der Maaten and Hinton [2008] Laurens van der Maaten and Geoffrey Hinton.  Visualizing data using t-SNE.  _Journal of Machine Learning Research_ , 9(86):2579–2605, 2008.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in Neural Information Processing Systems_ , 30, 2017.
  * Wang et al. [2018] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In Tal Linzen, Grzegorz Chrupała, and Afra Alishahi, editors, _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355, Brussels, Belgium, November 2018.
  * Warstadt et al. [2019] Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman.  Neural network acceptability judgments.  _Transactions of the Association for Computational Linguistics_ , 7:625–641, 2019.
  * Zhao et al. [2023] Xujiang Zhao, Jiaying Lu, Chengyuan Deng, Can Zheng, Junxiang Wang, Tanmoy Chowdhury, Li Yun, Hejie Cui, Zhang Xuchao, Tianjiao Zhao, et al.  Domain specialization as the key to make large language models disruptive: A comprehensive survey.  _arXiv preprint arXiv:2305.18703_ , 2023.
  * Zheng et al. [2024] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al.  Judging LLM-as-a-judge with MT-bench and chatbot arena.  _Advances in Neural Information Processing Systems_ , 36, 2024.

##  Appendix A Hyperparameters and Computing Resources

The hyperparameters used for the natural language understanding, natural
language generation and instruction tuning are provided in Table 6, 7 and 8.
All experiments were conducted on a server equipped with 8 A100 80GB GPUs.

Table 6: Hyperparameters and computing resources for natural language
understanding experiments on the GLUE benchmark. Training time and GPU memory
are reported as "query and value only" / "all linear modules". h: hour, m:
minute.

Model | Hyperparameter | SST-2 | MRPC | CoLA | QNLI | RTE | STS-B
---|---|---|---|---|---|---|---
| Optimizer | AdamW
| Warmup Ratio | 0.06
| LR Schedule | Linear
| Init. of the Vector Bank | $\mathcal{U}(-0.02,0.02)$
| Init. of the Logits | $\mathcal{N}(0,0.01)$
Base | # GPUs | 1
Epochs | 60 | 30 | 80 | 25 | 160 | 80
Learning Rate (Head) | 4E-3 | 4E-3 | 2E-2 | 1E-2 | 2E-2 | 2E-2
Learning Rate (Logits) | 1E-2
Learning Rate (Vector Bank) | 1E-3
Vector Bank Size | 90
Vector Length | 256
Rank | 4
Max Seq. Len. | 512
Batch Size Per GPU | 32
Training Time | 8h / 10h | 27m / 40m | 80m / 100m | 5h / 6.5h | 50m / 1h | 1h / 80m
GPU Memory | 24,552 MiB / 28,120 MiB
Large | # GPUs | 1
Epochs | 20 | 40 | 40 | 20 | 40 | 40
Learning Rate (Head) | 3E-3 | 3E-3 | 3E-3 | 2E-3 | 2E-3 | 6E-3
Learning Rate (Logits) | 1E-2
Learning Rate (Vector Bank) | 1E-3
Vector Bank Size | 90
Vector Length | 256
Rank | 4
Max Seq. Len. | 128
Batch Size Per GPU | 32
Training Time | 2h / 3h | 12m / 20m | 30m / 45m | 3h / 4.5h | 10m / 15m | 20m / 30m
GPU Memory | 9,804 MiB / 12,170 MiB

Table 7: Hyperparameters and computing resources on natural language generation experiments on the E2E dataset. Training time and GPU memory are reported as "query and value only" / "all linear modules". h: hour, m: minute. Hyperparameter | Medium | Large
---|---|---
# GPUs | 1
Optimizer | AdamW
Learning Rate Schedule | Linear
Weight Decay | 0.01
Batch Size | 8
Epochs | 5
Warmup Steps | 500
Label Smooth | 0.1
Rank | 4
Vector Length | 256
Vector Bank Size | 256 | 350
Learning Rate (Vector Bank) | 1E-3 | 1E-3
Learning Rate (Logits) | 1E-2 | 1E-2
Training Time | 3h | 3h
GPU Memory | 29,061 MiB | 29,282 MiB
Table 8: Hyperparameters and computing resources on instruction tuning on the Cleaned Alpaca Dataset. h: hour. 7B: llama2 7B, 13B: llama2 13B. Hyperparameter | LoRA, 7B | LoRA, 13B | VB-LoRA, 7B | VB-LoRA, 13B
---|---|---|---|---
# GPUs | 1
Optimizer | AdamW
Warmup Ratio | 0.1
Batch Size | 4
Accumulation Steps | 4
Epochs | 1
LR Schedule | Linear
Vector Length | N/A | N/A | 256 | 256
Rank | 64 | 64 | 4 | 6
Vector Bank Size | N/A | N/A | 2048 | 2048
Learning Rate (Vector bank) | N/A | N/A | 1E-3 | 1E-3
Learning Rate (Logits) | N/A | N/A | 1E-2 | 1E-2
Learning Rate (LoRA) | 4e-4 | 4e-4 | N/A | N/A
Training Time | 2h | 2.6h | 2.5h | 3h
GPU Memory | 8,467 MiB | 11,624 MiB | 6,872 MiB | 11,486 MiB

##  Appendix B Visualization of the Vector Selection

For visualization, we conducted experiments on the CoLA dataset using a
24-layer RoBERTa-large model with a vector bank of 30 vectors. We fine-tuned
the query and value modules, setting the rank to 2 and the vector length to
1024, resulting in 192 sub-vectors.

Figure 4 displays the vectors selected by sub-vectors at the initialization
(red) and at the end of training (blue), respectively. As we can see, most of
the final selections differ from the initial selections, demonstrating the
training dynamics of the vector selection process.

Figure 5 illustrates the sum of the top-k weights for each vector, grouped by
the first, middle, and last 8 layers. It shows that certain vectors are
favored by deeper layers, such as vectors #1 and #29, while some are favored
by shallower layers, such as vectors #20 and #26.

We then group the same data with respect to query and value modules, as well
as matrices A and B, shown in Figure 6. As we can see, some vectors are
predominantly utilized by specific module or matrix types. For instance,
vector #23 is heavily utilized in the formation of matrix A, while vector #29
is predominantly used in the formation of Query modules.

Figure 4: The x-axis represents the 192 sub-vectors formed by the vectors in
the vector bank, while the y-axis represents the 30 vectors in the vector
bank. The vectors initially selected by each sub-vector are shown in red, the
vectors finally selected are shown in blue, and the overlapping vectors are
shown in green. Figure 5: The sum of the top-$k$ weights for each vector,
grouped by the first, middle, and last 8 layers. The vectors in $\mathcal{B}$
are sorted by their norms. Figure 6: The sum of the top-$k$ weights for each
vector, grouped by query (Q) and value (V) modules, and matrices A and B. The
vectors in $\mathcal{B}$ are sorted by their norms.

##  Appendix C Visualization of the Vector Bank and the Sub-vectors

Figure 7 illustrates the positioning of the sub-vectors along the edge of the
simplex spanned by the vector bank. The vector bank is projected into a 2-D
space using T-SNE [van der Maaten and Hinton, 2008] for visualization.

Figure 7: Visualization of the learned vector bank and sub-vectors.

##  Appendix D Instruction-tuning Examples

Tables 9 and 10 provide examples of the responses generated by the finetuned
Llama2 13B model with LoRA and VB-LoRA.

Table 9: Example #1 of the responses generated by the finetuned Llama2 13B model with LoRA and VB-LoRA.  | LoRA | VB-LoRA
---|---|---
|  Draft a professional email seeking your supervisor’s feedback on the
’Quarterly Financial Report’ you prepared. Ask specifically about the data
analysis, presentation style, and the clarity of conclusions drawn. Keep the
email short and to the point.
Answer |  Dear [Supervisor’s Name], I am writing to request your feedback on the quarterly financial report I prepared. I would appreciate your input on the data analysis, presentation style, and the clarity of the conclusions drawn. I have attached the report for your review. Please let me know if you require any additional information. Thank you for taking the time to review my work. I am eager to receive your feedback and learn from your insights. |  Dear Supervisor, I am writing to request your feedback on the ’Quarterly Financial Report’ I prepared. Specifically, I would like to know your thoughts on the data analysis, presentation style, and the clarity of conclusions drawn. As a reminder, I used the most recent data available to analyze the company’s financial performance and provide insights into its overall health. I used charts and graphs to present the data in an easy-to-understand manner. Finally, I summarized the key findings and provided recommendations for future action. I look forward to your feedback and appreciate your time and consideration. Thank you. Your Name
Review |  The assistant’s response is very professional, concise, and to the point. It clearly communicates the user’s request for feedback on specific aspects of the report, namely the data analysis, presentation style, and clarity of conclusions. The assistant also shows respect for the supervisor’s time and expresses a willingness to learn from their feedback. The response is relevant, accurate, and detailed enough for the context. However, it lacks a bit of creativity, but given the professional context, this is not a major issue. |  The assistant’s response is highly relevant, accurate, and helpful. It provides a professional and concise email draft that meets the user’s requirements. The assistant has included all the necessary elements such as the specific areas where feedback is needed (data analysis, presentation style, and clarity of conclusions), a brief description of the work done, and a polite request for the supervisor’s time and feedback. The response is also creative and detailed, providing a clear context for the supervisor. Therefore, I would rate it highly.
Score | 9 | 10
Table 10: Example #2 of the responses generated by the finetuned Llama2 13B model with LoRA and VB-LoRA. Parts of answers are omitted due to their length. | LoRA | VB-LoRA
---|---|---
|  Compose an engaging travel blog post about a recent trip to Hawaii,
highlighting cultural experiences and must-see attractions.
Answer |  Sure! Here is a travel blog post about a recent trip to Hawaii: Hawaii is a paradise on earth, with its beautiful beaches, lush green landscapes, and warm tropical weather. My recent trip to the islands was filled with unforgettable experiences and breathtaking sights. The first stop on my itinerary was the city of Honolulu, the capital of Hawaii. I explored the historic landmarks, stunning beaches, and world-renowned cuisine. The Pearl Harbor Memorial was a must-visit, as it commemorates the tragic attack that occurred on December 7, 1941. The USS Arizona Memorial is located above the sunken battleship and is a solemn reminder of the sacrifices made during the war. From Honolulu, I ventured to the island of Oahu, where I spent a few days relaxing on the beautiful Waikiki Beach. The beach is lined with palm trees and crystal-clear waters, and is the perfect spot for swimming, sunbathing, and snorkeling.After exploring the island, I ventured to Maui, known as the "Valley Isle." Maui is home to some of the most beautiful beaches in Hawaii, including the world-famous Ka’anapali Beach. I spent the day soaking up the sun and enjoying the warm, turquoise waters. In conclusion, my trip to Hawaii was filled with unforgettable experiences and breathtaking sights. The islands are a paradise on earth, with something for everyone. Whether you’re looking for history, culture, or natural beauty, Hawaii has it all. I can’t wait to return and explore more of these beautiful islands. |  Dear reader, I recently had the opportunity to travel to the beautiful island of Hawaii, and I am thrilled to share my experience with you. This trip was full of cultural immersion, adventure, and breathtaking scenery. Here is a glimpse of my journey: On the first day, I landed in Honolulu, the capital city of Hawaii. I was immediately struck by the lush greenery and tropical weather. I spent the day exploring the city, visiting famous landmarks such as Pearl Harbor and the USS Arizona Memorial. The USS Arizona Memorial is a somber yet powerful tribute to the brave soldiers who lost their lives in the attack on Pearl Harbor. I spent the next few days relaxing on the beautiful beaches of Maui. I went snorkeling in the crystal-clear waters, and saw tropical fish, turtles, and even a few sharks. The snorkeling was truly an unforgettable experience. One of the highlights of my trip was a visit to the Polynesian Cultural Center. This is an interactive museum that showcases the culture and traditions of Polynesian islands. I got to try traditional dances, make fire, and even learn how to play the ukulele. In conclusion, my trip to Hawaii was truly unforgettable. I experienced cultural immersion, adventure, and breathtaking scenery. If you’re looking for a trip of a lifetime, I highly recommend visiting this beautiful island paradise.
Review |  The assistant’s response is a well-written and engaging travel blog post about a recent trip to Hawaii. It provides a detailed itinerary of the trip, highlighting the cultural experiences and must-see attractions in each location. The assistant also does a good job of describing the natural beauty and unique features of each island, which adds depth and interest to the post. The assistant’s response is relevant, accurate, and creative, making it a high-quality response to the user’s request. However, it could have included more about the cultural experiences, such as local food, music, or traditions. |  The assistant’s response is highly detailed, engaging, and relevant to the user’s request. It provides a comprehensive overview of a trip to Hawaii, highlighting cultural experiences and must-see attractions. The assistant’s use of descriptive language helps to paint a vivid picture of the experiences, making the blog post more engaging for readers. The assistant also provides a personal touch by sharing their own experiences and impressions, which adds depth to the response. The assistant’s response is accurate, as it mentions real places and experiences in Hawaii. Overall, the assistant’s response is highly creative and provides a high level of detail, making it an excellent travel blog post.
Score | 8.5 | 10