# Spectral Adapter: Fine-Tuning in Spectral Space

Fangzhao Zhang
Electrical Engineering
Stanford University
<EMAIL_ADDRESS>
&Mert Pilanci
Electrical Engineering
Stanford University
<EMAIL_ADDRESS>
###### Abstract

Recent developments in Parameter-Efficient Fine-Tuning (PEFT) methods for
pretrained deep neural networks have captured widespread interest. In this
work, we study the enhancement of current PEFT methods by incorporating the
spectral information of pretrained weight matrices into the fine-tuning
procedure. We investigate two spectral adaptation mechanisms, namely additive
tuning and orthogonal rotation of the top singular vectors, both are done via
first carrying out Singular Value Decomposition (SVD) of pretrained weights
and then fine-tuning the top spectral space. We provide a theoretical analysis
of spectral fine-tuning and show that our approach improves the rank capacity
of low-rank adapters given a fixed trainable parameter budget. We show through
extensive experiments that the proposed fine-tuning model enables better
parameter efficiency and tuning performance as well as benefits multi-adapter
fusion. The code will be open-sourced for reproducibility.

##  1 Introduction

Size of language and vision model undergoes a drastic explosion in recent days
and results in billions of parameters up to date. While fine-tuning has been
used a lot for adapting pretrained large models to various downstream tasks,
fine-tuning tasks become increasingly hard with current size of pretrained
models due to the huge demand of computing resource. Meanwhile, exchange and
storing of fine-tuned models are also expensive given their enormous size. To
alleviate these rising problems for fine-tuning large pretrained models, a
recent line of research has digged into the Parameter-Efficient Fine-Tuning
(PEFT) model family and harnessed great attention. A high-level philosophy
behind those PEFT methods is to train a reduced number of parameters compared
to full fine-tuning, which instantly saves computing resource and enables
light-weight fine-tuned model exchange. Among all PEFT methods, Low-Rank
Adaptation (LoRA) [20] model is a huge success attributed to its simplicity
and effectiveness. Specifically, LoRA proposes to tune an additive trainable
low-rank matrix and brings zero inference latency after merging the adapter
into pretrained model weights. Since its emergence, numerous variants of LoRA
have been developed. For instance, AdaLoRA [64], IncreLoRA [61], and DyLoRA
[53] propose to dynamically adjust LoRA rank distribution for improving tuning
efficiency, QLoRA [10] combines LoRA with model quantization to further save
computing resource, LoRA+ [16] and PrecLoRA [60] study the optimization
landscape of LoRA training, and more recent variant DoRA [32] decomposes
pretrained weights into magnitude and direction components and applies LoRA
for direction tuning, see Apppendix A for a more comprehensive review of
different LoRA variants. Other PEFT methods such as Orthogonal Fine-Tuning
(OFT) proposes to multiply pretrained weights by tunable orthogonal matrices
for preservation of hypersphere energy between pretrained neurons. Though
these different PEFT methods focus on improving fine-tuning efficiency with
reduced parameters, rare attention has been paid to utilize pretrained model
weights’ information beyond its magnitude in the fine-tuning procedure.

Figure 1: Training loss of fine-tuning Llama3 8B model with Orca Math dataset
[38] and evaluation score on GSM8K benchmark [7]. We follow experimental setup
in [52], see Appendix F.1 for details. All methods except full fine-tuning
maintain approximately $0.23\%$ trainable parameters.

Prior research in statistical machine learning such as [36] has studied the
Empirical Spectral Distribution (ESD) of deep models’ weight matrices and
found that the ESDs for larger model weights are usually more structured and
contain indicative information to distinguish between different training
stages. More recent work such as [3] investigates the "dark matter" effect of
bottom spectral space of model weights and recognizes its critical role in
attention sink phenomenon observed in [56]. Both work contributes to
decrypting spectral information of model weights and sheds light on building
insightful understanding of the connection between weight matrices’ spectral
information and model performance. In this work, we explore further the value
of model weights’ spectral pattern and unravel its effectiveness in enhancing
fine-tuning tasks. We showcase via extensive empirical observation that
integration of spectral information of pretrained model weights improves
current PEFT methods’ parameter efficiency, tuning effect, and arises as a
natural solution to multi-adapter fusion problems. Moreover, the suggested
fine-tuning model maintains better practicality compared to prior spectral
tuning models, which will be investigated further below.

Though any technique for weight fine-tuning can be directly applied to fine-
tune singular vector matrices of pretrained model weights, we investigate two
specific forms of such extension, namely additive tuning and orthogonal
rotating the top singular vector space, which we address as Spectral AdapterA
and Spectral AdapterR respectively in later content. The spectral adaptation
mechanisms being considered are formally depicted in Section 2. As a warmup,
to show that incorporating spectral information is indeed helpful, Figure 1
displays the training loss of fine-tuning Llama3 8B model on HuggingFace Orca
Math dataset and validation score on GSM8K benchmark, from which it can be
clearly observed that Spectral AdapterA performs superior to recent variants
of PEFT methods and behaves closest to full fine-tuning, here we follow
experimental setup in [52], see Appendix F.1 for details and more
investigation. In below, we first introduce the fine-tuning model being
studied in Section 2 and we then provide some theoretic insights in Section 3.
After that, we detail the advantage of our spectral adapter in enhancing fine-
tuning result, improving model’s parameter efficiency, and helping with multi-
adapter fusion as well as address any concern with respect to practicality
issues in Section 4. Conclusion and future work is discussed in Section 5. For
sake of page limitation, literature review is deferred to Appendix A.

To summarize, the proposed spectral adaptation mechanism demonstrates the
first attempt to fine-tune spectral space of pretrained model weights in a
parameter-efficient and storage-economic way which improves current PEFT
methods from aspects involving tuning results, parameter efficiency, and
multi-adapter fusion. We hope this work serves as a building block and
motivates further and deeper insightful investigation for exploring spectral
structure of pretrained model weights, which becomes increasingly meaningful
especially in current large model regime.

##  2 Spectral Adapter: Incorporating Spectral Information into Fine-Tuning

Figure 2: Compared to LoRA which proposes to add low-rank trainable matrices
to pretrained weights, we study two types of spectral adapters: Spectral
AdapterA considers additively tuning the top columns of singular vector
matrices and Spectral AdapterR considers orthogonally rotating the top columns
of singular vector matrices.

Motivated by the intrinsic low-rank of weight shifts in fine-tuning procedure
studied in [1], LoRA [20] proposes to add a low-rank factorized trainable
matrix to pretrained model weights and tune only these additive parameters for
downstream task adaptation, which usually injects far fewer trainable
parameters compared to full fine-tuning and results in light-weight tuned
adapters. LoRA serves as an outstanding representative of PEFT family and is
now widely-used for different fine-tuning tasks. Inspired by the parameter
efficiency of LoRA and the close connection between matrix rank and its
spectral representation, here we study two spectral fine-tuning mechanisms,
both are completed via first carrying out Singular Value Decomposition (SVD)
of pretrained model weights and then fine-tuning the top columns of singular
vector matrices obtained via the SVD. More precisely, consider a pretrained
weight matrix with its spectral representation of form $W=USV^{T}$, we define
additive spectral adapter as

$\textbf{Spectral
Adapter}^{A}(W):=[U_{1}+A_{U}~{}U_{2}]S[V_{1}+A_{V}~{}V_{2}],$

and correspondingly the rotational version

$\textbf{Spectral Adapter}^{R}(W):=[U_{1}R_{U}~{}U_{2}]S[V_{1}R_{V}~{}V_{2}],$

where $U_{1},V_{1}$ denote the top-$r$ columns of $U$ and $V$ and
$U_{2},V_{2}$ denote the rest of the columns. $A=(A_{U},A_{V})$ consists of
trainable matrices of shape same as $(U_{1},V_{1})$ and $R=(R_{U},R_{V})$
consists of two trainable orthogonal matrices of shape $r$ by $r$ such that
$R_{U}^{T}R_{U}=R_{V}^{T}R_{V}=I$. As we show in later sections, the
orthogonality constraint is efficiently handled with the Cayley
parameterization, see Section 4.3 for details. The proposed fine-tuning model
architecture can be visualized from Figure 2. Here Spectral AdapterA more
resembles LoRA as it is of additive form while Spectral AdapterR more
resembles prior Orthogonal Fine-Tuning (OFT) method which we compare further
in Section 4. To ensure zero initialization as often done for PEFT methods, we
initialize $A_{U}$ and $A_{V}$ both at zero. For rotational spectral adapter,
we initialize $R_{U}$ and $R_{V}$ as identity matrices.

A more thorough literature review suggests that prior work considering tuning
model weights’ spectral representation (FSGAN[46], SVDiff [15]) has been
proposed for alleviating overfitting when fine-tuning different vision models.
These methods only look at tuning the singular values of flattened CNN weights
and thus have fixed amount of trainable parameters. Moreover, these methods
require storing all $U,S$ and $V$ during training while only the diagonal
vector of $S$ is tuned, which nearly doubles the storage requirement compared
to pretraining when fine-tuning on downstream tasks. Contrarily, we consider
incorporating spectral information in generic fine-tuning procedure for
different layers (flattened CNN weights, dense linear weights, etc.) and our
method enables flexible parameter budget choices by varying values of $r$.
Methodology-wise, we consider tuning the top-$r$ columns of $U$ and $V$ by
additive and rotational tuning, both requiring only these top columns to be
stored additionally and the left part can be merged into a single weight
matrix. See Section 4.4 for more investigation on practicality of the proposed
method.

##  3 Theoretical Insights

After introducing the model architecture of spectral adapter we consider, the
main question now remains whether tuning the spectral representation of
pretrained weights is indeed an improvement over existing PEFT methods. Before
we step into our empirical observations, we first provide some theoretical
insights for the proposed spectral adaptation mechanism. In this section, we
show advantage of our spectral adapter method compared to LoRA from two
theoretic perspectives by analyzing both the rank capacity of the adapters
(Section 3.1) and the subspace alignment of pretrained weight matrices
(Section 3.2). Specifically, we will see that Spectral AdapterA has larger
rank capacity than LoRA adapter, which indicates the tuned weight has more
adaptation freedom and thus is more desirable. Moreover, the dominant spectral
direction of pretrained weight matrix identifies more ideal neuron alignment
under the setting we consider in Section 3.2, which justifies the robustness
of tuning top singular vectors in our spectral adapter. In Appendix D, we show
that Spectral AdapterA is approximately equivalent to DoRA [32] for vector-
form weights.

###  3.1 Adapter Rank Capacity

For any pretrained weight matrix $W$, suppose that the adapter is given by the
parameterization $f_{\theta}(W)$ where $\theta$ represents trainable weights.
For instance with LoRA adapter, $f_{\theta}(W)=W+AB^{T}$, where
$\theta=\\{A,B\\}$ is trainable. We define the _rank capacity_ of an adapter
$f_{\theta}(W)$ as follows:

$\mathcal{R}(f_{\theta};W):=\,\max_{\theta}\textbf{rank}(f_{\theta}(W))-\min_{\theta}\,\textbf{rank}(f_{\theta}(W)),$

which describes the range of matrix ranks the tuned weight can achieve given a
specific adapter form. Then, the following lemma shows that Spectral AdapterA
has twice the rank capacity of LoRA adapter under an equal number of trainable
parameters.

######  Lemma 3.1.

Suppose that $W\in\mathbb{R}^{n\times m}$ is an arbitrary full row-rank matrix
and $n\leq m$ without loss of generality. Consider rank-r LoRA and rank-r
additive spectral adapter, which have an equal number of trainable parameters.
We have

$\displaystyle\mathcal{R}(\mathrm{LoRA};W)$ $\displaystyle=r,$
$\displaystyle\mathcal{R}(\mathrm{Spectral~{}Adapter}^{A};W)$
$\displaystyle=2r.$

See Appendix B for proof. Therefore when pretrained model weight matrix is
close to full row-rank, as what has been observed in [20], Spectral AdapterA
has nearly double rank capacity compared to LoRA adapter. Furthermore, some
prior work explicitly imposes low-rank constraint when training original NNs
[49, 43, 65, 22, 67, 24, 9]. Using LoRA adapter to fine-tune such pretrained
model weights would destroy their rank constraints while applying spectral
adapter preserves the constraints.

Next we proceed to show that top spectral space of pretrained weight matrices
is more aligned with ideal neuron direction under a simple setting via
subspace decomposition analysis of pretrained model weights. This observation
corroborates our choice of tuning top singular vectors in our proposed
spectral adaptation mechanism. Empirically, we observe that tuning top
directions performs superior to tuning bottom ones, see Appendix F.3 and F.5.1
for related experiments.

###  3.2 Weight Subspace Alignment

Figure 3: Top singular vector of pretrained weight recognizes more ideal
neuron direction. Illustration plot for Section 3.2.

Consider two-layer ReLU network with $m$ hidden nodes and univariate output.
For squared loss objective, we can write out the training problem explicitly
as

$\min_{W^{(1)},W^{(2)}}\|(XW^{(1)})_{+}W^{(2)}-y\|_{2}^{2}+\beta(\|W^{(1)}\|_{F}^{2}+\|W^{(2)}\|_{2}^{2}),$

where $X\in\mathbb{R}^{n\times d}$ is the data matrix,
$(W^{(1)}\in\mathbb{R}^{d\times m},W^{(2)}\in\mathbb{R}^{m})$ are first and
second layer weights respectively and $y\in\mathbb{R}^{n}$ is the label
vector. For better visualization, we take $d=3.$ Consider the case that all
data points lie on $xy-$plane, which mimics the usual observation that data
points occupy a low-dimensional manifold. Then we can decompose each first
layer neuron $W_{j}^{(1)}\in\mathbb{R}^{d}$ into $W_{j}^{(1)}=w_{j1}+w_{j2}$
where $w_{j1}\in\mathcal{R}(X),w_{j2}\perp\mathcal{R}(X).$ With simple
algebra, for non-zero weight decay which is often the default setting for
current deep learning optimizers, one can derive $w_{j2}=0$ and thus
$W_{j}^{(1)}=w_{j1}\in\mathcal{R}(X)$. Therefore all optimal neurons lie also
in $xy-$plane. However, due to optimization errors, some of the trained
neurons might be slightly deviated from $xy-$plane, as illustrated in Figure
3, where $u_{i}$ indicates pretrained neuron directions, though most of them
lie in $xy-$plane, some might deviate (i.e., $u_{4}$). $u^{\star}$ indicates
the top singular vector direction of pretrained weight $W^{(1)}$ which here
recognizes the $xy-$plane orientation, and thus fine-tuning $u^{\star}$ is
noiseless and is expected to be more robust.

##  4 Empirical Results: The Impact of Spectral Information

We experiment our proposed spectral adapter with fine-tuning large language
models and diffusion models and compare against various recent PEFT methods.
From language model experiments, we observe that Spectral AdapterA performs
superior to various PEFT baselines and harnesses higher scores on different
benchmarks, which again verifies the effectiveness of incorporating spectral
information into the fine-tuning procedure, see Section 4.1 for details. For
diffusion model experiments, we will see that the advantage of spectral
adapter comes in two-fold: Spectral AdapterA offers a natural solution to
existing problems in multi-adapter fusion procedure and Spectral AdapterR
manifests finer-grained parameter budgets as well as better parameter
efficiency, see Section 4.2 and 4.3 respectively. For a fair comparison with
all baselines, we use their official implementation and follow hyperparameter
setting in their original reports as long as available. See each individual
section for corresponding experimental details. All experiments are done with
NVIDIA RTX A6000 GPU.

###  4.1 Language Model Fine-Tuning: Enhancing Fine-Tuning Results with
Spectral AdapterA

For large language model experiments, we present experimental results for
fine-tuning DeBERTaV3-base model (185M) and Mistral model (7B) on GLUE and
GSM8K tasks respectively. Our Spectral AdapterA method achieves superior
tuning results compared to various recent PEFT methods in most experiments.

DeBERTaV3-base Experiment. Table 1 shows fine-tuning results of DeBERTaV3-base
model on GLUE benchmarks with various PEFT methods. For a fair comparison, we
use official implementations for LoRA, DoRA, OFT and AdaLoRA in HuggingFace
PEFT library, with hyperparameter setting for LoRA [20] and AdaLoRA [64]
following their original reports. We use same hyperparameter setting as LoRA
for DoRA and follow the setting used in BOFT [33], a variant of OFT, for OFT
experiments. We abbreviate Spectral AdapterA as SpectralA for presentation
simplicity and we tune hyperparameters for Spectral AdapterA. See Appendix F.2
for hyperparameter details and F.3 for loss/validation plot comparison. We
fine-tune all $q,k,v$ matrices in attention layers. Our Spectral AdapterA
achieves highest average score and best scores for most tasks with fewest
trainable parameters.

Method $\\#$ Param GLUE MNLI SST-2 MRPC CoLA QNLI QQP RTE STS-B Avg. LoRAr=24
0.72$\%$ 90.15 95.87 90.20 67.25 94.14 91.38 81.95 90.67 87.70 DoRAr=24
$0.73$$\%$ 90.37 95.99 89.95 67.77 94.23 91.70 84.12 90.45 88.07 OFTr=4
$0.72$$\%$ 90.14 95.64 87.99 68.54 93.78 91.78 85.20 89.02 87.76 AdaLoRAr=24
$1.07$% 90.62 96.22 91.18 71.20 94.20 91.56 83.39 91.75 88.77
Spectral${}^{A}_{r=24}$ $0.72\%$ 90.44 96.67 91.67 71.91 94.29 91.88 88.09
91.54 89.56

Table 1: Accuracy comparison of fine-tuning DeBERTaV3-base with various PEFT
methods on GLUE benchmarks. SpectralA is abbreviation for Spectral AdapterA.
See Section 4.1 for experimental details.

Mistral 7B Experiment. We experiment our Spectral AdapterA with Mistral 7B
model [23] fine-tuned for GSM8K task [7]. Since all baseline model reports
include no fine-tuning tasks with the Mistral family, we use official
implementations of all baseline methods for comparison and we fix learning
rate to be $2.5e-5$ for all methods following [50].

---
Method |  $\\#$Param | GSM8K
Pre-Trained | $-$ | $38.82$
LoRAr=8 | $0.16\%$ | $46.25$
DoRAr=8 | $0.17\%$ | $45.79$
Spectral${}^{A}_{r=8}$ | $0.16\%$ | $51.1$
Table 2: Accuracy comparison of fine-tuning Mistral 7B model with different
PEFT methods on GSM8K benchmark. See Section 4.1 for experimental details.

We take $r=8$ for LoRA, DoRA and Spectral AdapterA to maintain approximately
same number of trainable parameters for all methods. Table 2 presents the
accuracy comparison where SpectralA stands for Spectral AdapterA. From the
result, we observe that our Spectral AdapterA scores higher than both LoRA and
DoRA by a large margin and increases the pretrained model baseline
significantly, which verifies the effectiveness of the proposed spectral
adaptation mechanism. See Appendix F.4 for more about experimental details.

###  4.2 Diffusion Model Fusion: Improving Multi-Object Fine-Tuning with
Spectral AdapterA

Figure 4: Distributing different concept tunings along different spectral
space helps with identity preservation in multi-adapter fusion, see Section
4.2 for details.

Multi-adapter fusion is a current bottleneck in diffusion model fine-tuning
tasks with LoRA adapters. Simply adding different LoRA adapters tuned for
distinct objects will result in problems involving identity loss and concept
binding [12]. To tackle this toughness, different methods emerge such as
Gradient Fusion [12] and Orthogonal Adaptation [42]. Specifically, Orthogonal
Adaptation method proposes to fix LoRA parameter $B$ to have orthogonal basis
and train $A$ solely. Experiments there show that merging LoRA weights with
such orthogonal basis helps preserving individual object characteristics
compared to its non-orthogonal counterpart. In Orthogonal Adaptation [42], the
authors maintain $B$ by manually keeping large orthogonal matrices for
different layer sizes and sample $r$ columns from corresponding orthogonal
matrix to form $B$ for each LoRA adapter. With knowledge from random matrix
theory, such sampled matrices are likely to have orthogonal basis.

Notably, our Spectral AdapterA naturally operates on orthogonal singular
vectors and thus introduces an elegant solution to multi-adapter fusion
problems by distributing different concept tunings along different columns of
singular vector matrices, which maps to wireless communications where the
signals are distributed over non-overlapping frequencies. A subtlety here lies
in the choice of column space for different fine-tuning tasks: (1) Sample-
based methods can be adopted if data privacy is considered and different
tuning tasks are done independently. In Appendix F.5, we show that tuning top
columns manifests better generation quality compared to both tuning bottom
columns and sampling random orthogonal basis as what has been done in
Orthogonal Adaptation [42]. Thus there is a trade-off between high-quality
generation and concept collapsing, i.e., sampling from top singular vectors is
more encouraged while column overlapping between concepts happens more often
compared to sampling from the whole set. (2) On the other hand, if fine-tuning
tasks are not isolated and can collaborate on the column scheduling, then more
deliberate tuning scheduling can be adopted, for example in a two-concept
tuning task with $r=4$, the first concept can allocate first to fourth columns
and the second concept then claims fifth to eighth columns. Figure 4
demonstrates steps for the same method for three-concept tuning task. Since we
expect fine-tuned weights to stay close to original weights, though both row
space and column space are tuned in spectral adapter, this adaptation
mechanism approximates orthogonal-basis tuning for different objects and thus
we expect it helps improving identity preservation for multi-adapter fusion.
In this section, we investigate this effect via extensive diffusion model
experiments.

Our experiments follow [42] and build on [12] which studies multi-LoRA fusion.
We experiment with multi-object tuning and face generation tasks. Due to space
limitation, we present some multi-object tuning results below and we leave the
rest to Appendix F.5. For all tasks, we compare against baselines including
Gradient Fusion [12], Orthogonal Adaptation [42], and FedAvg [37]. We start
with a simple review for these baseline methods.

#### Baseline Review

To merge different LoRA adapters, say we have a set of LoRA parameters
$\\{\Delta\theta_{1},\ldots,\Delta\theta_{n}\\}$ where
$\Delta\theta_{i}=A_{i}B_{i}^{T}$ and pretrained parameter $\theta_{0}$,
FedAvg [37] proposes to merge them in to a single parameter by taking a
weighted average as
$\theta_{\text{merged}}=\theta_{0}+\sum_{i}\lambda_{i}\Delta\theta_{i},$ where
$\lambda_{i}$ is the weight attached to parameter $\Delta\theta_{i}$ and is
usually taken to satisfy $\sum_{i}\lambda_{i}=1$, i.e.,
$\theta_{\text{merged}}$ is a convex combination of individual adapters.
Gradient Fusion [12] instead considers solving an auxiliary optimization
problem of form
$\theta_{\text{merged}}=\text{argmin}_{\theta}\sum_{i=1}^{n}\|(\theta_{0}+\Delta\theta_{i})X_{i}-\theta
X_{i}\|_{F}^{2}$ where $X_{i}$ represents the input activation of the $i$-th
concept. Orthogonal Adaptation [42] follows FedAvg method and replaces
original LoRA parameters with orthogonal-based LoRA adapters. For our method,
to merge different spectral adapters, let $\theta_{0}=U_{0}S_{0}V_{0}^{T}$
denote the spectral representation of pretrained model weight. Given a set of
spectral adapters $\\{(U_{i},V_{i}),\ldots,(U_{n},V_{n})\\}$ with zero-padding
to make the shape the same as $(U_{0},V_{0})$, we follow FedAvg and compute
$\theta_{\text{merged}}=(U_{0}+\sum_{i}\lambda_{i}U_{i})S_{0}(V_{0}+\sum_{i}\lambda_{i}V_{i})^{T}.$
In the following experiments, we take $\lambda_{i}=1/n$ as in [42] for all
FedAvg, Orthogonal Adaptation, and our Spectral AdapterA fusion. Notably, all
FedAvg, Orthogonal Adaptation, and our Spectral AdapterA fusion can be done
approximately instantly while Gradient Fusion usually takes around $10\sim 15$
minutes for solving its auxiliary optimization problems for all concept
adapters.

#### Multi-Object Generation

Figure 5: Generation results of Chilloutmix diffusion model [8] with different
fused adapters tuned on three custom animal concepts. See Section 4.2 for
details.

We follow default training setting in [12] and fine-tune the Chilloutmix
diffusion model [8] on three custom animal concepts, see original animals in
"reference" in Figure 5. For better spatial alignment, we adopt T2I-Adapter
[39] with sketch condition and we set guidance equal to one, see also
"reference" in Figure 5 for the sketch condition being used. LoRA rank $r=8$
is adopted. For baseline comparisons, we use original code for Gradient Fusion
[12] and Orthogonal Adaptation [42]. We adapt code of Gradient Fusion for
FedAvg method since there is no official implementation available. Custom
animal name is replaced with special token <$V_{\text{animal}}$> for fine-
tuning. For our Spectral AdapterA, we follow the method depicted in Figure 4
and tune first, second, and third top eighth columns of singular vector
matrices for different animal concepts. Figure 5 shows the generation results
with different methods for selected prompts. Notably, baseline methods
sometimes fail to capture the custom animal concepts while Spectral AdapterA
recognizes all custom animals and generates visually satisfactory images.

###  4.3 Diffusion Model Expressiveness: Improving Parameter Efficiency with
Spectral AdapterR

Spectral AdapterR is closely connected to prior Orthogonal Fine-Tuning (OFT )
[45] method which proposes to multiply the pretrained model weights by
trainable orthogonal matrices in the fine-tuning procedure. Motivation behind
OFT is to preserve hyperspherical energy which characterizes the pairwise
neuron relationship on the unit hypersphere. Unlike OFT which orthogonally
rotates neurons, Spectral AdapterR multiplies the top-$r$ columns of singular
vector space $U$ and $V$ by orthogonal trainable matrices. For our
implementation, several options are available for maintaining a trainable
orthogonal matrix such as adding an orthogonality penalty in the objective
function considered in [64] or via Cayley parameterization considered in [45].
We follow [45] and adopt Cayley parameterization which is supported by Pytorch
[44]. Specifically, the orthogonal matrix $R$ is constructed via
$R=(I+Q)(I-Q)^{-1}$ with a skew-symmetric matrix $Q$ maintained as
$(A-A^{T})/2$ where $A$ is our trainable parameter. Compared to adding an
auxiliary orthogonality penalty, this parametrization is exact and thus the
SVD form is preserved after tuning with Spectral AdapterR and can be adopted
directly for subsequent fine-tuning tasks, which we state formally as a lemma
below:

######  Lemma 4.1.

With the Cayley parametrization, Spectral AdapterR is an exact rotation
operation and thus preserves the structure of the SVD of the fine-tuned
weight. Subsequent fine-tunings can be applied consequently without
recomputing the SVD each time.

See Appendix C for the proof of above lemma. Unlike LoRA which requires number
of trainable parameters to scale with weight size, when tuning top-$r$ columns
of $U$ an $V$, Spectral AdapterR only requires two trainable matrices of size
$r\times r$ and thus can be more parameter-efficient especially for large
pretrained weight. For common weight size such as $W\in\mathbb{R}^{1024\times
1024}$, LoRA with only $r=1$ introduces same number of trainable parameters as
Spectral AdapterR with $r=32.$ For a thorough analysis on parameter efficiency
improvement brought by Spectral AdapterR, we here also compare with different
variants of LoRA which are proposed for trainable parameter savings. We review
all baselines in detail below.

#### Baseline Review

We compare our Spectral AdapterR with LoRA [20], SVDiff [15], LiDB [47], OFT
[45], and VeRA [25]. Though the other methods are proposed for vision model
tuning, VeRA is originally proposed for LLM tuning and we extend it here to
diffusion model tuning due to its parameter efficiency. Consider a pretrained
weight $W\in\mathbb{R}^{n\times n}$, SVDiff originally proposes to tune all
singular values of flattened CNN weights, here we extend it to tune all
singular values of text encoder and U-Net weights for our comparison, thus
trainable parameter attached to $W$ will be of size $n$ and is nonadjustable.
LiDB stands for Lightweight Dreambooth and proposes to cut down trainable
parameter budget by introducing auxiliary frozen matrix
$A_{\text{aux}}\in\mathbb{R}^{n\times a}$ and
$B_{\text{aux}}\in\mathbb{R}^{b\times n}$, then it mimics LoRA but uses
$A_{\text{aux}}AB^{T}B_{\text{aux}}$ in replace of $AB^{T}$ with trainable
$(A\in\mathbb{R}^{a\times r},B\in\mathbb{R}^{b\times r}).$ Thus with $a,b<n,$
LiDB requires $(a+b)r<2nr$ trainable parameters. In below, we use $a=50,b=100$
as default in [47]. OFT multiplies the weight matrix by a trainable orthogonal
matrix via Cayley parametrization discussed above, thus its complete version
requires $n^{2}$ trainable parameters. For parameter efficiency, OFT proposes
to use block-diagonal trainable matrix with all diagonal blocks being
orthogonal. Thus with $r$ diagonal blocks, the number of trainable parameter
will be $r\times(n/r)^{2}$.

---
Method | Granularity |  $\\#$Param | Auxiliary Param
LoRA | ☹ | $\infty$ | $2nr\propto n$ | no
SVDiff | ☹ | $1$ | $n\propto n$ | no
LiDB | ☹ | $\infty$ | $(a+b)r\propto r$ | yes
OFT | ☹ |  $\\#$ factors of $n$ 1 | $(n/r)^{2}\propto\frac{n}{r}$ | no
VeRA | ☹ | $\infty$ | $n+r\propto n$ | yes
Spectral AdapterR | ☺ | $n$ | $2r^{2}\propto r$ | no

  * 1

Ceiling operation is ignored for this count.

Table 3: Baseline methods comparison for parameter efficiency. Granularity
indicates number of trainable parameter budgets available. See Section 4.3 for
details.

Further reduction of trainable parameter is achieved via sharing the diagonal
blocks, which demands only $(n/r)^{2}$ parameters. In below comparison, we use
this shared block-diagonal version for best parameter efficiency of OFT. VeRA
proposes to use $\Lambda_{a}A\Lambda_{b}B^{T}$ in replace of $AB^{T}$ where
$\Lambda_{a}$ and $\Lambda_{b}$ are diagonal matrices of size $n\times n$ and
$r\times r$ respectively. Thus the total number of trainable parameters by
VeRA is $(n+r)\propto n$. Table 3 compares different properties across all
methods, where $n$ represents weight size and $r$ represents rank for all
methods except for OFT, where $r$ denotes number of diagonal blocks.

#### Parameter Efficiency

We fine-tune the Chilloumix diffusion model [8] with various PEFT methods on
custom vase concept and present the generation results for prompt "a
<$V_{\text{vase}}$>" in Figure 6 for various trainable parameter budgets,
where grey dash denotes that the corresponding parameter budget is
unobtainable with a given adapter no matter how the hyperparameter is chosen
and empty entry without grey dash represents that there is a way to achieve
the corresponding parameter budget though the generation result is skipped for
better visualization. We follow default LoRA implementation in [12] for LoRA
baseline and adjust it for all other methods. From Figure 6, it can be
observed that LoRA, OFT, and LiDB start to generate vase close to custom vase
with around $400\sim 500k$ trainable parameters.

Figure 6: Generation results for prompt “a <$V_{\text{vase}}$>” after fine-
tuning Chilloutmix diffusion model [8] on custom vase images with different
PEFT methods. See Section 4.3 for details.

SVDiff and VeRA generate unsatisfactory vase images. On the contrary, Spectral
AdapterR starts to recognize the custom vase concept with only $200\sim 300k$
trainable parameters and has finer-grained parameter choices compared to other
methods, i.e., notably Spectral AdapterR can have as few as $1k$ parameters
while other methods start with at least tens of thousands of trainable
parameters. In a word, Spectral AdapterR enjoys finer-grained parameter budget
choices and manifests better visual quality with fewer parameters, thus
achieves enhanced parameter efficiency compared to various other PEFT methods.
See Appendix F.6 for more generation results.

###  4.4 Final Note: A Closer Look at SVD Cost

Figure 7: Runtime and GPU storage cost plot. See Section 4.4 for details.

To alleviate the concerns with respect to online training cost and show that
our proposed method is very practical, we provide runtime and GPU storage cost
bar plot in Figure 7, which shows runtime and GPU storage cost for LoRA and
for our Spectral AdapterA when used for fine-tuning diffusion model in Section
4.2 and Mistral 7B model in Section 4.1. Here we adopt rank $r=8$ for both
LoRA and Spectral AdapterA. It can be observed that our Spectral AdapterA
introduces negligible runtime and storage overhead for current large model
size. Modern numerical tools such as randomized SVD [13] can also be exploited
for further runtime reduction and the SVD procedure can be parallelized when
multiple machines are available. See Appendix E for further investigation.

##  5 Conclusion and Future Work

In this work, we investigate the incorporation of spectral information of
pretrained model weights into current PEFT models by introducing a spectral
adaptation mechanism which updates only the top singular vectors of pretrained
weights. We investigate the additive and rotational variants of such spectral
adaptation mechanism. Theoretically, we show the motivation of tuning top
singular vectors by comparing the rank capacity of different fine-tuning
models and carrying out weight decomposition of pretrained model layers.
Empirically, we verify the superiority of our proposed spectral adaptation
method compared to various recent PEFT methods from different aspects via
extensive experiments. To our best knowledge, this is the first work
considering incorporating spectral information as a practical generic paradigm
for fine-tuning tasks and enhances fine-tuning results, parameter efficiency,
as well as benefits multi-adapter fusion of existing PEFT methods. For future
work, fine-tuning spectral representation of different components, i.e., only
the attention layer, of current large models is also worth studying. Other
PEFT methods such as AdaLoRA [64] can also be dynamically combined with
spectral adaptation.

##  6 Acknowledgement

This work was supported in part by the National Science Foundation (NSF) under
Grant DMS-2134248; in part by the NSF CAREER Award under Grant CCF-2236829; in
part by the U.S. Army Research Office Early Career Award under Grant
W911NF-21-1-0242; and in part by the Office of Naval Research under Grant
N00014-24-1-2164.

## References

  * [1] A. Aghajanyan, L. Zettlemoyer, and S. Gupta.  Intrinsic dimensionality explains the effectiveness of language model fine-tuning, 2020.
  * [2] A. Asai, M. Salehi, M. E. Peters, and H. Hajishirzi.  Attempt: Parameter-efficient multi-task tuning via attentional mixtures of soft prompts, 2022.
  * [3] N. Cancedda.  Spectral filters, dark signals, and attention sinks, 2024.
  * [4] A. Chavan, Z. Liu, D. Gupta, E. Xing, and Z. Shen.  One-for-all: Generalized lora for parameter-efficient fine-tuning, 2023.
  * [5] Y. Chen, D. Hazarika, M. Namazifar, Y. Liu, D. Jin, and D. Hakkani-Tur.  Empowering parameter-efficient transfer learning by recognizing the kernel structure in self-attention.  arXiv preprint arXiv:2205.03720, 2022.
  * [6] A. Chronopoulou, M. E. Peters, A. Fraser, and J. Dodge.  Adaptersoup: Weight averaging to improve generalization of pretrained language models, 2023.
  * [7] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman.  Training verifiers to solve math word problems, 2021.
  * [8] C. M. Creator.  Chilloutmix diffusion model.  https://civitai.com/models/6424/chilloutmix.
  * [9] M. Denil, B. Shakibi, L. Dinh, M. Ranzato, and N. de Freitas.  Predicting parameters in deep learning, 2014.
  * [10] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer.  Qlora: Efficient finetuning of quantized llms, 2023.
  * [11] A. Edalati, M. Tahaei, I. Kobyzev, V. P. Nia, J. J. Clark, and M. Rezagholizadeh.  Krona: Parameter efficient tuning with kronecker adapter, 2022.
  * [12] Y. Gu, X. Wang, J. Z. Wu, Y. Shi, C. Yunpeng, Z. Fan, W. Xiao, R. Zhao, S. Chang, W. Wu, Y. Ge, S. Ying, and M. Z. Shou.  Mix-of-show: Decentralized low-rank adaptation for multi-concept customization of diffusion models.  arXiv preprint arXiv:2305.18292, 2023.
  * [13] N. Halko, P.-G. Martinsson, and J. A. Tropp.  Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions, 2010.
  * [14] K. Hambardzumyan, H. Khachatrian, and J. May.  Warp: Word-level adversarial reprogramming, 2021.
  * [15] L. Han, Y. Li, H. Zhang, P. Milanfar, D. Metaxas, and F. Yang.  Svdiff: Compact parameter space for diffusion fine-tuning, 2023.
  * [16] S. Hayou, N. Ghosh, and B. Yu.  Lora+: Efficient low rank adaptation of large models, 2024.
  * [17] J. He, C. Zhou, X. Ma, T. Berg-Kirkpatrick, and G. Neubig.  Towards a unified view of parameter-efficient transfer learning, 2022.
  * [18] S. He, R.-Z. Fan, L. Ding, L. Shen, T. Zhou, and D. Tao.  Mera: Merging pretrained adapters for few-shot learning.  arXiv preprint arXiv:2308.15982, 2023.
  * [19] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. de Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly.  Parameter-efficient transfer learning for nlp, 2019.
  * [20] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen.  Lora: Low-rank adaptation of large language models, 2021.
  * [21] C. Huang, Q. Liu, B. Y. Lin, T. Pang, C. Du, and M. Lin.  Lorahub: Efficient cross-task generalization via dynamic lora composition, 2024.
  * [22] M. Jaderberg, A. Vedaldi, and A. Zisserman.  Speeding up convolutional neural networks with low rank expansions.  arXiv preprint arXiv:1405.3866, 2014.
  * [23] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed.  Mistral 7b, 2023.
  * [24] M. Khodak, N. Tenenholtz, L. Mackey, and N. Fusi.  Initialization and regularization of factorized neural layers, 2022.
  * [25] D. J. Kopiczko, T. Blankevoort, and Y. M. Asano.  Vera: Vector-based random matrix adaptation, 2024.
  * [26] T. Lei, J. Bai, S. Brahma, J. Ainslie, K. Lee, Y. Zhou, N. Du, V. Zhao, Y. Wu, B. Li, et al.  Conditional adapters: Parameter-efficient transfer learning with fast inference.  Advances in Neural Information Processing Systems, 36, 2024.
  * [27] B. Lester, R. Al-Rfou, and N. Constant.  The power of scale for parameter-efficient prompt tuning, 2021.
  * [28] X. L. Li and P. Liang.  Prefix-tuning: Optimizing continuous prompts for generation, 2021.
  * [29] Y. Li, Y. Yu, C. Liang, P. He, N. Karampatziakis, W. Chen, and T. Zhao.  Loftq: Lora-fine-tuning-aware quantization for large language models, 2023.
  * [30] Z. Lin, A. Madotto, and P. Fung.  Exploring versatile generative language model via parameter-efficient transfer learning.  arXiv preprint arXiv:2004.03829, 2020.
  * [31] Q. Liu, X. Wu, X. Zhao, Y. Zhu, D. Xu, F. Tian, and Y. Zheng.  Moelora: An moe-based parameter efficient fine-tuning method for multi-task medical applications, 2023.
  * [32] S.-Y. Liu, C.-Y. Wang, H. Yin, P. Molchanov, Y.-C. F. Wang, K.-T. Cheng, and M.-H. Chen.  Dora: Weight-decomposed low-rank adaptation, 2024.
  * [33] W. Liu, Z. Qiu, Y. Feng, Y. Xiu, Y. Xue, L. Yu, H. Feng, Z. Liu, J. Heo, S. Peng, Y. Wen, M. J. Black, A. Weller, and B. Schölkopf.  Parameter-efficient orthogonal finetuning via butterfly factorization, 2023.
  * [34] X. Liu, Y. Zheng, Z. Du, M. Ding, Y. Qian, Z. Yang, and J. Tang.  Gpt understands, too, 2023.
  * [35] R. K. Mahabadi, S. Ruder, M. Dehghani, and J. Henderson.  Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks, 2021.
  * [36] C. H. Martin and M. W. Mahoney.  Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning, 2018.
  * [37] H. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas.  Communication-efficient learning of deep networks from decentralized data, 2023.
  * [38] A. Mitra, H. Khanpour, C. Rosset, and A. Awadallah.  Orca-math: Unlocking the potential of slms in grade school math, 2024.
  * [39] C. Mou, X. Wang, L. Xie, Y. Wu, J. Zhang, Z. Qi, Y. Shan, and X. Qie.  T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models, 2023.
  * [40] mrm8488.  Lora finetune deberta-v3 huggingface blog, 2021.  Available at https://huggingface.co/mrm8488/deberta-v3-small-finetuned-mnli/commits/main.
  * [41] J. Pfeiffer, A. Kamath, A. Rücklé, K. Cho, and I. Gurevych.  Adapterfusion: Non-destructive task composition for transfer learning.  arXiv preprint arXiv:2005.00247, 2020.
  * [42] R. Po, G. Yang, K. Aberman, and G. Wetzstein.  Orthogonal adaptation for modular customization of diffusion models, 2023.
  * [43] D. Povey, G. Cheng, Y. Wang, K. Li, H. Xu, M. A. Yarmohammadi, and S. Khudanpur.  Semi-orthogonal low-rank matrix factorization for deep neural networks.  In Interspeech, 2018.
  * [44] pytorch group.  Pytorch orthogonal parameterization method implementation, 2023.
  * [45] Z. Qiu, W. Liu, H. Feng, Y. Xue, Y. Feng, Z. Liu, D. Zhang, A. Weller, and B. Schölkopf.  Controlling text-to-image diffusion by orthogonal finetuning, 2023.
  * [46] E. Robb, W.-S. Chu, A. Kumar, and J.-B. Huang.  Few-shot adaptation of generative adversarial networks, 2020.
  * [47] N. Ruiz, Y. Li, V. Jampani, W. Wei, T. Hou, Y. Pritch, N. Wadhwa, M. Rubinstein, and K. Aberman.  Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models, 2023.
  * [48] A. Rücklé, G. Geigle, M. Glockner, T. Beck, J. Pfeiffer, N. Reimers, and I. Gurevych.  Adapterdrop: On the efficiency of adapters in transformers, 2021.
  * [49] T. N. Sainath, B. Kingsbury, V. Sindhwani, E. Arisoy, and B. Ramabhadran.  Low-rank matrix factorization for deep neural network training with high-dimensional output targets.  In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 6655–6659, 2013.
  * [50] H. Skogström.  Lora finetune mistral 7b valohai blog, 2024.  https://valohai.com/blog/finetune-mistral/.
  * [51] A. Tang, L. Shen, Y. Luo, Y. Zhan, H. Hu, B. Du, Y. Chen, and D. Tao.  Parameter efficient multi-task model fusion with partial linearization, 2023.
  * [52] K. Turgutlu.  Answer.ai qdora report, 2024.  https://www.answer.ai/posts/2024-04-26-fsdp-qdora-llama3.html.
  * [53] M. Valipour, M. Rezagholizadeh, I. Kobyzev, and A. Ghodsi.  Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaptation, 2023.
  * [54] T. Vu, B. Lester, N. Constant, R. Al-Rfou, and D. Cer.  Spot: Better frozen model adaptation through soft prompt transfer.  arXiv preprint arXiv:2110.07904, 2021.
  * [55] Z. Wang, R. Panda, L. Karlinsky, R. Feris, H. Sun, and Y. Kim.  Multitask prompt tuning enables parameter-efficient transfer learning, 2023.
  * [56] G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis.  Efficient streaming language models with attention sinks, 2024.
  * [57] L. Xu, H. Xie, S.-Z. J. Qin, X. Tao, and F. L. Wang.  Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment, 2023.
  * [58] Y. Xu, L. Xie, X. Gu, X. Chen, H. Chang, H. Zhang, Z. Chen, X. Zhang, and Q. Tian.  Qa-lora: Quantization-aware low-rank adaptation of large language models, 2023.
  * [59] A. X. Yang, M. Robeyns, X. Wang, and L. Aitchison.  Bayesian low-rank adaptation for large language models, 2024.
  * [60] F. Zhang and M. Pilanci.  Riemannian preconditioned lora for fine-tuning foundation models, 2024.
  * [61] F. F. Zhang, L. Li, J.-C. Chen, Z. Jiang, B. Wang, and Y. Qian.  Increlora: Incremental parameter allocation method for parameter-efficient fine-tuning.  ArXiv, abs/2308.12043, 2023.
  * [62] L. Zhang, L. Zhang, S. Shi, X. Chu, and B. Li.  Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning, 2023.
  * [63] M. Zhang, H. Chen, C. Shen, Z. Yang, L. Ou, X. Yu, and B. Zhuang.  Loraprune: Pruning meets low-rank parameter-efficient fine-tuning, 2023.
  * [64] Q. Zhang, M. Chen, A. Bukharin, N. Karampatziakis, P. He, Y. Cheng, W. Chen, and T. Zhao.  Adalora: Adaptive budget allocation for parameter-efficient fine-tuning, 2023.
  * [65] Y. Zhang, E. Chuangsuwanich, and J. Glass.  Extracting deep neural network bottleneck features using low-rank matrix factorization.  In 2014 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 185–189. IEEE, 2014.
  * [66] H. Zhao, H. Tan, and H. Mei.  Tiny-attention adapter: Contexts are more important than the number of parameters, 2022.
  * [67] Y. Zhao, J. Li, and Y. Gong.  Low-rank plus diagonal adaptation for deep neural networks.  In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5005–5009. IEEE, 2016.
  * [68] Y. Zhu, J. Feng, C. Zhao, M. Wang, and L. Li.  Counter-interference adapter for multilingual machine translation, 2021.
  * [69] B. Zi, X. Qi, L. Wang, J. Wang, K.-F. Wong, and L. Zhang.  Delta-lora: Fine-tuning high-rank parameters with the delta of low-rank matrices, 2023.

## Appendix

##  Appendix A Prior Work

Here we provide an overview of recent PEFT methods. Dating back to 2019,
Houlsby et al. [19] develop the idea of parameter-efficient fine-tuning and
introduce Adapter model, which injects trainable components between pretrained
model layers, though the number of trainable parameters has been reduced due
to the small size of adapters, this method incurs inference latency and is
thus not desirable. Later improvement of Adapter fine-tuning focuses on
improving inference latency [48, 26], fusing multiple adapters [6, 41, 18],
modifying adapter model architecture [66], introducing parallelism [17, 68],
and creating task-specific and layer-specific adapter [35, 30]. Another line
of fine-tuning is prompt-tuning [27] which usually adds the trainable
components into the prompt. Variants of prompt-tuning involve WARP [14],
prefix-tuning [28], P-tuning [34], and ATTEMPT [2] which consider injecting
different forms of trainable components. Multitask prompt-tuning is considered
in [54, 55].

The more relevant PEFT methods to our spectral adaptation mechanism involves
LoRA [20] and OFT [45], which inspires our Spectral AdapterA and Spectral
AdapterR respectively. LoRA originates from the observation that model fine-
tuning is intrinsically low-rank [1]. Variants of LoRA involve different
methods proposing dynamic allocation of LoRA rank budgets [53, 61, 64, 5].
LoRA has been combined with model pruning [63] and quantization [10, 58, 29].
Some other variants further cut down the trainable parameter budget or
activation storage by modifying LoRA model [25, 11, 62]. DoRA [32] fixes
LoRA’s low-rank limitation by decomposing pretrained model weights and
isolating their magnitudes. Laplace-LoRA [59] incorporates Bayesian inference
into LoRA parameters to improve calibration. LoRAHub [21], MOELoRA [31], and
L-LoRA [51] consider multitask LoRA. Delta-LoRA [69] updates pretrained
weights simultaneously from information of LoRA parameters. GLoRA [4]
generalizes LoRA by introducing a prompt module. Another line of variants
focuses on analyzing the optimization scheme of LoRA model [60, 16]. OFT
studies the multiplicative fine-tuning and its variant BOFT [33] improves OFT
by utilizing butterfly parametrization for better information delivery
efficiency. [57] offers a comprehensive review of recent development of PEFT
methods.

##  Appendix B Rank Capacity Proof

###### Proof.

Consider weight matrix $W\in\mathbb{R}^{n\times m}$ with $n\leq m$ of full row
rank. For LoRA parameter $A\in\mathbb{R}^{m\times r},B\in\mathbb{R}^{n\times
r}$ with $n\geq r,$ final weight matrix $W+AB^{T}$ has rank in $[n-r,n].$ With
Spectral AdapterA parameters $A_{S}\in\mathbb{R}^{m\times
r},B_{S}\in\mathbb{R}^{n\times r}$ where $n\geq 2r$. Let $X_{r}$ denote the
first $r$ columns of any matrix $X$ and $X_{-r}$ denote the rest columns,
final weight matrix
$((U_{r}+A_{S})S_{r}(V_{r}+B_{S})^{T})+U_{-r}S_{-r}V_{-r}^{T}$ has rank in
$[n-2r,n].$ Therefore, $\mathcal{R}(\mathrm{LoRA};W)=r$ and
$\mathcal{R}(\mathrm{Spectral~{}Adapter}^{A};W)=2r$ can be derived trivially.
∎

##  Appendix C Cayley Parameterization Proof

###### Proof.

With any trainable square matrix $A,$ we set $Q=(A-A^{T})/2$ and thus
$Q=-Q^{T}$ and $Q$ is skew-symmetric thereby. Now we show that for any skew-
symmetric $Q$, $(I+Q)(I-Q)^{-1}$ is orthogonal. Let $O=(I+Q)(I-Q)^{-1}$, then

$\displaystyle O^{T}O$ $\displaystyle=((I+Q)(I-Q)^{-1})^{T}(I+Q)(I-Q)^{-1}$
$\displaystyle=(I-Q^{T})^{-1}(I+Q^{T})(I+Q)(I-Q)^{-1}$ by $Q$ skew-symmetric,
$\displaystyle=(I+Q)^{-1}(I-Q)(I+Q)(I-Q)^{-1}$ since $(I-Q)$ and $(I+Q)$ have
same eigen-basis and are commutable, $\displaystyle=I,$

which shows that the Cayley parametrization is exact and no re-SVD is needed
for orthogonality preservation. ∎

##  Appendix D Connection to DoRA

In DoRA [32], the authors observe that plain LoRA method tends to either
increase or decrease the magnitude and direction updates proportionally and
thus lacks ability to make slight direction change together with large
magnitude change, to come across this limitation, the authors propose to
decompose pretrained model weights into magnitude and direction and update
them separately. The magnitude is replaced with a trainable scalar and the
direction is updated with original LoRA method. Experiments in [32] show that
such decomposition helps improve effectiveness of LoRA significantly. Here we
show that our Spectral AdapterA is closely connected to the weight
decomposition trick used in DoRA when pretrained model weight is of vector
form. We note that in DoRA, after the weight decomposition, each column
becomes unit-length while in Spectral AdapterA, we also operates on matrices
with unit-length columns. Specifically, consider a pretrained model weight
$w_{0}\in\mathbb{R}^{n\times 1},$ then DoRA becomes

$w=\underline{w}\frac{w_{0}+\underline{b}\underline{a}}{\|w_{0}+\underline{b}\underline{a}\|_{2}},$

where $\underline{w}$ is a trainable scalar initialized at $\|w_{0}\|_{2}.$
$\underline{b}$ and $\underline{a}$ are trainable parameters of size $n\times
1$ and $1\times 1$ respectively, with $\underline{b}\underline{a}=0$ at
initialization. Comparably, Spectral AdapterA becomes

$w=(\frac{w_{0}}{\|w_{0}\|_{2}}+\underline{a}^{\prime})\|w_{0}\|_{2}(1+\underline{b}^{\prime}),$

with trainable vector $\underline{a}^{\prime}\in\mathbb{R}^{n\times 1}$ and
trainable scalar $\underline{b}^{\prime}$ both initialized at zero. We can
thus equivalently view $\|w_{0}\|_{2}(1+\underline{b}^{\prime})$ as a single
trainable scalar initialized at $\|w_{0}\|_{2},$ which then plays the role of
magnitude adapter as $\underline{w}$ in DoRA. $\underline{a}^{\prime}$ is
adopted for directional adaptation since it directly operates on the
normalized base vector.

##  Appendix E Cost Investigation (More Detailed)

Here we address the potential concern about the overhead of our proposed
spectral adaptation mechanism. Firstly, we note that spectral adapter
introduces similar number of trainable parameters and can be merged into
original model weights, thus it is lightweight for sharing and introduces no
additional inference latency, which preserves the strengths of additive fine-
tuning methods. Therefore, the major overhead concern exists in the runtime
and GPU storage overhead during online training. Note our method involves only
matrix multiplication in the forward procedure and thus should run as quick as
LoRA. Though the SVD procedure can bring additional runtime overhead, it needs
to be done only once for a single model and can be reused for later fine-
tuning on various downstream tasks. Besides, modern numerical tools such as
randomized SVD [13] can also be exploited and the SVD procedure can be
parallelized when multiple machines are available. As for GPU storage, unlike
SVDiff [15] where all SVD components are required for training procedure thus
introducing significant GPU storage burden, our method requires only the top
spectral space to be stored additionally and consumes similar GPU storage to
LoRA for relatively small tuning ranks (which is usually the case).

##  Appendix F Supplemental Materials for Experiments

###  F.1 Experimental Setup for Figure 1

For Figure 1 experiments, we follow QDoRA [52] experimental setup for fine-
tuning Llama3 8B model, where all k$\\_$proj, q$\\_$proj, v$\\_$proj,
up$\\_$proj, down$\\_$proj, and gate$\\_$proj weights are tuned. We adopt the
same data processing method and train on $10K$ Orca Math data (shuffled) as in
[52]. We fix learning rate as $1e-5$ for all methods as in QDoRA and train for
one epoch with batch size $8$. $r=8$ is adopted for LoRA, DoRA, AdaLoRA, and
Spectral AdapterA while for OFT, we set number of diagonal blocks to be $800$
to maintain similar amount of trainable parameters. LoRA alpha is set to be
$16$ following DoRA [32] convention and AdaLoRA hyperparameter is set
following what has been used for MNLI benchmark in the original AdaLoRA report
[64] with regularization set to $1e-3$ which we find works better. For
evaluation, we test on GSM8K [7] benchmark for exact matching. For more
comparisons, Figure 8 provides training loss for smaller rank $r=4$
(oft$\\_r=1600$) and larger rank $r=64$ (oft$\\_r=95$). All settings are the
same except that LoRA alpha is always kept as

Figure 8: More experiments with Llama3 8B model with different number of
trainable parameters. In the left plot, the training loss of LoRA and DoRA
overlaps. See Appendix F.1 for details.

twice as rank number. From Figure 8 we can observe that though increasing
trainable parameters closes the gap between different tuning methods, our
spectral adapter method is always superior to other PEFT methods and stays
closest to full fine-tuning.

###  F.2 Hyperparameter Setting for DeBERTaV3-base Experiment (Section 4.1)

---
Dataset | learning rate | batch size |  $\\#$epochs | optimizer | weight decay
MNLI | $5e-5$ | 32 | 7 | AdamW | 0.01
RTE | $2e-4$ | 32 | 15 | AdamW | 0.01
QNLI | $6e-5$ | 32 | 7 | AdamW | 0.01
MRPC | $7e-4$ | 32 | 13 | AdamW | 0.01
QQP | $1e-4$ | 32 | 8 | AdamW | 0.01
SST-2 | $1e-4$ | 32 | 10 | AdamW | 0.01
CoLA | $3e-4$ | 32 | 5 | AdamW | 0.01
STS-B | $2e-4$ | 32 | 12 | AdamW | 0.01
Table 4: Hyperparameters for DeBERTaV3-base model fine-tuning with Spectral
AdapterA in Section 4.1

Table 4 shows the hyperparameter setting for our Spectral AdapterA used for
fine-tuning DeBERTaV3-base model in Section 4.1. Rank $8$ is adopted for LoRA,
DoRA, AdaLoRA and Spectral Adapter${}^{A}.$ We set number of diagonal blocks
to be $4$ and enable block sharing for OFT to maintain similar amount of
trainable parameters.

###  F.3 More About DeBERTaV3-base Experiment

Left plot in Figure 9 presents the training loss and validation score
comparisons of LoRA, SVDiff and our Spectral AdapterA for fine-tuning
DeBERTaV3-base model on CoLA benchmark. We set learning rates for both LoRA
and Spectral AdapterA as what has been used in popular public blog [40] for
LoRA fine-tuning with DeBERTaV3-base model, which is not tuned in favor of our
method. For SVDiff, since it is originally proposed for vision model tuning,
we extend it to this experiment by tuning all singular values of pretrained
weights. We find the same learning rate leads to poor fine-tuning results with
SVDiff, we thus pick the best learning rate among $[1e-3,1e-4,1e-5]$ according
to validation performance and set learning rate to be $1e-3$. We use $r=8$ for
LoRA and Spectral Adapter${}^{A}.$ From Figure 9, it can be observed that
Spectral AdapterA achieves better training and validation performance compared
to both LoRA and SVDiff.

Figure 9: Left plot presents training loss and validation results for fine-
tuning DeBERTaV3-base model with LoRA, SVDiff, and Spectral AdapterA on CoLA
benchmark. Right plot compares the same statistics between LoRA and spectral
adapter with top ranks and bottom ranks tuned respectively.

Interestingly, in LoRA [20], the authors provide a correlation analysis
between the LoRA additive component $\triangle W=AB^{T}$ and original
pretrained weight matrix $W$ (see Section H.3 in [20]), and they find that the
additive component does not contain the top singular directions of $W$. The
authors therefore conclude that the learned LoRA component amplifies "task-
specific" directions which are not emphasized in the pretrained weight matrix.
Naively, this seems to suggest that tuning top singular subspace of pretrained
weights is not ideal and one should identify the desired "task-specific"
directions to improve LoRA. Here we show that this is not the case and fine-
tuning top directions provides a significant improvement to LoRA. In the right
plot of Figure 9 above, we experiment tuning the top eighth rank and the
bottom eighth rank of singular vector space in our Spectral AdapterA, which we
present as "Spectral Top" and "Spectral Bottom" respectively. Remarkably,
"Spectral Top" converges faster and scores higher than LoRA, which is then
superior to "Spectral Bottom". This result unravels the fact that tuning
different part of spectral space brings different tuning effect and tuning the
top columns of singular vector space improves LoRA tuning significantly. See
Section 3 for more theoretic insights.

###  F.4 Hyperparameter Setting for Mistral 7B Experiment (Section 4.1)

---
Method | lr | lora alpha | batch size |  $\\#$epochs | lora dropout | weight decay
LoRA | $2.5e-5$ | 16 | 4 | 5 | 0.05 | 0.01
DoRA | $2.5e-5$ | 16 | 4 | 5 | 0.05 | 0.01
Spectral AdapterA | $2.5e-5$ | - | 4 | 5 | - | 0.01
Table 5: Hyperparameters for Mistral 7B model fine-tuning task in Section 4.1

Table 5 shows training hyperparameter setting for fine-tuning Mistral 7B model
in Section 4.1. We train with bfloat16 precision and fine-tune all q$\\_$proj,
k$\\_$proj, v$\\_$proj, o$\\_$proj, and gate$\\_$proj weights. We evaluate
with lm-evaluation-harness [46].

###  F.5 Supplemental Materials for Multi-Adapter Fusion Experiment (Section
4.2)

####  F.5.1 Comparison of Single Object Generation

We present more experimental results to show that Spectral AdapterA with top
ranks tuned behaves at least as good as LoRA with same parameter budget and is
better than Orthogonal Adaptation [42], which is likely due to that Orthogonal
Adaptation fixes LoRA parameter $B$ and thus has limited expressiveness. We
also show that tuning bottom ranks in spectral adapter behaves worse than all
other methods. Figure 10 shows generation results for custom toy concept
tuning, where Orthogonal Adaptation and Spectral AdapterA (bottom) generate
inaccurate happy-face octopus, sad-face octopus, and green tortoise. Figure 11
shows generation results for custom animal concept tuning, where Orthogonal
Adaptation and Spectral AdapterA (bottom) sometimes miss first dog concept.

####  F.5.2 More Multi-Adapter Fusion Generation Results

Here we present more results for multi-adapter fusion generation. Figure 12
shows generation results for multi-object generation for custom toy concepts
and Figure 13 presents generation results for multi-character generation for
three computer scientists. See below for experimental details.

Figure 10: Generation results for single toy concept tuning with LoRA,
Orthogonal Adaptation, and Spectral AdapterA with top and bottom ranks tuned
respectively.

Multi-Object Generation. As in Section 4.2, we fine-tune Chilloutmix diffusion
model [8] on four custom toy concepts, see "reference" in Figure 12 for
original toy images. We use $r=8$ for all methods and tune first, second,
third, and fourth top eighth columns of singular vector space of pretrained
weights for first, second, third, and fourth toys in our Spectral
Adapter${}^{A}.$ We follow all default experimental settings in [12] and tune
all embedding layer, U-Net, and text-encoder. For better spatial alignment, we
employ T2I-Adapter with sketch condition listed in "reference" in Figure 12.
We randomly select three scenes and prompt fused-adapters for the results, see
"prompts" in Figure 12 for individual prompt being used. From Figure 12, it
can be observed that FedAvg and Orthogonal Adaptation generate

Figure 11: Generation results for single animal concept tuning with LoRA,
Orthogonal Adaptation, and Spectral AdapterA with top and bottom ranks tuned
respectively.

unsatisfactory happy-face octopus and green tortoise toys. On the contrary,
our spectral adapter generates high-quality images similar to Gradient Fusion
while saving much more time.

Multi-Character Generation. We also experiment fine-tuning Chilloutmix
diffusion model [8] with photos of three computer scientists Yoshua Bengio,
Yann LeCun, and Geoffrey Hinton. As in multi-object generation, we use $r=8$
for all methods and tune first, second, and third top eighth columns of
singular vector space of pretrained weights for Bengio, Lecun, and Hinton in
our Spectral Adapter${}^{A}.$ We use T2I-Adapter [39] with keypose condition.
See "reference" in Figure 13 for scientists’ photos and keypose condition
being used. Figure 13 shows generation results for prompt
"<$V_{\text{bengio}}$> and <$V_{\text{lecun}}$> and <$V_{\text{hinton}}$>,
standing near a lake, 4K, high quality, high resolution" with different fused
adapters, from which it can be observed that our spectral adapter generates
picture of most consistent styles across characters and renders all
scientists’ faces clearly.

###  F.6 Supplemental Materials for Parameter Efficiency Experiment (Section
4.3)

---
Method | learning rate
LoRA | $1e-5$
VeRA | $5e-3$
OFTA | $1e-5$
LiDB | $5e-4$
SVDiff | $1e-3$
Spectral AdapterR | $3e-4$
Table 6: Hyperparameters for diffusion model fine-tuning task in Section 4.3

In this section, we present more tuning results with various parameter budgets
for parameter efficiency experiment studied in Section 4.3, see Section 4.3
for baseline method explanation. Table 6 shows the learning rates used for
each method, the rest experimental settings are default as in [12]. Figure 14
presents generation results of Chilloutmix diffusion model [8] tuned on custom
chair concept with different methods under various parameter budgets. The
prompt used is "a yellow <$V_{\text{chair}}$>". See "reference" in Figure 14
for original chair images. From the generation results, it can be observed
that LoRA generates reasonable chairs for all rank $r=1,2,3$ though the
generation quality is not stable, i.e., $r=3$ is worse than $r=2$. OFT and
VeRA start to recognize custom chair with $>300k$ parameters, SVDiff generates
unsatisfactory image and has fixed trainable parameters. LiDB doesn’t generate
good results even for parameter budget $>800k.$ On the contrary, our Spectral
AdapterR generates reasonable images with parameters as few as $\sim 100k$ and
produces stable custom chairs since then.

Figure 12: Generation results of Chilloutmix diffusion model [8] tuned on four
custom toy concepts with different fused adapters. See Appendix F.5.2 for
details. Figure 13: Generation results of Chilloutmix diffusion model [8]
tuned on photos of three computer scientists with different fused adapters.
See Appendix F.5.2 for details.

Figure 15 shows generation results of Chilloutmix diffusion model [8] fine-
tuned on custom table concept with different methods under various parameter
budgets. The prompt used is “a <$V_{\text{table}}$>”. LoRA generates good
images for all rank $r=1,2,3$ though it starts with $273k$ parameters even if
rank is set to $1$. OFT also generates desirable images for all budgets though
the fewest number of parameters it can achieve is $29k$. VeRA and LiDB start
to generate reasonable images with $>300k$ trainable parameters and SVDiff
doesn’t recognize the custom table accurately. Meanwhile, our Spectral
AdapterR recognizes the shape of custom table with only $1k$ parameters and
produces ideal images since $20k$ parameters.

Figure 14: Generation results for prompt “a yellow <$V_{\text{chair}}$>” after
fine-tuning Chilloutmix diffusion model [8] on custom chair images with
different PEFT methods. SpectralR is abbreviation for Spectral
Adapter${}^{R}.$ See Appendix F.6 for details.

Figure 15: Generation results for prompt “a <$V_{\text{table}}$>” after fine-
tuning Chilloutmix diffusion model [8] on custom table images with different
PEFT methods. SpectralR is abbreviation for Spectral Adapter${}^{R}.$ See
Appendix F.6 for details.