# Bridging The Gap between Low-rank and Orthogonal Adaptation via Householder
Reflection Adaptation

Shen Yuan1 Haotian Liu1,2 Hongteng Xu1,3
1Gaoling School of Artificial Intelligence, Renmin University of China
2School of Computer Science and Technology, Beijing Institute of Technology
3Beijing Key Laboratory of Big Data Management and Analysis Methods
<EMAIL_ADDRESS>
This work is done when Haotian Liu works as a research intern of GSAI.Hongteng
Xu is the corresponding author of this work.

###### Abstract

While following different technical routes, both low-rank and orthogonal
adaptation techniques can efficiently adapt large-scale pre-training models in
specific tasks or domains based on a small piece of trainable parameters. In
this study, we bridge the gap between these two techniques, proposing a simple
but effective adaptation method based on Householder reflections. Given a pre-
trained model, our method fine-tunes its layers by multiplying each frozen
weight matrix with an orthogonal matrix constructed by a chain of learnable
Householder reflections (HRs). This HR-based orthogonal fine-tuning is
equivalent to an adaptive low-rank adaptation. Moreover, we show that the
orthogonality of the reflection planes corresponding to the HRs impacts the
model capacity and regularity. The analysis motivates us to regularize the
orthogonality of the HRs, leading to different implementations of the proposed
Householder reflection adaptation (HRA) method. Compared with state-of-the-art
methods, HRA achieves superior performance with fewer learnable parameters
when adapting large language models and conditional image generators. The code
is available at https://github.com/DaShenZi721/HRA

##  1 Introduction

In recent large foundation model competitions, “Scaling Laws” [18, 14, 35]
motivate researchers to increase model size continuously, which brings
significantly improved model capabilities in understanding, generation,
reasoning, and generalization but with more and more unbearable model
adaptation costs. For instance, the GPU memory for fine-tuning a LLaMA-65B
model with 16bit precision exceeds 780GB [7]. The adaptation of image
generative models (like the ControlNet in [53] did) may suffer the same issue
when applying large vision foundation models as backbones. Consequently, fine-
tuning large foundation models efficiently for adapting various downstream
tasks has become a challenge in practice.

(a) The scheme of our Householder reflection adaptation method

(b) Performance on GLUE Benchmark

Method | Param. Ratio | GSM8K | MATH
---|---|---|---
LLaMA2-7B | - | 14.6 | 2.5
$\text{LoRA\ }_{r=32}$ | 0.25% | 50.2 | 7.8
$\text{OFT\ }_{b=16}$ | 0.13% | 50.1 | 8.4
$\text{BOFT\ }^{m=2}_{b=8}$ | 0.12% | 50.6 | 8.6
PiSSA | 4.75% | 53.1 | 7.4
HRA ${}_{r=8,~{}\lambda=0}$ | 0.03% | 47.1 | 6.6
HRA ${}_{r=16,~{}\lambda=0}$ | 0.06% | 52.1 | 8.1
HRA ${}_{r=32,~{}\lambda=0}$ | 0.12% | 55.8 | 9.0
HRA ${}_{r=32,~{}\lambda=\infty}$ | 0.12% | 52.8 | 9.2
HRA ${}_{r=32,~{}\lambda=10^{-4}}$ | 0.12% | 56.3 | 9.3
(a) LLM adaptation for mathematical reasoning

Figure 1:  (a) An illustration of our HRA method. (b) Comparisons for various
methods on GLUE benchmark [45]. The x-axis corresponds to the number of
trainable parameters (M), and the y-axis corresponds to the average score (%).
(c) Comparisons for various methods on the ratio of trainable parameters and
accuracy (%) when adapting LLaMA2-7B [41] in mathematical reasoning tasks.

To overcome the above challenge, Parameter-Efficient Fine-Tuning (PEFT)
methods [48] provide promising solutions, which aim to reduce the trainable
parameters and memory consumption of fine-tuning while maintaining even
improving model adaptation performance. Among various PEFT methods, the
adapter-based fine-tuning [16, 33] attracts a lot because it only inserts
limited trainable parameters into existing models during fine-tuning but
without adding extra complexity or overhead in the inference phase. Currently,
given a parameter matrix of a pre-trained model, i.e.,
$\bm{W}\in\mathbb{R}^{d_{\text{out}}\times d}$, there are roughly two
strategies implementing the adapter-based fine-tuning. The mainstream strategy
is applying Low-Rank Adaptation (LoRA) [16] and its variants [7, 17, 21, 27,
29, 31, 43, 49, 53, 55], modifying the weight matrix by adding a trainable
low-rank decomposition matrix, i.e., $\bm{W}+\bm{AB}$, where
$\bm{A}\in\mathbb{R}^{d_{\text{out}}\times r}$ and
$\bm{B}\in\mathbb{R}^{r\times d}$, and $r\ll\min(d,d_{\text{out}})$ is the
intrinsic rank of the modification $\bm{AB}$. Another strategy is Orthogonal
Fine-Tuning (OFT) [33, 30], which multiplies the weight matrix with a
structured orthogonal matrix $\bm{R}\in\mathbb{R}^{d\times d}$ determined by
limited trainable parameters, i.e., $\bm{WR}$. Both strategies can reduce VRAM
usage because they merely leverage limited trainable parameters and do not
store the optimizer state of the original weight matrices. At the same time,
they achieve encouraging model adaptation performance in various vision and
NLP tasks.

Essentially, LoRA hypothesizes that the additive modifications of weight
matrices are intrinsically low-rank, while OFT preserves the pairwise angles
between neuron vectors and theoretically penalizes the discrepancy between
pre-trained and fine-tuned models. The difference between their principles
prevents us from building a unified adapter-based fine-tuning framework. To
bridge the gap between these two strategies, we propose a simple but effective
adapter-based fine-tuning method called Householder Reflection Adaptation
(HRA). This method provides a new perspective connecting LoRA to OFT and
achieves encouraging performance in various downstream tasks. As illustrated
in Figure 1(a), our method adapts a pre-trained model by multiplying each
frozen weight matrix with a chain of $r$ learnable Householder reflections
(HRs) [15]. HRA can be interpreted as either an OFT adapter or an adaptive
LoRA. Consequently, it harnesses the advantages of both strategies, reducing
parameters and computation costs while penalizing the loss of pre-training
knowledge.

Moreover, we show that the orthogonality of HR planes impacts the capacity and
regularity of HRA. Accordingly, we leverage an orthogonality regularizer of
the HR planes when applying HRA, achieving a trade-off between the model
capacity and regularity by controlling the strength of the regularizer. When
the weight of the regularizer (i.e., the $\lambda$ in Figure 1(a)) goes to
infinity, we constrain the orthogonality strictly by Gram-Schmidt
orthogonalization [2], resulting in a strictly-orthogonal HRA implementation.
We apply HRA to adapt different models, including large language models (LLMs)
and conditional image generators. Experiments show that HRA consistently
outperforms state-of-the-art adapters in various tasks, achieving better
performance with fewer trainable parameters. Figures 1(b)-1(a) highlight the
superiority of our method in natural language understanding and mathematical
reasoning tasks, and more results can be found in the following content.

##  2 Related Work and Preliminaries

The early PEFT methods [51, 39, 8] apply model fine-tuning, which keep model
architectures unchanged and only update a small portion of model parameters.
To achieve better performance, soft prompt fine-tuning [10, 24, 26, 44] is
proposed, introducing additional trainable parameters into inputs and/or
hidden layers when adapting models. Recently, adapter-based fine-tuning [16,
55] is proposed to improve model adaptation performance without changing model
architecture. As we mentioned before, it is often implemented based on the
following strategies.

###  2.1 Low-rank Adaptation (LoRA)

LoRA [16] formulates trainable parameters as decomposed low-rank matrices and
aggregates them to frozen weight matrices linearly during fine-tuning, which
achieves a trade-off between efficiency and effectiveness. Following LoRA,
many improved low-rank adaptation methods [7, 17, 21, 27, 29, 31, 43, 49, 53,
55] have been proposed, which can be coarsely categorized into three classes.

  * •

Structure Adjustment. The work in [43, 21, 55, 53] further reduces trainable
parameters by adjusting the structure of inserted low-rank matrices. VeRA [21]
incorporates frozen low-rank matrices shared across all layers with few
trainable scaling vectors. DyLoRA [43] learns low-rank matrices with different
ranks and determines optimal ranks automatically. AdaLoRA [55] prunes
trainable parameters based on the importance scores of the original weight
matrices.

  * •

Initialization Improvement. Some methods [29, 31] utilize matrix decomposition
methods on the original weight matrices to initialize parameters. DoRA [29]
decomposes each original weight matrix into magnitudes and directions for
fine-tuning. PiSSA [31] performs singular value decomposition (SVD) on each
original weight matrix, where the low-rank principal matrix serves as
trainable parameters, while the residual matrix is frozen.

  * •

Parameter Quantization. Some methods [49, 7, 27] quantize the pre-trained
model to further reduce computational costs in both training and inference.
For example, QA-LoRA [49] achieves a trade-off between quantization strength
and adaptation performance with the help of a group-wise quantization
operator.

These methods have empirically demonstrated decent performance in various
downstream tasks, however, they lack theoretical guarantees regarding the
retention of pre-training knowledge.

###  2.2 Orthogonal Fine-tuning (OFT)

LoRA and its variants have empirically demonstrated decent performance in
various downstream tasks. However, they lack theoretical guarantees regarding
the retention of pre-training knowledge. In order to address this issue,
orthogonal fine-tuning (OFT) [33, 30] is proposed, which transforms neuron
vectors within the same layer using the same set of orthogonal matrices. It
preserves the pairwise angles between neuron vectors and thus guarantees a
bounded discrepancy between pre-trained and fine-tuned models. For instance,
the OFT method in [33] adopts Cayley parameterization to generate the block-
diagonal orthogonal matrix. The BOFT [30] introduces butterfly factorization
to generate a denser orthogonal matrix from a chain of structured sparse
matrices, which improves OFT’s performance with fewer trainable parameters.

Note that, besides fine-tuning, the principle of imposing orthogonality
constraints on trainable parameters has been applied in designing robust and
training-efficient neural network architectures, e.g., convolution neural
networks (CNNs) [11, 23], recurrent neural networks (RNNs) [46, 22], and
Transformers [52]. In particular, by constraining the orthogonality of these
models’ weight matrices, we can ensure the models are 1-Lipschtize in theory
and thus make them achieve provable robustness against adversarial
perturbations [42, 25] and generalization bounds [38, 20].

##  3 Proposed Method

###  3.1 Model Adaptation via Learning A Chain of Householder Reflections

Denote $\mathbb{S}^{d-1}=\\{\bm{u}\in\mathbb{R}^{d}~{}|~{}\|\bm{u}\|_{2}=1\\}$
as a $d$-dimensional hypersphere. For each $\bm{u}\in\mathbb{S}^{d-1}$, we can
construct a Householder reflection matrix, denoted as $\bm{H}$, by
$\bm{I}-2\bm{u}\bm{u}^{\top}$, which corresponds to a specular reflection
hyperplane, denoted as $\mathcal{H}$. For any $\bm{x}\in\mathbb{R}^{d}$,
$\bm{Hx}$ corresponds to reflecting $\bm{x}$ across the hyperplane
$\mathcal{H}$, which reverses the component of $\bm{x}$ that is orthogonal to
the hyperplane.

Because $\bm{H}$ is an orthogonal matrix, it is natural for us to implement
orthogonal adaptation based on it — we can treat $\bm{H}$ as an adapter and
multiply it with the weight matrix of the pre-trained model. Moreover, since
the set of all $d\times d$ orthogonal matrices, denoted as
$\mathbb{O}_{d\times d}$, satisfies all the axioms of a group which is a
compact Lie group of dimension $d(d-1)/2$, the product of orthogonal matrices
is also an orthogonal matrix [1]. Therefore, we can enhance the capacity of
the adapter by constructing a chain of $r$ trainable Householder reflections,
leading to our HRA method. As shown in Figure 1(a), given a weight matrix
$\bm{W}\in\mathbb{R}^{d_{\text{out}}\times d}$ and an input
$\bm{x}\in\mathbb{R}^{d}$, the forward step of HRA is

$\displaystyle\begin{aligned}
\bm{z}=\bm{W}\bm{H}^{(r)}\bm{x}=\bm{W}\Bigl{(}\sideset{}{{}_{i=1}^{r}}{\prod}\bm{H}_{i}\Bigr{)}\bm{x}=\bm{W}\Bigl{(}\sideset{}{{}_{i=1}^{r}}{\prod}(\bm{I}-2\bm{u}_{i}\bm{u}_{i}^{\top})\Bigr{)}\bm{x},~{}\text{with}~{}\\{\bm{u}_{i}\in\mathbb{S}^{d-1}\\}_{i=1}^{r}.\end{aligned}$
(1)

Although equation 1 involves the chained product of $r+1$ dense matrices, we
can leverage the structure of Householder reflection to simplify the
computation. Let $\bm{x}^{(0)}=\bm{x}$ and
$\bm{x}^{(j+1)}=(\bm{I}-2\bm{u}_{r-j}\bm{u}_{r-j}^{\top})\bm{x}^{(j)}$ for
$j=0,...,r-1$. We implement equation 1 by the following two steps:

$\displaystyle\begin{aligned}
1)~{}\bm{x}^{(j+1)}=\bm{x}^{(j)}-2\langle\bm{u}_{r-j},\bm{x}^{(j)}\rangle\bm{u}_{r-j},~{}\text{for}~{}j=0,...,r-1.\quad
2)~{}\bm{z}=\bm{W}\bm{x}^{(r)}.\end{aligned}$ (2)

The first step involves $r$ vector inner products and $r$ scalar-vector
multiplications, whose complexity is $\mathcal{O}(rd)$. The second step
involves a matrix-vector multiplication, whose complexity is
$\mathcal{O}(d_{\text{out}}d)$. Therefore, the complexity of HRA can be as low
as $\mathcal{O}(d(r+d_{\text{out}}))$.

Table 1: Comparisons for various OFT-based adapters Method |  OFT [33] |  BOFT [30] | Our HRA
---|---|---|---
Implementation | $\bm{R}^{(b)}=\text{diag}(\\{\bm{R}_{i}\\}_{i=1}^{d/b})$ | $\bm{B}^{(m,b)}=\prod_{i=1}^{m}\bm{B}_{i}^{(b)}$ | $\bm{H}^{(r)}=\prod_{i=1}^{r}\bm{I}-2\bm{u}_{i}\bm{u}_{i}^{\top}$
Illustration |  |  |
#Parameters | $\frac{d(b-1)}{2}\sim db$ | $\frac{dm(b-1)}{2}\sim dmb$ | $rd$
Complexity | $\mathcal{O}(d(b^{2}+b+d_{\text{out}}))$ | $\mathcal{O}(d((b^{2}+b)m+d_{\text{out}}))\sim$ | $\mathcal{O}(d(r+d_{\text{out}}))$
$\mathcal{O}(d((b^{2}+d)m+d_{\text{out}}))$
Cover $\mathbb{O}_{d\times d}$ | $b=d$ | $\\{\bm{B}_{i}^{(m=\log d,b=2)}\\}_{i=1}^{d-1}$ | $\\{\bm{u}_{i}\\}_{i=1}^{d-1}$

###  3.2 Comparisons with Existing OFT Methods

Table 1 compares HRA with existing OFT [33] and BOFT [30] methods on their
implementations, numbers of parameters, computational complexity, and model
capacity.

  * •

The number of parameters. Both OFT and BOFT construct several orthogonal sub-
matrices in $\mathbb{O}_{b\times b}$ based on Cayley transformation [9], each
of which requires $\frac{b(b-1)}{2}\sim b^{2}$ trainable parameters111Cayley
transformation represents an orthogonal matrix in $\mathbb{O}_{b\times b}$ as
$\bm{R}=(\bm{I}+\bm{A})(\bm{I}-\bm{A})^{-1}$, where $\bm{A}$ is a trainable
skew-symmetric matrix. The computational complexity of $(\bm{I}-\bm{A})^{-1}$
is $\mathcal{O}(b^{3})$. Ideally, we only need $\frac{b(b-1)}{2}$ parameters
to determine $\bm{A}$. In practice, however, both OFT and BOFT leverage a
dense parameter matrix $\bm{P}\in\mathbb{R}^{b\times b}$ to construct $\bm{A}$
as $\frac{1}{2}(\bm{P}-\bm{P}^{\top})$, increasing the number of parameters to
$b^{2}$ for making the method friendly to GPU computation. and
$\mathcal{O}(b^{3})$ computational complexity. OFT constructs a block diagonal
matrix based on $\frac{d}{b}$ sub-matrices, i.e.,
$\bm{R}^{(b)}=\text{diag}(\\{\bm{R}_{i}\in\mathbb{O}_{b\times
b}\\}_{i=1}^{d/b})$. BOFT constructs $m$ sparse orthogonal matrices
$\\{\bm{B}_{i}^{d}\\}_{i=1}^{m}$ by scattering the elements of $\frac{dm}{b}$
orthogonal sub-matrices in a butterfly manner, such that when
$m=\log\frac{2d}{b}$, the product of the $m$ sparse matrices leads to a dense
orthogonal matrix. Therefore, according to the number of orthogonal sub-
matrices, the numbers of parameters of OFT and BOFT are $\frac{d(b-1)}{2}$ and
$\frac{dm(b-1)}{2}$ in theory while $db$ and $dmb$ in practice. To make the
number of parameters comparable to OFT, BOFT often applies a smaller block
size (e.g., $b=2$ or $4$). It is easy to find that when $r=b$ ($=mb$), HRA has
the same number of parameters as OFT (BOFT). Therefore, HRA is generally
comparable to OFT and BOFT regarding model size.

  * •

Computational complexity. For the forward step of OFT, i.e.,
$\bm{z}=\bm{W}\bm{R}^{(b)}\bm{x}$, the computational complexity is
$\mathcal{O}(d(b^{2}+b+d_{\text{out}}))$. Here, $\mathcal{O}(db^{2})$
corresponds to applying Cayley transformation to construct $\frac{d}{b}$
orthogonal sub-matrices, and $\mathcal{O}(db)$ and
$\mathcal{O}(dd_{\text{out}})$ correspond to the matrix block multiplication
used for $\bm{y}=\bm{R}^{(b)}\bm{x}$ and the matrix-vector multiplication for
$\bm{z}=\bm{Wy}$, respectively. For the forward step of BOFT, i.e.,
$\bm{z}=\bm{W}\bm{B}^{(m,b)}\bm{x}$, where
$\bm{B}^{(m,b)}=\prod_{i=1}^{m}\bm{B}_{i}^{(b)}$, the computational complexity
is
$\mathcal{O}(d(mb^{2}+mb+d_{\text{out}}))\sim\mathcal{O}(d(mb^{2}+md+d_{\text{out}}))$.
Similar to OFT, $\mathcal{O}(dmb^{2})$ means applying Cayley transformation to
construct $\frac{dm}{b}$ orthogonal sub-matrices,
$\mathcal{O}(dmb)\sim\mathcal{O}(d^{2}m)$ corresponds to the matrix-vector
multiplications of $m$ butterfly matrices222Because of their non-block
sparsity patterns, butterfly matrices are not friendly to modern hardware like
GPUs [4]. Ideally, we can apply sparse matrix-vector multiplication with
$\mathcal{O}(dmb)$ operations, but when applying GPUs, we often treat
butterfly matrices as ordinary dense matrices, which results in
$\mathcal{O}(d^{2}b)$ operations. to compute
$\bm{y}=\prod_{i=1}^{m}\bm{B}_{i}^{(b)}\bm{x}$, and
$\mathcal{O}(dd_{\text{out}})$ corresponds to $\bm{z}=\bm{Wy}$. When setting
$r\ll b^{2}+b$ ($\ll m(b^{2}+b)$), HRA can be more efficient than OFT (BOFT).

  * •

The trade-off between model capacity and regularity. The ranges of OFT, BOFT,
and HRA correspond to different subsets of $\mathcal{O}_{d\times d}$,
achieving a trade-off between model capacity and regularity. To cover the
whole $\mathcal{O}_{d\times d}$, OFT needs to set $b=d$ and compute a dense
orthogonal matrix with high complexity. With the help of the butterfly
structure, BOFT can derive a dense orthogonal matrix with fewer parameters.
However, we need to construct $\\{\bm{B}_{i}^{(m,b)}\\}_{i=1}^{d-1}$ with
$m=\log d$ and $b=2$, such that each
$\bm{B}_{i}^{(m,b)}(\bm{B}_{i}^{(m,b)})^{\top}$ can represent a Householder
reflection matrix [6], and accordingly, the chained product of the $d-1$
Householder reflections can represent an arbitrary orthogonal matrix in
$\mathbb{O}_{d\times d}$. Similarly, for HRA, we need to construct $d-1$
Householder reflections based on $\\{\bm{u}_{i}\\}_{i=1}^{d-1}$ to represent
an arbitrary orthogonal matrix in $\mathbb{O}_{d\times d}$. According to the
above analysis, BOFT and HRA are relatively easy to achieve a trade-off
between model capacity and regularity with mild computation costs — by setting
small $m,b$, and $r$, they can achieve dense orthogonal matrices that have
intrinsic low-dimensional manifold structures.

###  3.3 Connections with Low-rank Adaptation

Figure 2: A 2D illustration indicating that when the reflection planes
$\mathcal{H}_{1}$ and $\mathcal{H}_{2}$ are orthogonal, the distance
$\|\bm{H}_{2}\bm{H}_{1}\bm{w}-\bm{w}\|_{2}$ is maximized.

Different from OFT and BOFT, HRA can also be viewed as an adaptive low-rank
adapter. Specifically, we can rewrite the chain of HRs in equation 1 in the
following equivalent format:

$\displaystyle\begin{aligned}
\bm{H}^{(r)}=\sideset{}{{}_{i=1}^{r}}{\prod}(\bm{I}-2\bm{u}_{i}\bm{u}_{i}^{\top})=\bm{I}+\bm{U}_{r}\bm{\Gamma}_{r}\bm{U}_{r}^{\top},\end{aligned}$
(3)

where $\bm{U}_{r}=[\bm{u}_{1},...,\bm{u}_{r}]\in\mathbb{R}^{d\times r}$,
$\bm{\Gamma}_{r}=[\gamma_{ij}]\in\mathbb{R}^{r\times r}$ is a upper-triangular
matrix, and its upper-riangular element is

$\displaystyle\gamma_{ij}=\begin{cases}-2&i=j\\\
(-2)^{j-i+1}\sideset{}{{}_{i=1}^{j-1}}{\prod}\langle\bm{u}_{i},\bm{u}_{i+1}\rangle&i<j.\end{cases}$
(4)

Accordingly, we have

$\displaystyle\bm{WH}^{(r)}=\bm{W}+\bm{WU}_{r}\bm{\Gamma}_{r}\bm{U}_{r}^{\top}=\bm{W}+\bm{A}(\bm{W},\bm{B})\bm{B}.$
(5)

The above formulation can be viewed as an adaptive LoRA that inherits the
theoretical guarantee of OFT on the retention of pre-training knowledge. The
low-rank matrix $\bm{B}=\bm{U}_{r}$, is constructed by normalized vectors,
while the low-rank matrix $\bm{A}=[\bm{a}_{1},...,\bm{a}_{r}]$ is
parameterized as $\bm{WU}_{r}\bm{\Gamma}_{r}$, which can be treated as a
function of $\bm{W}$ and $\bm{B}$. Therefore, similar to OFT and BOFT, HRA
ensures that the columns of $\bm{WH}^{(r)}$ are always in the column space of
$\bm{W}$.

###  3.4 Enhancing The Orthogonality of Householder Reflections for Stronger
Regularity

Besides the number of Householder reflections, the orthogonality of them also
impacts the regularity of HRA. Specifically, the supreme change of the weight
matrix $\bm{W}$, i.e., $\sup_{\bm{H}^{(r)}}\|\bm{W}-\bm{W}\bm{H}^{(r)}\|_{F}$
(or, equivalently,
$\sup_{\bm{U}_{r}}\|\bm{W}\bm{U}_{r}\bm{\Gamma}_{r}\bm{U}_{r}^{\top}\|_{F}$),
is achieved when $\bm{U}_{r}$ consists of the top-$r$ right singular vectors
of $\bm{W}$. In such a situation, $\bm{U}_{r}$ is an orthogonal matrix, i.e.,
$\bm{U}_{r}^{\top}\bm{U}_{r}=\bm{I}_{r}$. When the orthogonality is not held,
$\|\bm{W}\bm{U}_{r}\bm{\Gamma}_{r}\bm{U}_{r}^{\top}\|_{F}$ is reduced, as
illustrated in Figure 2. In other words, when adapting the pre-trained model,
enhancing the orthogonality of $\bm{U}_{r}$ imposes stronger regularity on the
adapter — it encourages the discrepancy between the target model and the
original pre-trained model while restricting the feasible domain of the
adapter’s parameter matrix.

Motivated by the above analysis, we can implement HRA with an orthogonality
regularizer. Typically, given a pre-trained model $\mathcal{M}$, we can adapt
$L$ weight matrices of the model based on a dataset $\mathcal{D}$ by solving
the following optimization problem:

$\displaystyle\begin{aligned}
\sideset{}{{}_{\\{\bm{U}_{r}^{(l)}\\}_{l=1}^{L}}}{\min}\text{Loss}(\mathcal{D};\\{\bm{U}_{r}^{(l)}\\}_{l=1}^{L})+\lambda\sideset{}{{}_{l=1}^{L}}{\sum}\|\bm{I}_{r}-(\bm{U}_{r}^{(l)})^{\top}\bm{U}_{r}^{(l)}\|_{F}^{2},\end{aligned}$
(6)

where $\bm{U}_{r}^{(l)}$ denotes the parameters of HRA for the $l$-th weight
matrix. In equation 6, the first term denotes the loss function, while the
second term is the proposed regularizer that encourages the orthogonality of
all $\bm{U}_{r}^{(l)}$’s, whose significance is controlled by $\lambda>0$.
Because it does not change the forward step of HRA, this regularizer only
increases the adaptation cost slightly.

As shown in Figure 1(a), by controlling the strength of the orthogonality
regularizer, we can achieve a trade-off between the model capacity and
regularity. When $\lambda=0$, the feasible domain of $\bm{U}_{r}$ is the set
of column-normalized matrices, and accordingly, the model capacity is
maximized. In contrast, when $\lambda\rightarrow\infty$, the feasible domain
of $\bm{U}_{r}$ is the set of orthogonal matrices (i.e., $\mathbb{O}_{d\times
r}$), leading to the strongest regularity. When $\lambda=\infty$, we implement
a strictly-orthogonal HRA based on Gram-Schmidt (GS) orthogonalization. For
each layer’s HRA adapter, we initialize its parameter matrix as
$\bm{V}_{r}\in\mathbb{R}^{d\times r}$ and applying Gram-Schmidt
orthogonalization [2] to it, i.e., $\bm{U}_{r}=\text{GS}(\bm{V}_{r})$. As
shown in Figure 1(a), in such a situation, the forward step of each adapter
becomes $\bm{z}=\bm{W}(\bm{I}-2\bm{U}_{r}\bm{U}_{r}^{\top})\bm{x}$, and the
computational complexity becomes $\mathcal{O}(d(r^{2}+r+d_{\text{out}}))$,
where additional $\mathcal{O}(dr^{2})$ operations are used for Gram-Schmidt
orthogonalization. According to Table 1, the complexity of this strictly-
orthogonal HRA is comparable to OFT [33] when $r=b$.

Table 2: Results (%) of various methods on GLUE development set. The best results on each dataset are shown in bold, and the second best results are shown in underline. We report the matched accuracy for MNLI, Matthew’s correlation for CoLA and average correlation for STS-B. Method | #Param (M) | MNLI | SST-2 | CoLA | QQP | QNLI | RTE | MRPC | STS-B | All
---|---|---|---|---|---|---|---|---|---|---
Full Fine-tune | 184 | 89.90 | 95.63 | 69.19 | 92.40 | 94.03 | 83.75 | 89.46 | 91.60 | 88.25
BitFit | 0.10 | 89.37 | 94.84 | 66.96 | 88.41 | 92.24 | 78.70 | 87.75 | 91.35 | 86.20
H-Adapter | 1.22 | 90.13 | 95.53 | 68.64 | 91.91 | 94.11 | 84.48 | 89.95 | 91.48 | 88.28
P-Adapter | 1.18 | 90.33 | 95.61 | 68.77 | 92.04 | 94.29 | 85.20 | 89.46 | 91.54 | 88.41
$\text{LoRA\ }_{r=8}$ | 1.33 | 90.65 | 94.95 | 69.82 | 91.99 | 93.87 | 85.20 | 89.95 | 91.60 | 88.50
AdaLoRA | 1.27 | 90.76 | 96.10 | 71.45 | 92.23 | 94.55 | 88.09 | 90.69 | 91.84 | 89.46
$\text{OFT\ }_{b=16}$ | 0.79 | 90.33 | 96.33 | 73.91 | 92.10 | 94.07 | 87.36 | 92.16 | 91.91 | 89.77
$\text{BOFT\ }^{m=2}_{b=8}$ | 0.75 | 90.25 | 96.44 | 72.95 | 92.10 | 94.23 | 88.81 | 92.40 | 91.92 | 89.89
$\text{HRA\ }_{r=8,~{}\lambda=0}$ | 0.66 | 90.70 | 96.45 | 73.70 | 91.29 | 94.66 | 88.45 | 93.69 | 91.86 | 90.10
$\text{HRA\ }_{r=8,~{}\lambda=10^{-5}}$ | 0.66 | 90.43 | 96.79 | 71.91 | 91.02 | 94.44 | 89.53 | 94.10 | 91.74 | 90.00
$\text{HRA\ }_{r=8,~{}\lambda=\infty}$ | 0.66 | 90.52 | 95.87 | 70.71 | 90.71 | 94.12 | 87.00 | 92.59 | 91.54 | 89.13

##  4 Experiments

To demonstrate the effectiveness of HRA, we conduct comparative experiments
for HRA and state-of-the-art adaptation methods on three models oriented to
different tasks, including DeBERTaV3-base [12] for natural language
understanding, LLaMA2-7B [41] for mathematical reasoning, and Stable Diffusion
[36] for conditional text-to-image generation. Typical results are shown
below. More results and implementation details are provided in Appendices A
and B.

In each experiment, we set the number of HRs (i.e., $r$) to ensure that HRA
has comparable or fewer trainable parameters than existing adaptation methods
(including LoRA, OFT, and their variants). Setting $\lambda\in(0,\infty)$
leads to the proposed HRA method, and we demonstrate the robustness of HRA to
the $\lambda$ in a wide range. By default, we set
$\lambda\in[10^{-5},10^{-3}]$ in the experiments. Furthermore, to analyze the
trade-off between model capacity and regularity, we consider two variants of
HRA: $i)$ the HRA without the orthogonality regularization ($\lambda=0$) and
$ii)$ the strictly-orthogonal HRA using GS orthogonalization (i.e.,
$\lambda=\infty$). For convenience, we denote HRAr,λ as the HRA learning $r$
Householder reflections per layer with an orthogonality regularizer weighted
by $\lambda$.

###  4.1 Natural Language Understanding

Figure 3: The robustness of HRA ($r=8$) to $\lambda$ on MRPC.

We adapt DeBERTaV3-base [12] by different methods and test the performance of
the adapted models on the General Language Understanding Evaluation (GLUE)
benchmark [45]. Following AdaLoRA [55] and BOFT [30], we consider eight tasks
of GLUE in this experiment, including two single-sentence tasks, three
similarity and paraphrase tasks, and three inference tasks. The experimental
results in Figure 1(b) and Table 2 show that using fewer trainable parameters,
HRA achieves the best or comparable results across all datasets and thus leads
to the best average performance. These results demonstrate the efficiency and
effectiveness of HRA.

In this experiment, the HRA without the regularization (i.e., $\lambda=0$) and
that using weak regularization (i.e., $\lambda=10^{-5}$) achieve comparable
adaptation results, while imposing strong regularity by strict orthogonality
(i.e., $\lambda=\infty$) harms the model performance. This interesting
phenomenon implies that the adaptation tasks in GLUE are challenging enough to
apply the adapter with sufficient capacity. Figure 3 shows the performance of
HRA on MRPC when $\lambda\in[10^{-7},10^{-3}]$. We can find that HRA achieves
relatively stable performance when $\lambda$ changes in a wide range.

###  4.2 Mathematical Reasoning of LLM

Figure 4: The robustness of HRA ($r=8$) to $\lambda$ in mathematical reasoning
tasks.

We adapt LLaMA2-7B [41] on the MetaMathQA dataset [50] by different adaptation
methods and test the adaptation performance on the GSM8K [5] and MATH [50]
validation sets. Following LoRA [16], each method only adapts the query and
value projection matrices of LLaMA2-7B. The results in Figure 1(a) show that
HRA outperforms its competitors on both GSM8K and MATH when we set $r=32$ and
make it have the same number of trainable parameters as BOFT [30].
Furthermore, the HRA with $r=16$ utilizes only half the trainable parameters
of BOFT yet still surpasses its performance, which demonstrates the efficiency
of HRA. In addition, the HRA with the orthogonality regularization achieves a
trade-off between model capacity and regularity. In Figure 4, we test the
robustness of HRA to $\lambda$, demonstrating that the performance of the HRA
with $r=8$ is stable when $\lambda\in[10^{-5},10^{-3}]$. For the HRA using
more HRs (e.g., $r=16$ and $32$), we set $\lambda=10^{-4}$ based on this
robustness test result, balancing the performance on GSM8K and MATH. As shown
in Table 1(a), the best performance is achieved when $r=32$ and
$\lambda=10^{-4}$.

| a [V] clock on top of green grass with sunflowers around it
---|---
|  |  |  |
a red [V] clock
|  |  |  |
| a [V] teapot on top of a purple rug in a forest
|  |  |  |
a [V] teapot floating on top of water
|  |  |  |
Original Images | LoRA | OFT |  HRA7,0 |  HRA${}_{7,10^{-3}}$ |  HRA7,∞
Figure 5: Qualitative results on subject-driven generation.

###  4.3 Controllable Text-to-Image Diffusion Models

Following OFT [33] and BOFT [30], we evaluate HRA on adapting pre-trained
Stable Diffusion (SD) [36] for subject-driven generation and controllable
generation, respectively. For a fair comparison, we employ experimental
procedures and evaluation metrics as the same as OFT [33]:

  * •

Subject-driven generation. Given several images of a specific subject and a
textual prompt, subject-driven generation aims to generate images of the same
subject in a context aligning with the prompt. Taking SD as the backbone
model, we evaluate the generation performance of different model adaptation
methods, including DreamBooth [37], LoRA [16], OFT and its variant COFT [33],
and our HRA. Following DreamBooth [37], we train and evaluate on generating 25
subjects, each of which corresponds to 30 prompts.

  * •

Controllable generation. Controllable generation aims to generate images
aligning with a textual prompt and additional control signals (such as facial
landmark annotations, canny edges, and segmentation maps). We conduct
experiments on three challenging controllable generation tasks: Canny edge to
image (C2I) on the COCO dataset [28], landmark to face (L2F) on the CelebA-HQ
dataset [19, 47], and segmentation map to image (S2I) on the ADE20K dataset
[57]. In this experiment, we use DreamBooth [37], ControlNet [54], T2I-Adapter
[32], LoRA [16], OFT and its variant COFT [33], and BOFT [30] as baselines.

Table 3: Results of various methods on subject-driven generation and
controllable generation. For each evaluation metric, the best result is shown
in bold, and the second best result is shown in underline. For HRA and its
variants, we set $r=7$ and $8$ for subjective-driven generation and
controllable generation, respectively.

Method | #Param | Subject-driven generation | #Param | Controllable generation
---|---|---|---|---
(M) | DINO↑ | CLIP-I↑ | CLIP-T↑ | LPIPS↑ | (M) | C2I | S2I | L2F
IoU↑ | F1↑ | mIoU↑ | mAcc↑ | aAcc↑ | Error↓
Real Images | - | 0.764 | 0.890 | - | 0.562 | - | - | - | - | - | - | -
DreamBooth | 859.52 | 0.614 | 0.778 | 0.239 | 0.737 | 859.52 | 0.049 | 0.093 | 7.72 | 14.40 | 33.61 | 146.19
ControlNet | - | - | - | - | - | 361.30 | 0.189 | 0.317 | 20.88 | 30.91 | 61.42 | 7.61
T2I-Adapter | - | - | - | - | - | 77.00 | 0.078 | 0.143 | 16.38 | 26.31 | 51.63 | 23.75
LoRA | 0.8 | 0.613 | 0.765 | 0.237 | 0.744 | 1.25 | 0.168 | 0.286 | 22.98 | 35.52 | 58.03 | 7.68
$\text{COFT\ }_{b=4}$ | 23.3 | 0.630 | 0.783 | 0.235 | 0.744 | 26.40 | 0.195 | 0.325 | 26.92 | 40.08 | 62.96 | 6.92
$\text{OFT\ }_{b=4}$ | 23.3 | 0.632 | 0.785 | 0.237 | 0.746 | 26.40 | 0.193 | 0.323 | 27.06 | 40.09 | 62.42 | 7.07
$\text{BOFT\ }^{m=4}_{r=8}$ | - | - | - | - | - | 20.76 | - | - | 28.83 | 41.24 | 67.74 | 5.67
$\text{HRA\ }_{r=7,8~{}\lambda=0}$ | 0.69 | 0.670 | 0.803 | 0.238 | 0.758 | 0.89 | 0.213 | 0.350 | 29.45 | 42.02 | 66.83 | 5.56
$\text{HRA\ }_{r=7,8~{}\lambda=10^{-3}}$ | 0.69 | 0.661 | 0.799 | 0.255 | 0.760 | 0.89 | 0.205 | 0.339 | 29.27 | 40.89 | 67.86 | 5.46
$\text{HRA\ }_{r=7,8~{}\lambda=\infty}$ | 0.69 | 0.651 | 0.794 | 0.274 | 0.778 | 0.89 | 0.201 | 0.334 | 28.15 | 40.22 | 64.95 | 11.11

Table 3 shows the quantitative experimental results. In the subject-driven
generation task, we evaluate three crucial aspects of generated images:
subject fidelity (DINO [3], CLIP-I [34]), textual prompt fidelity (CLIP-T
[34]), and sample diversity (LPIPS [56]). It can be observed that HRA achieves
remarkable improvement across almost all metrics. In addition, we find that
without the orthogonality ($\lambda=0$), HRA achieves the highest subject
fidelity while sacrificing textual prompt fidelity and sample diversity to
some extent, while the strictly-orthogonal HRA ($\lambda=\infty$) shows
opposite tendencies. Applying the orthogonality regularization with
$\lambda=10^{-3}$ makes HRA balance the performance in all the metrics.
Similarly, in the three controllable generation tasks, HRA demonstrates
stronger and more precise control compared to the baselines. However, in these
tasks, the strictly-orthogonal HRA leads to suboptimal performance. It means
that these tasks require our adapter to have sufficient capacity, but the
strict orthogonality constrains its capacity too much. In both tasks, HRA
demonstrates the smallest number of trainable parameters among the compared
methods. Figures 5 and 6 provide typical qualitative results, demonstrating
that the images generated based on HRA have good visual effects and well-
aligned semantics.

##  5 Conclusion

In this study, we have proposed a simple but effective Householder reflection
adaptation method and have demonstrated its usefulness in various adaptation
tasks. The proposed HRA method bridges the gap between low-rank and orthogonal
adaptation strategies. It simplifies the implementation of OFT while
inheriting its theoretical guarantees on the retention of pre-training
knowledge. In addition, we show that controlling the orthogonality of the
Householder reflections can achieve the trade-off between HRA’s model capacity
and its regularity. In the future, we would like to improve HRA for practical
applications, including accelerating its computation, reducing its memory
cost, exploring other regularizers for parameter matrices, and adjusting the
weights of the regularizers automatically. We also plan to test HRA on
adapting more advanced LLMs, e.g., LLaMA3 and Grok-1.

Ref. Img | Control | LoRA | OFT |  HRA8,0 |  HRA${}_{8,10^{-5}}$ |  HRA8,∞
---|---|---|---|---|---|---
|  |  |  |  |  |
|  | Prompt: A baseball game being played.
|  |  |  |  |  |
|  | Prompt: A plate with a slice of orange on it.
|  |  |  |  |  |
|  | Prompt: A man smiling for the camera.
|  |  |  |  |  |
|  | Prompt: A young boy smiling for the camera.
|  |  |  |  |  |
|  | Prompt: A brick building.
|  |  |  |  |  |
|  | Prompt: A tree stump.
Figure 6: Comparisons for various adaptation methods on controllable
generation, in which the control signals include Canny edges, face landmarks,
and semantic segmentation results of reference images.

## References

  * [1] Teodor Banica and Roland Speicher.  Liberation of orthogonal lie groups.  Advances in Mathematics, 222(4):1461–1501, 2009.
  * [2] Åke Björck.  Numerics of gram-schmidt orthogonalization.  Linear Algebra and Its Applications, 197:297–316, 1994.
  * [3] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650–9660, 2021.
  * [4] Beidi Chen, Tri Dao, Kaizhao Liang, Jiaming Yang, Zhao Song, Atri Rudra, and Christopher Re.  Pixelated butterfly: Simple and efficient sparse training for neural network models.  In International Conference on Learning Representations, 2021.
  * [5] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al.  Training verifiers to solve math word problems.  arXiv preprint arXiv:2110.14168, 2021.
  * [6] Tri Dao, Nimit Sohoni, Albert Gu, Matthew Eichhorn, Amit Blonder, Megan Leszczynski, Atri Rudra, and Christopher Ré.  Kaleidoscope: An efficient, learnable representation for all structured linear maps.  In International Conference on Learning Representations, 2019.
  * [7] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer.  Qlora: Efficient finetuning of quantized llms.  Advances in Neural Information Processing Systems, 36, 2024.
  * [8] Zihao Fu, Haoran Yang, Anthony Man-Cho So, Wai Lam, Lidong Bing, and Nigel Collier.  On the effectiveness of parameter-efficient fine-tuning.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 12799–12807, 2023.
  * [9] Gene H Golub and Charles F Van Loan.  Matrix computations.  JHU press, 2013.
  * [10] Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May.  Warp: Word-level adversarial reprogramming.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4921–4933, 2021.
  * [11] Chuchu Han, Ruochen Zheng, Changxin Gao, and Nong Sang.  Complementation-reinforced attention network for person re-identification.  IEEE Transactions on Circuits and Systems for Video Technology, 30(10):3433–3445, 2019.
  * [12] Pengcheng He, Jianfeng Gao, and Weizhu Chen.  Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing.  arXiv preprint arXiv:2111.09543, 2021.
  * [13] Wu Hecong.  ControlLoRA: A Lightweight Neural Network To Control Stable Diffusion Spatial Information, 2 2023.
  * [14] Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al.  Scaling laws for autoregressive generative modeling.  arXiv preprint arXiv:2010.14701, 2020.
  * [15] Alston S Householder.  Unitary triangularization of a nonsymmetric matrix.  Journal of the ACM (JACM), 5(4):339–342, 1958.
  * [16] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al.  Lora: Low-rank adaptation of large language models.  In International Conference on Learning Representations, 2021.
  * [17] Nam Hyeon-Woo, Moon Ye-Bin, and Tae-Hyun Oh.  Fedpara: Low-rank hadamard product for communication-efficient federated learning.  In International Conference on Learning Representations, 2021.
  * [18] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.  Scaling laws for neural language models.  arXiv preprint arXiv:2001.08361, 2020.
  * [19] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of gans for improved quality, stability, and variation.  arXiv preprint arXiv:1710.10196, 2017.
  * [20] Hyunjik Kim, George Papamakarios, and Andriy Mnih.  The lipschitz constant of self-attention.  In International Conference on Machine Learning, pages 5562–5571. PMLR, 2021.
  * [21] Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki Markus Asano.  Vera: Vector-based random matrix adaptation.  arXiv preprint arXiv:2310.11454, 2023.
  * [22] Jitin Krishnan, Hemant Purohit, and Huzefa Rangwala.  Diversity-based generalization for neural unsupervised text classification under domain shift.  In ECML-PKDD, 2020.
  * [23] Mingu Lee, Jinkyu Lee, Hye Jin Jang, Byeonggeun Kim, Wonil Chang, and Kyuwoong Hwang.  Orthogonality constrained multi-head attention for keyword spotting.  In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 86–92. IEEE, 2019.
  * [24] Brian Lester, Rami Al-Rfou, and Noah Constant.  The power of scale for parameter-efficient prompt tuning.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059, 2021.
  * [25] Qiyang Li, Saminul Haque, Cem Anil, James Lucas, Roger Grosse, and Jörn-Henrik Jacobsen.  Preventing gradient attenuation in lipschitz constrained convolutional networks.  In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pages 15390–15402, 2019.
  * [26] Xiang Lisa Li and Percy Liang.  Prefix-tuning: Optimizing continuous prompts for generation.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582–4597, 2021.
  * [27] Yixiao Li, Yifan Yu, Chen Liang, Nikos Karampatziakis, Pengcheng He, Weizhu Chen, and Tuo Zhao.  Loftq: Lora-fine-tuning-aware quantization for large language models.  In The Twelfth International Conference on Learning Representations, 2023.
  * [28] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014.
  * [29] Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen.  Dora: Weight-decomposed low-rank adaptation.  arXiv preprint arXiv:2402.09353, 2024.
  * [30] Weiyang Liu, Zeju Qiu, Yao Feng, Yuliang Xiu, Yuxuan Xue, Longhui Yu, Haiwen Feng, Zhen Liu, Juyeon Heo, Songyou Peng, et al.  Parameter-efficient orthogonal finetuning via butterfly factorization.  In International Conference on Learning Representations, 2024.
  * [31] Fanxu Meng, Zhaohui Wang, and Muhan Zhang.  Pissa: Principal singular values and singular vectors adaptation of large language models.  arXiv preprint arXiv:2404.02948, 2024.
  * [32] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan.  T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4296–4304, 2024.
  * [33] Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf.  Controlling text-to-image diffusion by orthogonal finetuning.  Advances in Neural Information Processing Systems, 36, 2024.
  * [34] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [35] Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He.  Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale.  In International conference on machine learning, pages 18332–18346. PMLR, 2022.
  * [36] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  * [37] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500–22510, 2023.
  * [38] Jure Sokolić, Raja Giryes, Guillermo Sapiro, and Miguel RD Rodrigues.  Robust large margin deep neural networks.  IEEE Transactions on Signal Processing, 65(16):4265–4280, 2017.
  * [39] Yi-Lin Sung, Varun Nair, and Colin A Raffel.  Training neural networks with fixed sparse masks.  Advances in Neural Information Processing Systems, 34:24193–24205, 2021.
  * [40] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto.  Stanford alpaca: an instruction-following llama model (2023).  URL https://github. com/tatsu-lab/stanford_alpaca, 2023.
  * [41] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288, 2023.
  * [42] Yusuke Tsuzuku, Issei Sato, and Masashi Sugiyama.  Lipschitz-margin training: Scalable certification of perturbation invariance for deep neural networks.  Advances in neural information processing systems, 31, 2018.
  * [43] Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi.  Dylora: Parameter-efficient tuning of pre-trained models using dynamic search-free low-rank adaptation.  In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3274–3287, 2023.
  * [44] Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou, and Daniel Cer.  Spot: Better frozen model adaptation through soft prompt transfer.  In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5039–5059, 2022.
  * [45] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  arXiv preprint arXiv:1804.07461, 2018.
  * [46] Jiyao Wei, Jian Liao, Zhenfei Yang, Suge Wang, and Qiang Zhao.  Bilstm with multi-polarity orthogonal attention for implicit sentiment analysis.  Neurocomputing, 383:165–173, 2020.
  * [47] Weihao Xia, Yujiu Yang, Jing-Hao Xue, and Baoyuan Wu.  Tedigan: Text-guided diverse face image generation and manipulation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2256–2265, 2021.
  * [48] Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang.  Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment.  arXiv preprint arXiv:2312.12148, 2023.
  * [49] Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhengsu Chen, XIAOPENG ZHANG, and Qi Tian.  Qa-lora: Quantization-aware low-rank adaptation of large language models.  In The Twelfth International Conference on Learning Representations, 2023.
  * [50] Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu.  Metamath: Bootstrap your own mathematical questions for large language models.  arXiv preprint arXiv:2309.12284, 2023.
  * [51] Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel.  Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models.  In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1–9, 2022.
  * [52] Aston Zhang, Alvin Chan, Yi Tay, Jie Fu, Shuohang Wang, Shuai Zhang, Huajie Shao, Shuochao Yao, and Roy Ka-Wei Lee.  On orthogonality constraints for transformers.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, volume 2, pages 375–382. Association for Computational Linguistics, 2021.
  * [53] Feiyu Zhang, Liangzhi Li, Junhao Chen, Zhouqiang Jiang, Bowen Wang, and Yiming Qian.  Increlora: Incremental parameter allocation method for parameter-efficient fine-tuning.  arXiv preprint arXiv:2308.12043, 2023.
  * [54] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023.
  * [55] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao.  Adaptive budget allocation for parameter-efficient fine-tuning.  In The Eleventh International Conference on Learning Representations, 2023.
  * [56] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018.
  * [57] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba.  Scene parsing through ade20k dataset.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 633–641, 2017.

##  Appendix A Implementation Details

###  A.1 Natural Language Understanding

Following BOFT [30], we use the same batch size (i.e., $32$ for each task),
maximum sequence length, and tune the learning rate, number of training
epochs, warm-up steps, as well as $\lambda$ in the R-HRA method. Additionally,
the dropout rate is consistently set to 1E-01. We adapt every linear layer in
DeBERTaV3 and froze the pre-trained weights for all tasks. Both training and
testing are conducted on 7 NVIDIA GeForce RTX 3090 GPUs. Detailed
hyperparameter setups are presented in Table 4.

Table 4: The hyperparameters for DeBERTaV3-base on tasks included in the GLUE benchmark. Method | Dataset | MNLI | SST-2 | CoLA | QQP | QNLI | RTE | MRPC | STS-B
---|---|---|---|---|---|---|---|---|---
HRAr=8,λ=0 | Epochs | 8 | 10 | 34 | 12 | 12 | 11 | 60 | 39
Learning Rate | 1E-02 | 3E-03 | 9E-03 | 8E-03 | 1E-02 | 5E-03 | 6E-03 | 5E-03
Warm Up Steps | 1000 | 500 | 100 | 1000 | 500 | 50 | 50 | 50
Max Seq. Len. | 256 | 128 | 64 | 320 | 512 | 320 | 320 | 128
HRA${}_{r=8,\lambda=10^{-6}}$ | Epochs | 5 | 8 | 12 | 11 | 6 | 44 | 35 | 30
Learning Rate | 2E-03 | 2E-03 | 2E-03 | 2E-03 | 2E-03 | 1E-03 | 1E-03 | 9E-04
Warm Up Steps | 1000 | 500 | 100 | 1000 | 500 | 50 | 50 | 50
Max Seq. Len. | 256 | 128 | 64 | 320 | 512 | 320 | 320 | 128

###  A.2 Mathematical Reasoning

Following BOFT [30], we fine-tune the LLaMA2-7B model on the first generated
512 tokens, which is sufficient for these two tasks. In all experiments, we
fix the training epoch as 2 and use the cosine learning scheduler and the
warm-up ratio is set as 0.005. We follow the evaluation tools in MetaMathQA
[50], where they use the Alpaca [40] prompt and evaluate the model in zero-
shot. The generation temperature is set as 0 for both tasks. In our proposed
method, the learning rate of HRAλ=0 is set to 1E-05, HRA${}_{\lambda=10^{-4}}$
is set to 3E-05, and HRAλ=∞ is set to 1E-03. Both training and testing are
conducted on 8 NVIDIA L20 GPUs.

###  A.3 Subject-driven Generation

Following OFT [33], we use the same AdamW optimizer with a weight decay of
1E-02 and fine-tune the linear layers including $\bm{W}_{q}$, $\bm{W}_{k}$,
$\bm{W}_{v}$, and $\bm{W}_{o}$ in the UNet model. The learning rate is set to
7E-06, with a batch size of 1, and the number of training steps is
approximately 2000 steps. Both training and testing are conducted on 7 NVIDIA
GeForce RTX 3090 GPUs.

###  A.4 Controllable Generation

Our data processing procedure and architecture design are consistent with OFT
[33]. In addition to injecting trainable HRA weights into the stable diffusion
model, we employ a lightweight neural network [13] to encode the control
signals. We fine-tune the model for 11 epochs for C2I and 20 epochs for L2F
and S2I. The learning rate is set to 3E-05 with a batch size of 8 for all
three tasks. Both training and testing are conducted on 8 NVIDIA RTX A6000
GPUs.

##  Appendix B More Experimental Results

###  B.1 Subject-driven Generation

| a [V] backpack with a city in the background
---|---
|  |  |  |
a [V] backpack with a blue house in the background
|  |  |  |
| a [V] stuffed animal on the beach
|  |  |  |
a [V] stuffed animal on top of a wooden floor
|  |  |  |
| a [V] cat wearing a red hat
|  |  |  |
a [V] cat wearing pink glasses
|  |  |  |
Original Images | LoRA | OFT |  HRA7,0 |  HRA${}_{7,10^{-3}}$ |  HRA7,∞
Figure 7: More qualitative results on subject-driven generation. | a [V] dog wearing a santa hat
---|---
|  |  |  |
a [V] dog wearing a rainbow scarf
|  |  |  |
| a [V] sneaker with a tree and autumn leaves in the background
|  |  |  |
a [V] sneaker on top of a dirt road
|  |  |  |
| a [V] toy on top of pink fabric
|  |  |  |
a [V] toy on top of a white rug
|  |  |  |
Original Images | LoRA | OFT |  HRA7,0 |  HRA${}_{7,10^{-3}}$ |  HRA7,∞
Figure 8: More qualitative results on subject-driven generation. | a [V] backpack on a cobblestone street
---|---
|  |  |  |
a [V] backpack with the Eiffel Tower in the background
|  |  |  |
| a [V] vase in the snow
|  |  |  |
a [V] vase with a wheat field in the background
|  |  |  |
| a [V] cat on top of a wooden floor
|  |  |  |
a [V] cat with a city in the background
|  |  |  |
Original Images | LoRA | OFT |  HRA7,0 |  HRA${}_{7,10^{-3}}$ |  HRA7,∞
Figure 9: More qualitative results on subject-driven generation.

###  B.2 Controllable Generation

####  B.2.1 Canny Edge to Image

Ref. Img | Control | LoRA | OFT |  HRA8,0 |  HRA${}_{8,10^{-5}}$ |  HRA8,∞
---|---|---|---|---|---|---
|  |  |  |  |  |
|  | Prompt: Carrots on a cutting board.
|  |  |  |  |  |
|  | Prompt: A zebra.
|  |  |  |  |  |
|  | Prompt: A sheep crossing a dirt road.
|  |  |  |  |  |
|  | Prompt: A couple sitting on a bench.
|  |  |  |  |  |
|  | Prompt: A person throwing a frisbee.
|  |  |  |  |  |
|  | Prompt: A beach at sunset.
|  |  |  |  |  |
|  | Prompt: A table with a plate of food and a cup of coffee.
Figure 10: More qualitative results on canny edge to image.

####  B.2.2 Landmark to Face

Ref. Img | Control | LoRA | OFT |  HRA8,0 |  HRA${}_{8,10^{-5}}$ |  HRA8,∞
---|---|---|---|---|---|---
|  |  |  |  |  |
|  | Prompt: A man with sunglasses on.
|  |  |  |  |  |
|  | Prompt: A man with a tattoo on his arm.
|  |  |  |  |  |
|  | Prompt: A young man in a suit and tie.
|  |  |  |  |  |
|  | Prompt: A man with long hair.
|  |  |  |  |  |
|  | Prompt: A woman with blonde hair.
|  |  |  |  |  |
|  | Prompt: A man with long hair.
|  |  |  |  |  |
|  | Prompt: A woman with long blonde hair.
Figure 11: More qualitative results on landmark to face.

####  B.2.3 Segmentation to image

Ref. Img | Control | LoRA | OFT |  HRA8,0 |  HRA${}_{8,10^{-5}}$ |  HRA8,∞
---|---|---|---|---|---|---
|  |  |  |  |  |
|  | Prompt: A car parked in front of a house.
|  |  |  |  |  |
|  | Prompt: A field.
|  |  |  |  |  |
|  | Prompt: The coast.
|  |  |  |  |  |
|  | Prompt: A living room with two chairs and a table.
|  |  |  |  |  |
|  | Prompt: A man playing a game of pool.
|  |  |  |  |  |
|  | Prompt: A building with a car parked in front of it.
|  |  |  |  |  |
|  | Prompt: A castle in scotland.
Figure 12: More qualitative results on segmentation to image.

###  B.3 Case Studies in Mathematical Reasoning

Case Study 1 Question: James likes to check the coin return of the vending
machine for change. One day he finds a quarter, two nickels, and 7 dimes. How
much money in cents does James have? (Ground-truth answer is 105) Llama-2
Answer: Based on the information provided in the instruction, James has a
total of: * 1 quarter = 25 cents * 2 nickels = 50 cents * 7 dimes = 70 cents
Therefore, James has a total of 145 cents. The answer is: 145 HRA32,0 Answer:
First, we need to convert the coins to cents. One quarter is worth 25 cents.
Two nickels are worth 2 * 5 = 10 cents each. So, two nickels are worth 10
cents each. Seven dimes are worth 10 * 0.10 = 1 cents each. So, seven dimes
are worth 7 cents each. Now, we can add up the value of the coins. 25 cents +
10 cents + 7 cents = 42 cents. Therefore, James has 42 cents. The answer is:
42 HRA${}_{32,10^{-4}}$ Answer: 1 quarter = 25 cents 2 nickels = 2 * 5 = 10
cents 7 dimes = 7 * 10 = 70 cents So, James has 25 + 10 + 70 = 105 cents. The
answer is: 105 HRA32,∞ Answer: First, we need to convert the coins to cents. 1
quarter is equal to 25 cents. 2 nickels is equal to 2 * 5 = 10 cents. 7 dimes
is equal to 7 * 10 = 70 cents. Now, we can add up the total amount of money.
25 + 10 + 70 = 105 cents. Therefore, James has 105 cents. The answer is: 105

Case Study 2 Question: Jillian’s handbag cost $20 less than 3 times as much as
her shoes cost. If her shoes cost $80, how much did her bag cost? (Ground-
truth answer is 220) Llama-2 Answer: f Jillian’s shoes cost $80 and her
handbag cost $20 less than 3 times as much as her shoes cost, then we can set
up the following equation: 3x = $80 - $20 Solving for x, we get: x = $80 - $20
x = $60 So, Jillian’s handbag cost $60. The answer is: 60 HRA32,0 Answer: We
know that Jillian’s shoes cost $80. Her bag cost $20 less than 3 times as much
as her shoes cost. So, her bag cost $80 * 3 - $20 = $240 - $20 = $220.
Therefore, Jillian’s bag cost $220. The answer is: 220 HRA${}_{32,10^{-4}}$
Answer: Jillian’s shoes cost $80. Her bag cost $20 less than 3 times as much
as her shoes cost. So, her bag cost 3 * $80 - $20 = $240 - $20 = $220.
Therefore, her bag cost $220. The answer is: 220 HRA32,∞ Answer: Jillian’s
shoes cost $80. Her handbag cost $20 less than 3 times as much as her shoes
cost. So, her handbag cost 3 * $80 - $20 = $210 - $20 = $190. Therefore, her
handbag cost $190. The answer is: 190

Case Study 3 Question: Claire makes a 3 egg omelet every morning for
breakfast. How many dozens of eggs will she eat in 4 weeks? (Ground-truth
answer is 7) Llama-2 Answer: Claire makes 3 egg omelets every morning, so she
eats 3 x 21 (days in 4 weeks) = 63 eggs in 4 weeks. Therefore, Claire will eat
63 eggs in 4 weeks. So, the answer is 63 eggs. The answer is: 63 HRA32,0
Answer: Claire makes a 3 egg omelet every morning for breakfast. In 4 weeks,
there are 4 weeks x 7 days = 28 days. So, in 28 days, Claire will eat 28 x 3 =
84 eggs. Since there are 12 eggs in a dozen, Claire will eat 84 / 12 = 7 dozen
eggs. Therefore, Claire will eat 7 dozens of eggs in 4 weeks. The answer is: 7
HRA${}_{32,10^{-4}}$ Answer: Claire makes a 3 egg omelet every morning for
breakfast. In 4 weeks, there are 4 x 7 = 28 days. So, in 28 days, Claire will
eat 28 x 3 = 84 eggs. Since there are 12 eggs in a dozen, Claire will eat 84 /
12 = 7 dozen of eggs. The answer is: 7 HRA32,∞ Answer: Claire makes a 3 egg
omelet every morning for breakfast. In 4 weeksFirst, Marie ordered 1 chicken
meal that costs $12. Next, she ordered 5 packs of milk that cost $3 each, so 5
* $3 = $15 for the milk. Then, tshere a ordered 4 weeks x 7 days = 28 days.
So, in 28 days, Claire will eat 28 x 3 = 84 eggs. To find out how many dozens
of eggs she will eat, we divide 84 by 12. 84 / 12 = 7 Therefore, Claire will
eat 7 dozens of eggs in 4 weeks. The answer is: 7

Case Study 4 Question: There are 4 roses in the vase. There are 7 more dahlias
than roses in the vase. How many flowers are there in the vase in total?
(Ground-truth answer is 15) Llama-2 Answer: We know that there are 4 roses in
the vase, and there are 7 more dahlias than roses. This means that there are 4
+ 7 = 11 flowers in total in the vase. So, the answer is 11 flowers in total
in the vase. The answer is: 11 HRA32,0 Answer: There are 4 roses in the vase.
There are 7 more dahlias than roses, so there are 4 + 7 = 11 dahlias in the
vase. In total, there are 4 roses + 11 dahlias = 15 flowers in the vase. The
answer is: 15 HRA${}_{32,10^{-4}}$ Answer: There are 4 roses in the vase.
There are 7 more dahlias than roses in the vase, so there are 4 + 7 = 11
dahlias in the vase. In total, there are 4 roses + 11 dahlias = 15 flowers in
the vase. The answer is: 15 HRA32,∞ Answer: There are 4 roses in the vase.
There are 7 more dahlias than roses in the vase. So, there are 4 + 7 = 11
dahlias in the vase. In total, there are 4 roses + 11 dahlias = 15 flowers in
the vase. The answer is: 15

Case Study 5 Question: Andy plants 90 geraniums and 40 fewer petunias that
geraniums. How many flowers does he plant total? (Ground-truth answer is 140)
Llama-2 Answer: Based on the information provided, Andy plants 90 geraniums
and 40 fewer petunias than geraniums. This means he plants a total of: 90
geraniums + 40 fewer petunias = 130 flowers So, Andy plants 130 flowers in
total. The answer is: 130 HRA32,0 Answer: Andy plants 90 geraniums. He plants
40 fewer petunias than geraniums, so he plants 90 - 40 = 50 petunias. In
total, Andy plants 90 geraniums + 50 petunias = 140 flowers. The answer is:
140 HRA${}_{32,10^{-4}}$ Answer: Andy plants 90 geraniums. He plants 40 fewer
petunias than geraniums, so he plants 90 - 40 = 50 petunias. The total number
of flowers he plants is 90 + 50 = 140. The answer is: 140 HRA32,∞ Answer: Andy
plants 90 geraniums. He plants 40 fewer petunias than geraniums, so he plants
90 - 40 = 50 petunias. The total number of flowers he plants is 90 geraniums +
50 petunias = 140 flowers. The answer is: 140

Case Study 6 Question: Cody and Trevor had 12 sandwiches. Cody ate a third of
the sandwiches, and Trevor ate a quarter of the sandwiches. How many
sandwiches are left? (Ground-truth answer is 5) Llama-2 Answer: Cody ate a
third of the sandwiches, which means he ate 12 / 3 = 4 sandwiches. Trevor ate
a quarter of the sandwiches, which means he ate 12 / 4 = 3 sandwiches. So,
there are 4 + 3 = 7 sandwiches left. The answer is: 7 HRA32,0 Answer: Cody ate
1/3 of the sandwiches, so he ate 12/3 = 4 sandwiches. Trevor ate 1/4 of the
sandwiches, so he ate 12/4 = 3 sandwiches. After Cody and Trevor ate, there
are 12 - 4 - 3 = 5 sandwiches left. The answer is: 5 HRA${}_{32,10^{-4}}$
Answer: Cody ate a third of the sandwiches, so he ate 12/3 = 4 sandwiches.
Trevor ate a quarter of the sandwiches, so he ate 12/4 = 3 sandwiches. After
Cody and Trevor ate their sandwiches, there are 12 - 4 - 3 = 5 sandwiches
left. The answer is: 5 HRA32,∞ Answer: Cody ate 1/3 of the sandwiches, so he
ate 12/3 = 4 sandwiches. Trevor ate 1/4 of the sandwiches, so he ate 12/4 = 3
sandwiches. After Cody and Trevor ate their sandwiches, there are 12 - 4 - 3 =
5 sandwiches left. The answer is: 5

Case Study 7 Question: Gerald and Julia divided $100 in the ratio 3:2. If
Gerald spent $10 on a book, how much money did he have left? (Ground-truth
answer is 50) Llama-2 Answer: Gerald had $30 and spent $10 on a book. The
money he has left is subtracting the amount he spent from the amount he had:
$30 - $10 = $20. The answer is 20 HRA32,0 Answer: The ratio of Gerald’s share
to Julia’s share is 3:2. This means that for every 3 parts of the money,
Gerald gets 1 part. And for every 2 parts of the money, Julia gets 1 part. So,
Gerald’s share is 3/3 * $100 = $100. And Julia’s share is 2/3 * $100 = $66.67.
Gerald spent $10 on a book, so he has $100 - $10 = $90 left. The answer is: 90
HRA${}_{32,10^{-4}}$ Answer: The ratio 3:2 means that for every 3 parts that
Gerald receives, Julia receives 2 parts. So, Gerald received 3/5 * $100 = $60.
And Julia received 2/5 * $100 = $40. Gerald spent $10 on a book, so he has $60
- $10 = $50 left. The answer is: 50 HRA32,∞ Answer: Gerald and Julia divided
$100 in the ratio 3:2, which means Gerald received 3 parts and Julia received
2 parts. Gerald received 3/5 of the total amount, which is $100 * 3/5 = $60.
Julia received 2/5 of the total amount, which is $100 * 2/5 = $40. Gerald
spent $10 on a book, so he has $60 - $10 = $50 left. The answer is: 50

Case Study 8 Question: There were 50 cats on a rock. Four boats came and
carried away 5 cats each, and later, 3/5 of the remaining cats ran after a
mouse they’d seen. How many cats were left on the rock? (Ground-truth answer
is 12) Llama-2 Answer: There were 50 cats on the rock to start with. Four
boats came and carried away 5 cats each, so the number of cats left on the
rock is: 50 - 4 x 5 = 50 - 20 = 30 cats Later, 3/5 of the remaining cats
(which is 30 cats) ran after a mouse they’d seen. So, the number of cats left
on the rock is: 30 - 3/5 = 30 - 6 = 24 cats. The answer is 24 HRA32,0 Answer:
There were 50 cats on the rock. Four boats came and carried away 5 cats each,
so a total of 4 * 5 = 20 cats were carried away. The remaining number of cats
on the rock is 50 - 20 = 30 cats. Then, 3/5 of the remaining cats ran after a
mouse, so 3/5 * 30 = 24 cats ran away. The number of cats left on the rock is
30 - 24 = 6 cats. The answer is: 6 HRA${}_{32,10^{-4}}$ Answer: There were 50
cats on the rock. Four boats came and carried away 5 cats each, so 4 * 5 = 20
cats were carried away. After the cats were carried away, there were 50 - 20 =
30 cats left on the rock. Then, 3/5 of the remaining cats ran after a mouse,
so 3/5 * 30 = 18 cats ran away. Therefore, there were 30 - 18 = 12 cats left
on the rock. The answer is: 12 HRA32,∞ Answer: There were 50 cats on the rock.
Four boats came and carried away 5 cats each, so 4 x 5 = 20 cats were carried
away. The remaining cats on the rock are 50 - 20 = 30 cats. 3/5 of the
remaining cats ran after a mouse, so 3/5 x 30 = 18 cats ran away. The number
of cats left on the rock is 30 - 18 = 12 cats. The answer is: 12

Case Study 9 Question: Michael is traveling on the interstate at an average
speed of 50km/hr (taking into account all the necessary stops for gas,
relaxation, etc.). If he goes on like this for 3 days, and the distance
between Alaska and Texas is 6000 km, what percentage of this distance has he
covered? (Ground-truth answer is 60) Llama-2 Answer: Michael travels at an
average speed of 50 km/hr. To find the distance he covers in one day, we
multiply his speed by the number of hours in a day. Assuming an 8-hour driving
day, Michael covers: 50 km/hr x 8 hours = 400 km per day Since Michael travels
for 3 days, he covers a total distance of: 400 km/day x 3 days = 1200 km The
distance between Alaska and Texas is 6000 km. To find the percentage of this
distance that Michael has covered, we divide the distance he covered (1200 km)
by the total distance (6000 km) and multiply by 100: (1200 km ÷ 6000 km) x 100
= 20% So, Michael has covered 20% of the distance between Alaska and Texas.
The answer is 20 HRA32,0 Answer: Michael is traveling at an average speed of
50 km/hr. He goes on for 3 days, so he covers a distance of 50 km/hr * 3 days
= 150 km. The distance between Alaska and Texas is 6000 km. To find out what
percentage of the distance he has covered, we need to divide the distance he
has covered by the total distance and multiply by 100. So, 150 km / 6000 km *
100 = 2.5% Therefore, Michael has covered 2.5% of the distance between Alaska
and Texas. The answer is: 2.5 HRA${}_{32,10^{-4}}$ Answer: Michael is
traveling at an average speed of 50km/hr for 3 days. So, he covers a distance
of 50km/hr * 24 hours/day * 3 days = 3600km. The distance between Alaska and
Texas is 6000km. To find the percentage of the distance covered, we divide the
distance covered by the total distance and multiply by 100. So, the percentage
of the distance covered is (3600km / 6000km) * 100 = 60% The answer is: 60
HRA32,∞ Answer: Michael is traveling at an average speed of 50 km/hr for 3
days. So, he covers a distance of 50 km/hr x 24 hours/day x 3 days = 3600 km.
The distance between Alaska and Texas is 6000 km. To find the percentage of
the distance covered, we divide the distance covered by the total distance and
multiply by 100. So, (3600 km / 6000 km) x 100 = 60% Therefore, Michael has
covered 60% of the distance between Alaska and Texas. The answer is: 60

Case Study 10 Question: Julia’s boat sprang a leak while she was out on the
lake. The boat was taking on two liters of water for every ten feet she rowed
back towards shore. It took her sixteen seconds to row twenty feet. The shore
was 64 seconds away. How much water had the boat taken on by the time Julia
reached shore? (Ground-truth answer is 16) Llama-2 Answer: To find out how
much water the boat took on by the time Julia reached shore, we need to use
the information provided in the problem. We know that the boat takes on 2
liters of water for every 10 feet it rows back towards shore. So, if Julia
rowed 20 feet (which takes her 16 seconds), the boat took on 2 liters of
water. We also know that Julia rowed a total of 64 seconds to reach the shore.
So, if she rowed 20 feet in 16 seconds, she rowed a total of 40 feet (20 feet
x 2) in the remaining 44 seconds. This means the boat took on 8 liters of
water (2 liters per 10 feet x 40 feet). Therefore, by the time Julia reached
the shore, the boat had taken on 8 liters of water. The answer is 8 HRA32,0
Answer: Julia rowed 20 feet in 16 seconds, so she rowed 20/16 = 1.25 feet per
second. The boat takes on 2 liters of water for every 10 feet rowed, so it
takes on 2/10 = 0.2 liters of water per second. Since Julia rowed 1.25 feet
per second, the boat takes on 0.2 * 1.25 = 0.25 liters of water per second.
The shore is 64 seconds away, so the boat takes on 0.25 * 64 = 16 liters of
water by the time Julia reaches shore. The answer is: 16 HRA${}_{32,10^{-4}}$
Answer: Julia rowed 20 feet in 16 seconds, so she rowed at a rate of 20/16 =
1.25 feet per second. Since she was taking on 2 liters of water for every 10
feet she rowed, she was taking on 2/10 * 1.25 = 0.25 liters of water per
second. Since she was taking on 0.25 liters of water per second, and it took
her 64 seconds to reach shore, the boat had taken on 0.25 * 64 = 16 liters of
water by the time she reached shore. The answer is: 16 HRA32,∞ Answer: Julia
rowed 20 feet in 16 seconds. To find the rate at which the boat was taking on
water, we divide the amount of water by the time it took to row 20 feet. So,
the rate at which the boat was taking on water is 2 liters/10 feet * 16
seconds = 32 liters/10 feet. Since Julia rowed 20 feet, the boat took on 32
liters/10 feet * 20 feet = 640 liters of water. Since the shore was 64 seconds
away, the boat took on 640 liters of water in 64 seconds. Therefore, the boat
took on 640 liters of water by the time Julia reached shore. The answer is:
640

Case Study 11 Question: In a dance class of 20 students, 20% enrolled in
contemporary dance, 25% of the remaining enrolled in jazz dance, and the rest
enrolled in hip-hop dance. What percentage of the entire students enrolled in
hip-hop dance? (Ground-truth answer is 60) Llama-2 Answer: Total number of
students in the dance class = 20 Percentage of students enrolled in
contemporary dance = 20% of 20 = 4 Percentage of students enrolled in jazz
dance = 25% of 16 (since 20% are enrolled in contemporary dance) = 4
Percentage of students enrolled in hip-hop dance = Remaining students - 4 = 12
Therefore, 12% of the entire students in the dance class are enrolled in hip-
hop dance. The answer is 12 HRA32,0 Answer: First, we need to find out how
many students enrolled in contemporary dance. 20% of 20 students is 0.20 * 20
= 4 students. So, there are 20 - 4 = 16 students remaining. Next, we need to
find out how many students enrolled in jazz dance. 25% of 16 students is 0.25
* 16 = 4 students. So, there are 16 - 4 = 12 students remaining. Finally, we
need to find out how many students enrolled in hip-hop dance. The remaining 12
students are the ones who enrolled in hip-hop dance. Therefore, the percentage
of students enrolled in hip-hop dance is 12 / 20 * 100 = 60% The answer is: 60
HRA${}_{32,10^{-4}}$ Answer: 20% of the 20 students enrolled in contemporary
dance, so 20/100 * 20 = 4 students enrolled in contemporary dance. The
remaining students are 20 - 4 = 16 students. 25% of the remaining 16 students
enrolled in jazz dance, so 25/100 * 16 = 4 students enrolled in jazz dance.
The remaining students are 16 - 4 = 12 students. The rest of the students
enrolled in hip-hop dance, so 12 students enrolled in hip-hop dance. To find
the percentage of the entire students enrolled in hip-hop dance, we divide the
number of students enrolled in hip-hop dance by the total number of students
and multiply by 100. So, (12/20) * 100 = 60% The answer is: 60 HRA32,∞ Answer:
First, 20% of the 20 students enrolled in contemporary dance, which is 20/100
* 20 = 4 students. So, there are 20 - 4 = 16 students remaining. Then, 25% of
the remaining 16 students enrolled in jazz dance, which is 25/100 * 16 = 4
students. So, there are 16 - 4 = 12 students remaining. Finally, the rest of
the students enrolled in hip-hop dance, which is 12 students. Therefore, 12/20
* 100 = 60% of the entire students enrolled in hip-hop dance. The answer is:
60

Case Study 12 Question: A candle melts by 2 centimeters every hour that it
burns. How many centimeters shorter will a candle be after burning from 1:00
PM to 5:00 PM? (Ground-truth answer is 8) Llama-2 Answer: To determine how
many centimeters shorter a candle will be after burning from 1:00 PM to 5:00
PM, we need to first calculate the total time the candle will be burning
during that period. 1:00 PM + 5:00 PM = 8 hours Since the candle melts by 2
centimeters every hour, we can calculate the total distance it will melt
during the 8 hours as follows: 2 centimeters/hour x 8 hours = 16 centimeters
So, after burning from 1:00 PM to 5:00 PM, the candle will be 16 centimeters
shorter. The answer is 16 HRA32,0 Answer: The candle burns for a total of 4
hours. In the first hour, the candle melts by 2 centimeters. In the second
hour, the candle melts by 2 centimeters. In the third hour, the candle melts
by 2 centimeters. In the fourth hour, the candle melts by 2 centimeters. So,
in total, the candle melts by 2 centimeters x 4 hours = 8 centimeters.
Therefore, the candle will be 8 centimeters shorter after burning from 1:00 PM
to 5:00 PM. The answer is: 8 HRA${}_{32,10^{-4}}$ Answer: From 1:00 PM to 2:00
PM, the candle burns for 1 hour. So, the candle will melt by 2 centimeters x 1
hour = 2 centimeters. From 2:00 PM to 3:00 PM, the candle burns for 1 hour.
So, the candle will melt by 2 centimeters x 1 hour = 2 centimeters. From 3:00
PM to 4:00 PM, the candle burns for 1 hour. So, the candle will melt by 2
centimeters x 1 hour = 2 centimeters. From 4:00 PM to 5:00 PM, the candle
burns for 1 hour. So, the candle will melt by 2 centimeters x 1 hour = 2
centimeters. In total, the candle will melt by 2 centimeters + 2 centimeters +
2 centimeters + 2 centimeters = 8 centimeters. Therefore, the candle will be 8
centimeters shorter after burning from 1:00 PM to 5:00 PM. The answer is: 8
HRA32,∞ Answer: First, we need to find out how many hours the candle will burn
from 1:00 PM to 5:00 PM. There are 5 hours in total, so the candle will burn
for 5 - 1 = 4 hours. Next, we need to find out how many centimeters the candle
will melt in 4 hours. Since the candle melts by 2 centimeters every hour, it
will melt 2 x 4 = 8 centimeters in total. Finally, we need to subtract the
initial length of the candle from the total length it will melt to find out
how many centimeters shorter the candle will be. The initial length of the
candle is 0 centimeters, so the candle will be 8 centimeters shorter. The
answer is: 8

Case Study 13 Question: Marie ordered one chicken meal that costs $12, 5 packs
of milk that costs $3 each, 4 apples that cost $1.50 each, and some boxes of
pizza. Marie paid a total of $50. How many boxes of pizza did Marie order if
each box costs $8.50? (Ground-truth answer is 2) Llama-2 Answer: Based on the
information provided, we can calculate the number of boxes of pizza that Marie
ordered as follows: Total cost of Marie’s order = $12 (chicken meal) + $3 x 5
(packs of milk) + $4 x 4 (apples) = $50 If each box of pizza costs $8.50, then
the number of boxes of pizza that Marie ordered can be calculated as follows:
Total cost of pizza = $8.50 x number of boxes Since Marie paid a total of $50,
we can set up the following equation: $8.50 x number of boxes = $50 Now, we
can solve for the number of boxes of pizza that Marie ordered: number of boxes
= $50 / $8.50 number of boxes = 6 Therefore, Marie ordered 6 boxes of pizza.
The answer is 6 HRA32,0 Answer: The cost of the chicken meal is $12. The cost
of 5 packs of milk is 5 * $3 = $15. The cost of 4 apples is 4 * $1.50 = $6.
The total cost of the chicken meal, milk, and apples is $12 + $15 + $6 = $33.
If Marie paid a total of $50, then the cost of the pizza boxes is $50 - $33 =
$17. If each box of pizza costs $8.50, then the number of boxes of pizza is
$17 / $8.50 = 2. Therefore, Marie ordered 2 boxes of pizza. The answer is: 2
HRA${}_{32,10^{-4}}$ Answer: The chicken meal costs $12. The 5 packs of milk
cost $3 each, so 5 * $3 = $15. The 4 apples cost $1.50 each, so 4 * $1.50 =
$6. The total cost of the chicken meal, milk, and apples is $12 + $15 + $6 =
$33. Marie paid a total of $50, so the cost of the pizza boxes is $50 - $33 =
$17. Each box of pizza costs $8.50, so the number of boxes of pizza is $17 /
$8.50 = 2. Therefore, Marie ordered 2 boxes of pizza. The answer is: 2 HRA32,∞
Answer: First, Marie ordered 1 chicken meal that costs $12. Next, she ordered
5 packs of milk that cost $3 each, so 5 * $3 = $15 for the milk. Then, she
ordered 4 apples that cost $1.50 each, so 4 * $1.50 = $6 for the apples.
Finally, she ordered some boxes of pizza that cost $8.50 each, so let’s say
she ordered x boxes of pizza. So, the total cost of the chicken meal, milk,
apples, and pizza is $12 + $15 + $6 + $8.50x = $50. Simplifying the equation,
we get $33.50 + $8.50x = $50. Subtracting $33.50 from both sides, we get
$8.50x = $16.50. Dividing both sides by $8.50, we get x = 2. Therefore, Marie
ordered 2 boxes of pizza. The answer is: 2

##  Appendix C The Impacts of Orthogonality

Given a weight matrix $\bm{W}\in\mathbb{R}^{d_{\text{out}}\times d}$, we
assume that $d_{\text{out}}\geq d$. Denote the SVD of $\bm{W}$ as
$\bm{Q}\bm{\Sigma}\bm{V}^{\top}$, where
$\bm{\Sigma}=\text{diag}(\sigma_{1},...,\sigma_{d})$,
$\sigma_{1}\geq\sigma_{2}\geq...\geq\sigma_{d}$, and
$\bm{V}=[\bm{v}_{1},...,\bm{v}_{d}]$ is the right singular matrix.

When $r=1$, we have

$\displaystyle\|\bm{W}-\bm{W}\bm{H}^{(1)}\|_{F}^{2}=4\|\bm{W}\bm{u}_{1}\bm{u}_{1}^{\top}\|_{F}^{2}=4\text{tr}(\bm{u}_{1}^{\top}\bm{V}\bm{\Sigma}^{2}\bm{V}\bm{u}_{1}),~{}\text{for}~{}\bm{u}_{1}\in\mathbb{S}^{d-1}.$
(7)

Obviously, when $\bm{u}_{1}=\bm{v}_{1}$, the distance between $\bm{W}$ and
$\bm{W}\bm{H}^{(1)}$ is maximized.

Applying mathematical induction, we assume that
$\|\bm{W}-\bm{W}\bm{H}^{(r)}\|_{F}$ is maximized when
$\bm{U}_{r}=\bm{V}_{r}=[\bm{v}_{1},...,\bm{v}_{r}]$ for $r<d$.

Then, in the case of $r+1$, for $\bm{u}\in\mathbb{S}^{d-1}$, we have

$\displaystyle\|\bm{W}-\bm{W}\bm{H}^{(r+1)}\|_{F}^{2}=\|\bm{W}-\bm{W}\bm{H}^{(r)}(\bm{I}-2\bm{u}\bm{u}^{\top})\|_{F}^{2}=4\|\bm{W}\bm{V}_{r}\bm{V}_{r}^{\top}-\bm{W}\bm{u}\bm{u}^{\top}\|_{F}^{2},$
(8)

where the equations are based on the facts that $i)$ HR and the chain of HRs
are unitary matrices and $ii)$ the Frobenius norm is unitary-invariant.

Note that, $\bm{V}=[\bm{v}_{1},...,\bm{v}_{d}]=[\bm{V}_{r},\bm{V}_{d-r}]$
works as an orthonormal basis of $\mathbb{R}^{d}$, so we have
$\bm{u}=\bm{V}\bm{\beta}=\bm{V}_{r}\bm{\beta}_{r}+\bm{V}_{d-r}\bm{\beta}_{d-r}$,
where $\bm{\beta}=[\bm{\beta}_{r};\bm{\beta}_{d-r}]\in\mathbb{S}^{d-1}$.
Accordingly, we can rewrite equation 8 as

$\displaystyle\begin{aligned} &\|\bm{W}-\bm{W}\bm{H}^{(r+1)}\|_{F}^{2}\\\
=&4\|\bm{W}\bm{V}_{r}\bm{V}_{r}^{\top}-\bm{W}\bm{u}\bm{u}^{\top}\|_{F}^{2}\\\
=&4\|\bm{\Sigma}\bm{V}^{\top}\bm{V}_{r}\bm{V}_{r}^{\top}-\bm{\Sigma}\bm{V}^{\top}\bm{uu}^{\top}\|_{F}^{2}\\\
=&4\text{tr}([\bm{V}_{r},\bm{0}_{d\times(d-r)}]\bm{\Sigma}^{2}[\bm{V}_{r},\bm{0}_{d\times(d-r)}]^{\top})-8\text{tr}([\bm{V}_{r},\bm{0}_{d\times(d-r)}]\bm{\Sigma}^{2}\bm{V}^{\top}\bm{uu}^{\top})+4\text{tr}(\bm{u}^{\top}\bm{V}\bm{\Sigma}^{2}\bm{V}^{\top}\bm{u})\\\
=&4\sideset{}{{}_{i=1}^{r}}{\sum}\sigma_{i}^{2}+4\text{tr}(\bm{\beta}^{\top}\bm{\Sigma}^{2}\bm{\beta})-8\text{tr}(\bm{\beta}_{r}^{\top}\bm{\Sigma}_{r}^{2}\bm{\beta}_{r})\\\
=&4\sideset{}{{}_{i=1}^{r}}{\sum}\sigma_{i}^{2}+4\text{tr}(\bm{\beta}_{d-r}^{\top}\bm{\Sigma}_{d-r}^{2}\bm{\beta}_{d-r})-4\text{tr}(\bm{\beta}_{r}^{\top}\bm{\Sigma}_{r}^{2}\bm{\beta}_{r}),\end{aligned}$

where $\bm{\Sigma}=\text{diag}(\bm{\Sigma}_{r},\bm{\Sigma}_{d-r})$. Obviously,
when $\bm{\beta}_{r}=\bm{0}_{r}$ and
$\bm{\beta}_{d-r}=[1,0,..,0]^{\top}\in\mathbb{R}^{d-r}$, the distance between
$\bm{W}$ and $\bm{W}\bm{H}^{(r+1)}$ is maximized. In such a situation,
$\bm{u}=\bm{V\beta}=\bm{v}_{r+1}$, corresponds to the $(r+1)$-th right
singular vector, and $\bm{U}_{t+1}=\bm{V}_{t+1}$ accordingly.