# Understanding Linear Probing then Fine-tuning Language Models from NTK
Perspective

Akiyoshi Tomihari
Issei Sato
The University of Tokyo tomihari<EMAIL_ADDRESS>
###### Abstract

The two-stage fine-tuning (FT) method, linear probing then fine-tuning (LP-
FT), consistently outperforms linear probing (LP) and FT alone in terms of
accuracy for both in-distribution (ID) and out-of-distribution (OOD) data.
This success is largely attributed to the preservation of pre-trained
features, achieved through a near-optimal linear head obtained during LP.
However, despite the widespread use of large language models, the exploration
of complex architectures such as Transformers remains limited. In this paper,
we analyze the training dynamics of LP-FT for classification models on the
basis of the neural tangent kernel (NTK) theory. Our analysis decomposes the
NTK matrix into two components, highlighting the importance of the linear head
norm alongside the prediction accuracy at the start of the FT stage. We also
observe a significant increase in the linear head norm during LP, stemming
from training with the cross-entropy (CE) loss, which effectively minimizes
feature changes. Furthermore, we find that this increased norm can adversely
affect model calibration, a challenge that can be addressed by temperature
scaling. Additionally, we extend our analysis with the NTK to the low-rank
adaptation (LoRA) method and validate its effectiveness. Our experiments with
a Transformer-based model on natural language processing tasks across multiple
benchmarks confirm our theoretical analysis and demonstrate the effectiveness
of LP-FT in fine-tuning language models. Code is available at
https://github.com/tom4649/lp-ft_ntk.

##  1 Introduction

Fine-tuning pre-trained models for new tasks is a common practice across
various fields. However, simply fine-tuning the entire model can lead to
overfitting on training data, which may negatively impact generalization and
out-of-distribution (OOD) performance [32, 30]. To address this, the two-stage
approach known as linear probing then fine-tuning (LP-FT) [28] has
demonstrated high performance on both in-distribution (ID) and OOD data.
Initially, linear probing (LP) optimizes only the linear head of the model,
after which fine-tuning (FT) updates the entire model, including the feature
extractor and the linear head. This method has been extensively analyzed and
enhanced [50, 46, 16, 26].

The feature distortion theory, introduced by Kumar et al. [28], explains the
effectiveness of LP-FT on the basis of a theoretical analysis with a two-layer
linear model. This theory suggests that LP-FT minimizes changes to pre-trained
features by starting FT with an already optimized linear head from LP.
However, our understanding of LP-FT, particularly when applied to complex
architectures such as Transformers [51], remains incomplete. Thus, it is
crucial to further explore the training dynamics of LP-FT in more complex
models than the two-layer linear model.

In this paper, we apply the neural tangent kernel (NTK) theory [21] to clarify
the mechanisms underlying LP-FT, focusing on the training dynamics of
classification models. The NTK is a theoretical tool that analyzes training
dynamics by applying a first-order approximation to changes in the model
outputs with respect to its parameters. Therefore, the NTK is suited for
analyzing feature changes during FT dynamics [55, 34]. Our analysis reveals
that after LP, both more accurate predictions and increased norms of the
linear head compared to their initial values contribute to minimizing feature
changes. We then identify a significant increase in the linear head during LP
from the analysis of training with cross-entropy (CE) loss, which contributes
to small feature changes in the FT stage. On the other hand, we found that
this increase in the linear head norm can worsen calibration, causing
predicted probabilities to deviate from actual probabilities, which can be
corrected with temperature scaling [15]. Furthermore, we extend our analysis
based on the NTK to the low-rank adaptation (LoRA) method [20], a parameter-
efficient fine-tuning strategy, and validate its effectiveness.

Our contributions are summarized as follows:

  * •

We show that both accurate predictions and increased norms of the linear head
during LP reduce feature changes in LP-FT within the NTK regime (Section 4),
which is consistent with the feature distortion theory. (Corollary 1).

  * •

We find that norms of the linear head significantly affect the balance of the
NTK matrix components and influence the training dynamics of FT (1).

  * •

We also highlight that increased linear head norms can negatively affect model
calibration, and this can be fixed with temperature scaling.

  * •

We extend our analysis based on the NTK to the LoRA method and provide a
theoretical validation of its efficacy (2).

##  2 Related work

##### LP-FT

FT and LP are well-established transfer learning techniques with extensive
empirical and theoretical studies [60, 27, 49]. Kumar et al. [28] analyzed the
effectiveness of these techniques using a two-layer linear model. Then, they
proposed LP-FT that is a combined approach of LP then FT. Building on this
study, subsequent studies have explored LP-FT in more detail. Trivedi et al.
[50] investigated LP-FT through the lens of safety objectives, proposing
modifications to mitigate simplicity bias. Ren et al. [46] analyzed LP-FT from
the perspective of the initial discrepancy between predicted and actual
probabilities, emphasizing the importance of the number of probing epochs
during LP. Ha et al. [16] further improved LP-FT by aligning batch
normalization layers with the target domain. Kirichenko et al. [26]
highlighted the challenge that models depend on spurious features and proposed
last-layer retraining as a cost-effective strategy to improve model
robustness.

##### Other FT methods

Various FT strategies other than LP-FT have been proposed, including two-stage
approaches [59], regularization-based techniques [23], and parameter-efficient
fine-tuning methods [19, 17]. One prominent example of a parameter-efficient
method is LoRA, proposed by Hu et al. [20]. This approach draws inspiration
from the concept of intrinsic dimensions [1], suggesting that data can be
effectively represented in a lower-dimensional space. Zeng and Lee [58]
explored the expressive power of LoRA, and Jang et al. [22] provided a
theoretical analysis of its convergence properties. However, challenges remain
in parameter-efficient FT methods, including potential instability issues
identified by Chen et al. [6].

##### Neural tangent kernel (NTK)

The NTK, which was first introduced by Jacot et al. [21], has become a
valuable tool for analyzing the training dynamics of neural networks. Studies
by Lee et al. [29] and Arora et al. [2] used the NTK to gain insights into how
networks learn. Building on this foundation, Wei et al. [55] introduced the
concept of the empirical NTK, which extends the application of NTK to FT
scenarios. This approach replaces the randomly initialized parameters in the
standard NTK with the parameters of the pre-trained models. Further expanding
on the empirical NTK, Malladi et al. [34] conducted a theoretical and
experimental investigation and found that prompt-based fine-tuning exhibits
behavior consistent with the predictions of the kernel framework. Jang et al.
[22] extended this perspective to analyze LoRA.

##  3 Preliminary

In this section, we provide an overview of the FT methods used in this paper,
followed by a brief explanation of the NTK.

##### LP-FT

In standard FT, the parameters of the linear head, weight $\bm{V}$ and bias
$\bm{b}$, are initialized with random values. In contrast, in LP-FT, LP is
conducted before the FT stage, and the FT stage is started with the obtained
parameters. The performance of LP-FT is higher than that of LP and FT on both
ID and OOD data [28]. The original LP-FT paper [28] explains the reason behind
it as the feature distortion theory: the success of LP-FT stems from the
minimal feature changes because of starting the FT stage with the linear head
parameters which are close to the optimal solution. We analyze the training
process of LP-FT throughout this paper.

##### LoRA

LoRA [20] introduces trainable rank decomposition matrices into each layer of
the Transformer architecture. This approach, inspired by the concept of
“intrinsic dimensions" from Aghajanyan et al. [1], constrains updates to pre-
trained weight matrices via low-rank decomposition. The update of a pre-
trained weight matrix $\bm{W_{0}}\in\mathbb{R}^{q\times s}$ is approximated by
$\bm{W}+\Delta\bm{W}=\bm{W_{0}}+\bm{B}^{\text{LoRA}}\bm{A}^{\text{LoRA}}$,
where $\bm{B}^{\text{LoRA}}\in\mathbb{R}^{q\times r}$ and
$\bm{A}^{\text{LoRA}}\in\mathbb{R}^{r\times s}$ are the only matrices
optimized during fine-tuning. Here, $r\ll\min(q,s)$ represents the small
intrinsic rank of the weight matrix, reflecting the low-rank approximation.
The standard initialization of $\bm{B}^{\text{LoRA}}$ and
$\bm{A}^{\text{LoRA}}$ is $\bm{B}^{\text{LoRA}}=O$ and $\bm{A}^{\text{LoRA}}$
is drawn from a normal distribution with mean $0$.

##### Neural tangent kernel (NTK)

Jacot et al. [21] introduced the NTK, which captures the training dynamics
over time. They demonstrated that in the infinite width limit, the NTK remains
constant. In this limit, training dynamics are governed by a linear model
derived from a first-order Taylor expansion around the initial parameters of
the network, known as the linearized or NTK regime [29]. For networks with
finite width, this limiting kernel depends on the initialization parameters
and is known as the empirical NTK [55]. Although the empirical NTK differs
from the infinite width limit, it is valuable for analyzing the local training
dynamics of models [45, 13, 37, 55, 22], and has been used in FT [46, 34].

##  4 Analysis of LP-FT from NTK perspective

The original analysis of LP-FT by Kumar et al. [28] is based on a two-layer
linear model and proposes the feature distortion theory, which suggests that
minimal changes in pre-trained features are the reason behind the robust
performance of LP-FT. In this section, we use the NTK theory to analyze LP-FT
to better understand the training dynamics of LP-FT in complex models like
Transformers. After introducing the notation, we discuss the increase in the
classifier weight norm during training, followed by the training dynamics in
the NTK regime. We then extend our analysis to the LoRA method. These analyses
suggest the LP-FT reduces feature distortion with the increased norm of the
classifier weight and the near-optimal prediction after LP.

###  4.1 Notation

Let $\mathcal{X}=\\{\bm{x}_{1},\ldots,\bm{x}_{N}\\}\subseteq\mathbb{R}^{d}$
represent the training samples, paired with labels from the set
$\mathcal{Y}=\\{y_{1},\ldots,y_{N}\\}\subseteq\\{1,2,\ldots,C\\}$, where $d$,
$C$, and $N$ denote the dimensions of the input space, the number of classes,
and the number of training samples, respectively. This forms a training
dataset
$\\{(\bm{x}_{1},y_{1}),\ldots,(\bm{x}_{N},y_{N})\mid\bm{x}_{i}\in\mathcal{X},y_{i}\in\mathcal{Y}\\}$,
and we use $\bm{x}\in\mathbb{R}^{d}$ to denote both a training and a test
sample. We denote the $k$-th element of vector $\bm{a}$ as $[\bm{a}]_{k}$. We
use the Euclidean norm $\|\cdot\|$ for vectors and the Frobenius norm
$\|\cdot\|_{F}$ for matrices. $\langle\cdot,\cdot\rangle$ denotes the inner
product of two vectors. $\bm{e}_{k}$ represents the one-hot vector for class
$k$, and $\bm{I}_{C}$ is the identity matrix of size $C$.

The model function, denoted as
$\bm{f}(\cdot;\theta):\mathcal{X}\rightarrow\mathbb{R}^{C}$, is parameterized
by a set of parameters $\theta$, and sometimes abbreviated as $\bm{f}(\cdot)$.
The model includes a linear head, also referred to as the classifier, which
consists of a weight matrix $\bm{V}$ and a bias vector $\bm{b}$. The feature
extractor is denoted by
$\bm{\phi}(\cdot):\mathbb{R}^{h}\rightarrow\mathbb{R}^{C}$, where $h$
represents the hidden dimension. The output of the model is given by
$\bm{f}(\bm{x};\theta)=\bm{V}\bm{\phi}(\bm{x};\theta)+\bm{b}$. Parameters for
a function $g(\cdot)$ and matrix $\bm{A}$ are sometimes denoted as
$\theta^{g}$ and $\theta^{A}$, respectively. Subscripts represent iteration or
epoch, so $\bm{f}_{t}(\cdot)$ denotes the model at time $t$.

With the loss function
$\ell:\mathbb{R}^{C}\times\mathcal{Y}\rightarrow\mathbb{R}$, the training
objective is to minimize the empirical risk
$L(\bm{f}):=L(\bm{f}(\cdot;\theta))=\frac{1}{N}\sum_{i=1}^{N}\ell(\bm{f}(\bm{x}_{i};\theta),y_{i})$.
We use the CE loss,
$\ell(\bm{f}(\bm{x}),y)=-\log\left([\bm{\sigma}_{\text{SM}}(\bm{f}(\bm{x}))]_{y}\right)$,
where $\bm{\sigma}_{\text{SM}}:\mathbb{R}^{C}\rightarrow\mathbb{R}^{C}$ is the
softmax function with its $k$-th element given by
$[\bm{\sigma}_{\text{SM}}(\bm{f}(\bm{x}))]_{k}=\frac{\exp([\bm{f}(\bm{x})]_{k)})}{\sum_{k^{\prime}}\exp([\bm{f}(\bm{x})]_{k^{\prime}})}$.

###  4.2 Training dynamics in the NTK regime

We use the NTK [21], more specifically the empirical NTK [55, 34], to analyze
the training dynamics of both FT and LP-FT. The empirical NTK, defined as the
NTK with the parameters at the start of training, is a valuable tool for
understanding the neural network training process, particularly in the context
of FT [55, 34, 46]. The empirical NTK applies a first-order approximation to
changes in model outputs with respect to its parameters, so this is expected
to capture changes in features.

To investigate the feature distortion theory in FT and LP-FT, we decomposed
the updates into the following two parts. The part influenced by feature
updates, unique to FT and absent in LP, is termed the FT-effective component
of the NTK matrix, represented as $\bm{F}(\bm{x},\bm{x}_{i})$. In contrast,
the part not influenced by feature updates, common to both FT and LP,
determined by the pre-trained model, is termed the pre-train-effective
component, represented as $\bm{P}(\bm{x},\bm{x}_{i})$. This decomposition
highlights the distinct training dynamics of LP-FT in the NTK regime in the
following proposition.

######  Proposition 1 (FT in the NTK regime).

The NTK of a model $\bm{f}(\bm{x})=\bm{V}\bm{\phi}(\bm{x})+\bm{b}$, denoted by
$\Theta^{\bm{f}}$, can be decomposed as:

$\displaystyle\Theta^{\bm{f}}(\bm{x},\bm{x}_{i})=\bm{P}(\bm{x},\bm{x}_{i})+\bm{F}(\bm{x},\bm{x}_{i}),$
(1)

where the pre-train-effective component $\bm{P}(\bm{x},\bm{x}_{i})$ and the
FT-effective component $\bm{F}(\bm{x},\bm{x}_{i})$ are defined using the
classifier weight matrix $\bm{V}_{0}$ and the feature extractor
$\bm{\phi}_{0}$ at starting point of training as:

$\displaystyle\bm{P}(\bm{x},\bm{x}_{i})$
$\displaystyle:=(\langle\bm{\phi}_{0}(\bm{x}),\bm{\phi}_{0}(\bm{x}_{i})\rangle+1)\bm{I}_{C},$
(2) $\displaystyle\bm{F}(\bm{x},\bm{x}_{i})$
$\displaystyle:=\bm{V}_{0}\frac{\partial\bm{\phi}_{0}(\bm{x})}{\partial\theta^{\bm{\phi}}}\frac{\partial\bm{\phi}_{0}(\bm{x}_{i})}{\partial\theta^{\bm{\phi}}}^{\top}\bm{V}_{0}^{\top}.$
(3)

Consequently, assuming that one-epoch training within the NTK regime
approximates FT, the logits and feature vectors for a sample $\bm{x}$ after
FT, denoted as $\bm{f}^{\text{FT}}(\bm{x})$ and
$\bm{\phi}^{\text{FT}}(\bm{x})$, to the starting point of training,
$\bm{f}_{0}(\bm{x})$ and $\bm{\phi}_{0}(\bm{x})$, can be expressed as:

$\displaystyle\bm{f}^{\text{FT}}(\bm{x})-\bm{f}_{0}(\bm{x})$
$\displaystyle=\eta\sum_{i=1}^{N}\left(\bm{P}(\bm{x},\bm{x}_{i})+\bm{F}(\bm{x},\bm{x}_{i})\right)\bm{\delta}_{i},$
(4) $\displaystyle\bm{\phi}^{\text{FT}}(\bm{x})-\bm{\phi}_{0}(\bm{x})$
$\displaystyle=\eta\sum_{i=1}^{N}\Theta^{\bm{\phi}}(\bm{x},\bm{x}_{i})\bm{V}_{0}^{\top}\bm{\delta}_{i},$
(5)

where $\Theta^{\bm{\phi}}$ is the NTK matrix of the feature extractor
$\bm{\phi}$,
$\bm{\delta}_{i}:=\bm{e}_{y_{i}}-\bm{\sigma}_{\text{SM}}(\bm{f}_{0}(\bm{x}_{i}))$
represents the difference between the one-hot label for the class $y_{i}$ and
the predicted probability, and $\eta$ is the learning rate.

The proof of this proposition is included in the Appendix (Section 7.2.1). In
our decomposition of the NTK matrix, the pre-train-effective component
$\bm{P}(\bm{x},\bm{x}_{i})$ is a diagonal matrix and remains unchanged after
LP, while the FT-effective component $\bm{F}(\bm{x},\bm{x}_{i})$ is not a
diagonal matrix and does change after LP, resulting in distinct
characteristics for these components. The Frobenius norm of the classifier
weight matrix, $\|\bm{V}_{0}\|_{F}$, influences the balance between the pre-
train-effective and FT-effective components because it affects only the FT-
effective component. This indicates that the classifier weight norm
$\|\bm{V}_{0}\|_{F}$ has a significant impact on the training dynamics of FT.

##### Hypothesis on reduced feature changes in LP-FT

The above proposition provides insights into why LP-FT causes fewer feature
changes compared to FT:

  1. 1.

The impact of the classifier weight norm $\|\bm{V}_{0}\|_{F}$ differs in the
equations: it affects feature changes linearly (5) and affects logits
quadratically (4). This implies that a higher norm can result in significant
logit updates with relatively minor changes to the feature extractor, reducing
feature changes in LP-FT compared with FT due to the increased classifier
weight norm after LP.

  2. 2.

The magnitude of changes in both features and logits ((4) and (5)), is
proportional to $\bm{\delta}_{i}$, the difference between the predicted
probability and the one-hot label. This suggests that feature changes are less
pronounced in LP-FT than in FT since the difference $\bm{\delta}_{i}$ is
smaller after LP.

  3. 3.

The learning rate $\eta$, typically smaller in LP-FT than in FT [28, 46, 16],
helps moderate the direct influence of large classifier weight norms.

Prior studies [28, 46] have suggested that reduced feature changes in LP-FT
stem from the near-optimal linear head obtained during LP. However, our
analysis reveals that feature changes in LP-FT are also influenced by the
classifier weight norm $\bm{V}_{0}$ after LP. Our analysis focusing on
classifier weight norms provides a new perspective on the training dynamics of
LP-FT, highlighting the importance of the classifier weight norm in reducing
feature distortion.

###  4.3 Derivation of Lemma A.3 from Kumar et al. in the NTK regime

The analysis presented in the original LP-FT paper by Kumar et al. [28]
operates within a framework where the feature extractor is a linear function.
We define this framework in our context as follows:

######  Definition 1 (Linear model [28]).

A linear model is defined as $\bm{f}(\bm{x})=\bm{V}\bm{\phi}(\bm{x})+\bm{b}$,
where $\bm{\phi}(\bm{x})=\bm{B}\bm{x}$ denotes the feature extractor,
$\bm{V}\in\mathbb{R}^{C\times h}$ is the classifier weight matrix, and
$\bm{B}\in\mathbb{R}^{h\times d}$ is the weight matrix of the feature
extractor.

In this setting, we derive a corollary from 1 in our context, which is the
pivotal lemma in the original LP-FT analysis [28]:

######  Corollary 1 (Lemma A.3 from Kumar et al. in the NTK regime).

Within the context of the linear model (Definition 1), for any sample
$\bm{x}\in\operatorname{Span}(\mathcal{X})^{\bot}$, the orthogonal complement
of the subspace spanned by the training sample set $\mathcal{X}$, the features
after FT remain unchanged, expressed as:

$\displaystyle\bm{\phi}^{\text{FT}}(\bm{x})=\bm{\phi}_{0}(\bm{x}),$ (6)

where $\bm{\phi}^{\text{FT}}(\bm{x})$ and $\bm{\phi}_{0}(\bm{x})$ denote the
feature vectors after and before FT, respectively.

This corollary shows that feature vectors for the samples in the orthogonal
complement of training sample subspace are not updated. Therefore, given that
pre-trained features have characteristics beneficial to downstream tasks,
significant feature changes in FT, dependent on small training samples in LP,
lead to poor generalization and OOD performance. The proof of this lemma can
be found in the Appendix (Section 7.2.2).

###  4.4 Increase in the classifier weight norm

(a) LP

(b) FT

(c) After training

Figure 1: Increase in classifier weight norms during training on the RTE
dataset. (a) and (b) show the increase of the both accuracy and classifier
weight norms with training. (c) shows classifier weights norms after training.

The analysis in the previous section suggests that the classifier weight norm
affects both feature changes and logits. On the basis of this insight, we
examine classifier weight norms during training. Figure 1 shows that
classifier weight norms consistently increase over time for LP, standard FT,
and LoRA. As the training proceeds, norms of classifier bias and logits
increases, while training loss decreases. Notably, LP shows a significantly
larger increase in the norm compared to FT and LoRA.

Consider the transpose of the $k$-th row of matrix $\bm{V}$ denoted as
$\bm{v}_{k}\in\mathbb{R}^{h}$ for $1\leq k\leq C$, where $C$ is the number of
classes. Let $\tau_{ki}$ represent the angle between $\bm{\phi}(\bm{x_{i}})$
and $\bm{v}_{k}$, which expands
$\langle\bm{v}_{k},\bm{\phi}(\bm{x_{i}})\rangle$ to
$\|\bm{v}_{k}\|\|\bm{\phi}(\bm{x_{i}})\|\cos\tau_{ki}$. The probability that
class $k$ is chosen for sample $\bm{x}_{i}$ is given by the softmax function
$[\bm{\sigma}_{\text{SM}}\left(\bm{f}(\bm{x_{i}})\right)]_{k}=\frac{\exp(\langle\bm{v}_{k},\bm{\phi}(\bm{x_{i}})\rangle)}{\sum_{k^{\prime}}\exp(\langle\bm{v}_{k^{\prime}},\bm{\phi}(\bm{x_{i}})\rangle)}$.
Consequently, with the CE loss for an input $\bm{x}_{i}$ classified into class
$y_{i}$ defined as
$\ell(\bm{f}(\bm{x_{i}}),y_{i})=-\log\left([\bm{\sigma}_{\text{SM}}\left(\bm{f}(\bm{x_{i}})\right)]_{y_{i}}\right)$,
we have the following partial derivatives:

$\displaystyle\frac{\partial\ell(\bm{f}(\bm{x_{i}}),y_{i})}{\partial\cos\tau_{ki}}$
$\displaystyle=\begin{cases}[\bm{\sigma}_{\text{SM}}\left(\bm{f}(\bm{x_{i}})\right)]_{k}\|\bm{v}_{k}\|\|\bm{\phi}(\bm{x_{i}})\|&\text{if
}k\neq y_{i},\\\
-(1-[\bm{\sigma}_{\text{SM}}\left(\bm{f}(\bm{x_{i}})\right)]_{y_{i}})\|\bm{v}_{y_{i}}\|\|\bm{\phi}(\bm{x_{i}})\|&\text{if
}k=y_{i},\end{cases}$ (7)

where the derivative with respect to $\cos\tau_{y_{i}i}$ is negative and
positive for $k\neq y_{i}$. As training progresses, $\cos\tau_{y_{i}i}$ tends
to increase towards positivity, while $\cos\tau_{ki}$ for $k\neq y_{i}$ tends
to become negative for each $i$. The derivative with respect to
$\|\bm{v}_{k}\|$ is given by:

$\displaystyle\frac{\partial L(\bm{f})}{\partial\|\bm{v}_{k}\|}$
$\displaystyle=\sum_{i=1}^{N}\left(\sum_{y_{i}\neq
k}[\bm{\sigma}_{\text{SM}}\left(\bm{f}(\bm{x_{i}})\right)]_{k}\|\bm{\phi}(\bm{x_{i}})\|\cos\tau_{ki}-\sum_{y_{i}=k}(1-[\bm{\sigma}_{\text{SM}}\left(\bm{f}(\bm{x_{i}})\right)]_{y_{i}})\|\bm{\phi}(\bm{x_{i}})\|\cos\tau_{y_{i}i}\right).$
(8)

Therefore, with adequate training and $\cos\tau_{ki}<0$ and
$\cos\tau_{y_{i}i}>0$, the derivative with respect to $\|\bm{v}_{k}\|$ is
likely to become negative for each class $k$. The training of the model
proceeds so that the empirical risk $L$ decreases, so the norm
$\|\bm{v}_{k}\|$ tends to increase. This finding aligns with prior studies
[48, 24].

##### Remark: increase in classifier weight norms is more pronounced in LP
than in FT

In FT, particularly within an overparameterized setting, the model $\bm{f}$
may achieve perfect classification on the training dataset. That is,
$[\bm{\sigma}_{\text{SM}}\left(\bm{f}(\bm{x}_{i})\right)]_{k}$ becomes close
to $0$ for $k\neq y_{i}$ and $1$ for $k=y_{i}$. In this scenario, the
derivative in Eq. (8) becomes close to zero, or the training itself is
finished. Conversely, perfect classification is typically unattainable in LP
unless the training dataset is linearly separable, so the derivative continues
to be negative. In addition, while all parameters are updated in FT, only the
classifier is optimized in LP, so the change in the classifier weight needs to
be larger in LP than in FT to achieve the same classification performance.
Consequently, the classifier weight norm tends to increase more significantly
in LP than in FT, as shown in Figure 1 (c).

###  4.5 Training process of LoRA

We extend our analysis based on the NTK to the training process of LoRA. We
follow the linear model setting as in Definition 1 and analyze the training
dynamics of LoRA in the NTK regime.

######  Proposition 2 (LoRA approximates FT).

Consider the linear model setting (Definition 1) and let
$\bm{f}^{\text{LoRA}}$ and $\bm{f}^{\text{FT}}$ be the models obtained via
one-epoch training with LoRA and standard FT in the NTK regime. Let $r$ denote
the rank of the LoRA hyperparameter, and $\sigma^{2}$ represent the variance
of the low-rank weight matrix initialization. Assume the input samples
$\bm{x}$ satisfy $\|\bm{x}\|\leq c$. Then, for each sample pair
$\bm{x}_{i},\bm{x}_{j}\in\mathcal{X}$, the pre-train-effective components of
the NTK matrix for LoRA and FT, $\bm{P}^{\text{LoRA}}(\bm{x}_{i},\bm{x}_{j})$
and $\bm{P}^{\text{FT}}(\bm{x}_{i},\bm{x}_{j})$, are identical:

$\displaystyle\bm{P}^{\text{LoRA}}(\bm{x}_{i},\bm{x}_{j})$
$\displaystyle=\bm{P}^{\text{FT}}(\bm{x}_{i},\bm{x}_{j}).$ (9)

Moreover, with at least $1-4\exp(-(\epsilon^{2}-\epsilon^{3})r/4)$
probability, their FT-effective components,
$\bm{F}^{\text{LoRA}}(\bm{x}_{i},\bm{x}_{j})$ and
$\bm{F}^{\text{FT}}(\bm{x}_{i},\bm{x}_{j})$, satisfy:

$\displaystyle\|\bm{F}^{\text{LoRA}}(\bm{x}_{i},\bm{x}_{j})-\sigma^{2}r\bm{F}^{\text{FT}}(\bm{x}_{i},\bm{x}_{j})\|\leq
c\epsilon\|\bm{V}_{0}\bm{V}_{0}^{\top}\|.$ (10)

This proposition suggests that with high probability, the only difference of
the NTK matrix between LoRA and standard FT is a scalar factor of the FT-
effective component in the NTK matrix, and the scalar factor depends on the
hyperparameters of LoRA. This implies that when the hyperparameters of LoRA
are set appropriately, LoRA training is similar to standard FT training. This
is consistent with the analysis by Malladi et al. [34], where the NTK matrix
of LoRA and standard FT are close to each other. It is important to note that
the proposition is also valid for LP-FT and LP-LoRA (LP then LoRA). The proof
of this proposition is included in the Appendix (Section 7.2.3).

###  4.6 Discussion

An increased norm of the classifier weight reduces feature distortion and
enhances the contribution of the FT-effective component of the NTK matrix
during training. As a result, a higher classifier weight norm in LP-FT can be
advantageous. However, since the increased norm is dependent on LP training,
its optimality is not guaranteed. Specifically, during test time, although the
increased classifier weight norm does not influence accuracy, it affects the
calibration of the model. Calibration is defined as the alignment between the
predicted probabilities and the actual probabilities [15]. An excessively high
classifier weight norm can lead to overconfident predictions, which might be
detrimental in practical applications. Consequently, there is potential for
refining LP-FT by adjusting the classifier weight norm to enhance calibration.

Tuning the norm of the classifier after training can be effectively equated to
applying temperature scaling [15] at test time. Temperature scaling adjusts
the output logits with a temperature parameter $T$, thereby improving model
calibration. Specifically, temperature scaling with parameter $T$, expressed
as $\bm{f}(\bm{x})/T=\frac{\bm{V}}{T}\bm{\phi}(\bm{x})+\frac{\bm{b}}{T}$, can
be viewed as scaling the norm of classifier weight $\bm{V}$ and bias $\bm{b}$
by the temperature parameter $T$.

##  5 Numerical evaluation with transformer models

In this section, we numerically justify the following aspects obtained from
our analysis:

  * •

The changes in features during training is smaller in LP-FT than in FT, and
the norms of the classifier significantly increase during LP (Section 5.2).

  * •

The FT-effective component of the NTK matrix more effectively captures the
input data than the pre-train-effective component (Section 5.3) and is more
pronounced in LP-FT than FT.

  * •

A large classifier weight norm reduces the feature change during training, and
its negative effects on calibration can be improved by temperature scaling
(Section 5.4).

Details on the datasets, setup, and additional results, including performance
evaluations for the experimental and practical application, are available in
the Appendix (Sections 7.3 and 7.4).

###  5.1 Setup

##### Datasets and models

We used a total of $13$ classification datasets from various benchmarks:
SuperGLUE [53], GLUE [52], BOSS [57], and PubMed $20$k RCT [10]. The breakdown
of the datasets is as follows: five datasets from SuperGLUE (BoolQ, CB, RTE,
WiC, and WSC), three datasets from GLUE (CoLA, MRPC, and SST-2), four datasets
from BOSS (Amazon, Dynasent, SemEval, and SST-5), and PubMed $20$k RCT.
Following experimental settings in studies that analyze FT dynamics from NTK
perspectives [34, 22] and the study with similar settings [6], we employed the
RoBERTa-base model [33] as our Transformer-based model.

##### Implementation and training

We used the Transformers library [56] and AdapterHub [43] for our
implementation. Our training protocol followed the experimental setup
described by Chen et al. [6]. Hyperparameter tuning, especially for learning
rates during the FT stage of LP-FT, was conducted through a grid search based
on the validation set performance. For LP, we used logistic regression with L2
regularization on pre-trained features.

Table 1: Feature (F) changes and classifier (C) norms on the CB and RTE datasets. CS, Diff, FDR, and Norm represent cosine similarity, difference norm, Fisher’s discriminant ratio, and norm, respectively. Method | CB | RTE
---|---|---
CS(F) | Diff(F) | FDR(F) | Norm(C) | CS(C) | Diff(F) | FDR(F) | Norm(C)
Pre-trained | $0.997$ | $-$ | $8.14\times 10^{4}$ | $9.51\times 10^{-1}$ | $0.996$ | $-$ | $8.59\times 10^{1}$ | $7.76\times 10^{-1}$
LP | $0.997$ | $-$ | $8.14\times 10^{4}$ | $2.48\times 10^{1}$ | $0.996$ | $-$ | $8.59\times 10^{1}$ | $3.10\times 10^{1}$
FT | $0.336$ | $2.21\times 10^{1}$ | $7.39\times 10^{8}$ | $9.60\times 10^{-1}$ | $0.260$ | $2.16\times 10^{1}$ | $1.42\times 10^{4}$ | $7.84\times 10^{-1}$
LoRA | $0.499$ | $1.92\times 10^{1}$ | $8.91\times 10^{6}$ | $1.43\times 10^{0}$ | $0.759$ | $1.06\times 10^{1}$ | $2.97\times 10^{3}$ | $1.21\times 10^{0}$
LP-FT | $0.804$ | $1.20\times 10^{1}$ | $6.47\times 10^{6}$ | $2.48\times 10^{1}$ | $0.942$ | $4.70\times 10^{0}$ | $1.57\times 10^{2}$ | $3.10\times 10^{1}$
LP-LoRA | $0.837$ | $9.08\times 10^{0}$ | $2.10\times 10^{6}$ | $2.49\times 10^{1}$ | $0.924$ | $4.63\times 10^{0}$ | $2.06\times 10^{1}$ | $3.10\times 10^{1}$

###  5.2 Small feature changes during LP-FT and significant norm increase
during LP

LP-FT achieves notable performance with Transformer-based language models,
outperforming standard FT in both ID and OOD settings, as detailed in Appendix
(Sections 7.4.1 and 7.4.3). To understand the underlying reasons for these
results and validate small feature changes suggested by our analysis (Section
4.2), we analyzed changes in both the classifier and the features.

According to statistics presented in Table 1, the feature vectors of LP-FT
demonstrate smaller changes from those of the pre-trained model compared to
FT. Consequently, LP-FT maintains high cosine similarity among its features
and exhibits a low Fisher’s discriminant ratio (FDR) [12], which assesses
linear separability. Conversely, the classifier norms after LP and LP-FT is
substantially larger than those of the pre-trained model and after FT,
suggesting a significant increase in classifier weights during LP. A similar
trend is observed in training with LoRA.

###  5.3 Kernel analysis

We examined the overall NTK matrix and its pre-train-effective and FT-
effective components to understand their properties. Kernel regression was
performed on the train and test sets to evaluate the performance of each
kernel matrix.

Table 2: Kernel statistics on the CB dataset. FN, Acc, and FT Ratio denote the
Frobenius norm, kernel regression accuracy, and contribution of the FT-
effective component, respectively. Pre-train E and FT E refer to the pre-
train-effective and FT-effective components of the NTK matrix.

Method | Kernel | Rank | FN | Acc(train/test) | FT Ratio
---|---|---|---|---|---
- | Pre-train E | 18 | 51.0 | $87.11/79.17$ | -
FT | FT E | 608 | 13.9 | $84.74/79.76$ | $0.1987$
| NTK | 210 | 64.9 | $84.74/79.76$
LoRA | FT E | 500 | 0.0226 | $86.22/79.17$ | $0.0004$
| NTK | 20 | 51.0 | $92.15/84.52$
LP-FT | FT E | 344 | 7250 | $100.00/86.31$ | $1.0000$
| NTK | 344 | 7280 | $100.00/86.31$
LP-LoRA | FT E | 307 | 1.51 | $94.96/85.71$ | $1.0137$
| NTK | 188 | 62.6 | $95.11/85.71$

Figure 2: Singular value distribution normalized by the maximum value on the
CB dataset, showing the common pre-train-effective component (Pre-train E) and
the FT-effective components for each training option.

##### Analysis of NTK matrix components and effectiveness of LP-FT

In Table 2, the FT-effective component of the NTK matrix for LP-FT shows a
higher rank and greater kernel regression accuracy compared to the pre-train-
effective component, and the overall NTK matrix has intermediate properties.
Additionally, the FT-effective component contributes more significantly to the
overall kernel in LP-FT than in FT, as indicated by a higher FT Ratio. This
ratio, calculated as the average of
$\|\sum_{i=1}^{N}\bm{F}(\bm{x},\bm{x}_{i})\bm{\delta}_{i}\|/\|\sum_{i=1}^{N}\left(\bm{P}(\bm{x},\bm{x}_{i})+\bm{F}(\bm{x},\bm{x}_{i})\right)\bm{\delta}_{i}\|$
for the train set samples, reflects the enhanced influence of the FT-effective
component in LP-FT than in FT. These results suggest that the NTK matrix of
LP-FT better captures input data through the increased influence of the FT-
effective component.

##### Similarities between LoRA and FT

The ranks of the FT-effective components in LoRA and FT (or LP-LoRA and LP-FT)
are similar, as indicated in Table 2. Their distributions of singular values
normalized by the maximum singular value, also closely align, as shown in
Figure 2. These results suggest that the FT-effective components of the NTK
matrix in FT and LoRA differ only by a scalar factor. This consistency
demonstrates that our analysis (Section 4.2), originally based on a two-layer
linear model, is applicable to more complex Transformer-based models.

###  5.4 Analysis of classifier weight norms and temperature scaling

We experimentally verified significant effects of classifier weight norms in
training (Section 4.2) and at test time (Section 4.6) in the following.

##### Effects of classifier weight norms in training

We scaled the classifier weight norms at the start of the FT stage of LP-FT.
The results, shown in Figure 3, indicate that larger classifier weight norms
almost monotonically lead to smaller feature differences in both FT and LP-FT.
Notably, LP-FT consistently shows smaller feature differences than FT,
particularly when the classifier weight norms are large, validating our
analysis that larger classifier weight norms reduce feature changes.

Figure 3: Difference of features on SST-5 (OOD). The dashed vertical lines
indicate the original classifier weight norm after training.

Table 3: ECE and MCE with temperature scaling on the test set of the RTE
dataset. w/o TS and w/ TS denote without and with temperature scaling,
respectively, and Imp. represents the improvement because of temperature
scaling. We bold the best improvements.

Metric | Method | w/o TS | w/ TS | Imp.
---|---|---|---|---
ECE (%) | FT | $21.16$ | $5.13$ | $16.03$
LP-FT | $21.72$ | $5.48$ | $\bm{16.24}$
LoRA | $11.92$ | $6.17$ | $5.76$
LP-LoRA | $18.14$ | $5.72$ | $12.42$
MCE (%) | FT | $53.11$ | $25.87$ | $27.24$
LP-FT | $63.95$ | $13.94$ | $\bm{50.01}$
LoRA | $25.04$ | $13.75$ | $11.29$
LP-LoRA | $40.46$ | $18.82$ | $21.63$

##### Temperature scaling at test time

We implemented temperature scaling at test time, which is equivalent to
adjusting the classifier weight norms, as discussed in Section 4.6. We
optimized the temperature parameters on the validation sets based on CE loss,
following the methodology suggested by Guo et al. [15]. Table 3 presents the
results on the RTE datasets. We assessed the expected calibration error (ECE)
and maximum calibration error (MCE) [38], which quantify the absolute
differences between predicted and actual probabilities, with lower values
indicating better calibration. These results show that the improvements in
calibration with temperature scaling are the largest in LP-FT for both ECE and
MCE, with notably substantial improvements in MCE. This suggests that large
classifier weight norms contribute to substantial deviations in predictions in
LP-FT, which can be effectively mitigated through temperature scaling. These
results highlight the effectiveness of refining LP-FT by temperature scaling.

##  6 Conclusion

In this paper, we explored the LP-FT training dynamics in complex
classification models using the NTK to analyze feature changes. Our analysis
identified classifier weight norms at the start of the FT stage as a key
factor influencing FT dynamics. These norms balance the NTK matrix components
and help reduce feature changes. Our findings support the existing feature
distortion theory from an NTK perspective and emphasize the role of classifier
weight norms alongside prediction accuracy. We also found that increases in
classifier weight norms, characteristic of training with CE loss, may
negatively impact model calibration, and this can be mitigated by temperature
scaling. Additionally, the approximation effectiveness of LoRA is
theoretically validated in terms of the similarity of the NTK matrix
components. Empirical experiments with Transformer-based language models
supported our theoretical insights, validating our understanding of the NTK,
feature changes, and the benefits of temperature scaling. Overall, our study
substantiates the efficacy of LP-FT as a robust method for adapting pre-
trained complex models while preserving their well-trained features.

##### Limitations

The main limitation of our study is that it is based on the NTK regime, which
might not fully capture the training dynamics. Additionally, we consider just
one epoch of gradient descent in FT, which may not effectively represent the
overall training. In our experiments, we specifically focused on validating
the effectiveness of LP-FT on language models. Therefore, areas other than
natural language processing are outside the scope of our experiments.

## References

  * Aghajanyan et al. [2021] Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer.  Intrinsic dimensionality explains the effectiveness of language model fine-tuning.  In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 7319–7328, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.acl-long.568.  URL https://aclanthology.org/2021.acl-long.568.
  * Arora et al. [2019] Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang.  On exact computation with an infinitely wide neural net.  _Advances in neural information processing systems_ , 32, 2019.
  * Bar-Haim et al. [2006] Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, and Danilo Giampiccolo.  The second pascal recognising textual entailment challenge.  _Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment_ , 01 2006.
  * Bentivogli et al. [2009] Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo.  The fifth pascal recognizing textual entailment challenge.  _TAC_ , 7(8):1, 2009.
  * Burstein et al. [2019] Jill Burstein, Christy Doran, and Thamar Solorio.  Proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: Human language technologies, volume 1 (long and short papers).  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , 2019.
  * Chen et al. [2022] Guanzheng Chen, Fangyu Liu, Zaiqiao Meng, and Shangsong Liang.  Revisiting parameter-efficient tuning: Are we really there yet?  In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 2612–2626, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.emnlp-main.168.  URL https://aclanthology.org/2022.emnlp-main.168.
  * Clark et al. [2019] Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova.  BoolQ: Exploring the surprising difficulty of natural yes/no questions.  In Jill Burstein, Christy Doran, and Thamar Solorio, editors, _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2924–2936, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.  doi: 10.18653/v1/N19-1300.  URL https://aclanthology.org/N19-1300.
  * Dagan et al. [2005] Ido Dagan, Oren Glickman, and Bernardo Magnini.  The pascal recognising textual entailment challenge.  In _Machine learning challenges workshop_ , pages 177–190. Springer, 2005.
  * De Marneffe et al. [2019] Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser.  The commitmentbank: Investigating projection in naturally occurring discourse.  In _proceedings of Sinn und Bedeutung_ , volume 23, pages 107–124, 2019.
  * Dernoncourt and Lee [2017] Franck Dernoncourt and Ji Young Lee.  Pubmed 200k rct: a dataset for sequential sentence classification in medical abstracts, 2017.
  * Dolan and Brockett [2005] Bill Dolan and Chris Brockett.  Automatically constructing a corpus of sentential paraphrases.  In _Third international workshop on paraphrasing (IWP2005)_ , 2005.
  * Fisher [1936] Ronald A Fisher.  The use of multiple measurements in taxonomic problems.  _Annals of eugenics_ , 7(2):179–188, 1936.
  * Fort et al. [2020] Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M. Roy, and Surya Ganguli.  Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel, 2020.
  * Giampiccolo et al. [2007] Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan.  The third pascal recognizing textual entailment challenge.  In _Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing_ , pages 1–9, 2007.
  * Guo et al. [2017] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger.  On calibration of modern neural networks.  In Doina Precup and Yee Whye Teh, editors, _Proceedings of the 34th International Conference on Machine Learning_ , volume 70 of _Proceedings of Machine Learning Research_ , pages 1321–1330. PMLR, 06–11 Aug 2017.  URL https://proceedings.mlr.press/v70/guo17a.html.
  * Ha et al. [2024] Seokhyeon Ha, Sunbeom Jeong, and Jungwoo Lee.  Domain-aware fine-tuning: Enhancing neural network adaptability.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 38, pages 12261–12269, 2024.
  * He et al. [2022] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig.  Towards a unified view of parameter-efficient transfer learning.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=0RDcd5Axok.
  * Horace He [2021] Richard Zou Horace He.  functorch: Jax-like composable function transforms for pytorch.  https://github.com/pytorch/functorch, 2021.
  * Houlsby et al. [2019] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.  Parameter-efficient transfer learning for NLP.  In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, _Proceedings of the 36th International Conference on Machine Learning_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 2790–2799. PMLR, 09–15 Jun 2019.  URL https://proceedings.mlr.press/v97/houlsby19a.html.
  * Hu et al. [2022] Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  LoRA: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=nZeVKeeFYf9.
  * Jacot et al. [2018] Arthur Jacot, Franck Gabriel, and Clément Hongler.  Neural tangent kernel: Convergence and generalization in neural networks.  _Advances in neural information processing systems_ , 31, 2018.
  * Jang et al. [2024] Uijeong Jang, Jason D. Lee, and Ernest K. Ryu.  Lora training in the ntk regime has no spurious local minima, 2024.
  * Jiang et al. [2019] Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao.  Smart: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization.  _arXiv preprint arXiv:1911.03437_ , 2019.
  * Kim and Kim [2020] Byungju Kim and Junmo Kim.  Adjusting decision boundary for class imbalanced learning.  _IEEE Access_ , 8:81674–81685, 2020.
  * Kingma and Ba [2017] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization, 2017.
  * Kirichenko et al. [2023] Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson.  Last layer re-training is sufficient for robustness to spurious correlations.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=Zb6c8A-Fghk.
  * Kornblith et al. [2019] Simon Kornblith, Jonathon Shlens, and Quoc V Le.  Do better imagenet models transfer better?  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 2661–2671, 2019.
  * Kumar et al. [2022] Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang.  Fine-tuning can distort pretrained features and underperform out-of-distribution.  _arXiv preprint arXiv:2202.10054_ , 2022.
  * Lee et al. [2019] Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington.  Wide neural networks of any depth evolve as linear models under gradient descent.  _Advances in neural information processing systems_ , 32, 2019.
  * Lee et al. [2023] Yoonho Lee, Annie S Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn.  Surgical fine-tuning improves adaptation to distribution shifts.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=APuPRxjHvZ.
  * Levesque et al. [2012] Hector Levesque, Ernest Davis, and Leora Morgenstern.  The winograd schema challenge.  In _Thirteenth international conference on the principles of knowledge representation and reasoning_ , 2012.
  * Li et al. [2020] Hao Li, Pratik Chaudhari, Hao Yang, Michael Lam, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto.  Rethinking the hyperparameters for fine-tuning.  In _International Conference on Learning Representations_ , 2020.  URL https://openreview.net/forum?id=B1g8VkHFPH.
  * Liu et al. [2020] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Ro{bert}a: A robustly optimized {bert} pretraining approach, 2020.  URL https://openreview.net/forum?id=SyxS0T4tvS.
  * Malladi et al. [2023] Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora.  A kernel-based view of language model fine-tuning.  In _International Conference on Machine Learning_ , pages 23610–23641. PMLR, 2023.
  * Malladi et al. [2024] Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora.  Fine-tuning language models with just forward passes.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * McAuley and Leskovec [2013] Julian McAuley and Jure Leskovec.  Hidden factors and hidden topics: Understanding rating dimensions with review text.  In _Proceedings of ACM Conference on Recommender Systems_ , 2013.
  * Mohamadi and Sutherland [2023] Mohamad Amin Mohamadi and Danica J. Sutherland.  A fast, well-founded approximation to the empirical neural tangent kernel, 2023.  URL https://openreview.net/forum?id=HN0ehX-ov5Q.
  * Naeini et al. [2015] Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht.  Obtaining well calibrated probabilities using bayesian binning.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 29, 2015.
  * Nakov et al. [2016] Preslav Nakov, Alan Ritter, Sara Rosenthal, Fabrizio Sebastiani, and Veselin Stoyanov.  SemEval-2016 task 4: Sentiment analysis in Twitter.  In _Proceedings of International Workshop on Semantic Evaluation (SemEval)_ , 2016.
  * Novak et al. [2022] Roman Novak, Jascha Sohl-Dickstein, and Samuel S Schoenholz.  Fast finite width neural tangent kernel.  In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pages 17018–17044. PMLR, 17–23 Jul 2022.  URL https://proceedings.mlr.press/v162/novak22a.html.
  * Paszke et al. [2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library, 2019.
  * Pedregosa et al. [2011] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830, 2011.
  * Pfeiffer et al. [2020] Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych.  AdapterHub: A framework for adapting transformers.  In Qun Liu and David Schlangen, editors, _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 46–54, Online, October 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.emnlp-demos.7.  URL https://aclanthology.org/2020.emnlp-demos.7.
  * Potts et al. [2021] Christopher Potts, Zhengxuan Wu, Atticus Geiger, and Douwe Kiela.  DynaSent: A dynamic benchmark for sentiment analysis.  In _Proceedings of ACL-IJCNLP_ , 2021.
  * Ren et al. [2022] Yi Ren, Shangmin Guo, and Danica J. Sutherland.  Better supervisory signals by observing learning paths, 2022.
  * Ren et al. [2023] Yi Ren, Shangmin Guo, Wonho Bae, and Danica J. Sutherland.  How to prepare your task head for finetuning.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=gVOXZproe-e.
  * Socher et al. [2013] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of EMNLP_ , 2013.
  * Soudry et al. [2018] Daniel Soudry, Elad Hoffer, and Nathan Srebro.  The implicit bias of gradient descent on separable data.  In _International Conference on Learning Representations_ , 2018.  URL https://openreview.net/forum?id=r1q7n9gAb.
  * Tripuraneni et al. [2020] Nilesh Tripuraneni, Michael Jordan, and Chi Jin.  On the theory of transfer learning: The importance of task diversity.  _Advances in neural information processing systems_ , 33:7852–7862, 2020.
  * Trivedi et al. [2023] Puja Trivedi, Danai Koutra, and Jayaraman J. Thiagarajan.  A closer look at model adaptation using feature distortion and simplicity bias.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=wkg_b4-IwTZ.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc., 2017.  URL https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.
  * Wang et al. [2018] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  _arXiv preprint arXiv:1804.07461_ , 2018.
  * Wang et al. [2019] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.  Superglue: A stickier benchmark for general-purpose language understanding systems.  _Advances in neural information processing systems_ , 32, 2019.
  * Warstadt et al. [2019] Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman.  Neural network acceptability judgments.  _Transactions of the Association for Computational Linguistics_ , 7:625–641, 2019.  doi: 10.1162/tacl_a_00290.  URL https://aclanthology.org/Q19-1040.
  * Wei et al. [2022] Alexander Wei, Wei Hu, and Jacob Steinhardt.  More than a toy: Random matrix models predict how real-world neural representations generalize.  In _Proceedings of the 39th International Conference on Machine Learning_ , 2022.
  * Wolf et al. [2020] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online, October 2020. Association for Computational Linguistics.  URL https://www.aclweb.org/anthology/2020.emnlp-demos.6.
  * Yuan et al. [2023] Lifan Yuan, Yangyi Chen, Ganqu Cui, Hongcheng Gao, Fangyuan Zou, Xingyi Cheng, Heng Ji, Zhiyuan Liu, and Maosong Sun.  Revisiting out-of-distribution robustness in nlp: Benchmark, analysis, and llms evaluations.  _arXiv preprint arXiv:2306.04618_ , 2023.
  * Zeng and Lee [2024] Yuchen Zeng and Kangwook Lee.  The expressive power of low-rank adaptation.  In _The Twelfth International Conference on Learning Representations_ , 2024.  URL https://openreview.net/forum?id=likXVjmh3E.
  * Zhang et al. [2020] Jeffrey O Zhang, Alexander Sax, Amir Zamir, Leonidas Guibas, and Jitendra Malik.  Side-tuning: a baseline for network adaptation via additive side networks.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16_ , pages 698–714. Springer, 2020.
  * Zhuang et al. [2020] Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He.  A comprehensive survey on transfer learning.  _Proceedings of the IEEE_ , 109(1):43–76, 2020.

##  7 Appendix

###  7.1 Abbreviation and notation

Table 4 and Table 5 show our abbreviations and notations, respectively.

Table 4:  Table of abbreviations.  Abbreviation | Definition
---|---
FT | fine-tuning
LP | linear probing
LP-FT | linear probing then fine-tuning
NTK | neural tangent kernel
LoRA | low rank adaptation [20]
ECE | expected calibration error ([38])
MCE | maximum calibration error (Naeini et al. [38])
ID / OOD | in-distribution / out-of-distribution
FDR | Fisher’s discriminant ratio [12]
Table 5:  Table of notations.  Variable | Definition
---|---
$C~{}/~{}N$ | number of classes / training samples
$d~{}/~{}h~{}/~{}r$ | input dimension / hidden dimension / rank of LoRA
$\mathcal{X}~{}/~{}\mathcal{Y}$ | trainig samples / labels
$\bm{x}~{}/~{}y~{}$ | sample / label
$[\bm{a}]_{k}$ $k$-th element of vector $\bm{a}$ |
$\|\cdot\|~{}/~{}\|\cdot\|_{F}~{}/\langle\cdot,\cdot\rangle$ | Euclidean norm / Frobenius norm / inner product
$\bm{e}_{y}\in\\{0,1\\}^{C}$ | one-hot encoding of label $y$
$\bm{I}_{C}$ |  $C\times C$ identity matrix
$\ell(\bm{f}(\bm{x}),y)$ | loss function
$L$ | empirical risk
$\bm{\sigma}_{\text{SM}}$ | softmax function
$\bm{f}(\bm{x})$ | model output
$\bm{\phi}(\bm{x})$ | feature extractor
$\bm{V}~{}/~{}\bm{b}$ | classifier weight / bias
$~{}\bm{V}_{0}~{}/~{}\bm{\phi}_{0}$ | classifier weight / feature extractor at the start of training
$\bm{B}$ | feature extractor weight matrix in two-layer linear model
$\bm{A}^{\text{LoRA}}~{}/~{}\bm{B}^{\text{LoRA}}$ | low-rank weight matrices in LoRA
$\theta^{\bm{g}}~{}/~{}\theta^{\bm{A}}~{}/~{}\theta^{\bm{a}}$ | parameter of function $\bm{g}$ / matrix $\bm{A}$ / vector $\bm{a}$
$\Theta^{\bm{f}}~{}/~{}\Theta^{\bm{\phi}}$ | NTK matrix of model / feature extractor
$\bm{P}(\bm{x},\bm{x}_{i})~{}/~{}\bm{F}(\bm{x},\bm{x}_{i})$ | pre-train-effective / FT-effective component of NTK matrix
$\bm{\delta}_{i}$ | difference between one-hot label and predicted probability
$\eta$ | learning rate
$\otimes$ | kronecker product of two matrices

###  7.2 Proof of theoretical results

##### Additional notation

The parameters for a function $\bm{g}$, a weight matrix $\bm{A}$, and a vector
$\bm{a}$ is denoted as $\theta^{\bm{g}},\theta^{\bm{A}}$, and
$\theta^{\bm{a}}$. Given a function
$\bm{g}(\cdot;\theta^{\bm{g}}):\mathbb{R}^{d}\rightarrow\mathbb{R}^{s}$
trained on $N$ training samples
$\mathcal{X}=\\{\bm{x}_{1},\bm{x}_{2},\ldots,\bm{x}_{N}\\}\subseteq\mathbb{R}^{d}$,
we denote the NTK matrix of $\bm{g}$ at time $t$ as $\Theta^{\bm{g}}_{t}$,
which is defined as
$\Theta^{\bm{g}}_{t}:=\frac{\partial\bm{g}_{t}(\mathcal{X})}{\partial\theta^{\bm{g}}}\left(\frac{\partial\bm{g}_{t}(\mathcal{X})}{\partial\theta^{\bm{g}}}\right)^{\top}\in\mathbb{R}^{Ns\times
Ns}$, where
$\bm{g}_{t}(\mathcal{X}):=\operatorname{vec}\left(\bm{g}_{t}(\bm{x_{i}})\right)_{\bm{x_{i}\in\mathcal{X}}}$,
and
$\frac{\partial\bm{g}_{t}(\mathcal{X})}{\partial\theta^{\bm{g}}}\in\mathbb{R}^{Ns\times
p}$ with $p$ parameters. The sub-matrix
$\Theta^{\bm{g}}_{t}(\bm{x}_{i},\bm{x}_{j})$ is defined as
$\Theta^{\bm{g}}_{t}(\bm{x}_{i},\bm{x}_{j}):=\frac{\partial\bm{g}_{t}(\bm{x_{i}})}{\partial\theta^{\bm{g}}}\left(\frac{\partial\bm{g}_{t}(\bm{x}_{j})}{\partial\theta^{\bm{g}}}\right)^{\top}\in\mathbb{R}^{s\times
s}$, describing the relationship between training samples $\bm{x}_{i}$ and
$\bm{x}_{j}$ in $\mathcal{X}$. In the infinite width limit with NTK
parameterization and general assumptions, the NTK matrix converges to
$\Theta^{\bm{g}}:=\frac{\partial\bm{g}_{0}(\mathcal{X})}{\partial\theta^{\bm{g}}}\left(\frac{\partial\bm{g}_{0}(\mathcal{X})}{\partial\theta^{\bm{g}}}\right)^{\top}$
as shown by [21]. Subscripts represent iteration or epoch, so
$\bm{g}_{t}(\cdot)$ denotes the model $\bm{g}$ at time $t$. $\otimes$ denotes
the kronecker product of two matrices defined as

$\displaystyle\bm{A}\otimes\bm{B}:=\begin{bmatrix}a_{11}\bm{B}&a_{12}\bm{B}&\cdots&a_{1n}\bm{B}\\\
a_{21}\bm{B}&a_{22}\bm{B}&\cdots&a_{2n}\bm{B}\\\
\vdots&\vdots&\ddots&\vdots\\\
a_{m1}\bm{B}&a_{m2}\bm{B}&\cdots&a_{mn}\bm{B}\\\ \end{bmatrix},$ (11)

where $\bm{A}=[a_{ij}]$ is an $m\times n$ matrix and $\bm{B}$ is any matrix.

####  7.2.1 Proof of Proposition 1

######  1.

The NTK matrix of a model $\bm{f}(\bm{x})=\bm{V}\bm{\phi}(\bm{x})+\bm{b}$,
denoted by $\Theta^{\bm{f}}$, can be decomposed as:

$\displaystyle\Theta^{\bm{f}}(\bm{x},\bm{x}_{i})=\bm{P}(\bm{x},\bm{x}_{i})+\bm{F}(\bm{x},\bm{x}_{i}),$
(12)

where the pre-train-effective component $\bm{P}(\bm{x},\bm{x}_{i})$ and the
FT-effective component $\bm{F}(\bm{x},\bm{x}_{i})$ are defined using the
classifier weight matrix $\bm{V}_{0}$ and the feature extractor
$\bm{\phi}_{0}$ at starting point of training as:

$\displaystyle\bm{P}(\bm{x},\bm{x}_{i})$
$\displaystyle:=(\langle\bm{\phi}_{0}(\bm{x}),\bm{\phi}_{0}(\bm{x}_{i})\rangle+1)\bm{I}_{C},$
(13) $\displaystyle\bm{F}(\bm{x},\bm{x}_{i})$
$\displaystyle:=\bm{V}_{0}\frac{\partial\bm{\phi}_{0}(\bm{x})}{\partial\theta^{\bm{\phi}}}\frac{\partial\bm{\phi}_{0}(\bm{x}_{i})}{\partial\theta^{\bm{\phi}}}^{\top}\bm{V}_{0}^{\top}.$
(14)

Consequently, assuming that one-epoch training within the NTK regime
approximates FT, the logits and feature vectors for a sample $\bm{x}$ after
FT, denoted as $\bm{f}^{\text{FT}}(\bm{x})$ and
$\bm{\phi}^{\text{FT}}(\bm{x})$, to the starting point of training,
$\bm{f}_{0}(\bm{x})$ and $\bm{\phi}_{0}(\bm{x})$, can be expressed as:

$\displaystyle\bm{f}^{\text{FT}}(\bm{x})-\bm{f}_{0}(\bm{x})$
$\displaystyle=\eta\sum_{i=1}^{N}\left(\bm{P}(\bm{x},\bm{x}_{i})+\bm{F}(\bm{x},\bm{x}_{i})\right)\bm{\delta}_{i},$
(15) $\displaystyle\bm{\phi}^{\text{FT}}(\bm{x})-\bm{\phi}_{0}(\bm{x})$
$\displaystyle=\eta\sum_{i=1}^{N}\Theta^{\bm{\phi}}(\bm{x},\bm{x}_{i})\bm{V}_{0}^{\top}\bm{\delta}_{i},$
(16)

where
$\bm{\delta}_{i}:=\bm{e}_{y_{i}}-\bm{\sigma}_{\text{SM}}(\bm{f}_{0}(\bm{x}_{i}))$
represents the difference between the one-hot label and the predicted
probability, and $\eta$ is the learning rate.

##### Proof of 1

###### Proof.

The parameters of $\bm{f}$, denoted as $\bm{\theta}^{\bm{f}}$, consist of
$\bm{\theta}^{V}$, $\bm{\theta}^{b}$, and $\bm{\theta}^{\bm{\phi}}$. The
derivative of the model $\bm{f}$ with respect to each parameter is given by:

$\displaystyle\frac{\partial\bm{f}(\bm{x})}{\partial\bm{\theta}^{V}}$
$\displaystyle=\bm{\phi}(\bm{x})^{\top}\otimes\bm{I}_{C},$ (17)
$\displaystyle\frac{\partial\bm{f}(\bm{x})}{\partial\bm{\theta}^{b}}$
$\displaystyle=\bm{I}_{C},$ (18)
$\displaystyle\frac{\partial\bm{f}(\bm{x})}{\partial\bm{\theta}^{\bm{\phi}}}$
$\displaystyle=\bm{V}\frac{\partial\bm{\phi}(\bm{x})}{\partial\bm{\theta}^{\bm{\phi}}}.$
(19)

Therefore, the NTK matrix of $\bm{f}$, defined as
$\Theta^{\bm{f}}(\bm{x},\bm{x}_{i}):=\frac{\partial\bm{f}_{0}(\bm{x})}{\partial\bm{\theta}^{\bm{f}}}\left(\frac{\partial\bm{f}_{0}(\bm{x}_{i})}{\partial\bm{\theta}^{\bm{f}}}\right)^{\top}$,
can be expressed as:

$\displaystyle\Theta^{\bm{f}}(\bm{x},\bm{x}_{i})$
$\displaystyle=\frac{\partial\bm{f}_{0}(\bm{x})}{\partial\bm{\theta}^{\bm{f}}}\left(\frac{\partial\bm{f}_{0}(\bm{x}_{i})}{\partial\bm{\theta}^{\bm{f}}}\right)^{\top}$
(21)
$\displaystyle=\frac{\partial\bm{f}_{0}(\bm{x})}{\partial\bm{\theta}^{V}}\frac{\partial\bm{f}_{0}(\bm{x}_{i})}{\partial\bm{\theta}^{V}}^{\top}+\frac{\partial\bm{f}_{0}(\bm{x})}{\partial\bm{\theta}^{b}}\frac{\partial\bm{f}_{0}(\bm{x}_{i})}{\partial\bm{\theta}^{b}}^{\top}+\frac{\partial\bm{f}_{0}(\bm{x})}{\partial\bm{\theta}^{\bm{\phi}}}\frac{\partial\bm{f}_{0}(\bm{x}_{i})}{\partial\bm{\theta}^{\bm{\phi}}}^{\top}$
(22)
$\displaystyle=\left(\bm{\phi}_{0}(\bm{x})^{\top}\otimes\bm{I}_{C}\right)\left(\bm{\phi}_{0}(\bm{x}_{i})^{\top}\otimes\bm{I}_{C}\right)^{\top}+\bm{I}_{C}+\bm{V}\frac{\partial\bm{\phi}_{0}(\bm{x})}{\partial\bm{\theta}^{\bm{\phi}}}\left(\bm{V}\frac{\partial\bm{\phi}_{0}(\bm{x}_{i})}{\partial\bm{\theta}^{\bm{\phi}}}\right)^{\top}$
(23) $\displaystyle\hskip
227.62204pt(\because\text{Eqs.}~{}\eqref{eq:derivativeV},\eqref{eq:derivativeb},\eqref{eq:derivativePhi})$
(24)
$\displaystyle=\langle\bm{\phi}_{0}(\bm{x}),\bm{\phi}_{0}(\bm{x}_{i})\rangle\bm{I}_{C}+\bm{I}_{C}+\bm{V}_{0}\frac{\partial\bm{\phi}_{0}(\bm{x})}{\partial\bm{\theta}^{\bm{\phi}}}\left(\frac{\partial\bm{\phi}_{0}(\bm{x}_{i})}{\partial\bm{\theta}^{\bm{\phi}}}\right)^{\top}\bm{V}_{0}^{\top}$
(25)
$\displaystyle=(\langle\bm{\phi}_{0}(\bm{x}),\bm{\phi}_{0}(\bm{x}_{i})\rangle+1)\bm{I}_{C}+\bm{V}_{0}\frac{\partial\bm{\phi}_{0}(\bm{x})}{\partial\bm{\theta}^{\bm{\phi}}}\left(\frac{\partial\bm{\phi}_{0}(\bm{x}_{i})}{\partial\bm{\theta}^{\bm{\phi}}}\right)^{\top}\bm{V}_{0}^{\top}$
(26) $\displaystyle=\bm{P}(\bm{x},\bm{x}_{i})+\bm{F}(\bm{x},\bm{x}_{i}).$ (27)

For gradient descent, the update to the parameters $\bm{\theta}^{\bm{f}}$ at
time $t$ is given by:

$\displaystyle\bm{\theta}^{\bm{f}}_{t+1}-\bm{\theta}^{\bm{f}}_{t}$
$\displaystyle=-\eta\left(\frac{\partial
L(\bm{f}_{t})}{\partial\bm{\theta}^{\bm{f}}}\right)^{\top}$ (28)
$\displaystyle=-\eta\sum_{i=1}^{N}\left(\frac{\partial
L(\bm{f}_{t})}{\partial\left([\sigma_{\text{SM}}(\bm{f}(\bm{x}))]_{y_{i}}\right)}\frac{\partial\left([\sigma_{\text{SM}}(\bm{f}_{t}(\bm{x}))]_{y_{i}}\right)}{\partial\bm{f}(\bm{x}_{i})}\frac{\partial\bm{f}_{t}(\bm{x}_{i})}{\partial\bm{\theta}^{\bm{f}}}\right)^{\top}$
(29)
$\displaystyle=\eta\sum_{i=1}^{N}\left(\bm{\delta}_{i}^{\top}\frac{\partial\bm{f}_{t}(\bm{x}_{i})}{\partial\bm{\theta}^{\bm{f}}}\right)^{\top}$
(30)
$\displaystyle=\eta\sum_{i=1}^{N}\frac{\partial\bm{f}_{t}(\bm{x}_{i})}{\partial\bm{\theta}^{\bm{f}}}^{\top}\bm{\delta}_{i}.$
(31)

Assuming that one-epoch training approximates FT, the model is expressed as
$\bm{f}^{\text{FT}}=\bm{f}_{1}$. Therefore, the update to the model $\bm{f}$
in the linearized regime is given by:

$\displaystyle\bm{f}^{\text{FT}}(\bm{x})-\bm{f}_{0}(\bm{x})$
$\displaystyle=\bm{f}_{1}(\bm{x})-\bm{f}_{0}(\bm{x})\quad(\because\text{linearized
regime})$ (32)
$\displaystyle=\eta\sum_{i=1}^{N}\frac{\partial\bm{f}_{0}(\bm{x})}{\partial\bm{\theta}^{\bm{f}}}\left(\frac{\partial\bm{f}_{0}(\bm{x}_{i})}{\partial\bm{\theta}^{\bm{f}}}\right)^{\top}\bm{\delta}_{i}\quad(\because\text{Eq.}~{}\eqref{eq:updateThetaF})$
(33)
$\displaystyle=\eta\sum_{i=1}^{N}\left(\bm{P}(\bm{x},\bm{x}_{i})+\bm{F}(\bm{x},\bm{x}_{i})\right)\bm{\delta}_{i}.\quad(\because\text{Eq.}~{}\eqref{eq:NTKF})$
(34)

Finally, the same derivation with Eq. (31), the update to the parameters
$\bm{\theta}^{\bm{\phi}}$ at time $t$ is given by

$\displaystyle\bm{\theta}^{\bm{\phi}}_{t+1}-\bm{\theta}^{\bm{\phi}}_{t}$
$\displaystyle=\eta\sum_{i=1}^{N}\frac{\partial\bm{f}_{t}(\bm{x}_{i})}{\partial\bm{\theta}^{\bm{\phi}}}^{\top}\bm{\delta}_{i}.$
(35)

Therefore, the update to the feature extractor after FT, given by
$\bm{\phi}^{\text{FT}}=\bm{\phi}_{1}$ for the same assumption, is:

$\displaystyle\bm{\phi}^{\text{FT}}(\bm{x})-\bm{\phi}_{0}(\bm{x})$
$\displaystyle=\bm{\phi}_{1}(\bm{x})-\bm{\phi}_{0}(\bm{x})$ (36)
$\displaystyle=\frac{\partial\bm{\phi}_{0}(\bm{x})}{\partial\bm{\theta}^{\bm{\phi}}}(\bm{\theta}^{\bm{\phi}}_{1}-\bm{\theta}^{\bm{\phi}}_{0})\quad(\because\text{linearized
regime})$ (37)
$\displaystyle=\frac{\partial\bm{\phi}_{0}(\bm{x})}{\partial\bm{\theta}^{\bm{\phi}}}\eta\sum_{i=1}^{N}\left(\frac{\partial\bm{f}_{0}(\bm{x}_{i})}{\partial\bm{\theta}^{\bm{\phi}}}\right)^{\top}\bm{\delta}_{i}\quad(\because\text{Eq.}~{}\eqref{eq:updateThetaPhi})$
(38)
$\displaystyle=\frac{\partial\bm{\phi}_{0}(\bm{x})}{\partial\bm{\theta}^{\bm{\phi}}}\eta\sum_{i=1}^{N}\left(\bm{V}_{0}\frac{\partial\bm{\phi}_{0}(\bm{x}_{i})}{\partial\bm{\theta}^{\bm{\phi}}}\right)^{\top}\bm{\delta}_{i}\quad(\because\text{Eq.}~{}\eqref{eq:derivativePhi})$
(39)
$\displaystyle=\eta\sum_{i=1}^{N}\frac{\partial\bm{\phi}_{0}(\bm{x})}{\partial\bm{\theta}^{\bm{\phi}}}\left(\frac{\partial\bm{\phi}_{0}(\bm{x}_{i})}{\partial\bm{\theta}^{\bm{\phi}}}\right)^{\top}\bm{V}_{0}^{\top}\bm{\delta}_{i}$
(40)
$\displaystyle=\eta\sum_{i=1}^{N}\Theta^{\bm{\phi}}(\bm{x},\bm{x}_{i})\bm{V}_{0}^{\top}\bm{\delta}_{i}.$
(41)

This completes the proof. ∎

####  7.2.2 Proof of Corollary 1

######  Corollary 1.

Within the context of the linear model (Definition 1), for any sample
$\bm{x}\in\operatorname{Span}(\mathcal{X})^{\bot}$, the orthogonal complement
of the subspace spanned by the training sample set $\mathcal{X}$, the features
after FT remain unchanged, expressed as:

$\displaystyle\bm{\phi}^{\text{FT}}(\bm{x})=\bm{\phi}_{0}(\bm{x}),$ (42)

where $\bm{\phi}^{\text{FT}}(\bm{x})$ and $\bm{\phi}_{0}(\bm{x})$ denote the
feature vectors after and before FT, respectively.

##### Proof of Corollary 1

###### Proof.

The feature extractor is given by $\bm{\phi}(\bm{x})=\bm{B}\bm{x}$, where
$\bm{B}$ is the weight matrix. The derivative of the feature extractor with
respect to the parameters $\theta^{\bm{\phi}}=\theta^{\bm{B}}$ is:

$\displaystyle\frac{\partial\bm{\phi}(\bm{x})}{\partial\theta^{\bm{\phi}}}=\frac{\partial\bm{B}\bm{x}}{\partial\theta^{\bm{B}}}=\bm{x}\otimes\bm{I}_{h},$
(43)

so the empirical NTK matrix of the feature extractor becomes:

$\displaystyle\Theta^{\bm{\phi}}(\bm{x_{i}},\bm{x_{j}})$
$\displaystyle:=\frac{\partial\bm{\phi}_{0}(\bm{x_{i}})}{\partial\theta^{\bm{\phi}}}\frac{\partial\bm{\phi}_{0}(\bm{x_{j}})}{\partial\theta^{\bm{\phi}}}^{\top}$
(44) $\displaystyle=\langle\bm{x_{i}},\bm{x_{j}}\rangle\otimes\bm{I}_{h}$ (45)

where $\otimes$ denotes the kronecker product.

From the 1, the feature update is given by:

$\displaystyle\bm{\phi}^{\text{FT}}(\bm{x})-\bm{\phi}_{0}(\bm{x})$
$\displaystyle=\eta\sum_{i=1}^{N}\Theta^{\bm{\phi}}(\bm{x},\bm{x}_{i})\bm{V}_{0}^{\top}\bm{\delta}_{i}$
(46)
$\displaystyle=\eta\sum_{i=1}^{N}\langle\bm{x},\bm{x}_{i}\rangle\bm{V}_{0}^{\top}\bm{\delta}_{i},$
(47)

where
$\bm{\delta}_{i}=\bm{e}_{y_{i}}-\bm{\sigma}_{\text{SM}}(\bm{f}_{0}(\bm{x}_{i}))$,
$\bm{V}_{0}$ is the classifier weight matrix at the start of training, and
$\eta$ is the learning rate. For any sample
$\bm{x}\in\operatorname{Span}(\mathcal{X})^{\bot}$,
$\langle\bm{x},\bm{x}_{i}\rangle=0$ for all $\bm{x}_{i}\in\mathcal{X}$, so the
feature update is $0$ for OOD samples, namely:

$\displaystyle\bm{\phi}^{\text{FT}}(\bm{x})-\bm{\phi}_{0}(\bm{x})=0.$ (48)

This completes the proof. ∎

####  7.2.3 Proof of Proposition 2

######  2.

Consider the linear model setting (Definition 1) and let
$\bm{f}^{\text{LoRA}}$ and $\bm{f}^{\text{FT}}$ be the models obtained via
one-epoch training with LoRA and standard FT in the NTK regime. Let $r$ denote
the rank of the LoRA hyperparameter, and $\sigma^{2}$ represent the variance
of the low-rank weight matrix initialization. Assume the input samples
$\bm{x}$ satisfy $\|\bm{x}\|\leq c$. Then, for each sample pair
$\bm{x}_{i},\bm{x}_{j}\in\mathcal{X}$, the pre-train-effective components of
the NTK matrix for LoRA and FT,
$\bm{P}^{\text{LoRA}}(\bm{x}_{i},\bm{x}_{j})$ and
$\bm{P}^{\text{FT}}(\bm{x}_{i},\bm{x}_{j})$, are identical:

$\displaystyle\bm{P}^{\text{LoRA}}(\bm{x}_{i},\bm{x}_{j})=\bm{P}^{\text{FT}}(\bm{x}_{i},\bm{x}_{j}).$
(49)

Moreover, with at least $1-4\exp(-(\epsilon^{2}-\epsilon^{3})r/4)$
probability, their FT-effective components,
$\bm{F}^{\text{LoRA}}(\bm{x}_{i},\bm{x}_{j})$ and
$\bm{F}^{\text{FT}}(\bm{x}_{i},\bm{x}_{j})$, satisfy:

$\displaystyle\|\bm{F}^{\text{LoRA}}(\bm{x}_{i},\bm{x}_{j})-\sigma^{2}r\bm{F}^{\text{FT}}(\bm{x}_{i},\bm{x}_{j})\|\leq
c\epsilon\|\bm{V}_{0}\bm{V}_{0}^{\top}\|.$ (50)

##### Proof Approach

To prove this theorem, we use a lemma from distributional properties:

######  Lemma 1 (Corollary of the distributional Johnson-Lindenstrauss Lemma).

Given vectors $\bm{u},\bm{v}\in\mathbb{R}^{d}$ with $\|\bm{u}\|,\|\bm{v}\|\leq
c$, and a random matrix $\bm{A}\in\mathbb{R}^{k\times d}$ with i.i.d. entries
from a distribution with mean 0 and variance 1, for any $\epsilon>0$:

$\displaystyle\Pr\left[|(\bm{A}\bm{u})^{\top}(\bm{A}\bm{v})-\bm{u}^{\top}\bm{v}|\geq
c\epsilon\right]\leq 4\exp\left(-(\epsilon^{2}-\epsilon^{3})k/4\right).$ (51)

##### Proof of 2

###### Proof.

The feature vector of LoRA is given by
$\bm{\phi}^{\text{LoRA}}(\bm{x})=\bm{B}_{0}\bm{x}+\bm{B}^{\text{LoRA}}\bm{A}^{\text{LoRA}}\bm{x}$,
where pre-trained feature weight matrix $\bm{B}_{0}$ is fixed during training,
and $\bm{A}^{\text{LoRA}}\in\mathbb{R}^{r\times d}$ and
$\bm{B}^{\text{LoRA}}\in\mathbb{R}^{h\times r}$ are low-rank weight matrices
in LoRA. $\bm{A}^{\text{LoRA}}$ is initialized from a normal distribution with
mean 0 and variance $\sigma^{2}$, while $\bm{B}^{\text{LoRA}}$ is initialized
with zeros. The LoRA feature updates are represented as
$\bm{\phi}^{\text{LoRA}}(\bm{x})=\bm{B}_{0}\bm{x}+\bm{B}^{\text{LoRA}}\bm{A}^{\text{LoRA}}\bm{x}$,
with $\bm{B}_{0}$ fixed during training.

The pre-train-effective components of LoRA and FT, denoted as
$\bm{P}^{\text{LoRA}}(\bm{x},\bm{x}_{i})$ and
$\bm{P}^{\text{FT}}(\bm{x},\bm{x}_{i})$ respectively, are defined as:

$\displaystyle\bm{P}^{\text{LoRA}}(\bm{x},\bm{x}_{i})$
$\displaystyle=(\langle\bm{\phi}^{\text{LoRA}}_{0}(\bm{x}),\bm{\phi}^{\text{LoRA}}_{0}(\bm{x}_{i})\rangle+1)\bm{I}_{C},$
(52) $\displaystyle\bm{P}^{\text{FT}}(\bm{x},\bm{x}_{i})$
$\displaystyle=(\langle\bm{\phi}^{\text{FT}}_{0}(\bm{x}),\bm{\phi}^{\text{FT}}_{0}(\bm{x}_{i})\rangle+1)\bm{I}_{C},$
(53)

where $\bm{I}_{C}$ is the identity matrix of size $C$. These pre-train-
effective components are identical since:

$\displaystyle\bm{\phi}^{\text{LoRA}}_{0}(\bm{x})=\bm{B}_{0}\bm{x}+\bm{B}_{0}^{\text{LoRA}}\bm{A}_{0}^{\text{LoRA}}\bm{x}=\bm{B}_{0}\bm{x}=\bm{\phi}^{\text{FT}}_{0}(\bm{x}),$
(54)

for all $\bm{x}\in\mathcal{X}$ because $\bm{B}^{\text{LoRA}}$ is initialized
as a zero matrix i.e. $\bm{B}^{\text{LoRA}}_{0}=O$.

For the FT-effective component of the NTK matrix, consider the derivatives
concerning LoRA parameters $\bm{B}^{\text{LoRA}}$ and $\bm{A}^{\text{LoRA}}$:

$\displaystyle\frac{\partial\bm{\phi}^{\text{LoRA}}(\bm{x})}{\partial\theta^{\bm{B}^{\text{LoRA}}}}$
$\displaystyle=\bm{A}\bm{x}\otimes\bm{V},$ (55)
$\displaystyle\frac{\partial\bm{\phi}^{\text{LoRA}}(\bm{x})}{\partial\theta^{\bm{A}^{\text{LoRA}}}}$
$\displaystyle=\bm{x}\otimes\bm{V}\bm{B}^{\text{LoRA}}\bm{B}^{\text{LoRA}\top}\bm{V}^{\top}.$
(56)

Here, $\theta^{\bm{B}^{\text{LoRA}}}$ and $\theta^{\bm{A}^{\text{LoRA}}}$
denote the parameters of $\bm{B}^{\text{LoRA}}$ and $\bm{A}^{\text{LoRA}}$,
respectively.

The FT-effective component of the NTK matrix for LoRA, denoted as
$\bm{F}^{\text{LoRA}}(\cdot,\cdot)$, is derived by combining these partial
derivatives:

$\displaystyle\bm{F}^{\text{LoRA}}(\bm{x},\bm{x}_{i})=$
$\displaystyle\bm{V}_{0}\left(\frac{\partial\bm{\phi}_{0}^{\text{LoRA}}(\bm{x})}{\partial\theta^{\bm{B}^{\text{LoRA}}}}\frac{\partial\bm{\phi}_{0}^{\text{LoRA}}(\bm{x}_{i})}{\partial\theta^{\bm{B}^{\text{LoRA}}}}^{\top}+\frac{\partial\bm{\phi}_{0}^{\text{LoRA}}(\bm{x})}{\partial\theta^{\bm{A}^{\text{LoRA}}}}\frac{\partial\bm{\phi}_{0}^{\text{LoRA}}(\bm{x}_{i})}{\partial\theta^{\bm{A}^{\text{LoRA}}}}^{\top}\right)\bm{V}_{0}^{\top}$
$\displaystyle=$
$\displaystyle\bm{V}_{0}\left(\langle\bm{A}^{\text{LoRA}}_{0}\bm{x},\bm{A}^{\text{LoRA}}_{0}\bm{x}_{i}\rangle+\langle\bm{x},\bm{x}_{i}\rangle\bm{B}^{\text{LoRA}}_{0}\bm{B}^{\text{LoRA}\top}_{0}\right)\bm{V}_{0}^{\top}$
$\displaystyle=$
$\displaystyle\langle\bm{A}_{0}^{\text{LoRA}}\bm{x},\bm{A}_{0}^{\text{LoRA}}\bm{x}_{i}\rangle\bm{V}_{0}\bm{V}_{0}^{\top},$
(57)

where the last equality holds because $\bm{B}^{\text{LoRA}}_{0}$ is a zero
matrix.

Similarly, the FT-effective component of the NTK matrix for standard FT,
$\bm{F}^{\text{FT}}(\cdot,\cdot)$, is given by:

$\displaystyle\bm{F}^{\text{FT}}(\bm{x},\bm{x}_{i})=$
$\displaystyle\bm{V}_{0}\left(\frac{\partial\bm{\phi}^{\text{FT}}_{0}(\bm{x})}{\partial\theta_{\bm{B}}}\frac{\partial\bm{\phi}^{\text{FT}}_{0}(\bm{x}_{i})}{\partial\theta_{\bm{B}}}^{\top}\right)\bm{V}_{0}^{\top}$
$\displaystyle=$
$\displaystyle\langle\bm{x},\bm{x}_{i}\rangle\bm{V}_{0}\bm{V}_{0}^{\top}.$
(58)

Using the Johnson-Lindenstrauss lemma, with a probability of at least
$1-4\exp(-(\epsilon^{2}-\epsilon^{3})r/4)$:

$\displaystyle|\langle\bm{A}^{\text{LoRA}}\bm{x},\bm{A}^{\text{LoRA}}\bm{x}_{i}\rangle-\sigma^{2}r\langle\bm{x},\bm{x}_{i}\rangle|\leq
c\sigma^{2}r\epsilon,$ (59)

which implies:

$\displaystyle\|\bm{F}^{\text{LoRA}}(\bm{x},\bm{x}_{i})-\sigma^{2}r\bm{F}^{\text{FT}}(\bm{x},\bm{x}_{i})\|=$
$\displaystyle\|\langle\bm{A}^{\text{LoRA}}\bm{x},\bm{A}^{\text{LoRA}}\bm{x}_{i}\rangle\bm{V}_{0}\bm{V}_{0}^{\top}-\sigma^{2}r\langle\bm{x},\bm{x}_{i}\rangle\bm{V}_{0}\bm{V}_{0}^{\top}\|$
(60) $\displaystyle\leq$
$\displaystyle|\langle\bm{A}^{\text{LoRA}}\bm{x},\bm{A}^{\text{LoRA}}\bm{x}_{i}\rangle-\sigma^{2}r\langle\bm{x},\bm{x}_{i}\rangle|\|\bm{V}_{0}\bm{V}_{0}^{\top}\|$
(61) $\displaystyle\leq$ $\displaystyle
c\sigma^{2}r\epsilon\|\bm{V}_{0}\bm{V}_{0}^{\top}\|.$ (62)

This completes the proof. ∎

###  7.3 Experimental details

####  7.3.1 Datasets

From the SuperGLUE benchmark [53], we used the five datasets: BoolQ [7], CB
(CommitmentBank) [9], RTE (Recognizing Textual Entailment) [8, 3, 14, 4], WiC
(Words in Context) [5], and WSC (Winograd Schema Challenge) [31]. From the
GLUE benchmark [52], we used the three datasets: CoLA (Corpus of Linguistic
Acceptability) [54], MRPC (Microsoft Research Paraphrase Corpus) [11], and
SST-2 (Stanford Sentiment Treebank, version 2) [47]. Four datasets from BOSS
[57] were used in OOD evaluation: Amazon Reviews [36], Dynasent [44], SemEval
[39], and SST-5 [47]. Finally, we used the PubMed $20$k RCT dataset [10] for
validation in practical settings. The dataset statistics are detailed in Table
7.

For the datasets from the GLUE, SuperGLUE, and BOSS benchmarks, we divided the
original training set using a 9:1 training-to-validation ratio, using the
original validation set as the test set, in accordance with Chen et al. [6].
For PubMed $20$k RCT, we used the original training, validation, and test sets
for their respective purposes.

####  7.3.2 Implementation and training details

When applying LoRA, LoRA was applied only to the query and value projection
matrices of the attention mechanism in the Transformer architecture, following
the approach described in the original paper by Hu et al. [20]. The LoRA
settings were fixed at $\alpha=8$ and $r=8$ for all experiments.

The model was trained for $10$ epochs without early stopping, and the one
showing the best performance on the validation set was chosen for further
evaluation. We used the Adam optimizer [25]. Our code is built on PyTorch
[41], using the HuggingFace Transformers library [56] and AdapterHub [43]. All
experiments were run on a single NVIDIA A$100$ GPU. The results reported are
averages from 3 tuning seeds and 5 evaluation seeds.

For LP, cross-validation and automatic hyperparameter adjustment were used to
find the optimal L2 regularization strength, using scikit-learn [42] with its
standard training parameters.

Details on the hyperparameters for our experiments can be found in Table 6.

####  7.3.3 Details of each experiment

##### Experiments on the GLUE and SuperGLUE benchmarks

For the FT and LoRA methods, the learning rate and batch size were adopted
from Chen et al. [6], where these hyperparameters were optimized using grid
search on the validation set. For LP-FT and LP-LoRA, batch size is fixed at 8
and we tuned the learning rate.

##### Experiments on BOSS benchmark and the PubMed $20$k RCT dataset

For the experiments on BOSS benchmark and the PubMed $20$k RCT dataset, we
tuned the learning rate and batch size using grid search based on the
validation set performance.

##### Calculation of the NTK matrix

We computed the NTK matrix for FT, LoRA, LP-FT, and LP-LoRA as specified in
Eq. (4). We separately calculated the pre-train-effective and FT-effective
components of the NTK matrix. Following the methodology by Malladi et al.
[35], we used functorch [18] and forward-mode auto-differentiation [40] for
these calculations. To reduce computational costs, we randomly selected $10$%
of the parameters from the word embedding matrix for derivative calculations.
For datasets with more than $250$ samples, we used a subset of $250$ randomly
selected samples to compute the NTK matrix.

##### Solving the Kernel Regression

Following the methodology described by Malladi et al. [35], we treated each
output logit independently in our kernel regression model. This method is
based on the representer theorem, where the empirical risk minimizer is
expressed as a linear combination of kernel features from the training data:
$\bm{f}(\bm{x})=\sum_{i=1}^{NC}\alpha_{i}\bm{K}(\bm{x},x_{i})$, with $\bm{K}$
representing the NTK matrix or its component for a training set of size
$NC\times NC$. We solved this optimization using logistic regression with L2
regularization and used the resulting coefficients $\alpha_{i}$ to compute
logits on the test set via its corresponding NTK matrix.

##### Effects of classifier weight norms in training

We scaled the norms of the classifiers within the range of
$[0.1,0.5,1,2,5,10,50,100]$ before proceeding to the FT stage of training,
specifically after random initialization in FT and after LP training in LP-FT.
We conducted this experiment using the CB and RTE datasets and Boss benchmark.
We apply the LoRA method on the CB and RTE datasets. We averaged the results
over 5 seeds for the CB and RTE datasets and 3 seeds for the Boss benchmark,
plotting these with their standard deviations.

##### Temperature scaling

We applied temperature scaling [15] to the logits of the model at test time.
Following the methodology of the original paper [15], we tuned the temperature
parameter using the validation set to minimize the negative log-likelihood.
For implementation, we employed the Adam optimizer [25] with a learning rate
of $1\times 10^{-3}$, optimizing the temperature for $1\times 10^{5}$ steps.
We incorporated early stopping based on the negative log-likelihood, with a
patience of $10$ iterations starting from an initial temperature value of
$1.0$. The number of the bins to calculate ECE and MCE is set to $15$.

Table 6: Hyperparameter configurations. The settings include batch size (bs), learning rate (lr), alpha ($\alpha$), and rank ($r$). Method | Name | CB | RTE | BoolQ | WiC | WSC | CoLA | SST-2 | MRPC | Amazon | PubMed
---|---|---|---|---|---|---|---|---|---|---|---
FT | bs | 16 | 16 | 32 | 32 | 16 | 32 | 32 | 16 | 16 | 8
lr | $5e-5$ | $1e-5$ | $1e-5$ | $1e-5$ | $1e-3$ | $5e-5$ | $1e-5$ | $1e-5$ | $1e-5$ | $5e-6$
LoRA | bs | 16 | 16 | 32 | 16 | 16 | 16 | 32 | 32 | 16 | 8
lr | $1e-3$ | $1e-3$ | $5e-4$ | $1e-3$ | $1e-4$ | $1e-3$ | $5e-4$ | $5e-4$ | $1e-3$ | $5e-4$
$\alpha$ | 8
$r$ | 8
LP-FT | bs | 8
lr | $5e-6$ | $1e-5$ | $1e-5$ | $1e-5$ | $1e-3$ | $1e-5$ | $1e-5$ | $1e-5$ | $1e-6$ | $5e-6$
LP-LoRA | bs | 8
lr | $1e-4$ | $5e-4$ | $5e-4$ | $1e-3$ | $1e-4$ | $1e-3$ | $1e-3$ | $1e-3$ | $5e-4$ | $1e-3$
$\alpha$ | 8
$r$ | 8
Table 7: Dataset statistics. This table provides detailed counts of the classes, training, validation, and test samples for different datasets across various tasks including natural language inference (NLI), word sense disambiguation (WSD), question answering (QA), coreference resolution (coref.), sentiment analysis (sentiment), and sequential sentence classification (sequential). Dataset | Benchmark | Classes | Train | Val | Test | Task
---|---|---|---|---|---|---
CB | SuperGLUE | 3 | 225 | 25 | 57 | NLI
RTE | 2 | 2,241 | 249 | 277 | NLI
BoolQ | 2 | 8,484 | 943 | 3,270 | QA
WiC | 2 | 5,400 | 600 | 638 | WSD
WSC | 2 | 498 | 56 | 104 | coref.
CoLA | GLUE | 2 | 7,695 | 855 | 1,040 | acceptability
SST-2 | 2 | 60,614 | 6,735 | 872 | sentiment
MRPC | 2 | 3,301 | 367 | 408 | sentiment
Amazon | BOSS | 3 | 3,000 | 1,000 | 1,000 | sentiment
Dynasent | 3 | - | - | 1,000 | sentiment
SemEval | 3 | - | - | 1,000 | sentiment
SST-5 | 3 | - | - | 1,000 | sentiment
PubMed $20$k RCT | PubMed | 5 | 15,000 | 2,500 | 2,500 | sequential

###  7.4 Additional experimental results

####  7.4.1 Results on the SuperGLUE and GLUE benchmarks

Table 8 shows the test results for the SuperGLUE and GLUE benchmarks. We
report accuracy and its standard deviation on the test sets, except for the
CoLA dataset, which uses the Matthew’s correlation coefficient for the
performance metric.

Figure 4 shows the increase in the classifier weight norm during training on
the CB dataset. With more iterations or epochs, there is a noticeable increase
in both accuracy and the classifier weight norm.

Figure 5 and Figure 6 display t-SNE visualizations of the feature vectors from
the CB dataset. After FT, the features are distinctly separated by class. In
contrast, the classifier row vectors remain nearly identical to those of the
pre-trained model. After LP-FT, the features retain the structure of the pre-
trained model, but the classifier row vectors deviate from their initial
state. A similar pattern is observed with the LoRA method.

Table 8: Test results on the SuperGLUE and GLUE benchmarks. We report the accuracy and its standard deviation, other than the CoLA dataset, which is evaluated by the Matthew’s correlation coefficient. We take the average of five seeds. Dataset | LP | FT | LP-FT | LoRA | LP-LoRA
---|---|---|---|---|---
CB | $77.86\pm 4.24$ | $81.43\pm 3.91$ | $\bm{84.64}\pm\bm{2.40}$ | $77.50\pm 5.30$ | $75.71\pm 2.04$
RTE | $57.69\pm 1.10$ | $74.73\pm 3.04$ | $\bm{76.75}\pm\bm{0.87}$ | $72.85\pm 1.41$ | $74.08\pm 2.57$
SST-2 | $86.31\pm 0.10$ | $92.41\pm 0.32$ | $\bm{94.52}\pm\bm{0.26}$ | $50.92\pm 0.00$ | $94.22\pm 0.45$
WIC | $61.32\pm 0.28$ | $65.89\pm 1.15$ | $\bm{66.14}\pm\bm{1.83}$ | $62.70\pm 7.37$ | $64.29\pm 1.82$
CoLA | $46.27\pm 0.33$ | $\bm{58.75}\pm\bm{1.70}$ | $57.95\pm 1.95$ | $57.29\pm 2.98$ | $58.21\pm 1.55$
MRPC | $73.09\pm 0.86$ | $\bm{88.14}\pm\bm{0.73}$ | $87.60\pm 0.79$ | $68.38\pm 0.00$ | $87.79\pm 1.00$
WSC | $\bm{63.46}\pm\bm{0.00}$ | $\bm{63.46}\pm\bm{0.00}$ | $\bm{63.46}\pm\bm{0.00}$ | $\bm{63.46}\pm\bm{0.68}$ | $\bm{63.46}\pm\bm{0.00}$
BoolQ | $64.66\pm 0.08$ | $78.69\pm 0.27$ | $\bm{79.00}\pm\bm{0.42}$ | $77.59\pm 0.39$ | $77.67\pm 0.50$

(a) LP (CB)

(b) FT (CB)

(c) LoRA (CB)

(d) LoRA (RTE)

(e) After training (CB)

Figure 4: The increase in the norm of the classifier weight during training.

(a) FT

(b) LP-FT

Figure 5: Small changes in feature and large changes in classifier weight
during LP-FT. We visualize the t-SNE plot of the penultimate layer features
and the classifier row vector of the model trained on the CB dataset. (a) The
features after FT are clearly separated by class, while the classifier row
vectors are plotted nearly the same place as the pre-trained model. (b) The
features after LP-FT keep the structure of the pre-trained model, while the
classifier row vectors are changed from the initialization.

(a) FT

(b) LP-LoRA

Figure 6: The t-SNE plot of the penultimate layer features and the classifier
row vector of the model trained with LoRA on the CB dataset.

####  7.4.2 Results of NTK analysis

Table 9 displays the kernel statistics, while Figure 7 shows the distribution
of singular values. Figure 8 and Figure 9 visually depict the trace norms of
sub-matrices within the NTK matrix. For the kernel matrix
$\bm{K}\in\mathbb{R}^{NC\times NC}$, we calculated the trace norms of the sub-
matrix $\bm{K}(\bm{x}_{i},\bm{x}_{j})\in\mathbb{R}^{C\times C}$ for each
sample pair $(\bm{x}_{i},\bm{x}_{j})$ in the training sets.

Figure 8 reveals a consistent pattern in the FT-effective component of the NTK
matrix across all datasets: pairs of identical samples in diagonal positions
typically exhibit higher trace norms. This suggests that the FT-effective
component is more effective at capturing relationships among samples compared
to the pre-train-effective component. Additionally, in the CB dataset, certain
sample pairs, particularly in classes 1 and 3, show notably high trace norms,
indicating that the pre-trained model effectively differentiates between these
class samples.

Table 9: Kernel statistics on the RTE, BoolQ, and WiC datasets. FN, Acc, and FT Ratio denote the Frobenius norm, kernel regression accuracy, and contribution of the FT-effective component, respectively. Pre-train E and FT E refer to the pre-train-effective and FT-effective components of the NTK matrix. Dataset | Method | Kernel | Rank | FN | Acc (train/test) | FT Ratio
---|---|---|---|---|---|---
RTE | - | Pre-train E | $28$ | $4.70\times 10^{4}$ | $66.40/51.20$ | -
FT | FT E | $488$ | $1.29\times 10^{4}$ | $96.60/53.40$ | $0.2148$
NTK | $191$ | $5.98\times 10^{4}$ | $97.60/53.00$
LoRA | FT E | $432$ | $2.51\times 10^{1}$ | $70.80/54.60$ | $0.0005$
NTK | $30$ | $4.70\times 10^{4}$ | $59.60/54.80$
LP-FT | FT E | $250$ | $3.80\times 10^{6}$ | $100.00/51.20$ | $0.9918$
NTK | $251$ | $3.84\times 10^{6}$ | $100.00/51.20$
LP-LoRA | FT E | $243$ | $7.60\times 10^{3}$ | $84.80/51.20$ | $0.1942$
NTK | $103$ | $5.26\times 10^{4}$ | $88.00/51.20$
BoolQ | - | Pre-train E | $32$ | $4.48\times 10^{4}$ | $53.60/57.20$ | -
FT | FT E | $495$ | $1.24\times 10^{4}$ | $100.00/56.40$ | $0.2139$
NTK | $215$ | $5.67\times 10^{4}$ | $53.80/57.20$
LoRA | FT E | $448$ | $2.48\times 10^{1}$ | $53.60/57.20$ | $0.0005$
NTK | $34$ | $4.48\times 10^{4}$ | $53.60/57.20$
LP-FT | FT E | $247$ | $4.46\times 10^{6}$ | $100.00/61.60$ | $0.9921$
NTK | $248$ | $4.49\times 10^{6}$ | $100.00/61.20$
LP-LoRA | FT E | $237$ | $8.56\times 10^{3}$ | $68.80/63.60$ | $0.2118$
NTK | $99$ | $5.07\times 10^{4}$ | $86.00/59.20$
WiC | - | Pre-train E | $16$ | $4.81\times 10^{4}$ | $66.00/54.00$ | -
FT | FT E | $488$ | $1.45\times 10^{4}$ | $89.00/59.00$ | $0.2216$
NTK | $235$ | $6.17\times 10^{4}$ | $90.60/59.00$ | -
LoRA | FT E | $438$ | $2.58\times 10^{1}$ | $72.00/52.00$ | $0.0005$
NTK | $19$ | $4.81\times 10^{4}$ | $65.80/56.40$
LP-FT | FT E | $218$ | $7.77\times 10^{7}$ | $100.00/56.80$ | $0.9996$
NTK | $219$ | $7.77\times 10^{7}$ | $100.00/56.40$
LP-LoRA | FT E | $218$ | $1.09\times 10^{5}$ | $72.00/59.60$ | $0.7454$
NTK | $195$ | $1.47\times 10^{5}$ | $80.80/59.60$

(a) RTE

(b) BoolQ

(c) WiC

Figure 7: Singular value distribution normalized by the maximum singular value
on the RTE, BoolQ, and WiC datasets. Pre-train E denotes the pre-train-
effective component, and other plots denote the FT-effective component of NTK
matrix with each training option.

(a) Pre-train E (CB)

(b) FT E (FT, CB)

(c) FT E (LP-FT, CB)

(d) Pre-train E (CB)

(e) FT E (FT, RTE)

(f) FT E (LP-FT, RTE)

(g) Pre-train E (BoolQ)

(h) FT E (FT, BoolQ)

(i) FT E (LP-FT, BoolQ)

(j) Pre-train E (WiC)

(k) FT E (FT, WiC)

(l) FT E (LP-FT, WiC)

Figure 8: Heat map of NTK matrix on the CB, RTE, BoolQ, and WiC dataset. We
calculate the trace norm of the sub-matrix of the NTK matrix for each sample
pair and visualize them grouped by class. Pre-train E and FT E refer to the
pre-train-effective and FT-effective components of the NTK matrix.

(a) LoRA, CB

(b) LoRA, RTE

(c) LoRA, BoolQ

(d) LoRA, WiC

(e) LP-LoRA, CB

(f) LP-LoRA, RTE

(g) LP-LoRA, BoolQ

(h) LP-LoRA, WiC

Figure 9: Heat map of NTK matrix of FT-effective component with LoRA on the
CB, RTE, BoolQ, and WiC dataset with LoRA. We calculate the trace norm of the
sub-matrix of the NTK matrix for each sample pair and visualize them grouped
by class.

####  7.4.3 Experiments on BOSS benchmark

Table 10 shows indicate that LP-FT surpasses FT in OOD robustness and achieves
higher accuracy in ID settings than LoRA. This suggests that LP-FT is
effective in enhancing model robustness to OOD samples with reduced feature
changes.

Table 11 displays the statistics of feature and classifier changes on the
Amazon, Dynasent, SemEval, and SST-5 datasets. The FDR within the ID is lower
for LP-FT than for FT, whereas the FDR for OOD is higher for LP-FT than for
FT. This indicates that LP-FT is learning robust features that are less
sensitive to OOD data.

Table 10: Evaluation results on BOSS benchmark. We report the average accuracy and standard deviation over five seeds. The best results are highlighted in bold. Method | ID | OOD
---|---|---
Amazon | Dynasent | SemEval | SST-5
LP | $83.04\pm 0.01$ | $42.69\pm 0.05$ | $50.04\pm 0.01$ | $56.81\pm 0.11$
FT | $88.66\pm 1.62$ | $44.33\pm 1.11$ | $52.20\pm 1.82$ | $72.52\pm 1.28$
LoRA | $86.05\pm 2.16$ | $46.70\pm 1.68$ | $55.29\pm 2.93$ | $72.88\pm 1.84$
LP-FT | $88.89\pm 1.02$ | $45.41\pm 0.80$ | $51.96\pm 2.72$ | $73.78\pm 1.05$
LP-LoRA | $88.17\pm 1.97$ | $43.37\pm 1.50$ | $48.84\pm 3.20$ | $72.31\pm 1.30$
Table 11: Comparison of feature and classifier changes on the Amazon (ID),
Dynasent, SemEval, and SST-5 (OOD) datasets. CS, Diff, FDR, and Norm denote
cosine similarity, difference norm, Fisher’s discriminant ratio, and norm,
respectively. (F) and (C) indicate feature and classifier statistics. Averages
were calculated over five seeds.

Method | Amazon | Dynasent
---|---|---
CS(F) | Diff(F) | FDR(F) | Norm(C) | CS(F) | Diff(F) | FDR(F)
Pre-trained | $0.996$ | $-$ | $1.30\times 10^{0}$ | $9.51\times 10^{-1}$ | $0.996$ | $-$ | $1.94\times 10^{0}$
LP | $0.996$ | $-$ | $1.30\times 10^{0}$ | $1.20\times 10^{2}$ | $0.996$ | $-$ | $1.94\times 10^{0}$
FT | $0.691$ | $1.94\times 10^{1}$ | $3.74\times 10^{0}$ | $9.50\times 10^{-1}$ | $0.652$ | $1.80\times 10^{1}$ | $2.03\times 10^{0}$
LoRA | $0.848$ | $1.16\times 10^{1}$ | $3.38\times 10^{0}$ | $1.81\times 10^{0}$ | $0.855$ | $7.53\times 10^{0}$ | $2.06\times 10^{0}$
LP-FT | $0.999$ | $2.27\times 10^{0}$ | $3.00\times 10^{0}$ | $1.20\times 10^{2}$ | $0.998$ | $2.54\times 10^{0}$ | $2.20\times 10^{0}$
LP-LoRA | $0.999$ | $2.24\times 10^{0}$ | $3.01\times 10^{0}$ | $1.18\times 10^{2}$ | $0.999$ | $2.56\times 10^{0}$ | $2.04\times 10^{0}$

Method | SemEval | SST5
---|---|---
CS(F) | Diff(F) | FDR(F) | CS(F) | Diff(F) | FDR(F)
Pre-trained | $0.996$ | $-$ | $1.24\times 10^{0}$ | $0.998$ | $-$ | $1.69\times 10^{1}$
LP | $0.996$ | $-$ | $1.24\times 10^{0}$ | $0.998$ | $-$ | $1.69\times 10^{1}$
FT | $0.727$ | $1.68\times 10^{1}$ | $1.49\times 10^{0}$ | $0.604$ | $1.84\times 10^{1}$ | $2.26\times 10^{1}$
LoRA | $0.885$ | $6.74\times 10^{0}$ | $1.44\times 10^{0}$ | $0.837$ | $8.72\times 10^{0}$ | $2.01\times 10^{1}$
LP-FT | $0.997$ | $2.06\times 10^{0}$ | $1.45\times 10^{0}$ | $0.998$ | $1.86\times 10^{0}$ | $2.02\times 10^{1}$
LP-LoRA | $0.999$ | $2.08\times 10^{0}$ | $1.19\times 10^{0}$ | $0.998$ | $1.85\times 10^{0}$ | $1.95\times 10^{1}$

####  7.4.4 Change of feature and classifier norms

Table 12 shows the changes in features during the FT stage, indicating that
the changes are smaller during LP-FT compared to FT. Table 13 shows the
classifier norms, which increase during training, with a more noticeable
increase observed during LP than during FT.

Table 12: Feature change in FT stage. The change during LP-FT is smaller than during FT. Dataset | FT | LoRA | LP-FT | LP-LoRA
---|---|---|---|---
CB | $2.11\times 10^{1}$ | $2.07\times 10^{1}$ | $1.15\times 10^{1}$ | $7.85\times 10^{0}$
RTE | $2.12\times 10^{1}$ | $1.51\times 10^{1}$ | $3.33\times 10^{0}$ | $3.87\times 10^{0}$
COLA | $1.91\times 10^{1}$ | $1.10\times 10^{1}$ | $3.05\times 10^{0}$ | $2.75\times 10^{0}$
SST-2 | $2.31\times 10^{1}$ | $3.78\times 10^{0}$ | $6.95\times 10^{0}$ | $2.17\times 10^{0}$
MRPC | $2.11\times 10^{1}$ | $1.80\times 10^{0}$ | $1.84\times 10^{0}$ | $1.94\times 10^{0}$
BoolQ | $2.23\times 10^{1}$ | $1.55\times 10^{1}$ | $2.31\times 10^{0}$ | $1.95\times 10^{0}$
WiC | $2.08\times 10^{1}$ | $1.04\times 10^{1}$ | $2.28\times 10^{0}$ | $2.16\times 10^{0}$
WSC | $9.14\times 10^{0}$ | $2.44\times 10^{-1}$ | $7.33\times 10^{0}$ | $2.02\times 10^{-1}$
Amazon | $1.98\times 10^{1}$ | $1.35\times 10^{1}$ | $2.21\times 10^{0}$ | $2.28\times 10^{0}$
Dynasent | $1.94\times 10^{1}$ | $8.02\times 10^{0}$ | $2.47\times 10^{0}$ | $2.59\times 10^{0}$
SemEval | $1.83\times 10^{1}$ | $6.97\times 10^{0}$ | $1.99\times 10^{0}$ | $2.14\times 10^{0}$
SST-5 | $2.03\times 10^{1}$ | $9.08\times 10^{0}$ | $1.79\times 10^{0}$ | $1.89\times 10^{0}$
Table 13: The classifier weight norms. The classifier weight norms increase during training, and the increase is more pronounced in LP. Dataset | Pretrain | FT | LoRA | LP | LP-FT | LP-LoRA
---|---|---|---|---|---|---
CB | $9.47\times 10^{-1}$ | $9.51\times 10^{-1}$ | $1.56\times 10^{0}$ | $3.35\times 10^{1}$ | $3.35\times 10^{1}$ | $3.35\times 10^{1}$
RTE | $7.95\times 10^{-1}$ | $8.05\times 10^{-1}$ | $1.45\times 10^{0}$ | $2.86\times 10^{1}$ | $2.86\times 10^{1}$ | $2.85\times 10^{1}$
COLA | $7.95\times 10^{-1}$ | $7.88\times 10^{-1}$ | $1.06\times 10^{0}$ | $3.46\times 10^{1}$ | $3.46\times 10^{1}$ | $3.51\times 10^{1}$
SST2 | $7.95\times 10^{-1}$ | $7.20\times 10^{-1}$ | $1.96\times 10^{0}$ | $1.32\times 10^{2}$ | $1.09\times 10^{2}$ | $1.03\times 10^{2}$
MRPC | $7.95\times 10^{-1}$ | $7.98\times 10^{-1}$ | $1.35\times 10^{0}$ | $1.12\times 10^{1}$ | $1.12\times 10^{1}$ | $1.12\times 10^{1}$
BoolQ | $7.95\times 10^{-1}$ | $7.98\times 10^{-1}$ | $1.15\times 10^{0}$ | $1.27\times 10^{1}$ | $1.27\times 10^{1}$ | $1.25\times 10^{1}$
WiC | $7.95\times 10^{-1}$ | $7.98\times 10^{-1}$ | $1.14\times 10^{0}$ | $3.21\times 10^{1}$ | $3.25\times 10^{1}$ | $3.27\times 10^{1}$
WSC | $7.95\times 10^{-1}$ | $6.87\times 10^{-1}$ | $7.88\times 10^{-1}$ | $2.26\times 10^{-4}$ | $1.08\times 10^{-1}$ | $2.16\times 10^{-2}$
Amazon | $9.51\times 10^{-1}$ | $9.47\times 10^{-1}$ | $1.67\times 10^{0}$ | $1.21\times 10^{2}$ | $1.21\times 10^{2}$ | $1.20\times 10^{2}$

####  7.4.5 Effects of classifier weight norms in training

Figure 10 (Boss benchmark) and Figure 11 (the CB and RTE datasets) illustrate
the changes in features from the pre-trained models. Except for the CB
dataset, the change in features in LP-FT is generally smaller than in FT when
using large classifier norms. The CB dataset has a smaller sample size, which
could be an exception.

(a) Amazon (ID)

(b) Dynasent (OOD)

(c) SemEval (OOD)

Figure 10: Difference of features of the samples with scaling the classifier
weight norms on BOSS benchmark. The dashed vertical lines indicate the
original norms of the classifier weight.

(a) CB

(b) RTE

Figure 11: Difference of features of the samples with scaling the classifier
weight norms on the CB and RTE datasets. The dashed vertical lines indicate
the original norms of the classifier weight.

####  7.4.6 Temperature scaling

The result of temperature scaling on SuperGLUE and GLUE is presented in Tables
14 and 15.

Table 14: ECE and MCE with temperature scaling on SuperGLUE. w/o TS and w/ TS denote without and with temperature scaling, respectively, and Imp. represents the improvement because of temperature scaling. We bold the best improvements. We take 5 seeds and report the mean and the standard deviation. Dataset | Metric | Method | w/o TS | w/ TS | Imp.
---|---|---|---|---|---
CB | ECE (%) | FT | $15.60\pm 0.96$ | $14.64\pm 1.75$ | $0.95$
LP-FT | $13.93\pm 0.45$ | $13.13\pm 0.56$ | $0.80$
LoRA | $12.89\pm 0.41$ | $16.22\pm 0.55$ | $-3.34$
LP-LoRA | $14.78\pm 0.93$ | $13.51\pm 1.67$ | $1.27$
MCE (%) | FT | $75.99\pm 6.12$ | $69.99\pm 5.83$ | $6.01$
LP-FT | $76.78\pm 3.66$ | $70.28\pm 3.27$ | $6.50$
LoRA | $52.58\pm 4.72$ | $66.75\pm 7.96$ | $-14.16$
LP-LoRA | $68.16\pm 4.95$ | $60.80\pm 2.30$ | $7.36$
RTE | ECE (%) | FT | $21.16\pm 1.36$ | $5.13\pm 0.63$ | $16.03$
LP-FT | $21.72\pm 0.28$ | $5.48\pm 0.77$ | $16.24$
LoRA | $11.92\pm 2.23$ | $6.17\pm 0.20$ | $5.76$
LP-LoRA | $18.14\pm 0.99$ | $5.72\pm 0.48$ | $12.42$
MCE (%) | FT | $53.11\pm 8.51$ | $25.87\pm 6.30$ | $27.24$
LP-FT | $63.95\pm 7.70$ | $13.94\pm 1.80$ | $50.01$
LoRA | $25.04\pm 3.33$ | $13.75\pm 0.91$ | $11.29$
LP-LoRA | $40.46\pm 7.22$ | $18.82\pm 2.00$ | $21.63$
BoolQ | ECE (%) | FT | $13.63\pm 0.61$ | $1.83\pm 0.09$ | $11.81$
LP-FT | $18.93\pm 0.15$ | $2.41\pm 0.42$ | $16.51$
LoRA | $8.88\pm 0.38$ | $1.45\pm 0.18$ | $7.43$
LP-LoRA | $14.09\pm 0.92$ | $2.07\pm 0.19$ | $12.02$
MCE (%) | FT | $23.26\pm 1.48$ | $5.79\pm 0.90$ | $17.47$
LP-FT | $40.82\pm 1.94$ | $5.21\pm 0.53$ | $35.60$
LoRA | $13.96\pm 0.72$ | $3.85\pm 0.56$ | $10.11$
LP-LoRA | $24.60\pm 2.52$ | $5.51\pm 0.72$ | $19.09$
WiC | ECE (%) | FT | $25.88\pm 2.39$ | $8.85\pm 0.53$ | $17.03$
LP-FT | $29.47\pm 1.57$ | $7.68\pm 0.55$ | $21.78$
LoRA | $18.66\pm 4.39$ | $5.93\pm 1.42$ | $12.73$
LP-LoRA | $22.22\pm 1.98$ | $8.06\pm 0.60$ | $14.15$
MCE (%) | FT | $41.59\pm 5.39$ | $17.01\pm 2.87$ | $24.58$
LP-FT | $39.20\pm 2.74$ | $17.04\pm 1.50$ | $22.16$
LoRA | $27.95\pm 7.38$ | $11.40\pm 2.77$ | $16.54$
LP-LoRA | $30.99\pm 3.64$ | $14.45\pm 1.01$ | $16.54$
WSC | ECE (%) | FT | $6.26\pm 2.37$ | $7.97\pm 0.06$ | $-1.71$
LP-FT | $6.38\pm 1.78$ | $8.01\pm 0.06$ | $-1.63$
LoRA | $10.53\pm 1.35$ | $9.19\pm 0.60$ | $1.34$
LP-LoRA | $11.40\pm 0.23$ | $8.24\pm 0.01$ | $3.15$
MCE (%) | FT | $6.26\pm 2.37$ | $7.97\pm 0.06$ | $-1.71$
LP-FT | $6.38\pm 1.78$ | $8.01\pm 0.06$ | $-1.63$
LoRA | $13.27\pm 1.12$ | $11.12\pm 1.51$ | $2.15$
LP-LoRA | $11.40\pm 0.23$ | $8.24\pm 0.01$ | $3.15$
Table 15: ECE and MCE with temperature scaling on GLUE. w/o TS and w/ TS denote without and with temperature scaling, respectively, and Imp. represents the improvement because of temperature scaling. We bold the best improvements. We take 5 seeds and report the mean and the standard deviation. Dataset | Metric | Method | w/o TS | w/ TS | Imp.
---|---|---|---|---|---
CoLA | ECE (%) | FT | $15.08\pm 0.55$ | $4.46\pm 0.83$ | $10.61$
LP-FT | $15.74\pm 0.40$ | $9.53\pm 1.23$ | $6.21$
LoRA | $11.25\pm 1.32$ | $4.18\pm 0.40$ | $7.07$
LP-LoRA | $13.82\pm 0.48$ | $4.30\pm 0.43$ | $9.52$
MCE (%) | FT | $47.19\pm 5.15$ | $24.35\pm 3.33$ | $22.84$
LP-FT | $54.59\pm 2.94$ | $20.31\pm 1.37$ | $34.28$
LoRA | $31.01\pm 5.83$ | $15.23\pm 2.74$ | $15.78$
LP-LoRA | $38.36\pm 7.85$ | $15.36\pm 1.83$ | $23.00$
SST-2 | ECE (%) | FT | $4.61\pm 0.31$ | $2.26\pm 0.22$ | $2.35$
LP-FT | $5.67\pm 0.12$ | $2.00\pm 0.21$ | $3.66$
LoRA | $4.84\pm 0.13$ | $2.71\pm 0.16$ | $2.12$
LP-LoRA | $6.22\pm 0.10$ | $2.53\pm 0.08$ | $3.69$
MCE (%) | FT | $49.22\pm 4.78$ | $42.72\pm 5.24$ | $6.50$
LP-FT | $74.91\pm 1.72$ | $42.77\pm 5.75$ | $32.13$
LoRA | $54.20\pm 2.84$ | $36.58\pm 5.82$ | $17.63$
LP-LoRA | $71.12\pm 3.97$ | $32.47\pm 3.74$ | $38.65$
MRPC | ECE (%) | FT | $10.71\pm 0.39$ | $4.61\pm 0.24$ | $6.10$
LP-FT | $10.35\pm 0.14$ | $3.68\pm 0.10$ | $6.68$
LoRA | $6.58\pm 0.68$ | $4.04\pm 0.87$ | $2.54$
LP-LoRA | $9.03\pm 0.85$ | $3.89\pm 0.40$ | $5.14$
MCE (%) | FT | $61.84\pm 7.93$ | $32.72\pm 1.69$ | $29.12$
LP-FT | $74.43\pm 2.22$ | $22.73\pm 1.33$ | $51.70$
LoRA | $28.80\pm 5.05$ | $17.57\pm 2.00$ | $11.23$
LP-LoRA | $52.20\pm 6.64$ | $22.76\pm 7.60$ | $29.44$

####  7.4.7 PubMed 20k

In addition to the natural language understanding benchmarks, we also
evaluated LP-FT on the PubMed $20$k RCT dataset to evaluate its effectiveness
in practical applications. The PubMed $20$k RCT dataset, a subset of PubMed
$200$k [10], comprises 20,000 medical abstracts from randomized controlled
trials, categorized into five classes. Efficient tools for navigating
extensive medical literature are essential for the medical community.

The results are presented in Table 16. The LoRA model outperforms other
models, although the performance of FT, LP-FT, and LoRA models are relatively
similar.

Table 16: Test accuracy on PubMed $20$k. LP | FT | LP-FT | LoRA | LP-LoRA
---|---|---|---|---
$82.64\pm 0.02$ | $87.09\pm 0.17$ | $87.05\pm 0.11$ | $\bm{87.13\pm 0.09}$ | $86.85\pm 0.07$