(page )

# APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient
Training and Inference

Bowen Zhao, Hannaneh Hajishirzi, Qingqing Cao
University of Washington
bowen98<EMAIL_ADDRESS>

###### Abstract

Fine-tuning and inference with large Language Models (LM) are generally known
to be expensive. Parameter-efficient fine-tuning over pretrained LMs reduces
training memory by updating a small number of LM parameters but does not
improve inference efficiency. Structured pruning improves LM inference
efficiency by removing consistent parameter blocks, yet often increases
training memory and time. To improve both training and inference efficiency,
we introduce APT that adaptively prunes and tunes parameters for the LMs. At
the early stage of fine-tuning, APT dynamically adds salient tuning parameters
for fast and accurate convergence while discarding unimportant parameters for
efficiency. Compared to baselines, our experiments show that APT maintains up
to 98% task performance when pruning RoBERTa and T5 models with 40% parameters
left while keeping 86.4% LLaMA models’ performance with 70% parameters
remained. Furthermore, APT speeds up LMs’ fine-tuning by up to 8$\times$ and
reduces large LMs’ memory training footprint by up to 70%.

##  1 Introduction

Fine-tuning language models (LMs) (Devlin et al., 2019; Liu et al., 2019;
Raffel et al., 2020) is an essential paradigm to adapt them to downstream
tasks (Mishra et al., 2022; Wang et al., 2022b). Increasing the parameter
scale of LMs improves model performance (Kaplan et al., 2020), but incurs
significant training and inference costs. For instance, a 13B LLaMA model
(Touvron et al., 2023) costs about 100GB memory for fine-tuning and 30GB for
inference with float16 datatype. It is important to improve the training and
inference efficiency of LM for practical applications.

Figure 1: APT provides both training and inference efficiency benefits by
pruning and tuning pretrained LM parameters adaptively via the APT adapter. We
dynamically adjust (add/reduce) APT adapter input/output dimensions and the
rank ($r_{\text{apt}}$). Reducing adapter dimensions prunes frozen parameters,
making training and inference faster and more memory-efficient. Adding adapter
ranks helps recover the pruned LM’s task performance. In contrast, existing
adapters like LoRA allow efficient training but do not provide inference
efficiency since the model size is not reduced.

Parameter-efficient fine-tuning methods (PEFT, summarized in Table 1) (Houlsby
et al., 2019; Li and Liang, 2021) reduce LMs fine-tuning memory consumption
via updating a small number of parameters. However, PEFT models do not improve
inference efficiency because the LM size remains the same or even increases.
For instance, LoRA (Hu et al., 2021) tunes low-rank decomposed linear layers
parallel to frozen parameters to reduce training memory but takes longer to
converge (Ding et al., 2023). On the other hand, structured pruning Kwon et
al. (2022); Xia et al. (2022); Ma et al. (2023) improves inference efficiency
by removing blocks of parameters such as attention heads and feed-forward
neurons in Transformer LMs, showing more inference speedup than sparse
unstructured pruning methods (Han et al., 2015a, b; Sanh et al., 2020).
However, training pruned LMs takes extra time to converge and incurs high
memory.

Integrating structured pruning and PEFT could increase both training and
inference efficiency. However, existing research (Zhao et al., 2023) indicates
that simply combining PEFT and structured pruning, for instance, applying
structured pruning over LoRA-tuned models, leads to substantial performance
loss and extra training costs. It remains challenging to accurately prune LMs
using limited training resources.

Method | AP | AT | Training | Inference
---|---|---|---|---
T | M | T | M
PEFT | Adapter | ✗ | ✗ | $\Uparrow_{\text{High}}$ | $\Downarrow_{\text{Low}}$ | $\Uparrow_{\text{Low}}$ | $\Uparrow_{\text{Low}}$
LoRA | ✗ | ✗ | $\Uparrow_{\text{High}}$ | $\Downarrow_{\text{Low}}$ | = | =
AdaLoRA | ✗ | ✓ | $\Uparrow_{\text{High}}$ | $\Downarrow_{\text{Low}}$ | = | =
Pruning | MvP | ✗ | ✗ | $\Uparrow_{\text{High}}$ | $\Uparrow_{\text{Low}}$ | $\Downarrow_{\text{Low}}$ | $\Downarrow_{\text{Low}}$
BMP | ✗ | ✗ | $\Uparrow_{\text{High}}$ | $\Uparrow_{\text{Low}}$ | $\Downarrow_{\text{High}}$ | $\Downarrow_{\text{Low}}$
CoFi | ✗ | ✗ | $\Uparrow_{\text{High}}$ | $\Uparrow_{\text{Low}}$ | $\Downarrow_{\text{High}}$ | $\Downarrow_{\text{Low}}$
MT | ✗ | ✗ | = | = | $\Downarrow_{\text{High}}$ | $\Downarrow_{\text{Low}}$
Combined | SPA | ✗ | ✗ | $\Uparrow_{\text{High}}$ | $\Uparrow_{\text{Low}}$ | $\Downarrow_{\text{High}}$ | $\Downarrow_{\text{Low}}$
LRP | ✗ | ✗ | $\Uparrow_{\text{High}}$ | $\Downarrow_{\text{Low}}$ | $\Downarrow_{\text{High}}$ | $\Downarrow_{\text{Low}}$
APT | ✓ | ✓ | $\Uparrow_{\text{Low}}$ | $\Downarrow_{\text{Low}}$ | $\Downarrow_{\text{High}}$ | $\Downarrow_{\text{Low}}$

Table 1: Comparison of efficient methods to full fine-tuning. AP and AT
represent Adaptive Pruning and Tuning, where the total and tuning parameter
sizes are dynamically adjusted. We use training converge time or inference
time (T) and peak memory (M) to measure efficiency. $\Uparrow$ and
$\Downarrow$ denote change directions.

In this paper, we develop an efficient fine-tuning approach named APT that
Adaptively selects model parameters for Pruning and fine-Tuning. APT combines
the benefits of PEFT and structured pruning to make fine-tuning and inference
more efficient. Our intuition is that pre-trained LM parameters contain
general knowledge, but their importance to downstream tasks varies. Therefore,
we can remove the parameters irrelevant to the fine-tuning task in the early
training stage. Removing these parameters early improves training and
inference efficiency while not substantially hurting model accuracy (Frankle
et al., 2021; Shen et al., 2022a; Zhang et al., 2023c). Meanwhile,
continuously adding more parameters for fine-tuning could improve performance
because task-specific skills that live in a subset of LM parameters (Wang et
al., 2022a; Panigrahi et al., 2023).

More specifically, APT learns the pruning masks via an outlier-aware salience
scoring function to remove irrelevant LM parameter blocks and add more tuning
parameters during fine-tuning according to layer importance. To make training
more efficient, the salience scoring function is lightweight and causes little
runtime and memory overhead. Combined with our self-distillation technique
that shares teacher and student parameters, APT can accurately prune an LM
with less training time and lower memory usage.

Experimental results show that APT prunes RoBERTa and T5 base models 8$\times$
faster than the LoRA plus pruning baseline while reaching 98.0% performance
with 2.4 $\times$ speedup and 78.1% memory consumption during inference. When
pruning large LMs like LLaMA, APT costs only 30% memory compared to the state-
of-the-art pruning method and still maintains 86.4% performance with 70%
parameters. Our ablation study in Section 5.6 indicates the effectiveness of
adaptive pruning and tuning. It also demonstrates that efficient distillation
with APT adapter substantially recovers small LMs’ performance while outlier-
aware salience scoring prunes large LMs more accurately. Our analysis in
Section 5.7 demonstrates that controlled adaptive tuning with early pruning
during fine-tuning improves LM end-task accuracy better with less training
time and memory costs.

##  2 Related Works

###  2.1 Parameter-efficient Fine-tuning (PEFT)

PEFT methods aim to tune LMs with limited resources by updating a small number
of parameters (Lialin et al., 2023), mainly falling into three categories:
selective, additive, and dynamic. Selective methods focus on tuning a subset
of parameters in LMs with pre-defined rules (Ben Zaken et al., 2022) or
importance metrics (Sung et al., 2021; Guo et al., 2021). Additive methods
inject layer modules (Houlsby et al., 2019; Pfeiffer et al., 2020) or
embeddings (Lester et al., 2021; Li and Liang, 2021) for LM tuning. For
example, LoRA (Hu et al., 2021) tunes low-rank decomposed linear weights to
avoid inference cost overhead. However, LoRA keeps the tuning layer shapes
static in training without dynamic adjustments. Dynamic methods (He et al.,
2022) adjust tuning parameters during training. For instance, AdaLoRA (Zhang
et al., 2023b) gradually reduces tuning parameters but does not benefit
inference efficiency. Compared to these methods, APT adaptively adjusts the
pruning and tuning parameters simultaneously, improving training and inference
efficiency.

###  2.2 Model Compression

Model compression methods like quantization and pruning boost inference
efficiency. Quantization aims to reduce LMs’ memory consumption via converting
parameters to low-bit data types (Frantar et al., 2022; Dettmers et al., 2022;
Lin et al., 2023). However, despite effectively reducing LM’s memory
consumption, the speedup benefits of quantization require specific framework
support, limiting their practicality and adaptability.

Pruning (LeCun et al., 1989; Han et al., 2015a; Frankle and Carbin, 2018; Xu
et al., 2021) aims to discard unimportant parameters in LMs for inference
efficiency. Unstructured pruning (Sanh et al., 2020) prunes sparse parameters
in LMs, which requires dedicated hardware support for efficiency improvements.
Structured pruning (Lagunas et al., 2021; Xia et al., 2022), on the other
hand, prunes consistent blocks in transformer layers (MHA heads, FFN neurons,
and model dimensions) for effective and ubiquitous inference efficiency gains.
Such pruning often uses knowledge distillation (Hinton et al., 2015), which
causes more training costs. Post-training pruning (Kwon et al., 2022; Frantar
and Alistarh, 2023) aims to prune fine-tuned models with limited extra costs
but requires initialization from fully fine-tuned models. Moreover, task-
agnostic pruning (Sun et al., 2023; Ma et al., 2023) cannot achieve on-par
performance with task-specific pruning.

###  2.3 Combining Compression and PEFT

Combining model compression and PEFT might achieve both training and inference
efficiency improvements: QLoRA (Dettmers et al., 2023) and QA-LoRA (Xu et al.,
2023) bring quantization and LoRA together for large LM tuning. SPA (Hedegaard
et al., 2022) combines structured pruning and Compacter (Karimi Mahabadi et
al., 2021), yet suffers substantial performance loss. CPET (Zhao et al., 2023)
leverages different task-agnostic model compression methods together with LoRA
and knowledge distillation, but the performance loss becomes notable
specifically when structured pruning is applied. PST (Li et al., 2022) and LRP
(Zhang et al., 2023a) also explored the combination of LoRA and pruning, yet
their performance degradations are also substantial because their tuning
parameters are static. In contrast, APT identifies tuning and pruning
parameters based on their salience in fine-tuning, which can improve training
and inference efficiency under a new paradigm with minimal performance loss.

##  3 Problem Formulation

APT aims to improve the training and inference efficiency of pretrained LM
while maintaining task performance. Intuitively, tuning fewer parameters leads
to smaller training memory footprints and shorter time per training step;
models with fewer parameters also run faster with less memory footprint during
inference but come with task performance degradation. We aim to find the
optimal parameters for training and inference without sacrificing task
performance.

We formally define the problem objective as minimizing the task loss
$\mathcal{L}$ under the constraint that the total LM parameter size $\Theta$
reaches a target sparsity (defined as the ratio of the number of parameters
pruned to the original LM) $\gamma_{T}$ after $T$ training steps. For each
training step $t$, the sparsity of the LM remains above $\gamma_{t}$ while the
number of tuning parameters is below $\Delta_{t}$. We control the pruning
masks $\mathcal{M}_{t}$ and tuning ranks $\mathcal{R}_{t}$ to satisfy these
constraints. We describe the optimization process as:

$\displaystyle\operatorname*{argmin}_{\Theta_{T},\mathcal{M}_{T}}$
$\displaystyle\frac{1}{|\mathcal{D}|}\sum_{x,y\in\mathcal{D}}\mathcal{L}(x,y|\Theta_{T},\mathcal{M}_{T})$
(1) s.t. $\displaystyle
1-\frac{\mathcal{C}(\Theta_{t},\mathcal{M}_{t})}{\mathcal{C}(\Theta_{0},\mathcal{M}_{0})}\geq\gamma_{t},$
$\displaystyle\delta(\Theta_{t},\mathcal{M}_{t},\mathcal{R}_{t})\leq\Delta_{t},$
$\displaystyle\forall t\in\\{0,\ldots,T\\}.$

where $x,y$ are inputs and labels sampled from the task dataset $\mathcal{D}$,
while $\mathcal{C}$ and $\delta$ denotes total and tuning parameter numbers of
the LM, respectively.

Based on Eq. 1, a higher target sparsity $\gamma_{T}$ improves inference
efficiency with fewer FLOPs and memory usage but sacrifices performance.
Increasing $\gamma_{t}$ when $t\ll T$ also improves training efficiency. On
the other hand, tuning more parameters with larger $\Delta$ costs more
training memory but makes the model converge faster with better task
performance. Our formulation supports task performance improvements together
with training and inference efficiency by dynamically adjusting the LM
parameters during fine-tuning.

##  4 Adaptive Pruning and Tuning

Figure 2: APT adaptively identifies pruning and tuning parameters via APT
adapters during fine-tuning with little cost. APT gradually prunes LM
parameters with binary pruning masks learned from our lightweight outlier-
aware salience scoring function for training and inference efficiency. APT
also adds tuning parameters in salient layers in LM fine-tuning through
increasing dynamic ranks in APT adapters for performance recovery.

We design Adaptive Pruning and Tuning (APT) over LM parameters to allow
efficient training and inference while maintaining task performance.

Summarized in the left of Fig. 2, existing pruning methods often neglect
training costs where the number of tuning parameters is more than a parameter-
efficient threshold with
$\Delta_{t}\geq\mathcal{C}(\Theta_{t},\mathcal{M}_{t})$, resulting in long
training time and high memory consumption. Instead, to improve training
efficiency, we prune LM parameters (increase $\gamma_{t}$) during early
training when $t\ll T$ while keeping
$\Delta_{t}\ll\mathcal{C}(\Theta_{t},\mathcal{M}_{t})$ to reduce training
costs. In addition, we add tuning parameters (increase $\Delta_{t}$) in early
training to effectively mitigate the degradation of LM’s performance due to
pruning.

Overview. Fig. 2 shows the overview of our method that incorporates our new
APT adapter for pruning and tuning. Our intuition is that pruning LMs during
early fine-tuning will not hurt their task performance while reducing training
and inference costs. Meanwhile, unlike existing adapters like LoRA Hu et al.
(2021) that use fixed tuning parameters, APT adapters dynamically add tuning
parameters to accelerate LM convergence with superior task performance. We
first introduce the architecture of APT adapters in Section 4.1. We then
describe how we prune LM parameters at early fine-tuning with low cost in
Section 4.2 and adaptively tune LMs to recover task performance efficiently in
Section 4.3. Additionally, we explain our self-knowledge distillation
technique that improves pruned LM’s task performance with limited training
time and memory consumption in Section 4.4.

###  4.1 APT adapter

We build the APT adapter architecture over LoRA, but the key difference is
that APT adapter supports dynamic LM pruning and tuning. Assuming an APT
adapter projects the input activation with dimension $d_{i}$ to output
dimension $d_{o}$, we design binary pruning masks
($m_{i}\in\mathbb{R}^{d_{i}}$ for input and $m_{o}\in\mathbb{R}^{d_{o}}$ for
output) and dynamic ranks $r_{\text{apt}}$ in APT adapter to control the total
and tuning LM parameters during fine-tuning, respectively. Specifically, with
tuning parameters $W_{A}\in\mathbb{R}^{r_{\text{apt}}\times d_{i}}$ and
$W_{B}\in\mathbb{R}^{d_{o}\times r_{\text{apt}}}$, APT adapter
$H_{\text{apt}}$ is denoted as:

$H_{\text{apt}}(X)=m_{o}\circ(W+s\cdot W_{B}W_{A})X\circ m_{i}$ (2)

where $\circ$ denotes the block-wise Hadamard product between the masks and
their corresponding matrices. The parameter block is pruned when the
multiplying mask is set to 0 and retained when set to 1. In the meantime,
during fine-tuning, we dynamically increase $r_{\text{apt}}$ for the weight
matrices $W_{B}$ and $W_{A}$. Compared to LoRA, APT adapters can be more
efficient due to more adaptive pruning and tuning over LM parameters.

In transformer-based LM fine-tuning, we add APT adapters in queries and values
of multi-head attention (MHA) layers. We also add APT adapter in feed-forward
network (FFN) layers when fine-tuning smaller models like RoBERTa and T5 for
fast training convergence. In these cases, $m_{i}$ prunes transformers’ hidden
dimension and $m_{o}$ prunes attention heads in MHA and internal neurons in
FFN layers. By learning the pruning masks and adjusting the ranks dynamically
in the APT adapter, we can achieve the goal defined in Section 3 where the
tuning parameter number $\delta(\Theta_{t},\mathcal{M}_{t},\mathcal{R}_{t})$
increases to maintain task performance and the LM parameter size
$\mathcal{C}(\Theta_{t},\mathcal{M}_{t})$ decreases to support more efficient
training and inference. Next, we describe the adaptive pruning and tuning
procedures in detail.

###  4.2 Low-cost Adaptive LM Pruning ($\mathcal{A}_{\text{P}}$)

To benefit the efficiency of LM training and inference, APT adaptively prunes
LM parameters since the start of fine-tuning. The problem is finding the
parameters to be pruned and discarding them without hurting training
stability. Given a task, we compute the outlier-aware salience score of block
parameters at each training step during the early fine-tuning stage when $t\ll
T$. Afterward, we use a fast search algorithm to determine which parameter
blocks to prune, and then we update their binary pruning masks accordingly.
The upper-right of Fig. 2 shows this adaptive pruning procedure.

Outlier-aware salience scoring of LM parameters. When determining the
influence of pruning parameters on the LM performance for fine-tuning tasks,
the key idea is to compute the outlier-aware salience scores of LM activations
that consider both tuning and frozen parameters. In detail, salience is
defined as the magnitude of parameters’ weight-gradient production from
previous works (Sanh et al., 2020; Zhang et al., 2023b).

$S(W_{i,j})=|{W}_{i,j}\cdot\frac{\partial\mathcal{L}}{\partial{W}_{i,j}}|$ (3)

However, since the frozen weights’ gradients are unreachable in PEFT settings,
we compute the salience as the magnitude of the product of activations and
their gradients. Additionally, we compress the activation and gradients by
summing along batches before production to further reduce the training memory
consumption. On the other hand, block outlier parameters play a crucial role
in task-specific capabilities, as previous quantization methods suggest
(Dettmers et al., 2022; Lin et al., 2023). Such effects brought by outlier
parameters will be averaged if salience is only measured on the block level.
To keep more outlier parameters in the pruned LMs, we combine the salience
score above and the kurtosis111Representing the density of the outlier in a
distribution, the more the outliers are, the bigger the kurtosis will be of
the activation together:

$\displaystyle\textrm{Kurt}(X)$
$\displaystyle=\sum_{(x,y)\in\mathcal{D}}(\sum_{i}(\frac{X_{i}-\mu(X)}{\sigma(X)})^{4}-3)$
(4)
$\displaystyle\begin{split}\widetilde{S}_{t}(W_{:,j})&=\sum_{(x,y)\in\mathcal{D}_{t}}\sum_{i}|\frac{\partial\mathcal{L}(x,y|\Theta_{t},\mathcal{M}_{t})}{\partial
H_{j,i}}|\cdot\\\
&\quad\sum_{(x,y)\in\mathcal{D}_{t}}\sum_{i}|H_{j,i}|\end{split}$ (5)
$\displaystyle\hat{S}((W_{:,j})$
$\displaystyle=\widetilde{S}(W_{:,j})+(\textrm{Kurt}(O_{j,:}))^{\frac{1}{2}}$
(6)

where $H$ is the activations in the LM, $\mu$ and $\sigma$ denote the mean and
standard deviation, and $O_{:,j}=W_{:,j}\circ X_{j,:}^{\intercal}$ represents
the activation. We leave details of the salience scoring in Appendix B.

Efficient search of LM block parameters. Given the salience calculated in Eq.
6, the next step is to learn the binary pruning masks to increase the LM
sparsity above $\gamma_{t}$. Intuitively, we shall prune the blocks with less
salience score, which formulates a latency-saliency knapsack (Shen et al.,
2022b) task. For an LM with $n_{L}$ transformer layers, where layer $i$ has
$n_{h}^{i}$ MHA heads and $n_{f}^{i}$ FFN neurons, and all transformer layers’
hidden dimension sizes are $d_{m}$, the approximated222We ignore the model’s
layer norm and bias terms since their sizes are small, and we do not count
tuning parameters since they can be fully merged after training. number LM
parameter is:

$\mathcal{C}(\Theta_{t};\mathcal{M}_{t})\approx
d_{m}\sum_{i=1}^{n_{L}}(4n_{h}^{i}\cdot d_{h}+2n_{f}^{i})$ (7)

where $d_{h}$ is the dimension per attention head. To keep the constraint
denoted in Eq. 1, we need to prune MHA heads, FFN neurons, and the model
hidden dimension simultaneously by reducing $n^{i}_{h},n^{i}_{f}$, and
$d_{m}$. Hence, we first sort the blocks by their salience divided by the
parameter number. As the parameter size monotonically increases with the
number of blocks, we then use binary search to identify the top salient blocks
to be retained given the sparsity constraint $\gamma_{t}$. For simplicity, we
leave the implementation details in Appendix C.

###  4.3 Adaptive and Efficient LM Tuning ($\mathcal{A}_{\text{T}}$)

As using PEFT methods to fine-tune pruned LMs causes notable performance
decrease (illustrated in Table 2 and Table 4), we aim to dynamically add
tuning parameters in LM fine-tuning to improve the model’s end-task
performance. However, since more tuning parameters will consume extra training
time and memory, we want to add parameters in a controlled way, where new
parameters are only added to task-sensitive APT adapters. As a result, we can
recover pruned LMs’ performance with reasonable training costs. In detail, we
first calculate the salience of each APT adapter to determine their
importance. Next, we select the top-half APT adapters after sorting them with
salience and add their parameters by increasing their $r_{\text{apt}}$.

Salience scoring of APT adapter. Since gradients of tuning parameters
information are available when determining the layer salience, we can first
calculate each tuning parameter’s salience with Eq. 3. Then, we define the
salience of an APT adapter as the summation of the parameter salience scores
in $W_{B}$, denoted as
$\mathcal{I}(H_{\text{apt}})=\sum_{i,j}S({W_{B}}_{i,j})$, to represent each
tuning APT adapter’s importance333The salience scores calculated using $W_{B}$
and $W_{A}$ are equal, so using either of them will get the same result..
Given the calculated $\mathcal{I}(H_{\text{apt}})$ for each APT adapter, we
can then decide where to add new tuning parameters to efficiently improve the
pruned LM’s task accuracy.

Dynamically adding APT adapter parameters to recover task performance. With
the importance of APT adapters $\mathcal{I}(H_{\text{apt}})$ calculated, the
next step of adaptive tuning is to add tuning parameters by increasing the
salient tuning layers’ ranks $r_{\text{apt}}\in\mathcal{R}_{t}$ following
budget $\Delta_{t}$. Therefore, firstly, we sort all tuning layers according
to their importance score $\mathcal{I}(H_{\text{apt}})$ and linearly increase
the ranks of the top-half salient ones. More specifically, when increasing the
tuning parameter from $\Delta_{t}$ to $\Delta_{t^{\prime}}$, the salient
layer’s rank is changed from $r_{\text{apt}}$ to
$r_{\text{apt}}^{\prime}=\lfloor
r_{\text{apt}}\cdot\frac{\Delta_{t^{\prime}}}{\Delta_{t}}\rfloor$ where
$\lfloor\cdot\rfloor$ denotes the floor operation. For training stability,
when adding parameters and converting $W_{B}\in\mathbb{R}^{d_{o}\times
r_{\text{apt}}},W_{A}\in\mathbb{R}^{r_{\text{apt}}\times d_{i}}$ to
$W_{B}^{\prime}\in\mathbb{R}^{d_{o}\times
r_{\text{apt}}^{\prime}},W_{A}^{\prime}\in\mathbb{R}^{r_{\text{apt}}^{\prime}\times
d_{i}}$, we concatenate random Gaussian initialized parameters
$\mathcal{N}(0,\sigma^{2})$ in $W_{A}$ and zeros in $W_{B}$ same as the LoRA
initialization, so the layer’s output remains unchanged before and after new
parameters added.

###  4.4 Efficient Self-Knowledge Distillation

As shown in Table 4, training pruned LM without knowledge distillation causes
significant end-task performance drops. Therefore, we use knowledge
distillation in APT to effectively mitigate this performance loss of pruned
LM. Still, existing strategies require a fully trained teacher model being put
into the GPU together with the student during distillation, causing high
training time and memory. To avoid extra training costs, we keep duplicating
the tuning student layers as teachers during fine-tuning to reduce total
training time. Meanwhile, frozen parameters are shared between the student and
teacher model during training to reduce memory consumption. We edit the
distillation objective in CoFi (Xia et al., 2022) as

$\begin{split}\mathcal{L}&=\mu\mathcal{L}_{distill}+(1-\mu)\mathcal{L}_{ft}\\\
\mathcal{L}_{layer}&=\sum_{i=1}^{\mathcal{T}}\textrm{MSE}(\text{Tr}(H_{s}^{\phi(i)}),H_{t}^{i})\end{split}$
(8)

where $\mu$ is a moving term linearly scales from 0 to 1 during distillation
to encourage the pre-pruned model vastly fit to the training data,
$\mathcal{L}_{distill}$ is the distillation objective from CoFi, $\mathcal{T}$
is block-wise randomly sampled teacher layers following (Haidar et al., 2022),
$\phi(\cdot)$ is the teacher-student layer-mapping function that matches the
teacher layer to its closest, non-pruned student layer. Tr denotes the tunable
LoRA layer for layer transformation, initialized as an identical matrix
$\mathcal{I}$.

##  5 Experiments

To evaluate the training and inference efficiency gains of APT, we compare it
with the combined use of PEFT with pruning and distillation baselines. We
first describe the natural language understanding and generation tasks
targeting different LM backbones, then the setup of baselines and APT. We then
report task performance, speed, and memory usage for training and inference
costs.

###  5.1 Tasks

We apply APT to BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019),
T5(Raffel et al., 2020)444For fair comparisons, we use the t5-lm-adapt model,
which is only pre-trained on the C4 corpus to make sure the initial LM does
not observe downstream tasks in pre-training., and LLaMA (Touvron et al.,
2023). For BERT, RoBERTa, and T5 models, we train and evaluate on SST2 and
MNLI datasets from the GLUE benchmark (Wang et al., 2019) and report the dev
set accuracy. We also train and evaluate $\text{RoBERTa}_{\text{base}}$ on
SQuAD v2.0 (Rajpurkar et al., 2018) and report the dev set F1 score. For T5
models, we also fine-tune them on CNN/DM (Nallapati et al., 2016) and report
the ROUGE 1/2/L scores. Meanwhile, We use the GPT-4 generated Alpaca dataset
(Taori et al., 2023) to fine-tune large LLaMA models and evaluate them with
the lm-eval-harness package (Gao et al., 2021) on four tasks from the Open LLM
Leaderboard, namely 25-shot ARC (Clark et al., 2018), 10-shot HellaSwag
(Zellers et al., 2019), 5-shot MMLU (Hendrycks et al., 2021), and zero-shot
TruthfulQA (Lin et al., 2022).

###  5.2 Baselines

We validate the efficiency benefits of APT for both training and inference by
comparing with state-of-the-art PEFT, pruning, and distillation methods, along
with their combinations.

LoRA+Prune: a post-training pruning method over on LoRA-tuned LMs. We use Mask
Tuning (Kwon et al., 2022), a state-of-the-art post-training structured
pruning method based on fisher information. Due to that post-training pruning
performs poorly on high-sparsity settings, we retrain the pruned LM after
pruning to recover its performance.

Prune+Distill: knowledge distillation has been proved to be a key technique in
recovering pruned LMs’ task accuracy. In particular, we use the state-of-the-
art pruning plus distillation method called CoFi (Xia et al., 2022) which uses
$L_{0}$ regularization for pruning plus dynamic layer-wise distillation
objectives. We only compare APT to CoFi with RoBERTa models since the training
memory usage of CoFi is too high for larger LMs.

LoRA+Prune+Distill: to reduce the training memory consumption in pruning and
distillation, a simple baseline is to conduct CoFi pruning and distillation
but with LoRA parameters tuned only. More specifically, only the $L_{0}$
module and LoRA parameters are tunable under this setting.

LLMPruner (Ma et al., 2023): LLMPruner is the state-of-the-art task-agnostic
pruning method on LLaMA that prunes its blocks or channels based on salience
metrics while using LoRA for fast performance recovery. We compare APT to
LLMPruner with fine-tuning on the same GPT-4 generated Alpaca data for fair
comparisons.

We also compare APT to PST (Li et al., 2022) and LRP (Zhang et al., 2023a),
which are the state-of-the-art parameter-efficient unstructured and structured
pruning methods on BERT model. We leave these results in Appendix D.

###  5.3 Training Details

When pruning LMs with APT, following (Xia et al., 2022), we first prune and
train the LM with the self-distillation objective, and then fine-tune the
pruned LM to recover its end-task performance. Given $T$ pruning training
steps in total, we set a pre-determined target sparsity $\gamma_{T}$ (defined
as the ratio of pruned parameter size to the total parameter size) and use
cubic scheduling to control the LM parameter size, where
$\gamma_{t}=\gamma_{T}+(1-\gamma_{T})(1-\frac{t}{T})^{3}$. We adaptively
increase the tuning parameters in the pruning stage but restrict them to a
specific limit $\Delta_{t}$ at each training step $t$. Towards better training
stability in LM pruning, we gradually decrease the pruning masks of pruned
blocks by $\alpha<1$ instead of instantly setting them from ones to zeros. We
also use the exponential moving-averaged salience in (Zhang et al., 2023b)
when calculating the salience score during fine-tuning.

###  5.4 Evaluation Metrics

We evaluate APT and baselines on training and inference efficiency, measured
in runtime memory and time consumption as follows:

Training Efficiency Metrics: we report relative training peak memory (Train.
Mem.) and training speedup measured by time to accuracy (TTA555For instance,
97% TTA denotes the time spent reaching 97% of the fully fine-tuned model’s
performance) (Coleman et al., 2019). For fair comparisons, we consider the
training time of the teacher model plus the student for methods using
knowledge distillation.

Inference Efficiency Metrics: we report the inference peak memory (Inf. Mem.)
and the relative speedup (Inf. Speed) based on throughput (data processed per
second) for inference efficiency.

Both training and evaluation are conducted on a single A100 GPU. The inference
test batch size is 128 for small models while 32 and 4 for LLaMA 7B and 13B
models, respectively. We demonstrate detailed training and evaluation
setups/implementations in Appendix A.

###  5.5 Main Results

Model | Method | MNLI | SST2 | SQuAD v2 | CNN/DM | Train. Speed($\Uparrow$) | Train. Mem.($\Downarrow$) | Inf. Speed($\Uparrow$) | Inf. Mem.($\Downarrow$)
---|---|---|---|---|---|---|---|---|---
$\text{RoBERTa}_{\text{base}}$ | FT | 87.6 | 94.8 | 82.9 | - | 100.0% | 100.0% | 100.0% | 100.0%
LoRA | 87.5 | 95.1 | 83.0 | - | 4.7% | 60.5% | 100.0% | 100.0%
LoRA+Prune | 84.0 | 93.0 | 79.2 | - | 2.0% | 60.5% | 262.9% | 75.1%
Prune+Distill | 87.3 | 94.5 | - | - | 6.7% | 168.5% | 259.2% | 79.2%
LoRA+Prune+Distill | 84.2 | 91.9 | - | - | 1.5% | 141.4% | 253.7% | 82.3%
| APT | 86.4 | 94.5 | 81.8 | - | 16.9% | 70.1% | 241.9% | 78.1%
$\text{T5}_{\text{base}}$ | FT | 87.1 | 95.2 | - | 42.1/20.3/39.4 | 100.0% | 100.0% | 100.0% | 100.0%
LoRA | 87.0 | 95.0 | - | 38.7/17.2/36.0 | 39.1% | 62.0% | 100.0% | 100.0%
LoRA+Prune | 80.9 | 92.3 | - | 36.7/15.7/33.9 | 2.5% | 62.0% | 212.5% | 73.4%
APT | 87.0 | 95.0 | - | 38.6/17.0/35.8 | 20.6% | 73.9% | 134.1% | 81.5%

Table 2: RoBERTa and T5 pruning with APT compared to baselines under 60%
sparsity. We measure the training and inference efficiency with LMs pruned on
the SST2 task. Training speed is measured via 97% accuracy TTA. $\Uparrow$
denotes the metric is better when bigger, and $\Downarrow$ denotes smaller is
better.

Method | ARC | HellaSwag | MMLU | TruthfulQA | Avg. | Train. Speed($\Uparrow$) | Train. Mem.($\Downarrow$) | Inf. Speed($\Uparrow$) | Inf. Mem.($\Downarrow$)
---|---|---|---|---|---|---|---|---|---
LLaMA 2 7B | 53.1 | 77.7 | 43.8 | 39.0 | 53.4 | - | - | - | -
LoRA | 55.6 | 79.3 | 46.9 | 49.9 | 57.9 | 100.0% | 100.0% | 100.0% | 100.0%
LoRA+Prune | 46.8 | 65.2 | 23.9 | 46.2 | 45.5 | 55.3% | 100.0% | 115.5% | 68.9%
LLMPruner | 39.2 | 67.0 | 24.9 | 40.6 | 42.9 | 115.1% | 253.6% | 114.8% | 74.2%
APT | 45.4 | 71.1 | 36.9 | 46.6 | 50.0 | 94.3% | 75.8% | 117.0% | 67.2%

Table 3:  LLaMA 2 7B 30% sparsity pruning results with GPT4-generated Alpaca
dataset, evaluated on the Open LLM leaderboard few-shot tasks. Training speed
is measured via training time per step. We do not compare to distillation
baselines because the training cost of distillation is too large, and we also
compare APT to LLMPruner since it is dedicated to large LM pruning. $\Uparrow$
denotes the metric is better when bigger, and $\Downarrow$ denotes smaller is
better.

Overview We demonstrate the overall end-task performance of APT comparing to
fine-tuning (FT), LoRA-tuning (LoRA), and pruning baselines in Table 2 and
Table 3. Overall, APT does not hurt pruned LM’s accuracy compared to fine-
tuning and LoRA tuning, where 99% and 98% fine-tuned LM task accuracies are
maintained when pruning RoBERTa and T5 models leaving 40% parameters,
respectively. APT also only costs about 70% training memory than fine-tuning
for both RoBERTa and T5 model training. When pruning LLaMA2-7B models with 70%
parameters remained, APT can recover 86.4% task performance on average,
together with only 75.8% training memory consumption than LoRA tuning.
Furthermore, APT also significantly reduces training time and memory cost
compared to the pruning and distillation baselines, also with better end-task
performance with the same pruning sparsities. The detailed comparisons are
described as follows:

APT speeds up RoBERTa and T5 training 8$\times$ and reduces training memory
costs to 30% in LLaMA pruning compared to LoRA+Prune baseline. Shown in Table
2, when pruning with the 60% sparsity target, APT converges $8.4\times$ faster
in RoBERTa model pruning than the LoRA+Prune baseline with consuming similar
GPU memory costs. Moreover, when pruning RoBERTa models, APT converges
$3.6\times$ faster than the LoRA baseline without pruning. APT also prunes T5
models 8.2$\times$ faster than the LoRA+Prune baseline. The reason is that APT
adaptively prunes task-irrelevant parameters during training, reducing memory
and per-step training time. Adding parameters in salient tuning layers also
accelerates LM convergence. When pruning 30% parameters in billion-level LLMs
before tuning, APT significantly reduces memory usage. APT costs less than
24GB of memory for pruning and fine-tuning, which can be easily adapted to the
consumer-level GPUs. In contrast, LLM-Pruner costs about 80GB memory when
pruning the LLaMA 7B model666https://github.com/horseee/LLM-Pruner/issues/4.

APT achieves 2.5%-9.9% higher task performance than the LoRA+Prune baseline
with the same pruning sparsities. Presented in Table 2 and Table 3 , when
RoBERTa, T5, and LLaMA models, regardless of size, APT consistently reach
higher task performance than the LoRA+Prune baseline under the same 60%
sparsity. With the similar inference efficiency when pruning RoBERTa models
(2.4$\times$ speedup and 78.1% memory cost with APT, and 2.6$\times$ speedup
and 75.1% memory cost for LoRA+Prune for LoRA+Prune), APT reaches 2.5% more
end-task performance on average. When pruning T5 models under the 60%
sparsity, the task performance achieved by APT is 5.1% better than the
LoRA+Prune baseline, but the inference efficiency reached by APT (1.3$\times$
speedup and 81.5% memory cost) is worse than the LoRA+Prune baseline
(2.1$\times$ speedup and 73.4% memory cost). This is because APT can
adaptively prune more decoder parameters, which are also computationally
cheaper than encoder parameters (due to shorter output sequence length) but
relatively useless for classification tasks. For LLaMA2-7B model pruning with
70% sparsity, APT outperforms LLMPruner with 16.5% and the LoRA+Prune baseline
with 9.9%, where the inference efficiency improvements of APT (1.2$\times$
speedup and 67.2% memory cost) is slightly better than both LoRA+Prune and
LLMPruner baselines.

APT reaches on-par performance with the Prune+Distill baseline given the same
pruning sparsity but trained 2.5$\times$ faster and costs only 41.6% memory.
Compared to the Prune+Distill baseline, which achieves the best task
performance among all baseline methods, APT results in comparable task
accuracy, with only a 0.9 point drop in the MNLI dataset. At the same time,
with similar inference efficiency achieved, APT costs only 41.6% training
memory and converges 2.5$\times$ than the Prune+Distill baseline. This is
because of the self-distillation technique leveraged in APT where no separated
teacher model is required in pruned LM training. Moreover, APT achieves better
task performance than the LoRA+Prune+Distill baseline as well, with also
faster training speed and less training memory consumption. The result
demonstrates that simply combining LoRA with pruning and distillation hurts
pruned LM’s task accuracy and also does not substantially reduce training
memory consumption.

###  5.6 Ablation Study

We evaluate the impact of different components in APT by removing the adaptive
pruning ($\mathcal{A}_{\text{P}}$), adaptive tuning
($\mathcal{A}_{\text{T}}$), and self-distillation ($\mathcal{D}_{\text{S}}$).
Besides end-task performance, we also report the training efficiency metrics
for each ablation. We do not report the inference efficiency since they are
similar under the same sparsity.

Adaptive pruning ($\mathcal{A}_{\text{P}}$) We demonstrate the ablation of
adaptive pruning (w/o $\mathcal{A}_{\text{P}}$) for RoBERTa models in Table 4
and LLaMA models in Table 5. In these cases, we only train LMs with adaptive
tuning strategies as above with supervised fine-tuning objectives without
distillation. In such settings, APT can be recognized as a PEFT method with
dynamic tuning parameters adaptively changed in LM fine-tuning. Hence, the
inference efficiency of the trained LMs are the same as full fine-tuning and
LoRA. Without pruning, the task performance of RoBERTa reaches 94.4 for SST2
and 87.5 for MNLI (99.8% fine-tuned LM performance on average). The average
performance of the LLaMA model also achieves 96.6% to its LoRA-tuned
counterpart. In addition, we surprisingly find that the RoBERTA training speed
with APT w/o $\mathcal{A}_{\text{P}}$ is even 21% faster than full fine-tuning
while costing only 62.2% memory. This result indicates that adaptive tuning is
effective in accurately and vastly fine-tuning LMs with minimal training
memory cost. In the meantime, training memory cost will be higher than LoRA
without $\mathcal{A}_{\text{P}}$ will be higher in LLaMA pruning since the
total LM parameter number is fixed, yet the tuning parameter number will grow
larger than static LoRA-tuning. This ablation demonstrates that adaptive
pruning is essential in reducing the training memory consumption of LLaMA
model fine-tuning, besides benefiting model inference efficiency.

Adaptive tuning ($\mathcal{A}_{\text{T}}$) In Table 4, we show results of
ablating adaptive tuning (w/o $\mathcal{A}_{\text{T}}$) where the tuning
parameters are static when pruning RoBERTa models. Without
$\mathcal{A}_{\text{T}}$, the model’s performance decreases to 93.2/84.4,
leading to a similar performance as the LoRA+Prune baseline (93.0/84.0).
Moreover, equally increasing parameters across all layers instead of adding
parameters based on salience would notably hurt the task accuracy (84.4 on
MNLI compared to 86.4). At the same time, $\mathcal{A}_{\text{T}}$ helps the
model converge 16% faster than static LoRA training. When ablating
$\mathcal{A}_{\text{T}}$ in LLaMA model pruning, shown in Table 5, we observe
that $\mathcal{A}_{\text{T}}$ recovers the model performance under 50% pruning
setting (38.2 compared to 35.8) but the difference under 70% pruning is
insignificant. Meanwhile, if calculating the pruning parameter salience
without using kurtosis to consider outliers parameters, the pruned LM
performance substantially drops from 50.0 to 38.1. We conclude that
$\mathcal{A}_{\text{T}}$ substantially improves LM training speed and end-task
performance. For large LLaMA-based LM pruning, and outlier parameters are
essential to recovering the pruned large LLaMA-based models’ capabilities.

Method | SST2 | MNLI | Train. Speed($\Uparrow$) | Train. Mem.($\Downarrow$)
---|---|---|---|---
APT | 94.5 | 86.4 | 16.9% | 70.1%
w/o $\mathcal{A}_{\text{P}}$ | 94.4 | 87.5 | 121.0% | 62.2%
w/o salience | 94.3 | 84.7 | 16.4% | 65.0%
w/o $\mathcal{A}_{\text{T}}$ | 93.2 | 84.5 | 14.6% | 64.4%
w/o $\mathcal{D}_{\text{S}}$ | 92.9 | 85.3 | 20.7% | 61.9%

Table 4: Results of ablating salience-based allocation strategy and APT
adapter with RoBERTa-base model, with relative training efficiency metrics to
fine-tuning.

| Sparsity | T.M. | ARC | HellaSwag | MMLU | TruthfulQA | Avg.
---|---|---|---|---|---|---|---
APT | 30% | 75.8% | 45.4 | 71.1 | 36.9 | 46.6 | 50.0
w/o $\mathcal{A}_{\text{P}}$ | 100% | 102.4% | 53.8 | 79.1 | 46.9 | 48.4 | 57.1
w/o kurtosis | 30% | 75.9% | 47.2 | 39.7 | 23.0 | 42.3 | 38.1
w/o $\mathcal{A}_{\text{T}}$ | 30% | 76.1% | 44.2 | 70.1 | 40.8 | 45.1 | 50.0
APT | 50% | 60.2% | 29.8 | 48.9 | 26.7 | 47.6 | 38.2
w/o $\mathcal{A}_{\text{T}}$ | 50% | 60.1% | 27.9 | 46.2 | 24.5 | 44.7 | 35.8

Table 5: LLaMA 2 7B model ablation results under 30% and 50% sparsity
settings. T.M. denotes relative training memory compare to LoRA-tuning.

Self-distillation ($\mathcal{D}_{\text{S}}$) Shown in Table 4, tuning APT
adapters dynamically without distillation objectives gets 1.35 worse task
accuracy on average. However, pruning RoBERTa models without self-distillation
is 22.5% faster and costs 11.7% less training memory. This result indicates
the effectiveness of leveraging knowledge distillation to recover pruned LM
performance, but conducting distillation will result in extra training costs
regarding both time and memory. Detailed comparisons of self-distillation and
traditional, static distillation strategies are shown in Appendix G

###  5.7 Adaptive Pruning and Tuning Analysis

Figure 3: Comparison of different initial ranks of LoRA layers pruning with
APT on RoBERTa with SST2 task accuracy, relative training peak memory and
speed to 97% fine-tuning accuracy to the fine-tuning model.  Figure 4:
Training initial sparsity trade-off with 30% target sparsity model’s relative
performances to the LoRA-tuned LLaMA2-7B and 13B models.

Effects of adaptive tuning strategies on end-task performance and training
efficiency. As the trajectories shown in Fig. 3, simply enlarging the initial
tuning parameter number in APT will not improve or even hurt the model’s final
performance. Moreover, the training memory consumption grows even higher than
fine-tuning when the tuning layer ranks become extremely large (initial ranks
set as 256). Therefore, this result proves that adding tuning parameters
according to layer salience is better than uniformly increasing them before
tuning.

Effects of early pruning on task accuracy and training memory in LLaMA
pruning. Fig. 4 shows the effect of the initial density on LLaMA models’ task
performance under the 30% sparsity pruning setting. We find that densely-
trained models only perform better in TruthfulQA with fewer parameters pruned
before tuning. The accuracy reaches 48.6 and 47.4 when not pruning before
tuning, compared to 46.6 and 44.7 when directly pruning to the target sparsity
for both 7B and 13B models. Training the LM densely harms the model
performance while costing extra memory for all other tasks. These results
demonstrate that pruning during training hurts large LM performance under
distillation-free settings, and we hypothesize this is due to the training
instability issue when parameters are set to zeros during fine-tuning.

##  6 Conclusion

We propose APT to adaptively identify LMs’ pruning and tuning parameters
during fine-tuning, aiming for both training and inference efficiency. APT
prunes small LMs faster while pruning large LMs with less memory consumption.
With using similar memory costs as LoRA, APT prunes small LMs 8 $\times$
faster than the LoRA plus pruning baseline. In large LM pruning, APT maintains
87% performance with only 30% pruning memory usage when 70% LM parameter
retained. We conclude that APT shapes a new paradigm of pruning LMs in fine-
tuning for people with limited computational resources, opening the
possibilities for the wider usage of LMs in practical applications with
restrictive hardware constraints. In future works, we would adapt APT to more
PEFT architectures (for example, parallel adapters and Prefix-tuning) while
targeting better performance recovery for billion-level large LMs. Meanwhile,
we also hope future research will continue to find efficient and accurate
techniques to identify salient structures in LMs based on our formulated
setting.

##  7 Limitation and Discussion

Towards better performance gain and inference speedup of large LM in limited
resource settings. Comparing Table 2 Table 3, we can notice that the
performance gap between billion-level LLMs is larger than smaller LMs.
Meanwhile, instead of time, memory reduction is a more urgent need in
practical LM-based usage to make more LLMs possible to be utilized. Therefore,
when pruning LLMs, APT focuses on the distillation-free setting. At the same
time, we only propose a simple but effective strategy. Yet, we believe that
superior performance-efficiency trade-offs can be reached with better memory-
efficient distillation strategies, parameter sharing, and re-allocation.
Furthermore, because of the unique features of Ampere-architecture GPUs, layer
dimensions divisible by 8 for FP16 and divisible by 16 for Int8 would reach
better speedup while APT only focuses on the conventional structured pruning
paradigm. Towards better speedup, a higher level of structured pruning
(grouped neuron and dimension) in LLMs shall also be considered.

Training could be unstable because of parameter shape changes. Since we
conduct parameter re-allocation during training, there are new parameters to
be initialized while also existing parameters to be pruned. To avoid stability
issues, we reset the optimizer every time after each parameter allocation, yet
this strategy might also cause the training unstable. Meanwhile, the selection
of teacher checkpoints during training also highly affects the pruned model’s
performance, whereas either non-converged or sparse teachers don’t help as
much in performance recovery. We chose a simple strategy where only
checkpointing the weights if the current student model’s performance grows
better, yet we also found that some experiment results show that an even more-
tuned but less performant teacher model would help the distillation better.

Could non-linear adapters perform better for performance recovery? To avoid
inference time and memory overhead, we adapt APT adapter to LoRA specifically.
However, low-rank decomposition does not add more complexity to a LM, whereas
the model’s overall representation capacity doesn’t increase. The adaptation
with a wider range of adapters, such as Prefix-tuning, HAdapters, and
Parallel-adapters, could be better explored.

## References

  * Ben Zaken et al. (2022) Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022.  BitFit: Simple parameter-efficient fine-tuning for transformer-based masked language-models.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 1–9, Dublin, Ireland. Association for Computational Linguistics.
  * Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018.  Think you have solved question answering? try arc, the ai2 reasoning challenge.  _arXiv preprint arXiv:1803.05457_.
  * Coleman et al. (2019) Cody Coleman, Daniel Kang, Deepak Narayanan, Luigi Nardi, Tian Zhao, Jian Zhang, Peter Bailis, Kunle Olukotun, Chris Ré, and Matei Zaharia. 2019.  Analysis of dawnbench, a time-to-accuracy machine learning performance benchmark.  _SIGOPS Oper. Syst. Rev._ , 53(1):14–25.
  * Dettmers et al. (2022) Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022.  Llm.int8(): 8-bit matrix multiplication for transformers at scale.  _ArXiv_ , abs/2208.07339.
  * Dettmers et al. (2023) Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023.  Qlora: Efficient finetuning of quantized llms.  _arXiv preprint arXiv:2305.14314_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ding et al. (2023) Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. 2023.  Parameter-efficient fine-tuning of large-scale pre-trained language models.  _Nature Machine Intelligence_ , 5(3):220–235.
  * Frankle and Carbin (2018) Jonathan Frankle and Michael Carbin. 2018.  The lottery ticket hypothesis: Finding sparse, trainable neural networks.  In _International Conference on Learning Representations_.
  * Frankle et al. (2021) Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. 2021\.  Pruning neural networks at initialization: Why are we missing the mark? In _International Conference on Learning Representations_.
  * Frantar and Alistarh (2023) Elias Frantar and Dan Alistarh. 2023.  Sparsegpt: Massive language models can be accurately pruned in one-shot.  _ArXiv_ , abs/2301.00774.
  * Frantar et al. (2022) Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022.  Gptq: Accurate post-training quantization for generative pre-trained transformers.  _ArXiv_ , abs/2210.17323.
  * Gao et al. (2021) Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2021.  A framework for few-shot language model evaluation.
  * Guo et al. (2021) Demi Guo, Alexander Rush, and Yoon Kim. 2021.  Parameter-efficient transfer learning with diff pruning.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4884–4896, Online. Association for Computational Linguistics.
  * Haidar et al. (2022) Md Akmal Haidar, Nithin Anchuri, Mehdi Rezagholizadeh, Abbas Ghaddar, Philippe Langlais, and Pascal Poupart. 2022.  Rail-kd: Random intermediate layer mapping for knowledge distillation.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 1389–1400.
  * Han et al. (2015a) Song Han, Huizi Mao, and William J. Dally. 2015a.  Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding.  _arXiv: Computer Vision and Pattern Recognition_.
  * Han et al. (2015b) Song Han, Jeff Pool, John Tran, and William Dally. 2015b.  Learning both weights and connections for efficient neural network.  _Advances in neural information processing systems_ , 28.
  * He et al. (2022) Shwai He, Liang Ding, Daize Dong, Jeremy Zhang, and Dacheng Tao. 2022.  SparseAdapter: An easy approach for improving the parameter-efficiency of adapters.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 2184–2190, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Hedegaard et al. (2022) Lukas Hedegaard, Aman Alok, Juby Jose, and Alexandros Iosifidis. 2022.  Structured Pruning Adapters.  ArXiv:2211.10155 [cs].
  * Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021.  Measuring massive multitask language understanding.  In _International Conference on Learning Representations_.
  * Hinton et al. (2015) Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015.  Distilling the knowledge in a neural network.  _ArXiv_ , abs/1503.02531.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.  Parameter-efficient transfer learning for nlp.  In _International Conference on Machine Learning_ , pages 2790–2799. PMLR.
  * Hu et al. (2021) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021.  LoRA: Low-Rank Adaptation of Large Language Models.  ArXiv:2106.09685 [cs].
  * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020.  Scaling laws for neural language models.  _arXiv preprint arXiv:2001.08361_.
  * Karimi Mahabadi et al. (2021) Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. 2021.  Compacter: Efficient low-rank hypercomplex adapter layers.  _Advances in Neural Information Processing Systems_ , 34:1022–1035.
  * Kwon et al. (2022) Woosuk Kwon, Sehoon Kim, Michael W Mahoney, Joseph Hassoun, Kurt Keutzer, and Amir Gholami. 2022.  A fast post-training pruning framework for transformers.  In _Advances in Neural Information Processing Systems_ , volume 35, pages 24101–24116. Curran Associates, Inc.
  * Lagunas et al. (2021) François Lagunas, Ella Charlaix, Victor Sanh, and Alexander Rush. 2021.  Block Pruning For Faster Transformers.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 10619–10629, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * LeCun et al. (1989) Yann LeCun, John S. Denker, and Sara A. Solla. 1989.  Optimal brain damage.  In _NIPS_.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.  The power of scale for parameter-efficient prompt tuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3045–3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021.  Prefix-tuning: Optimizing continuous prompts for generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4582–4597.
  * Li et al. (2022) Yuchao Li, Fuli Luo, Chuanqi Tan, Mengdi Wang, Songfang Huang, Shen Li, and Junjie Bai. 2022.  Parameter-efficient sparsity for large language models fine-tuning.  In _Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22_ , pages 4223–4229. International Joint Conferences on Artificial Intelligence Organization.  Main Track.
  * Lialin et al. (2023) Vladislav Lialin, Vijeta Deshpande, and Anna Rumshisky. 2023.  Scaling down to scale up: A guide to parameter-efficient fine-tuning.  _arXiv preprint arXiv:2303.15647_.
  * Lin et al. (2023) Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. 2023\.  Awq: Activation-aware weight quantization for llm compression and acceleration.  _arXiv preprint arXiv:2306.00978_.
  * Lin et al. (2022) Stephanie Lin, Jacob Hilton, and Owain Evans. 2022.  TruthfulQA: Measuring how models mimic human falsehoods.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Ma et al. (2023) Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023.  Llm-pruner: On the structural pruning of large language models.  _arXiv preprint arXiv:2305.11627_.
  * Mishra et al. (2022) Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022.  Cross-task generalization via natural language crowdsourcing instructions.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3470–3487, Dublin, Ireland. Association for Computational Linguistics.
  * Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Cícero Nogueira dos Santos, Çaglar Gülçehre, and Bing Xiang. 2016.  Abstractive text summarization using sequence-to-sequence rnns and beyond.  In _Conference on Computational Natural Language Learning_.
  * Panigrahi et al. (2023) Abhishek Panigrahi, Nikunj Saunshi, Haoyu Zhao, and Sanjeev Arora. 2023.  Task-specific skill localization in fine-tuned language models.  _arXiv preprint arXiv:2302.06600_.
  * Pfeiffer et al. (2020) Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2020.  Adapterfusion: Non-destructive task composition for transfer learning.  _ArXiv_ , abs/2005.00247.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21:1–67.
  * Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018.  Know what you don’t know: Unanswerable questions for SQuAD.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 784–789, Melbourne, Australia. Association for Computational Linguistics.
  * Sanh et al. (2020) Victor Sanh, Thomas Wolf, and Alexander Rush. 2020.  Movement Pruning: Adaptive Sparsity by Fine-Tuning.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 20378–20389. Curran Associates, Inc.
  * Shen et al. (2022a) Maying Shen, Pavlo Molchanov, Hongxu Yin, and Jose M. Alvarez. 2022a.  When to prune? a policy towards early structural pruning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 12247–12256.
  * Shen et al. (2022b) Maying Shen, Hongxu Yin, Pavlo Molchanov, Lei Mao, Jianna Liu, and Jose M. Alvarez. 2022b.  Structural pruning via latency-saliency knapsack.  In _Advances in Neural Information Processing Systems_ , volume 35, pages 12894–12908. Curran Associates, Inc.
  * Sun et al. (2023) Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. 2023.  A simple and effective pruning approach for large language models.  _arXiv preprint arXiv:2306.11695_.
  * Sung et al. (2021) Yi-Lin Sung, Varun Nair, and Colin A Raffel. 2021.  Training neural networks with fixed sparse masks.  _Advances in Neural Information Processing Systems_ , 34:24193–24205.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Wang et al. (2019) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _International Conference on Learning Representations_.
  * Wang et al. (2022a) Xiaozhi Wang, Kaiyue Wen, Zhengyan Zhang, Lei Hou, Zhiyuan Liu, and Juanzi Li. 2022a.  Finding skill neurons in pre-trained transformer-based language models.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 11132–11152, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Wang et al. (2022b) Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. 2022b.  Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 5085–5109, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Xia et al. (2022) Mengzhou Xia, Zexuan Zhong, and Danqi Chen. 2022.  Structured pruning learns compact and accurate models.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1513–1528, Dublin, Ireland. Association for Computational Linguistics.
  * Xu et al. (2021) Dongkuan Xu, Ian En-Hsu Yen, Jinxi Zhao, and Zhibin Xiao. 2021.  Rethinking network pruning–under the pre-train and fine-tune paradigm.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2376–2382.
  * Xu et al. (2023) Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhensu Chen, Xiaopeng Zhang, and Qi Tian. 2023.  Qa-lora: Quantization-aware low-rank adaptation of large language models.  _arXiv preprint arXiv:2309.14717_.
  * Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019.  HellaSwag: Can a machine really finish your sentence? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4791–4800, Florence, Italy. Association for Computational Linguistics.
  * Zhang et al. (2023a) Mingyang Zhang, Chunhua Shen, Zhen Yang, Linlin Ou, Xinyi Yu, Bohan Zhuang, et al. 2023a.  Pruning meets low-rank parameter-efficient fine-tuning.  _arXiv preprint arXiv:2305.18403_.
  * Zhang et al. (2023b) Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023b.  Adaptive budget allocation for parameter-efficient fine-tuning.  In _The Eleventh International Conference on Learning Representations_.
  * Zhang et al. (2023c) Zhengyan Zhang, Zhiyuan Zeng, Yankai Lin, Chaojun Xiao, Xiaozhi Wang, Xu Han, Zhiyuan Liu, Ruobing Xie, Maosong Sun, and Jie Zhou. 2023c.  Emergent modularity in pre-trained transformers.  _arXiv preprint arXiv:2305.18390_.
  * Zhao et al. (2023) Weilin Zhao, Yuxiang Huang, Xu Han, Zhiyuan Liu, Zhengyan Zhang, and Maosong Sun. 2023.  Cpet: Effective parameter-efficient tuning for compressed large language models.  _arXiv preprint arXiv:2307.07705_.

##  Appendix A Hyperparameter and Training Setups

Our hyper-parameter settings are shown in Table 6. For GLUE task fine-tuning,
we follow the hyper-parameter setting of CoFi (Xia et al., 2022), separating
the tasks into big (MNLI, SST2, QNLI, QQP) and small (MRPC, CoLA, RTE, STSB)
based on the dataset size. For instruction tuning on the Alpaca dataset, we
train the pruned model for 15 epochs after the pre-tuning pruning process to
make sure they converge. However, in practice, such training epochs can be
reduced. To adaptively increase the tuning parameters in the LM, at the start
of fine-tuning, we initialize adapter ranks to 8, with salient layers’ ranks
linearly increased. The scaling factors are set as 2 statically. Since
evaluating billion-level LLaMA models during instruction tuning with all
evaluation tasks would be time-consuming, we did not do the TTA evaluation as
small models. We do not conduct any hyper-parameters search for any training
for fair comparison.

Hypeparameter | GLUE-small | GLUE-big | SQuAD | CNN/DM | Alpaca
---|---|---|---|---|---
Learning rate | 2e-4 | 2e-4 | 2e-4 | 1e-4 | 1e-4
Batch size | 32 | 32 | 32 | 16 | 32
Epochs | 40 | 40 | 40 | 16 | 15
Distill epochs | 20 | 20 | 20 | 6 | -

Table 6: Hyperparameters used in APT experiments

##  Appendix B Block salience calculation and correlations

As addressed in Section 4.1, we use the compressed weight-gradient production
as the salience metric for identifying the tuning and pruning parameter blocks
in LMs. Previous works (Sanh et al., 2020) use salience score defined as the
magnitude of the parameters’ weight-gradient production, where given a linear
layer $H=WX$ (we omit the bias term here for simplicity) in model parameters
$\Theta$ trained on the objective $\mathcal{L}$, the salience scoring function
$S$ is defined as:

$\begin{split}S(W_{i,j})&=\sum_{(x,y)\in\mathcal{D}}s(W_{i,j},x,y)\\\
&=\sum_{(x,y)\in\mathcal{D}}|\frac{\partial\mathcal{L}(x,y|\Theta)}{\partial
W_{i,j}}\cdot W_{i,j}|\\\
S(W_{:,j})&=\sum_{(x,y)\in\mathcal{D}}\sum_{i}|\frac{\partial\mathcal{L}(x,y|\Theta)}{\partial
W_{i,j}}\cdot W_{i,j}|\\\
&=\sum_{(x,y)\in\mathcal{D}}(\sum_{i}|\frac{\partial\mathcal{L}(x,y|\Theta)}{\partial
X_{j,i}}\cdot X_{j,i}|)\end{split}$ (9)

where $x,y$ are the inputs and labels sampled from the training batch
$\mathcal{D}$. $S(W_{i,j})$ denotes the unstructured, sparse parameter’s
salience, and $S(W_{:,j})$ denotes the salience score of a block in the weight
$W$ (for example, rows, columns, attention heads, etc.).

When applying this equation to APT adapter layers as defined in Eq. 2, there
are three different consistent dimensions, namely input dimension $j$, output
dimension $i$, and tuning rank dimension $k$. Therefore, the combined salience
(including tuning low-rank weights and the frozen weight) of the parameter
block shall be calculated as follows:

$\begin{split}S(H,i)&=\sum_{l}\frac{\partial\mathcal{L}(x,y|\Theta)}{\partial
H(X)_{i,l}}\cdot H(X)_{i,l}\\\
&=\sum_{p}\frac{\partial\mathcal{L}(x,y|\Theta)}{\partial W_{i,p}}\cdot
W_{i,p}\\\
&+s\cdot\sum_{q}\frac{\partial\mathcal{L}(x,y|\Theta)}{\partial{W_{B}}_{i,q}}\cdot{W_{B}}_{i,q}\\\
S(H,j)&=\sum_{l}\frac{\partial\mathcal{L}(x,y|\Theta)}{\partial X_{j,l}}\cdot
X_{j,l}\\\ &=\sum_{p}\frac{\partial\mathcal{L}(x,y|\Theta)}{\partial
W_{p,j}}\cdot W_{p,j}\\\
&+s\cdot\sum_{q}\frac{\partial\mathcal{L}(x,y|\Theta)}{\partial{W_{A}}_{q,j}}\cdot{W_{A}}_{q,j}\\\
S(H,k)&=s\cdot\sum_{l}\frac{\partial\mathcal{L}(x,y|\Theta)}{\partial{W_{A}}_{k,l}}\cdot{W_{A}}_{k,l}\\\
&=s\cdot\sum_{l}\frac{\partial\mathcal{L}(x,y|\Theta)}{\partial{W_{B}}_{l,k}}\cdot{W_{B}}_{l,k}\\\
\end{split}$ (10)

Therefore, we can notice that the real block-wise salience of the LoRA layer
shall be the sum of the block-wise frozen weight salience and the
corresponding tuning weight. Hence, the existing work (Zhang et al., 2023a)
that only uses the tuning block salience as layer salience leads to sub-
optimal pruning results. Meanwhile, we shall also notice the correlation
between the input-, output-dimension, and tuning rank dimensions, which are
the summation of the weight-gradient production of parameters on different
dimensions.

##  Appendix C Adaptive Pruning and Tuning Details

Algorithm 1 Adaptive Pruning and Tuning

1: Input: Model $f$; Training dataset $\mathcal{D}$; total training steps $T$;
Adjustment step set $\mathcal{T}$; Training target $\mathcal{L}$; Initial
parameters and masks $\Theta_{0},M_{0}$, training memory budget $\Delta$;
Parameter number constraint $\gamma$; Hyperparameters $\alpha\,\beta$.

2: for $t=1,\dots,T$ do

3: Forward pass: $L\leftarrow\mathcal{L}(f(\Theta_{t},D_{t}))$

4: Cache the batch-sequence summed hidden states:
$\widetilde{H}\leftarrow\sum_{i,j}(|H|)_{ij}$

5: Backward pass:
$\nabla_{\Theta_{t}}L\leftarrow\frac{\partial\mathcal{L}(f(\Theta_{t},D_{t}))}{\partial\Theta_{t}}$

6: Calculate approximated salience:
$\widetilde{S}(m_{i})\leftarrow\widetilde{H}\cdot\sum_{i,j}(|\nabla_{H}L|)_{ij}$

7: Update global scores:
$\overline{S}^{(t)}(m)\leftarrow\beta\overline{S}^{(t-1)}(m)+(1-\beta)\widetilde{S}(m)$;

8: Select blocks: $M_{1},M_{0}\leftarrow$ Binary search against constraint Eq.
7, with scores $\overline{S}^{(t)}(m)$;

9: Update masks: $M^{(t)}_{1}\leftarrow min(1,M^{(t-1)}_{1}+\alpha)$,
$M^{(t)}_{0}\leftarrow max(0,M^{(t-1)}_{0}-\alpha)$;

10: Update parameters:
$\Theta_{t+1}\leftarrow\Theta_{t}-\alpha\nabla_{\Theta_{t}}L$

11: end for

12: Output: Parameters and masks $\Theta^{(T)},M^{(T)}$.

We show the detailed algorithm description of our Lightweight Parameter
Adjustment as described in Section 4.1 in Algorithm 1. For the details of the
algorithm, we first sort all blocks by the salience density, defined as the
block salience divided by the number of parameters in the block. For instance,
given a RoBERTa-base model with the hidden dimension $d_{m}=768$, the number
of transformer layers $n_{L}=12$, the number of attention heads $n_{h}=12$,
and the number of FFN neurons $n_{f}=3072$, we have:

$\displaystyle\mathcal{C}_{\text{head}}$ $\displaystyle=4\times d_{m}\times
d_{m}/n_{h}=196608$ (11) $\displaystyle\mathcal{C}_{\text{neuron}}$
$\displaystyle=2\times d_{m}=1536$ (12)
$\displaystyle\mathcal{C}_{\text{dimension}}$
$\displaystyle=n_{L}\times(4d_{m}+2n_{f})=110592$ (13)

We also omit the bias term for density calculation since it takes up less than
1% of LM’s parameters. Since the number of heads, neurons, and hidden
dimensions is ever-changing during pruning, we re-calculate the density after
executing each parameter size change. Meanwhile, for T5 and LLaMA-like models,
the FFN layers are gated, consisting of up-, gate-, and down-projection linear
layers. Therefore, the number of layers in FFN shall be three instead of two
in these LMs. Furthermore, for encoder-decoder LMs like T5, the cross-
attention layers in the decoder shall also be counted.

After sorting the blocks by salience density, as LM’s parameter size
monotonically increases with the number of MHA heads, FFN neurons, and hidden
dimensions, we conduct a binary search algorithm to identify the blocks shall
be retained as LM’s parameter size monotonically increases with the number of
MHA heads, FFN neurons, and hidden dimensions. Specifically, given a sorted
list of $N$ blocks $B=\\{b_{1},b_{2},...,b_{N}\\}$ and function $f$ for
identifying the block’s category where

$f(b_{i})=\begin{cases}0&\text{if }b_{i}\text{ is a head}\\\ 1&\text{if
}b_{i}\text{ is a neuron}\\\ 2&\text{if }b_{i}\text{ is a dimension}\\\
\end{cases}$ (14)

given any index $i$, we can calculate the parameter number of the LM
consisting of the top-$i$ blocks by:

$\begin{split}\mathcal{C}_{\text{top-}i}&=(4d_{h}^{\prime}\cdot
n_{h}^{\prime}+2n_{f}^{\prime})\cdot d_{m}^{\prime}\\\
n_{h}^{\prime}&=\sum_{j=0}^{i-1}\delta(0,f(b_{j}))\\\
n_{f}^{\prime}&=\sum_{j=0}^{i-1}\delta(1,f(b_{j}))\\\
d_{m}^{\prime}&=\sum_{j=0}^{i-1}\delta(2,f(b_{j}))\\\ \end{split}$ (15)

where $\delta(i,j)$ is the Kronecker delta function that valued 1 if $i=j$ and
otherwise 0. Hence, we can use binary search to get the top-$i$ salient
blocks, which shall be retained given a parameter constraint, and the rest of
the block shall be pruned. In our implementation, for training stability, we
do not set the pruned blocks’ corresponding masks to 0 directly but gradually
decrease their values by $\alpha=0.01$.

Density | Method | MNLI | QQP | QNLI | SST2 | CoLA | STS-B | MRPC | RTE | GLUE Avg.
---|---|---|---|---|---|---|---|---|---|---
50% | MaP | 83.6 |  87.8 | 91.5 | 91.0 | 60.1 | 89.8 | 90.7 | 67.2 |  82.7
MvP | 82.3 | 87.3 |  90.8 | 90.8 | 57.7 |  89.4 |  91.1 | 67.2 | 82.1
PST | 81.0 | 85.8 | 89.8 |  91.3 | 57.6 | 84.6 | 90.7 | 67.9 | 81.0
LRP | 82.4 | 87.2 | 89.6 | 90.9 | 54.1 | 88.7 | 89.8 |  69.3 | 82.2
APT |  82.8 | 90.1 | 90.1 | 92.7 |  59.6 | 88.3 | 91.8 | 70.4 | 83.2
10% | MaP | 78.2 | 83.2 | 84.1 | 85.4 | 27.9 | 82.3 | 80.5 | 50.1 | 71.4
MvP | 80.1 | 84.4 | 87.2 | 87.2 | 28.6 |  84.3 | 84.1 | 57.6 | 74.2
PST |  79.6 |  86.1 |  86.6 | 89.0 | 38.0 | 81.3 | 83.6 |  63.2 |  75.9
LRP | 79.4 | 86.0 | 85.3 |  89.1 |  35.6 | 83.3 |  84.4 | 62.8 | 75.7
APT | 78.8 | 89.4 | 85.5 | 90.0 | 30.9 | 86.3 | 88.2 | 65.3 | 76.8

Table 7: Comparison of APT to existing unstructured pruning baseline with using PEFT in conjunction. The best results are bold while the second-best ones are underlined. Sparsity | Method | MNLI | QQP | QNLI | SST2 | CoLA | MRPC | RTE | GLUE Avg.
---|---|---|---|---|---|---|---|---|---
0% | FT | 87.6 | 91.9 | 92.8 | 95.2 | 91.2 | 90.2 | 78.7 | 89.7
LoRA | 87.5 | 90.8 | 93.3 | 95.0 | 63.4 | 89.7 | 72.1 | 84.5
40% | LoRA+Distill | 84.2 | 88.3 | 90.1 | 91.9 | 49.9 | 86.8 | 68.6 | 80.0
APT | 86.4 | 90.9 | 92.3 | 94.5 | 56.5 | 92.3 | 74.4 | 83.9
Table 8: Detailed results of RoBERTa pruning with APT compared to the
LoRA+Distill baseline. We ignore the evaluation results of the STS-B task
since it cannot be successfully reproduced with CoFi (the distillation
backbone).

##  Appendix D Additional Baseline Comparisons

In this section, we further compare APT to existing parameter-efficient
pruning methods, such as PST and LRP. In the meantime, we also show detailed
results of APT pruning compared to the LoRA+Distill baseline with more tasks
in the GLUE benchmark and LLaMA-2 13B model pruning results.

###  D.1 Comparison to PST and LRP

We compare APT with the state-of-the-art joint use of unstructured pruning (Li
et al., 2022) and structured pruning (Zhang et al., 2023a) with PEFT on
$\text{BERT}_{\text{base}}$ model, showing in Table 7. We can see that APT
outperforms existing baselines in both 50% and 10% pruning density settings
with a notable margin. The performance gain is credited to our more accurate
pruning strategy considering frozen and tuning parameters. At the same time,
our efficient self-distillation technique used in conjunction with salient
parameters added in training also boosts performance recovery.

###  D.2 Further Comparison to LoRA+Distill

We show the detailed comparison between APT and the LoRA+Distill baseline in
Table 8. APT reaches superior task performance compared to the baseline in all
seven GLUE tasks listed in the table, with on average 93.5% fine-tuned LM
performance maintained, notably outperforming the joint use of LoRA and
knowledge distillation. In particular, the results of STS-B cannot be
reproduced when conducting CoFi distillation with LoRA parameters tuned only,
so we exclude the comparison on STS-B. Among the other seven tasks in the GLUE
benchmark, we find that tasks with relatively smaller dataset sizes, namely
CoLA, MRPC, and RTE, reach superior performance gain when using APT. We
conclude that this is because, compared to simple fine-tuning, knowledge
distillation with salient parameters added in training is more robust and not
prone to overfitting the training data.

Figure 5: The performance-efficiency tradeoff of APT compared to baseline
methods. All metrics are normalized using LoRA tuning w/o pruning as the
baseline. The circular dots with vertical axes on the left indicate training
speed v.s. performance, with their sizes denoting the peak training memory
usage. The squared dots with axes on the right indicate inference speedup v.s.
performance, with sizes denoting inference memory usage.

###  D.3 LLaMA-2 13B Pruning Results

Method | ARC | HellaSwag | MMLU | TruthfulQA | Avg.
---|---|---|---|---|---
LLaMA2 7B | 53.1 | 77.7 | 43.8 | 39.0 | 53.4
LoRA | 55.6 | 79.3 | 46.9 | 49.9 | 57.9
LoRA+Prune | 46.8 | 65.2 | 23.9 | 46.2 | 45.5
LLMPruner | 39.2 | 67.0 | 24.9 | 40.6 | 42.9
APT | 45.4 | 71.1 | 36.9 | 46.6 | 50.0
LLaMA2 13B | 59.4 | 82.1 | 55.8 | 37.4 | 58.7
LoRA | 60.8 | 82.8 | 56.0 | 46.5 | 61.5
LoRA+Prune | 56.4 | 79.1 | 50.7 | 42.1 | 57.1
LLMPruner | 46.8 | 74.0 | 24.7 | 34.8 | 45.1
APT | 49.5 | 75.8 | 52.5 | 44.7 | 55.6
Table 9: LLaMA2 7B and 13B 30% sparsity pruning results with GPT4-generated
Alpaca dataset, evaluated on the Open LLM leaderboard few-shot tasks.

As shown in Table 9, when pruning LLaMA-2 13B models, APT maintains 90.0%
performance of the unpruned LoRA-tuned baseline. Compared to the pruning
result on 7B models that maintain 86.4% dense model performance, better
accuracies can be recovered in larger models (13B). At the same time, under
the same pre-tuning pruning settings, APT performs better than the LLMPruner
baseline on all four evaluation tasks, indicating the effectiveness of
considering outlier parameters in large LM pruning. Nonetheless, the
LoRA+Prune baseline reaches slightly better results than APT when pruning 13B
models, illustrating that there is still room for improving pre-tuning pruning
methods in future works. More specifically, among the four tasks we use for
evaluating large LMs, TruthfulQA benefits the most from Alpaca fine-tuning. We
can see that APT reaches superior results on TruthfulQA than existing
baselines regardless of model size. The LM’s capabilities on ARC and HellaSawg
downgrade the most when pruning large LM before fine-tuning, implying
possibilities of catastrophic forgetting in this paradigm.

##  Appendix E Efficiency and Performance Tradeoff Analysis

We use Fig. 5 to clearly show the LMs’ end-task performance and efficiency
tradeoffs between different tuning, pruning, and distillation baselines. We
add several extra baselines to conduct more detailed comparisons between APT
with existing PEFT, pruning, and distillation methods:

LoRA+Prune w/distill: we first use LoRA to fully converge a model on the task
dataset, and then use Mask-Tuning (Kwon et al., 2022) to prune the LM.
Afterward, we utilize the converged model before pruning as the teacher model
and distill its knowledge to the pruned student model with static knowledge
distillation objectives.

LoRA+Prune w/o retrain: we use Mask-Tuning to prune a LoRA-tuned converged
model but do not conduct any retraining to recover the pruned models’
performance. Therefore, the LM’s training time will be reduced, yet its
performance is lower than the LoRA+Prune baseline.

With the same target sparsity in RoBERTa and LLaMA pruning setups, APT
achieves on-par end-task performance with full fine-tuning and LoRA tuning
baselines. Meanwhile, APT-tuned models reach similar or even better inference
time and memory efficiency than existing baselines. APT-pruned T5 LMs’
inference efficiency is slightly worse because more decoder parameters (with
less computations happening) are pruned than the baselines. Moreover, when
pruning RoBERTa and T5 models, APT achieves faster training time than all
pruning and distillation baselines. Specifically, the training speed of APT in
RoBERTa models is even higher than LoRA tuning without pruning. In LLaMA model
pruning, APT costs significantly less training memory than both LLMPruner and
LoRA+Prune baselines.

##  Appendix F Pruning Sparsity Analysis

Figure 6: Task performance v.s. relative inference efficiency on RoBERTa, T5,
and LLaMA-2 7B models with APT and baselines.

We further show the task performance changing trajectory with different
pruning sparsities in Fig. 6. APT achieves superior inference speedup and less
inference memory consumption than baselines targeting the same task
performance. Compared to the LoRA+Prune baseline, when pruning RoBERTa models
targeting similar task accuracy, APT gains 21.8% more inference speedup and 7%
more memory reduction. For T5 model pruning with 97% dense model performance
maintained, APT results in 62.7% more inference speedup with 24.8% more
inference memory reduced compared to the LoRA+Prune baseline. When pruning
large LLaMA2-7B models, APT prunes gets 6.7% more speedup and 9.2% more
inference memory reduction than the LoRA+Prune baseline, with about 85% dense
model performance maintained.

##  Appendix G Distillation Strategy Comparison

| SST2 | Train. Speed($\Uparrow$) | Train. Mem.($\Downarrow$)
---|---|---|---
APT | 94.5 | 16.9% | 70.1%
w/o $\mathcal{L}_{layer}$ | 93.7 | 17.4% | 69.8%
w/o self-distillation | 92.9 | 20.7% | 69.2%
FT teacher | 94.3 | 7.9% | 111.8%
LoRA teacher | 93.7 | 1.7% | 96.1%

Table 10: Ablation study of distillation strategies and comparison to non-
efficient distillation techniques. The training speed and memory are relative
metrics compared to fine-tuning the dense model.

We show the further analysis in Table 10 to compare the self-distillation
technique we use in APT and traditional knowledge distillation methods. When
ablating the dynamic layer mapping strategy in our self-distillation approach,
the LM performance decreased by 0.8% with similar training time and memory
consumption. When training without distillation objectives (w/o self-
distillation), the LM performance drops by 1.7%. Nonetheless, the training is
slightly faster with less memory costs. These results present that using
distillation objectives for better LM task performance will sacrifice training
efficiency as a tradeoff. Furthermore, we also demonstrate the comparisons
with existing static knowledge distillation strategies, using the converged
full-parameter fine-tuned LM (FT teacher) and LoRA-tuned LM (LoRA teacher) as
the teacher model. We calculate the time consumption for both teacher and
student training when using these distillation baselines. As shown in Table
10, using fully fine-tuned models as the teacher will incur more memory cost
than dense model fine-tuning, while APT only consumes 70%. In the meantime,
the training convergence speed of APT training is two times faster than the
traditional knowledge distillation method with a fine-tuned teacher.
Furthermore, using a LoRA-tuned model as the teacher will result in extremely
slow training speed. In addition, simply tuning the LoRA layers with knowledge
distillation objectives doesn’t help reduce the training memory consumption,
as the memory consumption is still 96.1% than full fine-tuning.