# Revisiting the Parameter Efficiency of Adapters from the Perspective of
Precision Redundancy

Shibo Jie Haoqing Wang Zhi-Hong Deng
School of Intelligence Science and Technology, Peking University
<EMAIL_ADDRESS>Corresponding Author.

###### Abstract

Current state-of-the-art results in computer vision depend in part on fine-
tuning large pre-trained vision models. However, with the exponential growth
of model sizes, the conventional full fine-tuning, which needs to store a
individual network copy for each tasks, leads to increasingly huge storage and
transmission overhead. Adapter-based Parameter-Efficient Tuning (PET) methods
address this challenge by tuning lightweight adapters inserted into the frozen
pre-trained models. In this paper, we investigate how to make adapters even
more efficient, reaching a new minimum size required to store a task-specific
fine-tuned network. Inspired by the observation that the parameters of
adapters converge at flat local minima, we find that adapters are resistant to
noise in parameter space, which means they are also resistant to low numerical
precision. To train low-precision adapters, we propose a computational-
efficient quantization method which minimizes the quantization error. Through
extensive experiments, we find that low-precision adapters exhibit minimal
performance degradation, and even 1-bit precision is sufficient for adapters.
The experimental results demonstrate that 1-bit adapters outperform all other
PET methods on both the VTAB-1K benchmark and few-shot FGVC tasks, while
requiring the smallest storage size. Our findings show, for the first time,
the significant potential of quantization techniques in PET, providing a
general solution to enhance the parameter efficiency of adapter-based PET
methods. Code: https://github.com/JieShibo/PETL-ViT

##  1 Introduction

Large pre-trained vision models have demonstrated exceptional performance on
various visual tasks via fine-tuning on task-specific data. In the traditional
fine-tuning paradigm, the entire model is updated for each downstream task,
resulting in the need to store a fine-tuned model separately for each task.
However, with the remarkable scalability of modern vision models, the size of
pre-trained vision models is increasing exponentially to achieve superior
performance. As a result, the storage cost of the full fine-tuning paradigm
becomes prohibitive in multi-task scenarios.

Figure 1: Average accuracy _vs._ size of trainable parameters in backbones
(log scale) on VTAB-1K benchmark. Our low-precision adapter-based methods
outperform other baselines.

_Parameter-Efficient Tuning_ (PET) has recently emerged as a promising
approach for fine-tuning a limited number of parameters while attaining
performance comparable to full fine-tuning on downstream tasks. Adapter-based
methods [63, 25, 59, 6, 51, 31, 26, 32] are among the techniques proposed for
PET and have gained considerable attention due to their effectiveness.
Adapters are typically small subnetworks with bottleneck architecture
comprising two fully-connected (FC) layers inserted into pre-trained models.
Adapter-based methods freeze pre-trained weights and update only the adapters,
whose parameter efficiency is achieved through their small hidden dimension.

Although the bottleneck adapters have been already lightweight (_e.g_., 0.5
MB/task for ViT-B [12]), the storage costs remain considerable when dealing
with a huge number of tasks (_e.g_., platform that provides customized models
for millions of users). To address this issue, recent studies have shown that
the parameter efficiency of adapters can be further improved. For example,
[51, 32, 22] explore the low-rank structure in adapters, reparameterizing the
weight of adapters into smaller subspace with Kronecker, Tensor-Train, or
Tucker factorization. Additionally, [21] leverages network pruning to train
sparse adapters. We find that these methods actually have a common motivation
– reducing the redundancy (_e.g_., rank redundancy, density redundancy) in
adapters. Also motivated by this, we pose a question, _whether there is any
other kind of redundancy that can be utilized to better improve the efficiency
of adapters._

In this paper, we begin by exploring the loss landscape of adapters and
observe that the local minima of adapters are much flatter than that of the
fully fine-tuned models. The flatness of local minima indicates that the
trained adapters possess greater resilience to noise in parameter space, such
that adapters with low-precision parameters should perform equally well as
their high-precision counterparts. Therefore, we infer that adapters are
redundant in numerical precision. Since previous work on adapters all employs
full-precision (FP32) data type, the impact of precision on adapters has not
been investigated yet.

To reduce the precision redundancy, we propose an approach that involves
training and storing adapters in low-bit parameter space. Through empirical
analysis, we observe that the parameters of each adapter weight approximately
follow a Gaussian distribution. Under this assumption, we quantize the adapter
parameters by minimizing the quantization loss. Inspired by previous work of
neural network quantization [17], we adopt quantization-aware training and
train the low-bit adapters with straight-through estimator (STE). Our
experiments, conducted on extensive datasets, reveal several key findings:
_1)_ Unlike quantizing the entire model, quantizing only the adapters results
in negligible performance degradation, even in the 1-bit setting; _2)_ With a
fixed storage budget, 1-bit quantized (_i.e_., binary) adapters achieve
superior performance among all precision settings; _3)_ Our 1-bit adapter can
outperform all previous PET methods, including low-rank factorization methods,
while using the smallest storage size.

Our contributions are summarized as follows:

  * •

From the investigation on the flat local minima of adapters, we infer the
existence of precision redundancy in the parameters of adapters, which can be
leveraged to improve their parameter efficiency.

  * •

Based on empirical observations of the distribution of adapter parameters, we
propose an efficient quantization-aware training method for learning low-bit
adapters while minimizing the quantization error.

  * •

Extensive experiments and comparisons verify that lowering the bit-width
brings significant efficiency improvement to adapters. Our proposed method
achieves new state-of-the-art results in terms of both performance and
parameter efficiency.

Figure 2: _Left:_ Illustration of adapters. “Pre-Trained OP” denotes
operations in pre-trained models, such as the FFN blocks or QKV
transformations in ViTs. _Right:_ Loss landscape visualization of full fine-
tuning and adapter-based tuning [6, 26] on ViT-B. Figure 3: Accuracy
degradation under different intensity of Gaussian noise. Adapters converge at
flatter local minima and are more resistant to disturbation.

##  2 Related Work

Parameter-Efficient Tuning. Parameter-Efficient Tuning (PET) aims to adapt
pre-trained vision backbone to downstream tasks by tuning only a small number
of parameters. Most work about PET focuses on tuning transformer-based
networks, _e.g_., Vision Transformers (ViTs) [12]. Prompt-based methods [30,
78, 79, 65, 49, 74] concatenate trainable tokens to the sequential inputs of
transformers as prompts, adapting the models by tuning the prompts. However,
since the computational cost of self-attention is proportional to the square
of the length of inputs, prompt-based methods are not as computation-efficient
as the original network [30, 6]. Adapter-based methods [63, 25, 59, 6, 51, 31,
26, 32, 19, 50] insert small adapters into the pre-trained model, adjusting
the intermediate representations of the network to fit the downstream data.
Some of them [26, 32] can be absorbed into the pre-trained weights during
inference, which ensures the computational cost is not increased. Besides,
there are also methods that tune bias parameters [73], modify the intermediate
features via affine transformation [42], fit the change in the network outputs
by a small side-network [76], or combine multiple methods automatically [77,
5]. Among them, adapter-based methods have attracted much attention for their
competitive performance, generality to different backbones, and scalability.

Efficient Designs of Adapters.  As illustrated in Figure 2 (left), adapters
are commonly subnetworks composed of two FC layers with nonlinear activation
in between. Adapter-P [59] places the adapters after the Feed-Forward Network
(FFN) blocks, and AdaptFormer [6] uses adapters parallel to the FFN blocks of
ViT. LoRA [26] uses two low-rank matrices to fit the change in the query and
value transformation of Multi-Head Self-Attention (MHSA). The formulation of
LoRA is equivalent to two FC layers without bias parameters and activation,
and can be regarded as special adapters in parallel with the query and value
weights.

Besides, some work focuses on more compact designs for adapters. Compacter
[51] and KAdaptation [22] regard the weights of adapters as the Kronecker
product of two smaller matrices, one of which is shared among adapters. FacT
[32] tensorizes the network as a tensor, and reparameterizes its change as
several factors according to Tensor-Train or Tucker format that are updated
end-to-end. Similar to LoRA, FacT is not proposed as an adapter-based method,
but it can also be viewed as reparameterized adapters with partially shared
weights. Besides, SparseAdapter [21] prunes the dense weights of adapters
before fine-tuning. These designs reduce the rank and density redundancy in
adapters, but we focus on a neglected but more effective direction – precision
redundancy.

Network Quantization.  Network quantization [17] compresses networks by
reducing the bit-width of weight and activation. Current quantization methods
include Post-Training Quantization [1, 55, 69, 47, 72, 27, 29], which performs
quantization on trained model without re-training; and Quantization-Aware
Training [34, 13, 3, 41], which introduce quantization during the training
process by approximating the gradient of the non-differentiable quantization
operator. The former paradigm does not require access to the entire training
data during quantization and has shown almost lossless performance using FP16
and INT8 data type, while the latter yields quantized models with better
performance and can work in extremely low-bit settings, _e.g_., binary
quantization [60, 46, 43].

##  3 Preliminaries

In this paper, we mainly focus on ViTs as pre-trained backbone following
previous work [30, 6, 77, 31]. We start with a concise formalization of the
commonly used adapters.

AdaptFormer [6] uses bottleneck FFN composed of two FC layers with in-between
ReLU activation as adapters. The weights of an adapter are
$\boldsymbol{W}_{down}\in\mathbb{R}^{d\times h}$ and
$\boldsymbol{W}_{up}\in\mathbb{R}^{h\times d}$, where $h<<d$. Adapters are
inserted into networks as shortcuts bypassing the FFN blocks, _i.e_., given an
input $\boldsymbol{X}\in\mathbb{R}^{N\times d}$, the computation is formulated
as

$\boldsymbol{X}^{\prime}=\underbrace{\boldsymbol{X}+\textit{FFN}(\boldsymbol{X})}_{\text{Frozen}}+\underbrace{s\cdot\textit{ReLU}(\boldsymbol{X}\boldsymbol{W}_{down})\boldsymbol{W}_{up}}_{\text{Adapter}}$
(1)

where $s$ is a hyper-parameter, $\boldsymbol{X}$ is the input of FFN blocks.

LoRA [26] learns the low-rank approximation of change in $\boldsymbol{W}_{q}$
and $\boldsymbol{W}_{v}$. Formally, it reparameterizes
$\Delta\boldsymbol{W}_{q/v}$ into $\boldsymbol{A}_{q/v}\boldsymbol{B}_{q/v}$,
where $\boldsymbol{A}_{q/v}\in\mathbb{R}^{d\times
h},\boldsymbol{B}_{q/v}\in\mathbb{R}^{h\times d}$ and $h<<d$. The query and
value of MHSA are computed as

$\boldsymbol{Q/V}=\underbrace{\boldsymbol{XW}_{q/v}}_{\text{Frozen}}+\underbrace{s\cdot\boldsymbol{XA}_{q/v}\boldsymbol{B}_{q/v}}_{\text{Adapter}}$
(2)

in which $s$ is a scaling hyper-parameter, and $\boldsymbol{X}$ is the input
of MHSA blocks. LoRA is equivalent to using AdaptFormer-style adapters with
identity activation, whose weights are
$\boldsymbol{A}_{q},\boldsymbol{B}_{q},\boldsymbol{A}_{v},\boldsymbol{B}_{v}$.

Figure 4: Illustration of the proposed quantization method with $b=2$. Figure
5: Parameter frequency histogram visualization of the 24 weight matrices in
all the 12 adapters of AdaptFormer fine-tuned on Caltech101. The parameters
(blue histograms) are roughly subject to Gaussian distribution (red curves).

##  4 Methodology

###  4.1 Precision Redundancy in Adapters

It has been extensively studied that the property of a neural network is
highly correlated with the flatness of its loss landscape, _e.g_., the flatter
the local minima, the better the generalization [40, 64, 36, 15, 7, 20].
Inspired by them, we here investigate the loss landscape of adapters in vision
models to explore their property. Following [40], we plot the loss landscape
of full fine-tuning, AdaptFormer, and LoRA when adapting pre-trained ViT-B
[12]. As shown in Figure 2 (right), AdaptFormer and LoRA obviously converge at
much flatter regions than full fine-tuning.

The flat local minima of visual adapters indicate that they generalize better,
providing an explanation for their superior performance over full fine-tuning
on small and medium-size datasets [30, 31]. Moreover, if the parameters
converge at flatter local minima, there are wide low-loss areas around these
points. Therefore, when adding noise to the converged parameters, we can
expect that the loss will not increase significantly. In other words, the
model is resistant to disturbation in parameter space.

As shown in Figure 3, we add Gaussian noise
$\mathcal{N}(0,\sigma_{noise}^{2})$ with different $\sigma_{noise}$ to the
fine-tuned weights, and find that adding noise to adapter-tuned models leads
to much less accuracy degradation than fully fine-tuned models. Adapters still
retain most of the performance even if the noise has equivalent variance to
the weights (_i.e_., $\sigma_{noise}=\sigma_{weight}$). Since numerical error
can be also viewed as a type of noise, we conjecture that the adapters would
not suffer from lower numerical precision.

###  4.2 Trading Precision for Efficiency

In view of the existence of precision redundancy, a natural idea is to trading
the redundant precision for much needed efficiency. Previous work on
quantization [18, 71, 9] has demonstrated that clustering is a reliable
direction for quantization of arbitrary bit-width, so we also adopt a
clustering-based quantization strategy for adapters.

As illustrated in Figure 3, the smaller the noise, the less the performance
degradation. The object of adapter quantization is to minimize the noise
involved, _i.e_., minimize the quantization error. The $b$-bit quantization
process can be viewed as dividing $\mathbb{R}$ into $B=2^{b}$ non-overlapping
sets $\left\\{\mathcal{U}_{1},...,\mathcal{U}_{B}\right\\}$, which correspond
to a codebook with $B$ codes $\\{c_{1},...,c_{B}\\}$. The quantization
function quantizes all values in $\mathcal{U}_{j}$ to $c_{j}$,

$\mathcal{Q}(w)=c_{j}\ \text{if}\ w\in\mathcal{U}_{j}$ (3)

Then we minimize the quantization error as follows,

$\displaystyle\underset{c_{1},...,c_{B},\mathcal{U}_{1},...,\mathcal{U}_{B}}{\text{minimize}}$
$\displaystyle\sum_{i=1}^{m}{|w_{i}-\mathcal{Q}(w_{i})|}^{p}$ (4)

in which $w_{i}$ is an element of a weight $\boldsymbol{W}$ of the adapters,
and $m$ is the number of elements in $\boldsymbol{W}$. This problem is
equivalent to 1D clustering, which can be addressed via clustering algorithm
such as $k$-means ($p=2$) and $k$-medians ($p=1$).

Low-bit quantization, particularly 1-bit quantization suffers catastrophically
poor performance in the absence of quantization-aware training (QAT). In QAT,
the weights are ever-changing, so the clustering algorithm has to be rerun in
each forward propagation during tuning. An appropriate clustering algorithm is
supposed to have negligible computational cost, but an iterative algorithm
like $k$-means and $k$-medians is not efficient enough. Moreover, since the
cluster assignment in $k$-means and $k$-medians is not differentiable, this
process cannot be end-to-end optimized in QAT. Therefore, although previous
work [18] has applied $k$-means into post-training quantization, it is not a
suitable choice for QAT on adapters.

To find an efficient and differentiable clustering method, we visualize the
frequency histogram of the parameters in the weights of adapters. As shown in
Figure 5, we find that the parameters in full-precision adapters are subject
to a bell-shaped distribution with tails. For simplicity, we suppose the
parameters of each weight are always Gaussian, so that the clustering
algorithm can be simplified considerably.

Before tuning, we perform clustering on a standard Gaussian distribution to
calculate $\\{c_{1},...,c_{B}\\}$ and
$\left\\{\mathcal{U}_{1},...,\mathcal{U}_{B}\right\\}$. We suppose $p=1$ in
Eq. (4) and use $k$-medians for simplicity. As illustrated in Figure 4, in
each training step, we first standardize the weights by the means and
variances of their parameters,

$w_{i}^{\prime}=\frac{w_{i}-\mu}{\sigma}$ (5)

where
$\mu=\textit{MEAN}(\\{w_{i}\\}_{i=1}^{m}),\sigma=\textit{STD}(\\{w_{i}\\}_{i=1}^{m})$.
According to the Gaussian assumption, the parameters in each standardized
weight are subject to standard Gaussian distribution. Then we quantize each
standardized weight with the pre-calculated $\\{c_{1},...,c_{B}\\}$ and
$\\{\mathcal{U}_{1},...,\mathcal{U}_{B}\\}$,

$\hat{w_{i}}^{\prime}=\mathcal{Q}(w_{i}^{\prime})=c_{j}\ \text{if}\
w_{i}^{\prime}\in\mathcal{U}_{j}$ (6)

Finally, we de-standardize the weights to their original means and variances,

$\hat{w_{i}}=\hat{w_{i}}^{\prime}\cdot\sigma+\mu$ (7)

and then feed the inputs to perform the forward and backward propagation.

In the whole quantization process, only the quantization operation
$\mathcal{Q}$ is not differentiable, so we use straight-through estimator
(STE) to approximate the gradient, _i.e_.,
$\frac{\partial\mathcal{Q}(w_{i}^{\prime})}{\partial w_{i}^{\prime}}=1$. Then
$\forall w_{i},w_{k}\in\boldsymbol{W}$ the overall gradient is calculated as

$\frac{\partial\hat{w_{i}}}{\partial
w_{k}}=\begin{cases}1+\frac{w_{i}^{\prime}(\hat{w_{i}}^{\prime}-w_{i}^{\prime})}{m}\
\quad\text{if}\ i=k\\\
\frac{w_{k}^{\prime}(\hat{w_{i}}^{\prime}-w_{i}^{\prime})}{m}\quad\quad\quad\text{otherwise}\end{cases}$
(8)

During tuning, the pre-trained weights are always frozen, and only the
adapters as well as the classification head are updated. The full-precision
weights are maintained in training, and updated via end-to-end gradient
descent. Since PET only focuses on boosting parameter efficiency, we still use
full-precision activation for better performance. After tuning, we store
necessary information for reproducing the quantized weights instead of the
full-precision adapters, _i.e_., the $b$-bit quantization indexes $j$ of
adapters’ parameters ($b$ bits per parameter) and the mean $\mu$ and standard
deviation $\sigma$ of each full-precision weight matrix in adapters (128 bits
per adapter). $\\{c_{1},...,c_{B}\\}$ and
$\\{\mathcal{U}_{1},...,\mathcal{U}_{B}\\}$ can be recalculated before
inference. At inference time, the weights are reconstructed as

$\hat{w_{i}}=c_{j}\cdot\sigma+\mu$ (9)

which are directly used for inference.

Method |  Bit-width |  Avg. Acc. |  Size (KB)
---|---|---|---
AdaptFormer |  32 (FP) |  76.70 |  576.0
16 (FP) |  76.70 (- 0.00) |  288.0
8 (INT) |  76.69 ($\downarrow$ 0.01) |  144.0
4 |  76.76 ($\uparrow$ 0.06) |  72.3
2 |  76.64 ($\downarrow$ 0.06) |  36.2
1 |  76.41 ($\downarrow$ 0.29) |  18.2
LoRA |  32 (FP) |  76.42 |  1152.0
16 (FP) |  76.42 (- 0.00) |  576.0
8 (INT) |  76.42 (- 0.00) |  288.0
4 |  76.33 ($\downarrow$ 0.09) |  144.4
2 |  76.27 ($\downarrow$ 0.15) |  72.4
1 |  76.40 ($\downarrow$ 0.02) |  36.4

Table 1: Average accuracy on VTAB-1K benchmark. We fix $h=8$ for AdaptFormer
and LoRA to 8 and change the bit-width. “Size” denotes the size of adapters
per task.

Method |  Bit-width |  Dimension |  Avg. Acc.
---|---|---|---
AdaptFormer |  32 (FP) |  1 |  75.29
8 (INT) |  4 |  76.34 ($\uparrow$ 1.05)
4 |  8 |  76.76 ($\uparrow$ 1.47)
2 |  16 |  76.89 ($\uparrow$ 1.60)
1 |  32 |  76.97 ($\uparrow$ 1.68)
LoRA |  32 (FP) |  1 |  75.70
8 (INT) |  4 |  76.08 ($\uparrow$ 0.38)
4 |  8 |  76.33 ($\uparrow$ 0.63)
2 |  16 |  76.70 ($\uparrow$ 1.00)
1 |  32 |  76.72 ($\uparrow$ 1.02)

Table 2: Average accuracy on VTAB-1K benchmark under certain storage budget.
Lower bit-width and higher hidden dimension lead to better performance.

| Natural | Specialized | Structured
---|---|---|---
|  |  Size (MB)
---
|  Avg. Acc.
---
|  Cifar100
---
|  Caltech101
---
|  DTD
---
|  Flower102
---
|  Pets
---
|  SVHN
---
|  Sun397
---
|  Camelyon
---
|  EuroSAT
---
|  Resisc45
---
|  Retinopathy
---
|  Clevr-Count
---
|  Clevr-Dist
---
|  DMLab
---
|  KITTI-Dist
---
|  dSpr-Loc
---
|  dSpr-Ori
---
|  sNORB-Azim
---
|  sNORB-Ele
---
_Conventional Fine-Tuning_
Full |  327 |  68.9 |  68.9 |  87.7 |  64.3 |  97.2 |  86.9 |  87.4 |  38.8 |  79.7 |  95.7 |  84.2 |  73.9 |  56.3 |  58.6 |  41.7 |  65.5 |  57.5 |  46.7 |  25.7 |  29.1
Linear |  0 |  57.6 |  64.4 |  85.0 |  63.2 |  97.0 |  86.3 |  36.6 |  51.0 |  78.5 |  87.5 |  68.5 |  74.0 |  34.3 |  30.6 |  33.2 |  55.4 |  12.5 |  20.0 |  9.6 |  19.2
_PET methods_
VPT-Deep [30] |  2.03 |  72.0 |  78.8 |  90.8 |  65.8 |  98.0 |  88.3 |  78.1 |  49.6 |  81.8 |  96.1 |  83.4 |  68.4 |  68.5 |  60.0 |  46.5 |  72.8 |  73.6 |  47.9 |  32.9 |  37.8
NOAH† [77] |  1.37 |  75.5 |  69.6 |  92.7 |  70.2 |  99.1 |  90.4 |  86.1 |  53.7 |  84.4 |  95.4 |  83.9 |  75.8 |  82.8 |  68.9 |  49.9 |  81.7 |  81.8 |  48.3 |  32.8 |  44.2
LoRA [26] |  1.13 |  76.4 |  72.0 |  91.2 |  71.6 |  99.1 |  91.3 |  88.9 |  56.4 |  87.2 |  94.6 |  83.9 |  74.9 |  83.7 |  64.0 |  52.3 |  81.2 |  84.8 |  53.3 |  38.1 |  43.4
SSF [42] |  0.78 |  75.7 |  69.0 |  92.6 |  75.1 |  99.4 |  91.8 |  90.2 |  52.9 |  87.4 |  95.9 |  87.4 |  75.5 |  75.9 |  62.3 |  53.3 |  80.6 |  77.3 |  54.9 |  29.5 |  37.9
Adapter-P [59] |  0.56 |  75.5 |  73.2 |  90.1 |  69.6 |  99.2 |  91.1 |  84.9 |  56.0 |  86.6 |  94.8 |  82.5 |  75.8 |  82.9 |  63.9 |  49.7 |  79.7 |  81.7 |  55.5 |  31.6 |  42.2
AdaptFormer [6] |  0.56 |  76.7 |  73.8 |  92.3 |  72.7 |  99.3 |  91.6 |  89.1 |  56.5 |  87.8 |  95.5 |  84.9 |  75.2 |  83.3 |  62.5 |  52.4 |  81.7 |  86.2 |  55.9 |  34.4 |  40.2
BitFit [73] |  0.39 |  65.2 |  72.8 |  87.0 |  59.2 |  97.5 |  85.3 |  59.9 |  51.4 |  78.7 |  91.6 |  72.9 |  69.8 |  61.5 |  55.6 |  32.4 |  55.9 |  66.6 |  40.0 |  15.7 |  25.1
FacT-TT [32] |  0.30 |  76.7 |  73.4 |  91.0 |  72.4 |  99.2 |  91.4 |  90.1 |  56.6 |  87.3 |  94.7 |  84.5 |  75.8 |  83.0 |  64.9 |  51.3 |  81.4 |  87.4 |  53.2 |  33.5 |  44.3
VPT-Shallow [30] |  0.24 |  67.8 |  77.7 |  86.9 |  62.6 |  97.5 |  87.3 |  74.5 |  51.2 |  78.2 |  92.0 |  75.6 |  72.9 |  50.5 |  58.6 |  40.5 |  67.1 |  68.7 |  36.1 |  20.2 |  34.1
Compacter [51] |  0.15 |  74.2 |  71.9 |  89.0 |  69.7 |  99.1 |  90.7 |  82.7 |  56.1 |  86.0 |  93.5 |  82.4 |  75.3 |  80.2 |  63.4 |  47.4 |  77.2 |  78.1 |  53.5 |  27.3 |  39.8
Bi-LoRA (Ours) |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
$h=32$ |  0.14 |  76.7 |  72.1 |  91.7 |  71.2 |  99.1 |  91.4 |  90.2 |  55.8 |  87.0 |  95.4 |  85.5 |  75.5 |  83.1 |  64.1 |  52.2 |  81.3 |  86.4 |  53.5 |  36.7 |  44.4
$h=1$ |  0.0048 |  75.4 |  72.6 |  90.4 |  71.8 |  99.0 |  91.3 |  87.0 |  56.0 |  86.1 |  94.1 |  82.1 |  75.4 |  81.0 |  64.2 |  50.5 |  79.7 |  83.0 |  53.7 |  29.7 |  42.9
Bi-AdaptFormer (Ours) |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
$h=32$ |  0.071 |  77.0 |  74.1 |  92.4 |  72.1 |  99.3 |  91.6 |  89.0 |  56.3 |  88.2 |  95.2 |  86.0 |  76.2 |  83.9 |  63.6 |  53.0 |  81.4 |  86.2 |  54.8 |  35.2 |  41.3
$h=1$ |  0.0024 |  75.0 |  73.3 |  91.0 |  72.1 |  99.1 |  91.4 |  86.0 |  56.2 |  87.0 |  94.6 |  82.9 |  76.0 |  79.6 |  62.8 |  50.1 |  78.6 |  76.6 |  53.9 |  27.4 |  38.6

Table 3: Full results on the VTAB-1K benchmark. “Avg. Acc.” denotes the
average results over three groups. “Size” denotes the average size of
trainable parameters in backbones per task, _i.e_., classification heads (0.14
MB/task in average) are not counted. † denotes results from [77] using
normalized inputs.

##  5 Experiments

###  5.1 Datasets

We use more than 20 image classification tasks to evaluate the performance of
different PET methods.

VTAB-1K benchmark. VTAB-1K [75] contains 19 image classification tasks from
diverse fields, which can be categorized into three groups: Natural,
Specialized, and Structured. These tasks cover a large range of possible
domains where downstream tasks come, so the performance of different methods
on this benchmark largely reflects their ability to transfer learning. Each
dataset contains 800 samples for training and 200 for validation. Following
previous work [30, 77, 42, 32, 31], we tune the pre-trained model with all the
1,000 training and validation samples and report results evaluated on test-
set. Following [30, 42], we use _unnormalized inputs_ that are consistent with
the VTAB paper [75]. Note that some previous methods [77, 32] normalize the
images with ImageNet’s mean and standard deviation, so we re-implement some of
them for a fair comparison.

Few-shot fine-grained visual recognition (FGVC). We use five FGVC datasets to
evaluate the capability of PET methods in the low-data regime. The five
datasets are FGVC-Aircraft [52], Oxford-Pets [58], Food-101 [4], Stanford Cars
[37], and Oxford-Flowers102 [57]. Experiments are conducted in 1, 2, 4, 8, and
16-shot settings.

###  5.2 Performance of Low-Precision Adapters

We first address the most critical question in this paper: _is reducing
precision redundancy a good choice for improving the parameter efficiency of
adapter-based PET methods?_ To investigate the role of numerical precision in
adapters, we make comparisons across different bit-widths. We use AdaptFormer
[6] and LoRA [26] with $h=8$ to adapt ViT-B/16 [12] pre-trained on supervised
ImageNet-21K [11]. The 32-bit adapters are trained using FP32 without
quantization. 16-bit (FP16) and 8-bit (INT8) adapters are directly converted
from fine-tuned FP32 adapters. Others are fine-tuned using the proposed QAT
method. Table 1 presents the accuracy and adapter size on VTAB-1K.

We notice that using $b$-bit adapters leads to about $\frac{32}{b}\times$ more
parameter efficiency than full-precision adapters. However, the performance
degradation resulting from quantization is very slight and sometimes
negligible, even in the 1-bit setting. Note that quantizing the entire model
to a very low bit-width usually causes significant performance degradation,
but our observation indicates that low-bit quantization only on adapters is
reliable and much less damaging.

Moreover, we explore the best bit-width given a certain storage budget. Since
low-precision adapters are more lightweight, we can augment their performance
by using higher hidden dimension to utilize the saved space. The size of a
$b$-bit $h$-dimension adapter is about $2dbh$ bits where $d$ is the feature
dimension, so we fix $bh=32$ and compare different combinations of $b$ and
$h$. As shown in Table 2, the lower $b$ and higher $h$ yield better
performance on LoRA and AdaptFormer. 1-bit adapters perform the best across
different combinations. Overall, we find that the parameter efficiency gains
of the low-bit adapters far outweigh their performance damage, demonstrating
the feasibility and necessity to trade precision for efficiency.

Figure 6: Accuracy of few-shot learning on FGVC datasets. The average size
(MB) of trainable parameters in backbones is shown in parentheses. Bi-
AdaptFormer outperforms other baselines on average accuracy using the fewest
trainable parameters. Results are averaged over three trials with different
seeds.

Method |  $h$ |  Binary Head |  Avg. Acc. |  Ckpt Size (KB)
---|---|---|---|---
Full |  - |  $\times$ |  68.9 |  3.4$\times$105
Linear |  - |  $\times$ |  57.6 |  140.8
Bi-AdaptFormer |  32 |  $\times$ |  76.97 |  212.9
32 |  $\checkmark$ |  76.89 ($\downarrow$ 0.08) |  76.6
1 |  $\times$ |  74.96 |  143.2
1 |  $\checkmark$ |  73.81 ($\downarrow$ 1.15) |  6.8
Bi-LoRA |  32 |  $\times$ |  76.72 |  285.1
32 |  $\checkmark$ |  76.31 ($\downarrow$ 0.41) |  148.8
1 |  $\times$ |  75.39 |  145.6
1 |  $\checkmark$ |  74.56 ($\downarrow$ 0.83) |  9.2

(a) Classification head quantization. “Ckpt Size” denotes the average size of
checkpoint including classification heads.

Backbone |  Method |  Avg. Acc. |  Size (MB)
---|---|---|---
ConvNeXt-B |  Full |  74.03 |  334.0
Linear |  63.58 |  0
VPT-Deep |  68.71 |  0.017
AdaptFormer |  78.86 |  1.102
Bi-AdaptFormer |  79.07 ($\uparrow$ 0.21) |  0.138
Swin-B |  Full |  74.99 |  332.2
Linear |  62.60 |  0
VPT-Deep |  71.55 |  0.622
AdaptFormer |  77.22 |  0.734
Bi-AdaptFormer |  77.24 ($\uparrow$ 0.02) |  0.092

(b) Performance on other backbones. We use ConvNeXt-B and Swin-B as backbones.

Method |  Training Time (ms/batch) |  Inference Time (ms/batch)
---|---|---
Full |  342.0 |  110.8
Linear |  115.2 |  110.8
VPT-Deep |  407.9 |  174.2
FacT-TT |  296.0 |  110.8
AdaptFormer |  252.3 |  115.3
Bi-AdaptFormer |  265.0 ($\uparrow$ 12.7) |  115.5 ($\uparrow$ 0.02)
LoRA |  275.7 |  110.8
Bi-LoRA |  293.2 ($\uparrow$ 17.5) |  110.8

(c) Average training and inference time. Measured on a single GeForce RTX 3090
GPU with batch size 64.

Method |  Bit-width |  Avg. Acc.
---|---|---
AdaptFormer |  2 (PTQ) |  74.89
1 (PTQ) |  67.28
2 (Ours) |  76.64 ($\uparrow$ 1.75)
1 (Ours) |  76.41 ($\uparrow$ 9.13)
LoRA |  2 (PTQ) |  74.22
1 (PTQ) |  67.38
2 (Ours) |  76.27 ($\uparrow$ 2.05)
1 (Ours) |  76.40 ($\uparrow$ 9.02)

(d) QAT _vs._ PTQ. “PTQ” denotes directly quantizing fine-tuned FP32 adapters
using $k$-means.

Method |  # Block |  Avg. Acc. |  Size (KB)
---|---|---|---
Bi-AdaptFormer |  1 |  76.97 |  72.2
8 |  76.96 |  73.5
32 |  76.92 |  78.0
Bi-LoRA |  1 |  76.72 |  144.4
8 |  76.69 |  147.0
32 |  76.66 |  156.0

(e) Block-wise quantization. We use Bi-AdaptFormer and Bi-LoRA with $h=32$.

Table 4: Supplementary results on VTAB-1K benchmark.

###  5.3 Comparison with the State-of-the-Art

####  5.3.1 VTAB-1K benchmark

We compare our methods with full fine-tuning, linear probing (_i.e_., only
training the classification head), VPT [30], NOAH [77], SSF [42], Adapter-P
[59], BitFit [73], AdaptFormer [6], LoRA [26], Compacter [51], and FacT [32]
on VTAB-1K. All baselines use FP32 by default. The hidden dimension $h$ is set
to 8 for Adapter-P, AdaptFormer, and LoRA. The number of Kronecker products
and hidden dimensions are 4 and 32 for Compacter, respectively. For FacT, we
use FacT-TT with rank searched from {8, 16, 32} to adapt the MHSA blocks. The
settings of other baselines follow their original papers. As for our low-
precision adapters, we quantize the bit-width of AdaptFormer and LoRA to 1,
named Bi-AdaptFormer and Bi-LoRA, and report results with hidden dimensions
$h=1$ and $32$. All these methods use a ViT-B/16 [12] pre-trained on
supervised ImangeNet-21K as backbone. We train the models for 100 epochs with
AdamW optimizer.

Table 3 shows the full results on VTAB-1K. Since 1-bit adapters are much more
storage-efficient than their full-precision counterparts, Bi-AdaptFormer and
Bi-LoRA can use a larger hidden dimension while maintaining a smaller size.
Our Bi-AdaptFormer with $h=32$ beats all previous PET methods while using a
smaller storage size. Notably, Bi-AdaptFormer and Bi-LoRA achieve better
performance than Compacter and FacT-TT while being more parameter-efficient,
indicating that precision redundancy is more significant than rank redundancy
in adapters and thus quantization is a better solution than low-rank
parameterization for designing efficient adapters. Moreover, Bi-AdaptFormer
and Bi-LoRA with $h=1$ only store less than 5 KB of backbone parameters for
each task, while reaching performance better than VPT, BitFit, Compacter, and
full fine-tuning.

####  5.3.2 Few-shot learning on FGVC

On few-shot FGVC datasets, we compare Bi-AdaptFormer, the best-performing
quantized adapter in the experiments above, with other competitive baselines:
VPT-Deep, Adapter-P, LoRA, AdaptFormer, NOAH, and FacT-TT. The hidden
dimensions of Adapter-P, LoRA, and AdaptFormer, as well as the prompt length
of VPT-Deep, are all set to 8. The rank of FacT-TT is set to 16, and NOAH
follows the best recipes in [77]. As for Bi-AdaptFormer, we use a hidden
dimension of 32. Other settings are the same as in the VTAB-1K experiments.
Per-dataset results as well as the average results in the five settings are
shown in Figure 6.

Overall, our Bi-AdaptFormer outperforms all baselines on 5-task average
accuracy with the smallest size of trainable parameters. On FGVC-Aircraft,
Oxford-Pets, and Stanford Cars, Bi-AdaptFormer exhibits significant
performance improvement over the previously state-of-the-art PET methods. Only
on Food-101, Bi-AdaptFormer performs worse than FacT-TT and NOAH. Note that
Bi-AdaptFormer is about 3$\times$ and 19$\times$ more storage-efficient than
FacT and NOAH, respectively, and thus is more competitive under strict storage
restrictions.

###  5.4 Further Analysis

####  5.4.1 Quantizing classification head

As the size of adapters is compressed, the classification heads take up most
of the storage space, hindering further improvements in storage efficiency.
For example, on VTAB-1K, the average size of the classification heads is 0.14
MB, much larger than that of Bi-AdaptFormer modules. As shown in Table 4(a),
by quantizing the classification heads, Bi-AdaptFormer keeps state-of-the-art
results (76.89 _vs._ AdaptFormer’s 76.70) with checkpoint size smaller than
linear probing (76.7 KB _vs._ 140.8 KB). Note that linear probing is usually
considered as the efficiency lower bound of adaptation. Furthermore, Bi-
AdaptFormer and Bi-LoRA with $h=1$ and binary head achieve better performance
than full fine-tuning, linear probing, and VPT, but the average size of the
total checkpoints is only 6.8 KB and 9.2 KB, respectively, which are dozens of
times more storage-efficient than linear probing.

####  5.4.2 Computational efficiency

One of the design principles behind our quantization method is to ensure the
quantization operation has negligible computational cost during QAT. To
evaluate the efficiency of our proposed method, we conducted experiments to
study the training and inference time of different tuning methods, as
summarized in Table 4(c). For all baselines, we use the same settings as in
the VTAB-1K experiments. As for our Bi-AdaptFormer and Bi-LoRA, we set a
larger hidden dimension $h=32$. We find that the QAT and larger $h$ slightly
increase the training time of adapters. However, Bi-AdaptFormer and Bi-LoRA
are still faster than VPT, FacT, and full fine-tuning. At inference time,
since (Bi-)LoRA, and FacT can be re-parameterized and absorbed into the pre-
trained backbone, they do not incur additional computation.

####  5.4.3 Performance on other backbones

Note that our proposed quantization method is a plug-in strategy that can be
applied in any backbones and any adapters. Besides ViTs [12], there are also
other commonly used backbone networks in vision, such as hierarchical
transformers like Swin [44] and convolutional networks like ConvNeXt [45]. In
Table 4(b), we apply Bi-AdaptFormer to Swin-B and ConvNeXt-B, and compare it
with other baselines that can also be extended to these backbones. We notice
that Bi-AdaptFormer still achieves state-of-the-art results on VTAB-1K. Bi-
AdaptFormer with $h=32$ offers on-par or better performance than AdaptFormer
with $h=8$ while only using about $\frac{1}{8}$ of the storage size, which
verifies the generalization ability of binary adapters.

####  5.4.4 Ablation studies

We perform further ablation experiments on our low-bit adapters. The low-bit
adapters are fine-tuned via QAT, which has been proven to work better in low-
bit settings. To illustrate this, we compare our method with a PTQ method,
_i.e_., directly quantizing fine-tuned full-precision adapters using
$k$-means. We set $h=8$ for AdaptFormer and LoRA. As shown in Table 4(d), PTQ
obviously underperforms QAT, especially in 1-bit setting.

Moreover, since each weight matrix can be divided into several sub-matrices as
blocks to perform block-wise quantization, _i.e_., standardizing the
parameters and storing the $\mu$ and $\sigma$ of each block, we here compare
the performance of 1-bit adapters across different numbers of blocks. We set
$h=32$ for all methods. As shown in Table 4(e), since block-wise quantization
methods (# block $>1$) store more $\mu$ and $\sigma$ than our methods (# block
$=1$), block-wise quantization uses a larger storage size. However, block-wise
quantization does not demonstrate superiority over our methods.

##  6 Conclusion

In this work, we systematically revisit the parameter efficiency of adapter-
based PET through the lens of precision redundancy. Based on our observations,
we propose a plug-in strategy to train low-precision counterparts for existing
adapter-based methods. Through extensive experiments on more than 20 datasets,
we empirically verify the superiority of 1-bit adapters in terms of both
performance and parameter efficiency. Surprisingly, we find that 2.4 KB
parameters in backbone is almost sufficient to describe the difference between
the pre-trained ViT-B and a task-specific fine-tuned ViT-B, suggesting that
the intrinsic dimension of visual datasets is much smaller than what we used
to believe. Our work also brings quantization to PET, providing a general
solution to largely enhance the parameter efficiency of adapter-based PET
methods.

## References

  * [1] Ron Banner, Yury Nahshan, and Daniel Soudry.  Post training 4-bit quantization of convolutional networks for rapid-deployment.  In Proceedings of NeurIPS, 2019.
  * [2] Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, et al.  Deepmind lab.  arXiv preprint, arXiv:1612.03801, 2016.
  * [3] Yash Bhalgat, Jinwon Lee, Markus Nagel, Tijmen Blankevoort, and Nojun Kwak.  LSQ+: improving low-bit quantization through learnable offsets and better initialization.  In Proceedings of CVPR, 2020.
  * [4] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool.  Food-101–mining discriminative components with random forests.  In Proceedings of ECCV, 2014.
  * [5] Arnav Chavan, Zhuang Liu, Deepak K. Gupta, Eric P. Xing, and Zhiqiang Shen.  One-for-all: Generalized lora for parameter-efficient fine-tuning.  arXiv preprint, arXiv:2306.07967, 2023.
  * [6] Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo.  Adaptformer: Adapting vision transformers for scalable visual recognition.  In Proceedings of NeurIPS, 2022.
  * [7] Xiangning Chen, Cho-Jui Hsieh, and Boqing Gong.  When vision transformers outperform resnets without pre-training or strong data augmentations.  In Proceedings of ICLR, 2022.
  * [8] Gong Cheng, Junwei Han, and Xiaoqiang Lu.  Remote sensing image scene classification: Benchmark and state of the art.  Proc. IEEE, 2017.
  * [9] Minsik Cho, Keivan Alizadeh-Vahid, Saurabh Adya, and Mohammad Rastegari.  DKM: differentiable k-means clustering layer for neural network compression.  In Proceedings of ICLR, 2022.
  * [10] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, , and A. Vedaldi.  Describing textures in the wild.  In Proceedings of CVPR, 2014.
  * [11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In Proceedings of CVPR, 2009.
  * [12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In Proceedings of ICLR, 2021.
  * [13] Steven K. Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S. Modha.  Learned step size quantization.  In Proceedings of ICLR, 2020.
  * [14] Li Fei-Fei, Rob Fergus, and Pietro Perona.  Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories.  In Proceedings of CVPR workshops, 2004.
  * [15] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur.  Sharpness-aware minimization for efficiently improving generalization.  In Proceedings of ICLR, 2021.
  * [16] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun.  Vision meets robotics: The kitti dataset.  The International Journal of Robotics Research, 2013.
  * [17] Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W. Mahoney, and Kurt Keutzer.  A survey of quantization methods for efficient neural network inference.  arXiv preprint, arXiv:2103.13630, 2021.
  * [18] Song Han, Huizi Mao, and William J. Dally.  Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding.  In Proceedings of ICLR, 2016.
  * [19] Tianxiang Hao, Hui Chen, Yuchen Guo, and Guiguang Ding.  Consolidator: Mergable adapter with group connections for visual adaptation.  In Proceedings of ICLR, 2023.
  * [20] Ruidan He, Linlin Liu, Hai Ye, Qingyu Tan, Bosheng Ding, Liying Cheng, Jia-Wei Low, Lidong Bing, and Luo Si.  On the effectiveness of adapter-based tuning for pretrained language model adaptation.  In Proceedings of ACL/IJCNLP, 2021.
  * [21] Shwai He, Liang Ding, Daize Dong, Jeremy Zhang, and Dacheng Tao.  Sparseadapter: An easy approach for improving the parameter-efficiency of adapters.  In Findings of EMNLP, 2022.
  * [22] Xuehai He, Chunyuan Li, Pengchuan Zhang, Jianwei Yang, and Xin Eric Wang.  Parameter-efficient model adaptation for vision transformers.  In Proceedings of AAAI, 2023.
  * [23] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth.  Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
  * [24] Irina Higgins, Loïc Matthey, Arka Pal, Christopher P. Burgess, Xavier Glorot, Matthew M. Botvinick, Shakir Mohamed, and Alexander Lerchner.  beta-vae: Learning basic visual concepts with a constrained variational framework.  In Proceedings of ICLR, 2017.
  * [25] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.  Parameter-efficient transfer learning for NLP.  In Proceedings of ICML, 2019.
  * [26] Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  LoRA: Low-rank adaptation of large language models.  In Proceedings of ICLR, 2022.
  * [27] Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry.  Accurate post training quantization with small calibration sets.  In Proceedings of ICML, 2021.
  * [28] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In Proceedings of ICML, 2015.
  * [29] Yongkweon Jeon, Chungman Lee, Eulrang Cho, and Yeonju Ro.  Mr.biq: Post-training non-uniform quantization based on minimizing the reconstruction error.  In Proceedings of CVPR, 2022.
  * [30] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge J. Belongie, Bharath Hariharan, and Ser-Nam Lim.  Visual prompt tuning.  In Proceedings of ECCV, 2022.
  * [31] Shibo Jie and Zhi-Hong Deng.  Convolutional bypasses are better vision transformer adapters.  arXiv preprint, arXiv:2207.07039, 2022.
  * [32] Shibo Jie and Zhi-Hong Deng.  Fact: Factor-tuning for lightweight adaptation on vision transformer.  In Proceedings of AAAI, 2023.
  * [33] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick.  Clevr: A diagnostic dataset for compositional language and elementary visual reasoning.  In Proceedings of CVPR, 2017.
  * [34] Sangil Jung, Changyong Son, Seohyung Lee, JinWoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi.  Learning to quantize deep networks by optimizing quantization intervals with task loss.  In Proceedings of CVPR, 2019.
  * [35] Kaggle and EyePacs.  Kaggle diabetic retinopathy detection.  2015\.
  * [36] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang.  On large-batch training for deep learning: Generalization gap and sharp minima.  In Proceedings of ICLR, 2017.
  * [37] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In Proceedings of CVPR workshops, 2013.
  * [38] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [39] Yann LeCun, Fu Jie Huang, and Leon Bottou.  Learning methods for generic object recognition with invariance to pose and lighting.  In Proceedings of CVPR, 2004.
  * [40] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein.  Visualizing the loss landscape of neural nets.  In Proceedings of NeurIPS, 2018.
  * [41] Yanjing Li, Sheng Xu, Baochang Zhang, Xianbin Cao, Peng Gao, and Guodong Guo.  Q-vit: Accurate and fully quantized low-bit vision transformer.  In Proceedings of NeurIPS, 2022.
  * [42] Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang.  Scaling & shifting your features: A new baseline for efficient model tuning.  In Proceedings of NeurIPS, 2022.
  * [43] Xiaofan Lin, Cong Zhao, and Wei Pan.  Towards accurate binary convolutional neural network.  In Proceedings of NIPS, 2017.
  * [44] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In Proceedings of ICCV, 2021.
  * [45] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.  A convnet for the 2020s.  In Proceedings of CVPR, 2022.
  * [46] Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang-Ting Cheng.  Reactnet: Towards precise binary neural network with generalized activation functions.  In Proceedings of ECCV, 2020.
  * [47] Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao.  Post-training quantization for vision transformer.  In Proceedings of NeurIPS, 2021.
  * [48] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In Proceedings of CVPR, 2015.
  * [49] Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian.  Prompt distribution learning.  In Proceedings of CVPR, 2022.
  * [50] Gen Luo, Minglang Huang, Yiyi Zhou, Xiaoshuai Sun, Guannan Jiang, Zhiyu Wang, and Rongrong Ji.  Towards efficient visual adaption via structural re-parameterization.  arXiv preprint, arXiv:2302.08106, 2023.
  * [51] Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder.  Compacter: Efficient low-rank hypercomplex adapter layers.  In Proceedings of NeurIPS, 2021.
  * [52] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi.  Fine-grained visual classification of aircraft.  arXiv preprint, arXiv:1306.5151, 2013.
  * [53] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew B. Blaschko, and Andrea Vedaldi.  Fine-grained visual classification of aircraft.  arXiv preprint, arXiv:1306.5151, 2013.
  * [54] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan L. Yuille.  The role of context for object detection and semantic segmentation in the wild.  In Proceedings of CVPR, 2014.
  * [55] Markus Nagel, Rana Ali Amjad, Mart van Baalen, Christos Louizos, and Tijmen Blankevoort.  Up or down? adaptive rounding for post-training quantization.  In Proceedings of ICML, 2020.
  * [56] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng.  Reading digits in natural images with unsupervised feature learning.  In Proceedings of NIPS Workshops, 2011.
  * [57] M-E Nilsback and Andrew Zisserman.  A visual vocabulary for flower classification.  In Proceedings of CVPR, 2006.
  * [58] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar.  Cats and dogs.  In Proceedings of CVPR, 2012.
  * [59] Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych.  Adapterfusion: Non-destructive task composition for transfer learning.  In Proceedings of EACL, 2021.
  * [60] Haotong Qin, Ruihao Gong, Xianglong Liu, Xiao Bai, Jingkuan Song, and Nicu Sebe.  Binary neural networks: A survey.  Pattern Recognit., 2020.
  * [61] Haotong Qin, Ruihao Gong, Xianglong Liu, Mingzhu Shen, Ziran Wei, Fengwei Yu, and Jingkuan Song.  Forward and backward information retention for accurate binary neural networks.  In Proceedings of CVPR, 2020.
  * [62] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi.  Xnor-net: Imagenet classification using binary convolutional neural networks.  In Proceedings of ECCV, 2016.
  * [63] Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi.  Learning multiple visual domains with residual adapters.  In Proceedings of NIPS, 2017.
  * [64] Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry.  How does batch normalization help optimization?  In Proceedings of NeurIPS, 2018.
  * [65] Sheng Shen, Shijia Yang, Tianjun Zhang, Bohan Zhai, Joseph E. Gonzalez, Kurt Keutzer, and Trevor Darrell.  Multitask vision-language prompt tuning.  arXiv preprint, arXiv:2211.11720, 2022.
  * [66] Robin Strudel, Ricardo Garcia Pinel, Ivan Laptev, and Cordelia Schmid.  Segmenter: Transformer for semantic segmentation.  In Proceedings of ICCV, 2021.
  * [67] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In Proceedings of ICML, 2021.
  * [68] Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling.  Rotation equivariant CNNs for digital pathology.  arXiv preprint, arXiv:1806.03962, 2018.
  * [69] Peisong Wang, Qiang Chen, Xiangyu He, and Jian Cheng.  Towards accurate post-training network quantization via bit-split and stitching.  In Proceedings of ICML, 2020.
  * [70] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba.  Sun database: Large-scale scene recognition from abbey to zoo.  In Proceedings of CVPR, 2010.
  * [71] Jiwei Yang, Xu Shen, Jun Xing, Xinmei Tian, Houqiang Li, Bing Deng, Jianqiang Huang, and Xian-Sheng Hua.  Quantization networks.  In Proceedings of CVPR, 2019.
  * [72] Zhihang Yuan, Chenhao Xue, Yiqi Chen, Qiang Wu, and Guangyu Sun.  Ptq4vit: Post-training quantization for vision transformers with twin uniform quantization.  In Proceedings of ECCV, 2022.
  * [73] Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel.  Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models.  In Proceedings of ACL, 2022.
  * [74] Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy.  Unified vision and language prompt learning.  arXiv preprint, arXiv:2210.07225, 2022.
  * [75] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, André Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby.  The visual task adaptation benchmark.  arXiv preprint, arXiv:1910.04867, 2019.
  * [76] Jeffrey O. Zhang, Alexander Sax, Amir Roshan Zamir, Leonidas J. Guibas, and Jitendra Malik.  Side-tuning: A baseline for network adaptation via additive side networks.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Proceedings of ECCV, 2020.
  * [77] Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu.  Neural prompt search.  arXiv preprint, arXiv:2206.04673, 2022.
  * [78] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Learning to prompt for vision-language models.  arXiv preprint, arXiv:2109.01134, 2021.
  * [79] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Conditional prompt learning for vision-language models.  In Proceedings of CVPR, 2022.

## Appendix

##  1 Algorithm

Input: Bit-width $b$

Output: Sets $\mathcal{U}_{1},...,\mathcal{U}_{2^{b}}$ and codes
$c_{1},...,c_{2^{b}}$

1

2Initialize all $c_{i}$ to 0;

3 Set $c_{0}=-\infty,c_{2^{b}+1}=+\infty$;

4 while  _not converged_ do

5 $\forall
i,\mathcal{U}_{i}=(\frac{c_{i-1}+c_{i}}{2},\frac{c_{i}+c_{i+1}}{2}]$;

6 $\forall i,c_{i}=$ median of standard Gaussian within $\mathcal{U}_{i}$;

7

8 end while

9

10return _$\mathcal{U}_{1},...,\mathcal{U}_{2^{b}},c_{1},...,c_{2^{b}}$_ ;

11

12

Algorithm 1 $k$-medians on Standard Gaussian

##  2 Derivation of Eq. (8)

If $i=k$,

$\displaystyle\frac{\partial\hat{w_{i}}}{\partial w_{k}}$
$\displaystyle=\frac{\partial(\hat{w_{i}}^{\prime}\cdot\sigma+\mu)}{\partial
w_{i}}$ $\displaystyle=\sigma\cdot\frac{\partial\hat{w_{i}}^{\prime}}{\partial
w_{i}}+\hat{w_{i}}^{\prime}\cdot\frac{\partial\sigma}{\partial
w_{i}}+\frac{\partial\mu}{\partial w_{i}}$
$\displaystyle=\sigma\cdot\frac{\partial{w_{i}}^{\prime}}{\partial
w_{i}}+\hat{w_{i}}^{\prime}\cdot\frac{w_{i}^{\prime}}{m}+\frac{1}{m}$
$\displaystyle=\sigma\cdot\frac{\partial\frac{w_{i}-\mu}{\sigma}}{\partial
w_{i}}+\hat{w_{i}}^{\prime}\cdot\frac{w_{i}^{\prime}}{m}+\frac{1}{m}$
$\displaystyle=\sigma\cdot\frac{m-1-w_{i}^{\prime
2}}{m\sigma}+\hat{w_{i}}^{\prime}\cdot\frac{w_{i}^{\prime}}{m}+\frac{1}{m}$
$\displaystyle=1+\frac{w_{i}^{\prime}(\hat{w_{i}}^{\prime}-w_{i}^{\prime})}{m}$

Otherwise,

$\displaystyle\frac{\partial\hat{w_{i}}}{\partial w_{k}}$
$\displaystyle=\frac{\partial(\hat{w_{i}}^{\prime}\cdot\sigma+\mu)}{\partial
w_{k}}$ $\displaystyle=\sigma\cdot\frac{\partial\hat{w_{i}}^{\prime}}{\partial
w_{k}}+\hat{w_{i}}^{\prime}\cdot\frac{\partial\sigma}{\partial
w_{k}}+\frac{\partial\mu}{\partial w_{k}}$
$\displaystyle=\sigma\cdot\frac{\partial{w_{i}}^{\prime}}{\partial
w_{k}}+\hat{w_{i}}^{\prime}\cdot\frac{w_{k}^{\prime}}{m}+\frac{1}{m}$
$\displaystyle=\sigma\cdot\frac{\partial\frac{w_{i}-\mu}{\sigma}}{\partial
w_{k}}+\hat{w_{i}}^{\prime}\cdot\frac{w_{k}^{\prime}}{m}+\frac{1}{m}$
$\displaystyle=\sigma\cdot\frac{-1-w_{i}^{\prime}w_{k}^{\prime}}{m\sigma}+\hat{w_{i}}^{\prime}\cdot\frac{w_{k}^{\prime}}{m}+\frac{1}{m}$
$\displaystyle=\frac{w_{k}^{\prime}(\hat{w_{i}}^{\prime}-w_{i}^{\prime})}{m}$

##  3 Supplementary Experiments

###  3.1 Performance on Full CIFAR100

In the VTAB-1K benchmark, each task only contains 1,000 training samples. We
also conduct experiments on the full CIFAR100 [38] dataset, which has a larger
60,000-image training set. Following [42], we use a ViT-B/16 supervisedly pre-
trained on ImageNet-21K with AugReg as backbone, and train the model for 100
epochs with batch size 128. We use $h=8$ for AdaptFormer and $h=32$ for Bi-
AdaptFormer. All other settings are the same as in [42]. As shown in Table 5,
Bi-AdaptFormer still outperforms other baselines while using the smallest
storage size. Compared to AdaptFormer, Bi-AdaptFormer brings 0.4% performance
improvement and 8$\times$ more storage efficiency.

Method |  Top-1 Acc. |  Size (MB)
---|---|---
Full† |  93.82 |  334.0
Linear† |  88.70 |  0
BitFit† |  93.39 |  0.39
VPT-Shallow† |  90.38 |  0.59
VPT-Deep† |  93.17 |  1.76
SSF† |  93.99 |  0.78
AdaptFormer |  93.55 |  0.56
Bi-AdaptFormer |  93.95 ($\uparrow$ 0.40) |  0.071

Table 5: Accuracy on full CIFAR100. † denotes results reported in [42].

###  3.2 Semantic Segmentation

As for the dense prediction, we apply our method on Segmenter [66]. We use
DeiT-B/16384 [67] pre-trained on ImageNet-1K as encoder. Since each
segmentation task tunes an individual decoder upon the pre-trained encoder, we
use a single FC layer as a decoder which is much more lightweight than FCN
[48] and MaskTransformer [66]. We conduct experiments on Pascal-Context [54].
We evaluate three tuning paradigms: full fine-tuning, Adaptformer with $h=8$,
and Bi-AdaptFormer with $h=32$. The models are trained for 50 epochs with
batch size 64. As shown in Table 6, Bi-AdaptFormer still outperform
AdaptFormer in terms of both performance and efficiency.

Method |  mIoU (SS) |  Size (MB)
---|---|---
Full |  52.61 |  335.1
AdaptFormer |  51.57 |  0.56
Bi-AdaptFormer |  51.75 ($\uparrow$ 0.18) |  0.071

Table 6: Semantic segmentation on Pascal-Context. “Size” denotes the size of
trainable parameters in encoders. We report mIoU of single-scale inference on
validation set. Each method also has a decoder of 0.17MB.

###  3.3 Comparison with Other Quantization Methods

We compare our quantization method with existing binary neural networks –
XNOR-Net [62] and IR-Net [61]. Similar to Bi-AdaptFormer, we use the
quantization strategy of XNOR-Net and IR-Net to quantize (_i.e_., binarize)
the weights of adapters to 1 bit, and keep the full-precision activation,
called XNOR-AdaptFormer and IR-AdaptFormer, respectively. In experiments, we
find that IR-AdaptFormer cannot be trained stably without Batch Normalization
(BN) [28], so we add BN after each FC layer of the adapters. We set the hidden
dimension $h=8$ for AdaptFormer, and $h=32$ for Bi-AdaptFormer, XNOR-
AdaptFormer, and IR-AdaptFormer.

As shown in Table 7, since IR-AdaptFormer uses additional BN and XNOR-
AdaptFormer uses channel-wise scaling factors, their storage sizes are larger
than that of Bi-AdaptFormer. IR-AdaptFormer results in significant performance
degradation compared to AdaptFormer. We conjunct this is because IR-Net is
designed for traditional convolutional networks equiped with BN and is not
suitable for plug-in adapters in modern large-size vision architecture. XNOR-
AdaptFormer also underperforms AdaptFormer, demonstrating the necessity of
tailoring quantization strategy for adapters.

Method |  Avg. Acc. |  Size (MB)
---|---|---
Full |  68.9 |  334.0
Linear |  57.6 |  0
AdaptFormer |  76.70 |  0.56
IR-AdaptFormer |  72.13 ($\downarrow$ 4.57) |  0.14
XNOR-AdaptFormer |  76.34 ($\downarrow$ 0.36) |  0.11
Bi-AdaptFormer |  76.97 ($\uparrow$ 0.27) |  0.071

Table 7: Average accuracy on VTAB-1K.

##  4 Experimental Details

###  4.1 Datasets

See Table 9.

###  4.2 Pre-Trained Backbones

Model |  Pre-Training Dataset |  Size (M) |  Pre-Trained Weights
---|---|---|---
ViT-B/16 [12] |  ImageNet-21K |  85.8 |  checkpoint
Swin-B [44] |  ImageNet-21K |  86.7 |  checkpoint
ConvNeXt-B [45] |  ImageNet-21K |  87.6 |  checkpoint
AugReg ViT-B/16 [12] |  ImageNet-21K |  85.8 |  checkpoint
DeiT-B/16384 [45] |  ImageNet-1K |  86.1 |  checkpoint

Table 8: Pre-Trained backbones.

###  4.3 Code Implementation

We use _PyTorch_ and _timm_ to implement all experiments on NVIDIA RTX 3090
GPUs.

###  4.4 Data Augmentation

####  4.4.1 VTAB-1K

Following [30], we just resize the images to $224\times 224$.

####  4.4.2 Few-shot learning

Following [77], for training samples, we use color-jitter and
RandAugmentation; for validation/test samples, we resize them to $256\times
256$, crop them to $224\times 224$ at the center, and then normalize them with
ImageNet’s mean and standard deviation.

####  4.4.3 Full CIFAR100

Following [42], we use a strong augmentation in the fine-tuning setting of
[12]. Please refer to the official code of [12].

####  4.4.4 Semantic Segmentation

We completely follow the setting used in [66], which does mean substraction,
random resizing, random left-right flipping, and randomly crops large images
and pad small images to $480\times 480$.

###  4.5 Hyper-parameters

$s$ of (Bi-)AdaptFormer, (Bi-)LoRA, and FacT is searched from {0.01, 0.1, 1,
10, 100}. See Table 10 for other hyper-parameters. We basically follow the
hyper-parameters used by [77].

| Dataset | # Classes | Train | Val | Test
---|---|---|---|---|---
VTAB-1K [75]
Natural | CIFAR100 [38] | 100 | 800/1,000 | 200 | 10,000
Caltech101 [14] | 102 | 6,084
DTD [10] | 47 | 1,880
Oxford-Flowers102 [57] | 102 | 6,149
Oxford-Pets [58] | 37 | 3,669
SVHN [56] | 10 | 26,032
Sun397 [70] | 397 | 21,750
Specialized | Patch Camelyon [68] | 2 | 800/1,000 | 200 | 32,768
EuroSAT [23] | 10 | 5,400
Resisc45 [8] | 45 | 6,300
Retinopathy [35] | 5 | 42,670
Structured | Clevr/count [33] | 8 | 800/1,000 | 200 | 15,000
Clevr/distance [33] | 6 | 15,000
DMLab [2] | 6 | 22,735
KITTI-Dist [16] | 4 | 711
dSprites/location [24] | 16 | 73,728
dSprites/orientation [24] | 16 | 73,728
SmallNORB/azimuth [39] | 18 | 12,150
SmallNORB/elevation [39] | 18 | 12,150
Few-shot learning
| Food-101 [4] | 101 | 1/2/4/8/16 per class | 20,200 | 30,300
| Stanford Cars [37] | 196 | 1,635 | 8,041
| Oxford-Flowers102 [57] | 102 | 1,633 | 2,463
| FGVC-Aircraft [53] | 100 | 3,333 | 3,333
| Oxford-Pets [58] | 37 | 736 | 3,669
Supplementary experiments
| CIFAR100 (Full) [38] | 100 | 60,000 | - | 10,000
| Pascal-Context [54] | 59 | 4,996 | 5,104 | -

Table 9: Statistics of used datasets.

| optimizer | batch size | learning rate | weight decay | # epochs | lr decay | # warm-up epochs
---|---|---|---|---|---|---|---
VTAB-1K | AdamW | 64 | 1e-3 | 1e-4 | 100 | cosine | 10
Few-shot learning | AdamW | 64 | 5e-3 | 1e-4 | 100 | cosine | 10

Table 10: Hyper-parameters.