11institutetext: 1Intelligent Computing and Machine Learning Lab, School of
ASEE
Beihang University, Beijing 100191, China
2 Samsung R&D Institute China Beijing
18F TaiTangGong Plaza, Beijing, 100028, China
<EMAIL_ADDRESS>,
<EMAIL_ADDRESS>

# Auto-painter: Cartoon Image Generation from Sketch by Using Conditional
Generative Adversarial Networks

Yifan Liu1,2 Zengchang Qin1 Zhenbo Luo2 Hua Wang 2

###### Abstract

Recently, realistic image generation using deep neural networks has become a
hot topic in machine learning and computer vision. Images can be generated at
the pixel level by learning from a large collection of images. Learning to
generate colorful cartoon images from black-and-white sketches is not only an
interesting research problem, but also a potential application in digital
entertainment. In this paper, we investigate the sketch-to-image synthesis
problem by using conditional generative adversarial networks (cGAN). We
propose the auto-painter model which can automatically generate compatible
colors for a sketch. The new model is not only capable of painting hand-draw
sketch with proper colors, but also allowing users to indicate preferred
colors. Experimental results on two sketch datasets show that the auto-painter
performs better that existing image-to-image methods.

##  1 Introduction

Human beings possess a great cognitive capability of comprehend black-and-
white cartoon sketches. Our mind can create realistic colorful images when see
black-and-white cartoons. It needs a great artistic talent to choose
appropriate colors, also with proper changes in light and shade to create
vivid cartoon images. It is not easy for ordinary people to do so. How to
automatically paint the sketch into colorful cartoon images is a useful
application for digital entertainment. In this work, we are interested in
solving this problem by employing deep neural networks with designed
constraints to transfer the line draft into specific cartoon style.
Practically, the new model can make up for ordinary people’s artistic talent
and even inspire the artists to create new images. Ideally, readers may have
the freedom to choose to generate different style of cartoons based on their
own tastes of colors.

Cartoon image generation from sketches can be regarded as an image synthesis
problem. Previously, many non-parametric models were proposed [8, 1, 6] by
matching the sketch to a database of existing image fragments. Recently,
numerous image synthesis methods based on deep neural networks have emerged
[20, 13, 7, 16]. These methods can generate detailed images of the real-world,
such as faces, bedrooms, chairs and handwritten numbers. As the photo-
realistic images are full of sharp details, the results may suffer from being
blurry [12], noisy [7] and objects being wobbly [2]. What’s more, the outputs
of the network are hard to be controlled because the generator samples from a
random vector in low dimension and the model has too much freedom. Several
recent approaches have explored the applicability of controllable deep
synthesis in different applications, for instance, the super resolution
problem [11], semantic labels tagging for objects [4], grayscale image
colorization [25]. Especially the sketch-to-image problem [21], the control
signals are relatively sparse, more ill-posed than colorization problem based
on grayscale, it requires a model to synthesize image details beyond what is
contained in the input. The network should learn the high frequency textures
for the details of the scene elements as well as high-level image styles.

In this paper, we propose a learning model called _auto-painter_ that can
automatically generate painted cartoon images from a sketch based on
conditional Generative Adversarial Networks (cGANs). The cartoon images have
more artistic color collocations (e.g. green hair, purple eyes) that may
require more constraints in modeling. Constraints including _total variance
loss_ , _pixel loss_ and _feature loss_ are used in training the generator in
order to generate more artistic color collocations. We also introduce the
color control for the auto-painter based on [21] to allow users paint their
favorite colors. Figure 1 shows an example of generated cartoon images from a
sketch, the results of auto-painter (with and without color control) are
compared to the ground truth in the middle.

Figure 1: Examples of painted cartoon images generated from a sketch by auto-
painter.

##  2 Related Work

Generative Adversarial Networks Generative adversarial networks (GANs) were
recently regarded as a breakthrough in machine learning [7, 16], it consists
of two ‘adversarial’ modules: a generative module $G$ that captures the data
distribution, and a discriminative module $D$ that estimates the probability
that a sample came from the training data rather than $G$. Both $G$ and $D$
could be deep neural networks. In image synthesis with GANs, the generator
attempts to produce a realistic image from an input random vector to fool the
simultaneously adversarial trained discriminator, which tries to distinguish
whether its input image is from the training set or the generated set. It
corresponds to a minimax two-player game. The generator has benefited from
convolutional decoder networks, it can go back to the work using deep
convolutional decoder networks to generate realistic images.
Conditional GANs GANs are unconditioned generative models that learn a mapping
from random noise vector $z$ to output $y$: $G:z\rightarrow y$. In contrast,
conditional GANs (cGANs) learn a mapping from observed input $x$ and random
noise vector $z$, to $y$: $G:\\{x,z\\}\rightarrow y$ [14]. Several works
studied different cGAN where the generator is conditioned on inputs such as
text [17], labels [5] and other forms of the images [10] to generate ‘fake’
images. The discriminator needs to distinguish the authentic and fake pairs of
images [17]. In this paper, both of generator and discriminator are
conditioned on the input sketch in order to get a better supervised
performance.
Image-to-image The ‘pix2pix’ method proposed in [10] is a ‘U-net’ architecture
[18] to deal with general image-to-image transfer which allows the decoder to
be conditioned on encoder layers to get more information. They investigated
different kinds of image-to-image transfer tasks that include transforming the
image of daylight to night, produce city images from map, and even synthesize
shoes and handbags from designer’s sketches. Sangkloy _et al._[21] proposed
the ‘scribbler’ by allowing users to scribble over the sketch to indicate
preferred color for objects. In our model, we also introduce the function of
interactive color control to generate different style of cartoon images.

##  3 Method

Auto-painter is a supervised learning model, given a black-and-white sketch,
the model can generate a painted colorful image based on given sketch-image
pairs in the training data. We employ a feed-forward deep neural network as
generator to get a quick response in test. The generator takes the sketch as
input and outputs a colorful cartoon image of the same resolution at pixel-
level.

###  3.1 Network Structure

Many previous solutions [3, 21, 23] used a pure encoder-decoder network. The
input goes through a series of down sampling steps to a lower dimension, then
gets some non-liner transformation with a fully connected layer, and finally
gets up sampling to the present output size. Such a structure may cause
information loss when passing through layers. Especially in the sketch-to-
image problem, we need to keep the edges as the most important information
from the input to ensure the quality of the output image. Instead of the
encoder-decoder structure, we employ the ‘U-net’ [18], by concatenating layers
in encoder to the corresponding layers of decoder. As we can see from Figure
2, in order to decoding out low-level information of sketch, we concatenate
encoder layer $A$ to the decoding layer $A^{\prime}$ for generating the final
colorful cartoon image, where $A$ contains the sketch edge information and
$A^{\prime}$ mainly contains trained color painting information.

Figure 2: Left-hand side: the U-net structure of generator: the yellow blocks
represent layers in encoder, and blue blocks are decoder layers. In each layer
of decoding, the corresponding layers of encoder are concatenated to current
layer to decode the next layer. E.g., $B$ and $B^{\prime}$ are concatenated to
obtain $A^{\prime}$ through deconvolution. The right-hand side figure is the
patchGAN discriminator: input of discriminator is either the pair of sketch
(yellow block) and real target image (red block), or the pair of sketch and
generated image (blue block).

The discriminator only has the encoder units comparing to the generator, it
aims to classify whether the input sketch-image pair is ‘real’ or ‘fake’. The
network is trained to maximize the classification accuracy. As we can see from
Figure 2 (right), the output of the discriminator is a matrix of probability,
in which each element gives the probability of being real for a pair of
correspond patches sampled using Markov random field or PatchGAN [10]. In our
research, we take the patch size of $70\times 70$ and output a probability
matrix of $30\times 30$. The convolutional layers between the input and the
output extract the high-level features of the input pairs.

###  3.2 Loss Function

The task of auto-painter is similar to colorization [9] but much harder,
because a gray scale image has more information than a simple line sketch.
Facing this ill-posed problem, we may need more constraints for better
performance. The cGANs learn a mapping from observed $x$ and random noise
vector $z$ to the output image $y$. The objective (loss) function of general
cGANs can be expressed by:

$\begin{split}\mathop{{\rm{min}}}\limits_{G}\mathop{\max}\limits_{D}V(G,D)=&{\mathbb{E}_{x,y\thicksim{p_{data}}(x,y)}}[\log
D(x,y)]+\\\
&{\mathbb{E}_{x\thicksim{p_{data}}(x),z\thicksim{p_{data}}(z)}}[\log(1-D(x,G(x,z)))]\end{split}$
(1)

where the generator $G$ tries to minimize the objective function while an
adversarial $D$ tries to maximize it. As for the auto-painter, we update the
weights of the encoder-unit in discriminator to maximize $V(G,D)$, while $x$
is the input sketch, $y$ is the target (the colored cartoon image) and
$G(x,z)$ is the generated image. The discriminator $D$ outputs the probability
to classify the ‘real’ and ‘fake’ input pair as mentioned in Section 3.1.

The generator is simultaneous trained to minimize the generative loss:

$L_{G}={\mathbb{E}_{x\thicksim{p_{data}}(x),z\thicksim{p_{data}}(z)}}[\log(1-D(x,G(x,z)))]$
(2)

where we provide a Gaussian noise $z$ as an input of generator, in addition to
$x$. The adversarial training encourages more variations and vividness in
generated images. Previous approaches of conditional GANs [10, 15] have found
it beneficial to mix the GAN objective with a more traditional loss functions.
In this paper, we also use the $L_{1}$ distance to describe the pixel-level
loss ${L_{p}}$ in our model:

${L_{p}}={\mathbb{E}_{x,y\thicksim{p_{data(x,y)},z\thicksim{p}_{{}_{data}}(z)}}}\left[\left\lVert
y-G(x,z)\right\rVert_{1}\right]$ (3)

${L_{p}}$ represents for the difference between the generate image and the
ground truth at pixel level. Except for the pixel loss, we also employ a pre-
trained Visual Geometry Group net (VGGnet) [22] to extract high-level
information of the image. The feature loss ${L_{f}}$ is defined as the $L_{2}$
distance in a feature space:

${L_{f}}={\mathbb{E}_{x,y\thicksim{p_{data(x,y)},z\thicksim{p_{data}}(z)}}}\left[\left\lVert{\phi_{j}}(y)-{\phi_{j}}(G(x,z))\right\rVert_{2}\right]$
(4)

${\phi_{j}}$ is the activation function of the $j$th layer of the 16-layer VGG
network (VGG16) pre-trained on the ImageNet dataset [19]. In this paper,
through the trail study, we choose the outputs of the fourth layer of VGG16
($j=4$). In order to avoid a color mutation in the output, we also add a total
variation loss ${L_{tv}}$. For the ${L_{tv}}$ can constrain the pixel changes
in the generated results and encourages smoothness [11].

${L_{tv}}=\sqrt{{{({y_{i+1,j}}-{y_{i,j}})}^{2}}+{{({y_{i,j+1}}-{y_{i,j}})}^{2}}}$
(5)

Finally, the objective function is defined as the following:

$L={w_{p}}{L_{P}}+{w_{f}}{L_{f}}+{w_{G}}{L_{G}}+{w_{tv}}{L_{tv}}$ (6)

where ${w_{p}}$, ${w_{f}}$, ${w_{G}}$ and ${w_{tv}}$ are the weights for loss
functions we discussed above. We adjust them to control the importance of each
part. This loss function ensures that the generated image contains both pixel
level details of sketch as well as high-level information of painted colors.
By minimizing this objective function, we can learn complex correlations
between sketch and its associated painting style.

##  4 Experimental Studies

###  4.1 Datasets

In order to train the auto-painter model, we collect a large number of cartoon
pictures from the Internet with a crawler. Previous researches [21, 25] prefer
lower resolution for the synthetic images. To be more practical as most
cartoons are in higher resolution, our training images are all resized to
$512\times 512$. This will increase the difficulty of training, but we deepen
the neural network and use more constraints to stabilize the training. It is
not easy to find paired sketch-image cartoons for training, we employ the
boundary detection filter XDoG [24] to extract sketches. By adjusting
parameters $\gamma$ of the XDoG, we can obtain sketches with different levels
of details (e.g, see Figure 3). We include all four types of sketches in
training. All the data will be available to public at the final version of
this paper. After preprocessing (resize, truncate to squared images and
extract sketches), two cartoon datasets are obtained: _Minions_ and
_Japanimation_. _Minions_ contains 1100 pictures of different colored minions.
_Japanimation_ contains 60000 pictures of Japanese anime, and most of them are
characters. We take 90 percent of data for training and 10 percent for test.

Figure 3: Generated black-white sketches using XDoG filter [24]. From left to
the right: the parameter $\gamma$ is set to 0.96, 0.97, 0.98, 0.99 in order to
obtain sketches with different level of high-frequency information.

###  4.2 Evaluation Metrics

It is well-known that evaluating the quality of a synthesized image is an open
and difficult problem [25]. The traditional metrics such as per-pixel mean-
squared error can not assess high-level features of the output image. In order
to evaluate the visual quality of the auto-painter and compare to the pix2pix
model, we design a test called ‘like vs dislike’ inspired by the ‘real vs
fake’ test for evaluating realistic image synthesis models in [10, 25].
Because the ultimate goal of the auto-painter is to produce favorite cartoon
images. Volunteers were presented with a series of painted images generated by
different algorithms and asked to choose the best one and the worst one. In
this research, we design this evaluation metrics mainly to analyze the effect
of the different parts of the loss function. Unlike [25], we post four
pictures simultaneously for two seconds and complete the evaluation of all
four algorithms instead of evaluating them one by one. This allows the
participants to conclude after a more comprehensive comparison. No feedbacks
are given during the test because all images are generated ‘fake’ ones, and
there is no prejudice on auto-painter’ s results. In our paper, 55 volunteers
were asked to take part in the evaluation.

In order to quantify the evaluation, we define the _popularity index_ as the
following: for the $i^{th}$ picture generated by the $j^{th}$ algorithm, the
popularity $pop_{ij}$ is define by:
${pop_{ij}}=\log(\frac{{n^{like}_{ij}+c}}{{n^{dislike}_{ij}+c}})$, where
$i\in[1,40]$, $j\in[1,4]$, $c=1$ in our test and
${n^{like}_{ij}}({n^{dislike}_{ij}})$ represent the number of subjects who
choose ‘like(dislike)’ on this ${picture_{ij}}$. $c$ is set for smoothness.
Therefore, the popularity of ${j^{th}}$ algorithm is:
${pop_{j}}=\log(\frac{{\sum\limits_{i}{n_{ij}^{like}}+c}}{{\sum\limits_{i}{n_{ij}^{dislike}}+c}})$
We consider both ${pop_{j}}$ and ${pop_{ij}}$ because for different kinds of
sketches, a specific algorithm may output different quality images. The
variance of ${pop_{ij}}$ represents for the stability of an algorithm $j$.

###  4.3 Experimental Results

In order to verify the importance of constraints ($L_{f}$ and $L_{tv}$)
introduced in Section 3.2, we make ablation studies to isolate the effect of
each term. The Pix2pix model which has no feature loss and total variation
(tv) loss is used as the baseline method to compare to the results of the
auto-painter. All the parameters of training details are the same including
the random seeds, learning rate, epoches and the batch size. The only
difference are the objective function and the detail weights are shown in
Figure 4. It also gives qualitative results of these variations on the
Minions. Without tv loss (${w_{tv}}=0$), the result images tend to dissolve to
background and look messy. If we set ${w_{f}}=0$, which means we do not
consider the feature loss, the details of result image become blurry. The
relative importance of ${L_{p}}$ and ${L_{G}}$ have already been discussed in
[10]. Considering both terms together may produce quality results.

Figure 4: Results of auto-painter and pix2pix with different parameter
settings.

Table 1 show result comparisons of the pix2pix model [10], pix2pix $+$tv loss,
pix2pix $+$feature loss and the auto-painter. From the table, we can see that
auto-painter is the most popular algorithm among all candidates while pix2pix
is the worst. Adding a tv loss improves the popularity but decreases the
stability as the variance is larger. Adding feature loss improves the ability
of the system to adapt for diversity of sketches. However, combine the tv loss
and the feature loss together we will have the largest popularity and the
smallest variance. In summary, auto-painter considers all these constraints to
fit all sorts of sketches (with or without background, hand-draw or modified)
and it can achieve the best performance.

Method | pix2pix [10] | tv loss | feature loss | auto-painter
---|---|---|---|---
${n^{like}}$ | 249 | 304 | 687 | 960
${n^{dislike}}$ | 1147 | 698 | 219 | 136
${pop_{j}}$ | -1.524 | -0.829 | 1.140 | 1.948
${variance(pop_{ij})}$ | 1.319 | 1.519 | 1.110 | 0.888
${mean(pop_{ij})}$ | -1.549 | -0.675 | 1.227 | 1.873
Table 1: Results of _‘like vs dislike’_ test with 55 volunteers.

Given an input image of resolution $512\times 512$, the auto-painter can
generate a painted colorful output image within 1 second, which enables
instant feedback to design an interactive image editing tools. The auto-
painter trained on the Minions provides a tool for users to design virtual
images of ‘minions style’. As it shown in the left-hand side of Figure 5,
based on the given initial sketch, one can modify the original sketch, the
glass or the gesture, etc. It is interesting that even on the simple hand draw
sketches, the auto-painter can result in ‘minions style’ images (right-hand
side of Figure 5).

Figure 5: Cartoon images generated from modified and amateur hand-draw
sketches.

###  4.4 Color Controlled Sketch Synthesis

When a gray-scale sketch is transferred to a real-colored photo, the color
scheme is relatively fixed because it shows the color of the real-world, such
as grass is green, sky is blue and sand is golden. There may be a mass of
alternative color scheme in a complex training set. Especially in the cartoon
image dataset, the color of a girl’s eyes may be green or purple, which is
uncommon in the real world. If we take the example of the minions, the model
can learn the correlation between the minion shapes and the yellow color. Such
correlations are too hard to learn on the dataset like _Japanimation_. When
observing a complex cartoon sketch, different users may prefer different color
schemes. But the generator trained with black-and-white sketch can only choose
a specific color schemes. So we train a color control model to satisfy
different users aesthetic needs. In order to train the auto-painter to
recognize color control signals, we can add color block to sketches. We blur
the ground-truth image with a Gaussian filter and sample a random number of
points at random start locations and grow them to become a color block along
the diagonal. It is obvious that each particular color block should not across
different color regions , so we put a constraint when growing the block that
if the difference between the mean color of the current block and the next
sample block exceeds a threshold, we stop the growing.

In the experiment on the _Japanimation_ data set, we need the model to
generate more colorful images comparing to the Minions data. Follow the
description above, we generate sketches with color control blocks of
_Japanimation_. Figure 6 shows the results of reconstructing _Japanimation_
based on synthesis color control blocks. We can see that for the complicated
task, the auto-painter identify the edges of the images successfully and make
feasible color schemes based on the color block. The color of the control
block is rendered around in a particular region naturally. We also test the
model with user input and the colors of the blocks deviate a lot from the
colors in the ground-truth image (see Figure 7). Nevertheless, the auto-
painter is able to paint the color in object boundaries and distinguish
different parts of the sketch.

Figure 6: Results on the _Japanimation_. The color blocks are generated by
random sampling.  Figure 7: Examples of painted sketch by auto-painter with
color control: (a), (c), (e), (g) are generated cartoon images with
corresponding inputs with indicated color (b), (d), (f), (h). The last row is
the ground truth from the training data.

##  5 Conclusion

In this paper, we proposed the auto-painter model to solve the sketch-to-image
problem. Our approach was based on conditional GAN with ‘U-net’ structure that
allows the output image has both low level information of sketch as well as
learned high-level color information. We introduce more constraints based on
the pix2pix model to obtain better painting performance. We also trained the
auto-painter to adapt to color control, so that our network can adapt the
synthesis results to satisfy users with different color taste. Empirical
results show that the auto-painter can generate high-quality cartoon images on
two given datasets, and it has the best popularity measure based on a
subjective test with volunteers. Despite the promising results, our current
system suffers from the difficulties of adjusting parameters just like other
deep learning models. Complex network structure may result in low training
speed. For example, on our _Japanimation_ dataset, it takes 2 to 3 days to get
fairly good results with a single Tesla K80 GPU. In the future work, we will
focus on improving the system performance as well the speed in order to build
an interactive toolkit for users.

## References

  * [1] Barnes, C., Shechtman, E., Finkelstein, A., Dan, B.G.: Patchmatch:a randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 28(3), 1–11 (2009)
  * [2] Denton, E.L., Chintala, S., Szlam, A., Fergus, R.: Deep generative image models using a laplacian pyramid of adversarial networks. In: NIPS (2015)
  * [3] Dong, C., Chen, C.L., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. ECCV 8692, 184–199 (2014)
  * [4] Dong, H., Neekhara, P., Wu, C., Guo, Y.: Unsupervised image-to-image translation with generative adversarial networks. CoRR abs/1701.02676 (2017)
  * [5] Eigen, D., Fergus, R.: Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In: IEEE International Conference on Computer Vision. pp. 2650–2658 (2015)
  * [6] Freeman, W.T., Jones, T.R., Pasztor, E.C.: Example-based super-resolution. IEEE Computer Graphics & Applications 22(2), 56–65 (2002)
  * [7] Goodfellow, I.J., Pougetabadie, J., Mirza, M., Xu, B., Wardefarley, D., Ozair, S., Courville, A., Bengio, Y., Ghahramani, Z., Welling, M.: Generative adversarial nets. Advances in Neural Information Processing Systems 3, 2672–2680 (2014)
  * [8] Hays, J., Efros, A.A.: Scene completion using millions of photographs. Commun. ACM 51, 87–94 (2007)
  * [9] Iizuka, S., Simo-Serra, E., Ishikawa, H.: Let there be color!: joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification. ACM Trans. Graph. 35, 110:1–110:11 (2016)
  * [10] Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. CoRR abs/1611.07004 (2016)
  * [11] Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: ECCV (2016)
  * [12] Kingma, D.P., Welling, M.: Auto-encoding variational bayes. In: Conference proceedings: papers accepted to the International Conference on Learning Representations (ICLR) 2014 (2014)
  * [13] Lee, H., Grosse, R., Ranganath, R., Ng, A.Y.: Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In: International Conference on Machine Learning, ICML 2009, Montreal, Quebec, Canada,. pp. 609–616 (2009)
  * [14] Mirza, M., Osindero, S.: Conditional generative adversarial nets. CoRR abs/1411.1784 (2014)
  * [15] Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: Feature learning by inpainting. CVPR pp. 2536–2544 (2016)
  * [16] Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. CoRR abs/1511.06434 (2015)
  * [17] Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., Lee, H.: Generative adversarial text to image synthesis. In: International Conference on Machine Learning (2016)
  * [18] Ronneberger, O., Fischer, P., Brox, T.: U-Net: Convolutional Networks for Biomedical Image Segmentation (2015)
  * [19] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M.: Imagenet large scale visual recognition challenge. International Journal of Computer Vision 115(3), 211–252 (2015)
  * [20] Salakhutdinov, R., Hinton, G.: Deep boltzmann machines. Journal of Machine Learning Research 5(2), 1967 – 2006 (2009)
  * [21] Sangkloy, P., Lu, J., Fang, C., Yu, F., Hays, J.: Scribbler: Controlling deep image synthesis with sketch and color. CoRR abs/1612.00835 (2016)
  * [22] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556 (2014)
  * [23] Wang, X., Gupta, A.: Generative image modeling using style and structure adversarial networks. CoRR abs/1603.05631 (2016)
  * [24] Winnemöller, H., Kyprianidis, J.E., Olsen, S.C.: Xdog: An extended difference-of-gaussians compendium including advanced image stylization. Computers & Graphics 36, 740–753 (2012)
  * [25] Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. CoRR abs/1603.08511 (2016)


# Sensitivity-Aware Visual Parameter-Efficient Fine-Tuning

Haoyu He1 Jianfei Cai1 Jing Zhang2 Dacheng Tao2 Bohan Zhuang1
1 ZIP Lab, Monash University 2 The University of Sydney
Corresponding author. E-mail: $\tt<EMAIL_ADDRESS>
###### Abstract

Visual Parameter-Efficient Fine-Tuning (PEFT) has become a powerful
alternative for full fine-tuning so as to adapt pre-trained vision models to
downstream tasks, which only tunes a small number of parameters while freezing
the vast majority ones to ease storage burden and optimization difficulty.
However, existing PEFT methods introduce trainable parameters to the same
positions across different tasks depending solely on human heuristics and
neglect the domain gaps. To this end, we study where to introduce and how to
allocate trainable parameters by proposing a novel Sensitivity-aware visual
Parameter-efficient fine-Tuning (SPT) scheme, which adaptively allocates
trainable parameters to task-specific important positions given a desired
tunable parameter budget. Specifically, our SPT first quickly identifies the
sensitive parameters that require tuning for a given task in a data-dependent
way. Next, our SPT further boosts the representational capability for the
weight matrices whose number of sensitive parameters exceeds a pre-defined
threshold by utilizing existing structured tuning methods, e.g., LoRA [23] or
Adapter [22], to replace directly tuning the selected sensitive parameters
(unstructured tuning) under the budget. Extensive experiments on a wide range
of downstream recognition tasks show that our SPT is complementary to the
existing PEFT methods and largely boosts their performance, e.g., SPT improves
Adapter with supervised pre-trained ViT-B/16 backbone by 4.2% and 1.4% mean
Top-1 accuracy, reaching SOTA performance on FGVC and VTAB-1k benchmarks,
respectively. Source code is at https://github.com/ziplab/SPT.

##  1 Introduction

Figure 1: (a) Existing PEFT methods, such as Adapter [22] introduce trainable
parameters to the same positions for all downstream tasks. These methods
design task-agnostic positions to employ trainable parameters relying on
heuristics and neglect consideration of the distinct domain gaps and
characteristics for the downstream tasks. (b) Our Sensitivity-aware visual
Parameter-efficient fine-Tuning (SPT) introduces trainable parameters to the
task-specific important positions and allocates them with both unstructured
and structured tuning granularities, simultaneously. For structured tuning,
SPT can exploit any existing structured tuning methods, such as LoRA [23] or
Adapter [22]. Red lines and blocks represent trainable parameters and modules,
while blue lines represent frozen parameters.

To effectively adapt the pre-trained representations to the downstream tasks,
the de-facto choice is full fine-tuning, which initializes the model with the
pre-trained weights and tunes all the parameters. However, vanilla full fine-
tuning needs to store a separate instance of parameters for each task and each
deployment scenario. It can be extremely storage-intensive as the storage cost
grows linearly with the number of possible cases, considering there are vast
varieties of downstream tasks and dynamic deployment environments, especially
when deploying the large vision models [13, 34, 58] to mobile systems. For
example, even storing a single large pre-trained ViT-H [13] model on a local
disk requires at least 2.3GB, while the Top-10 U.S. apps required only
collectively 2.2GB in May 2021.111https://sensortower.com/blog/ios-app-size-
growth-2021

Notably, an emerging solution is to replace vanilla fine-tuning with visual
Parameter-Efficient Fine-Tuning (PEFT) [24, 10, 65, 25], which only tunes a
small number of trainable parameters while freezing the vast majority ones
that are shared by multiple tasks. As PEFT approaches exhibit less than 1% of
trainable parameters, the storage burden is largely alleviated. Another
attractive property of PEFT is that tuning fewer parameters eases the
optimization difficulty and mitigates the overfitting issue when adapting
large pre-trained models on the target dataset, thereby achieving comparable
or even better performance than vanilla fine-tuning [24]. Although promising,
the existing PEFT approaches introduce trainable parameters to the same
positions for all downstream tasks, relying on human heuristics and neglecting
task-specific domain gaps and characteristics, which limits their performance.
For instance, in a task-agnostic manner, Prompt Tuning-deep [24] and Adapter
[22] respectively add trainable parameters to multi-head self-attention and
feed-forward network layers for all distinct tasks as depicted in Figure 1
(a).

To address this fundamental challenge, we explore where to introduce and how
to allocate trainable parameters under a desired parameter budget by
presenting a novel Sensitivity-aware visual Parameter-efficient fine-Tuning
(SPT) scheme that identifies the task-specific important positions to
adaptively allocate trainable parameters. Since the pre-trained weights at
distinct positions have varying contributions for different downstream tasks
[60, 28, 39], we first propose a new criterion to quickly identify the task-
specific sensitive parameters that require tuning in a data-dependent way.
Inspired by model pruning metrics [46, 37, 3, 4], we propose to measure the
parameter sensitivity with the loss reduction when being tuned, which can be
approximated by a first-order Taylor expansion derived within a single forward
and backward pass ahead of fine-tuning in one-shot. Our sensitivity criterion
is simple and effective, which can identify the task-specific important
positions to introduce trainable parameters for any backbone quickly. For
instance, calculating the sensitivity for ViT-B/16 backbone takes only 5.5
seconds with a single GPU on any of the VTAB-1k datasets.

With our criterion, we empirically observe that the proportions of the
sensitivity parameters for each block indeed vary markedly across different
tasks in Section 4.4. To allocate the trainable parameters under a desired
trainable parameter budge, an intuitive solution is to directly tune the most
sensitive weight connections, which we name unstructured tuning. Despite its
simplicity and flexibility, unstructured tuning only tunes a few parameters
which still lacks representational capability and is challenging to bridge the
domain gap. To this end, we propose to further incorporate structured tuning
to replace unstructured tuning at the sensitive weight matrices whose numbers
of sensitive parameters exceed a pre-defined threshold to improve the
representational capability under a similar parameter budget. Structured
tuning can be implemented by any parameter-efficient structured tuning methods
[23, 10, 25, 24] that directly adjust the hidden representations, _e.g_.,
inserting an adapter module sequentially after the sensitive weight matrices.
Therefore, our SPT adaptively combines both unstructured and structured tuning
granularity and allocates trainable parameters with high flexibility and
representational capability for each distinct downstream task.

This paper has the following key contributions. 1) We make the pioneering
exploration to identify the task-specific important positions under the PEFT
setting, which is fast, effective, versatile to be applied to various
backbones with different pre-training strategies, and orthogonal to the
existing PEFT methods. 2) Based on the sensitivity criterion, we propose a
trainable parameter allocation strategy that adaptively combines both
unstructured and structured tuning under a desired parameter budget to achieve
high flexibility, large capacity, and favorable trade-off between parameter
efficiency and accuracy. 3) Extensive experiments on a total of 24 downstream
recognition tasks with both plain and hierarchical vision Transformer
backbones under supervised and self-supervised pre-trainings show that our SPT
is complementary to the existing PEFT methods and boosts their performance by
large margins. For instance, SPT improves Adapter [22] by 4.2% mean Top-1
accuracy, outperforming the SOTA PEFT methods on the FGVC benchmark.

##  2 Related Work

Parameter-efficient fine-tuning. Full fine-tuning is the most predominant
approach when adapting a large-scale pre-trained model to downstream tasks,
where the model is initialized from the pre-trained weights with all
parameters trainable. Yet, when a model becomes larger, parameter-efficient
fine-tuning [29, 30] is highly desirable, which tunes only a tiny portion of
parameters to alleviate the storage burden. The general PEFT approaches can be
categorized into addition-based PEFT methods and reparameterization-based PEFT
methods.

_Addition-based PEFT_ attaches additional trainable parameters to the backbone
and only tunes these parameters. Apart from Prompt tuning [24] and Adapter
[22], recent addition-based methods study connecting or combining existing
PEFT methods. For instance, He _et al_. [19] connect Prompt tuning and Adapter
and provide a unified view that all PEFT approaches share the same design to
adjust the hidden representations. Zhang _et al_. [65] search for the optimal
configurations to combine multiple PEFT approaches following once-for-all
scheme [6, 56]. Since the additional parameters require extra computations
compared to full fine-tuning, a few recent works [48, 50] design specific
architectures to avoid storing the intermediate activations, thereby
alleviating the fine-tuning memory cost. However, it is noteworthy that
enhancing training efficiency is not the primary objective of our work.

_Reparameterization-based PEFT_ aims to avoid extra computational costs by
tuning parameters that are inherently in or can be reparameterized into the
backbone during inference. Prior works select the parameters that are
inherently in the backbone, including the bias terms [61], the last several
layers [60, 5], and weight connections [15, 66]. To reparameterize new
parameters into the backbone [18, 31], representative work LoRA [23] optimizes
two low-rank matrices which can be further merged into the weight matrices. In
contrast to the aforementioned works, we argue the importance of tuning
parameters at task-specific important positions and quickly identify them with
our proposed parameter sensitivity criterion before tuning, which is
complementary to and provides valuable guidance for the existing PEFT methods.
Moreover, our SPT can also be inference-efficient when implementing structured
tuning with any reparameterization-based structured tuning method. Recently,
SSF [31] is proposed to introduce trainable scaling and shifting parameters
that can be absorbed into the previous linear layers. However, it cannot scale
to higher trainable parameter budgets and requires a complex and time-
consuming hyper-parameter search for learning rate, weight decay, and drop-
path rate on each individual dataset, thus is not directly comparable to our
method.

Task-specific transfer learning. The effectiveness of transferring pre-trained
models to downstream tasks strongly depends on the relationship between the
source and target tasks [45, 55, 28, 42]. This has motivated the community to
explore the optimal pre-training data [12, 59], model [49, 40], and weights
[17, 57] for the target task. To seek suitable _task-specific pre-training
data_ , Cui  _et al_. [12] select the source domain data from the top-k most
similar classes measured by Earth Mover’s Distance; Yoon  _et al_. [59] weight
each class in the source domain with reinforcement learning; and Puigcerver
_et al_. [43] first train a diverse set of experts and then select the most
relevant expert for each target task. Another line of work selects a suitable
_pre-trained model for the target task_ ahead of fine-tuning by measuring the
transferability of pre-trained models to the target domain with interclass
covariance between the source data and target classes [2] or conditional
cross-entropy [49] between the source and target labels. Considering the
transferability of the feature representations at distinct layers for the same
pre-trained model is different [60, 39], recent works [16, 47] endeavour
_transfer task-specific weights_ by freezing some pre-trained weights and
fine-tuning the rest. For example, the task-specific fine-tuned weights are
selected by learning a policy network with Gumbel-Softmax [17], optimizing a
sparse mask with $L_{0}$ norm [15], and learning binary gates for each
parameter [66]. Our SPT also adaptively selects task-specific parameters. In
contrast to the previous work, we 1) derive task-specific important positions
prior to fine-tuning with only a single forward and backward pass, which is
computationally efficient; 2) mask the gradients for insensitive parameters in
unstructured tuning with fixed binary masks, thereby having more affordable
fine-tuning memory than optimizing learnable binary masks in [15, 66].
Moreover, we are pioneering work to adaptively allocate task-specific
trainable parameters with both fine-grained unstructured and coarse-grained
structured tuning granularities to achieve both high flexibility and
representational capability.

##  3 Method

Our sensitivity-aware visual parameter-efficient fine-tuning consists of two
stages. In the first stage, SPT measures the task-specific sensitivity for the
pre-trained parameters (Section 3.1). Based on the parameter sensitivity and a
given parameter budget, SPT then adaptively allocates trainable parameters to
task-specific important positions (Section 3.2).

###  3.1 Task-specific Parameter Sensitivity

Recent research has observed that pre-trained backbone parameters exhibit
varying feature patterns [44, 38] and criticality [64, 9] at distinct
positions. Moreover, when transferred to downstream tasks, their efficacy
varies depending on how much pre-trained features are reused and how well they
adapt to the specific domain gap [60, 28, 39]. Motivated by these
observations, we argue that not all parameters contribute equally to the
performance across different tasks in PEFT and propose a new criterion to
measure the sensitivity of the parameters in the pre-trained backbone for a
given task.

Specifically, given the training dataset ${\mathcal{D}}_{t}$ for the $t$-th
task and the pre-trained model weights
${\bm{w}}=\left\\{w_{1},w_{2},\ldots,w_{N}\right\\}\in{\mathbb{R}}^{N}$ where
$N$ is the total number of parameters, the objective for the task is to
minimize the empirical risk: $\min_{{\bm{w}}}E({\mathcal{D}}_{t},{\bm{w}})$.
We denote the parameter sensitivity set as
${\mathcal{S}}=\\{s_{1},\ldots,s_{N}\\}$ and the sensitivity $s_{n}$ for
parameter $w_{n}$ is measured by the empirical risk difference when tuning it:

$\displaystyle
s_{n}=E({\mathcal{D}}_{t},{\bm{w}})-E({\mathcal{D}}_{t},{\bm{w}}\mid
w_{n}=w_{n}^{*}),$ (1)

where $w_{n}^{*}=\underset{w_{n}}{\rm argmin}(E({\mathcal{D}}_{t},{\bm{w}}))$.
We can reparameterize the tuned parameters as
$w_{n}^{*}=w_{n}+\Delta_{w_{n}}$, where $\Delta_{w_{n}}$ denotes the update
for $w_{n}$ after tuning. Here we individually measure the sensitivity of each
parameter, which is reasonable given that most of the parameters are frozen
during fine-tuning in PEFT. However, it is still computationally intensive to
compute Eq. (1) for two reasons. Firstly, getting the empirical risk for $N$
parameters requires forwarding the entire network $N$ times, which is time-
consuming. Secondly, it is challenging to derive $\Delta_{w_{n}}$, as we have
to tune each individual $w_{n}$ until convergence.

Algorithm 1 Computing task-specific parameter sensitivities

Input: Pre-trained model with network parameters ${\bm{w}}$, training set
${\mathcal{D}}_{t}$ for the $t$-th task, and number of training samples $C$
used to calculate the parameter sensitivities

Output: Sensitivity set ${\mathcal{S}}=\\{s_{1},\ldots,s_{N}\\}$

Initialize ${\mathcal{S}}=\\{0\\}^{N}$

for $i\in\\{1,\ldots,C\\}$ do

Get the $i$-th training sample of ${\mathcal{D}}_{t}$

Compute loss $E$

Compute gradients ${\bm{g}}$

for $n\in\\{1,\ldots,N\\}$ do

Update sensitivity for the $n$-th parameter: $s_{n}=s_{n}+g_{n}^{2}$

end for

end for

Figure 2: Overview of our trainable parameter allocation strategy. With the
parameter sensitivity set ${\mathcal{S}}$, we first get the top-$\tau$
sensitive parameters. Instead of directly tuning these sensitive parameters,
we also boost the representational capability by replacing unstructured tuning
with structured tuning at sensitive weight matrices that have a large number
of sensitive parameters, which can be implemented by an existing structured
tuning method, _e.g_., LoRA [23] and Adapter [22]. Red lines and blocks
represent trainable parameters and modules, while blue lines represent frozen
parameters.

To overcome the first barrier, we simplify the empirical loss by approximating
$s_{n}$ in the vicinity of ${\bm{w}}$ by its first-order Taylor expansion

$\displaystyle s_{n}^{(1)}=-g_{n}\Delta_{w_{n}},$ (2)

where the gradients ${\bm{g}}=\partial E/\partial{\bm{w}}$, and $g_{n}$ is the
gradient of the $n$-th element of ${\bm{g}}$. To address the second barrier,
following [33, 8], we take the one-step unrolled weight as the surrogate for
$w_{n}^{*}$ and approximate $\Delta_{w_{n}}$ in Eq. (2) with a single step of
gradient descent. We can accordingly get $s_{n}^{(1)}\approx
g_{n}^{2}\epsilon$, where $\epsilon$ is the learning rate. Since $\epsilon$ is
the same for all parameters, we can eliminate it when comparing the
sensitivity with the other parameters and finally get

$\displaystyle s_{n}^{(1)}\approx g_{n}^{2}.$ (3)

Therefore, the sensitivity of a parameter can be efficiently measured by its
potential to reduce the loss on the target domain. Note that although our
criterion draws inspiration from pruning work [37], it is distinct from it.
[37] measures the parameter importance by the squared change in loss when
removing them, _i.e_.,
$\left(E({\mathcal{D}}_{t},{\bm{w}})-E({\mathcal{D}}_{t},{\bm{w}}\mid
w_{n}=0)\right)^{2}$ and finally derives the parameter importance by
$\left(g_{n}w_{n}\right)^{2}$, which is different from our formulations in
Eqs. (1) and (3).

In practice, we accumulate ${\mathcal{S}}$ from a total number of $C$ training
samples ahead of fine-tuning to generate accurate sensitivity as shown in
Algorithm 1, where $C$ is a pre-defined hyper-parameter. In Section 4.3, we
show that employing only 400 training samples is sufficient for getting
reasonable parameter sensitivity, which requires only 5.5 seconds with a
single GPU for any VTAB-1k dataset with ViT-B/16 backbone [13].

###  3.2 Adaptive Trainable Parameters Allocation

Our next step is to allocate trainable parameters based on the obtained
parameter sensitivity set ${\mathcal{S}}$ and a desired parameter budget
$\tau$. A straightforward solution is to directly tune the top-$\tau$ most
sensitive unstructured connections (parameters) while keeping the rest frozen,
which we name unstructured tuning. Specifically, we select the top-$\tau$ most
sensitive weight connections in ${\mathcal{S}}$ to form the sensitive weight
connection set ${\mathcal{T}}$. Then, for a weight matrix
${\bm{W}}\in{\mathbb{R}}^{d_{\rm in}\times d_{\rm out}}$, we can get a binary
mask ${\bm{M}}\in{\mathbb{R}}^{d_{\rm in}\times d_{\rm out}}$ computed by

$\vspace{-0.5em}{\begin{array}[]{ll}\small\begin{aligned}
{\bm{M}}^{j}=\left\\{\begin{array}[]{ll}1~{}~{}~{}~{}~{}{\bm{W}}^{j}\in{\mathcal{T}}\\\
0~{}~{}~{}~{}~{}{\bm{W}}^{j}\notin{\mathcal{T}}\end{array}\right.\end{aligned},\small\end{array}}$
(4)

where ${\bm{W}}^{j}$ and ${\bm{M}}^{j}$ are the $j$-th element in ${\bm{W}}$
and ${\bm{M}}$, respectively. Accordingly, we can train the sensitive
parameters by gradient descent and the updated weight matrix can be formulated
as
${\bm{W}}^{\prime}\leftarrow{\bm{W}}-\epsilon{\bm{g}}_{{\bm{W}}}\odot{\bm{M}}$,
where ${\bm{g}}_{{\bm{W}}}$ is the gradient for ${\bm{W}}$.

However, considering PEFT approaches generally limit the proportion of
trainable parameters to less than 1%, tuning only a small number of
unstructured weight connections might not have enough representational
capability to handle the downstream datasets with large domain gaps from the
source pre-training data. Therefore, to improve the representational
capability, we propose to replace unstructured tuning with structured tuning
at the sensitive weight matrices that have a high number of sensitive
parameters. To preserve the parameter budget, we can implement structured
tuning with an existing efficient structured tuning PEFT method [23, 10, 22,
25] that learns to directly adjust all hidden dimensions at once. We depict an
overview of our trainable parameter allocation strategy in Figure 2. For
example, we can employ the low-rank reparameterization trick LoRA [23] to the
sensitive weight matrices and the one-step update for ${\bm{W}}$ can be
formulated as

$\vspace{-0.4em}{\begin{array}[]{ll}\small\begin{aligned}
{\bm{W}}^{\prime}=\left\\{\begin{array}[]{ll}{\bm{W}}+{\bm{W}}_{\rm
down}{\bm{W}}_{\rm up}&~{}~{}\text{ if }~{}~{}\sum_{j=0}^{d_{\rm in}\times
d_{\rm out}}{\bm{M}}^{j}\geq\sigma_{\rm opt}\\\
{\bm{W}}-\epsilon{\bm{g}}_{{\bm{W}}}\odot{\bm{M}}&~{}~{}{\rm
otherwise}\end{array}\right.\end{aligned},\small\end{array}}$ (5)

where ${\bm{W}}_{\rm down}\in{\mathbb{R}}^{d_{\rm in}\times r}$ and
${\bm{W}}_{\rm up}\in{\mathbb{R}}^{r\times d_{\rm out}}$ are two learnable
low-rank matrices to approximate the update of ${\bm{W}}$ and rank $r$ is a
hyper-parameter where $r\ll{\rm min}(d_{\rm in},d_{\rm out})$. In this way, we
perform structured tuning on ${\bm{W}}$ when its number of sensitive
parameters exceeds $\sigma_{\rm opt}$, whose value depends on the pre-defined
type of structured tuning method. For example, since implementing structured
tuning with LoRA requires $2\times d_{\rm in}\times d_{\rm out}\times r$
trainable parameters for each sensitive weight matrix, we set $\sigma_{\rm
LoRA}\leftarrow 2\times d_{\rm in}\times d_{\rm out}\times r$ to ensure that
the number of trainable parameters introduced by structured tuning is always
equal to or lower than the number of sensitive parameters.

In this way, our SPT adaptively incorporates both structured and unstructured
tuning granularities to enable higher flexibility and stronger
representational power, simultaneously. In Section 4.3, we show that
structured tuning is important for the downstream tasks with larger domain
gaps and both unstructured and structured tuning contribute clearly to the
superior performance of our SPT.

ViT-B/16 |  Total | FGVC | VTAB-1k
---|---|---|---
(85.8M) |  params | Tuned / Total | Mean Acc. | Tuned / Total | Natural | Specialized | Structured | Mean Acc.
Full | 24.02$\times$ | 100% | 88.5 | 100% | 75.9 | 83.4 | 47.6 | 69.0
Addition-based methods
Mlp-3 | 1.35$\times$ | 1.50% | 79.8 | 1.42% | 67.8 | 72.8 | 30.6 | 57.1
Prompt-shallow | 1.04$\times$ | 0.31% | 84.6 | 0.13% | 76.8 | 79.7 | 47.0 | 67.8
Prompt-deep | 1.18$\times$ | 0.98% | 89.1 | 1.14% | 78.5 | 82.4 | 55.0 | 72.0
Adapter-8 | 1.06$\times$ | 0.39% | 85.5 | 0.23% | 79.0 | 84.1 | 58.5 | 73.9
Adapter-32 | 1.19$\times$ | 0.95% | 85.6 | 0.71% | 79.6 | 84.0 | 58.3 | 74.0
AdaptFormer | 1.09$\times$ | 0.44% | 85.1 | 0.36% | 80.6 | 85.4 | 58.5 | 74.8
NOAH | - | - | - | 0.52% | 80.2 | 84.9 | 61.3 | 75.5
SPT-Adapter (Ours) | 1.08$\times$ | 0.41% |  89.5 | 0.30% | 81.3 | 85.3 | 60.8 | 75.8
SPT-Adapter (Ours) | 1.10$\times$ | 0.47% |  89.8 | 0.44% | 82.0 | 85.8 | 61.4 | 76.4
Reparameterization-based methods
Linear | 1.02$\times$ | 0.12% | 79.3 | 0.04% | 68.9 | 77.2 | 26.8 | 57.6
Partial-1 | 3.00$\times$ | 8.38% | 82.6 | 8.30% | 69.4 | 78.5 | 34.2 | 60.7
Bias | 1.05$\times$ | 0.13% | 88.4 | 0.13% | 73.3 | 78.3 | 44.1 | 65.2
LoRA-8 | 1.07$\times$ | 0.55% | 86.0 | 0.23% | 79.5 | 84.6 | 60.5 | 74.9
LoRA-16 | 1.18$\times$ | 0.90% | 84.8 | 0.69% | 79.8 | 84.9 | 60.2 | 75.0
SPT-LoRA (Ours) | 1.08$\times$ | 0.41% |  89.3 | 0.31% | 81.5 | 85.6 | 60.7 | 75.9
SPT-LoRA (Ours) | 1.15$\times$ | 0.60% |  90.1 | 0.63% | 81.9 | 85.9 | 61.3 | 76.4

Table 1: Comparisons on FGVC and VTAB-1k [62] benchmarks using supervised pre-
trained ViT-B/16 backbone pre-trained on ImageNet-21k. “Total params” denotes
the ratio of the total number of parameters needed for all downstream tasks
relative to the one for the pre-trained backbone, and “Tuned/Total” denotes
the fraction of trainable parameters. Top-1 accuracy (%) is reported. The best
result is in bold, and the second-best result is underlined.

##  4 Experiments

###  4.1 Experimental Setup

Datasets and metrics. We evaluate our SPT on total $24$ downstream tasks in
two groups following [24]. 1) FGVC is a benchmark for fine-grained visual
classification, including CUB-200-2011 [54], NABirds [52], Oxford Flowers
[41], Stanford Cars [14], and Stanford Dogs [26] datasets. Each FGVC dataset
contains between 55 to 200 classes and a few thousand images for train,
validation, and test. We follow the validation splits in [24] if the
validation set is unavailable. 2) VTAB-1k [62] is a large-scale transfer
learning benchmark consisting of a collection of 19 visual classification
tasks. VTAB-1k can further be divided into three groups, including Natural
tasks with natural images, Specialized tasks with images captured by
specialized equipment, e.g., medical images, and Structured tasks with images
mostly generated from synthetic environments. Each of the VTAB-1k dataset has
only 800 training and 200 validation samples, while the test set sizes vary.
We use top-1 accuracy (%) averaged within each group as our main metric
following [24].

Pre-trained backbones. We conduct experiments on the plain vision Transformer
backbone ViT-B/16 [13] that is pre-trained on ImageNet [27] with different
pre-training strategies following [24], including supervised pre-training and
self-supervised pre-training with MAE [20] and MoCo v3 [11] following [24]. We
also conduct experiments on the representative hierarchical vision Transformer
backbone Swin-B [34] under supervised pre-training.

Contenders. We categorize the baseline methods into addition-based and
reparameterization-based PEFT methods as introduced in Section 2. Unless
specified, all baseline methods keep the backbone frozen. Addition-based
methods require extra computations during inference, including Mlp-$k$,
Prompt-shallow [24], Prompt-deep [24], Adapter-$k$ [22], AdaptFormer [10], and
NOAH [65]. Reparameterization-based methods have no additional computational
overhead during inference, including Linear, Partial-$k$, Bias [61], and
LoRA-$k$ [23]. Here $k$ represents the number of bottleneck dimension in
Adapter-$k$ and LoRA-$k$. We also compare with full fine-tuning which is
denoted by Full. We introduce the details of these methods in the
supplementary material.

We also introduce two variants of our SPT: addition-based SPT-Adapter and
reparameterization-based SPT-LoRA. SPT-Adapter directly adjusts the hidden
representations that are computed by sensitive weight matrices following [22],
while SPT-LoRA approximates updating the sensitive weight matrices following
[23]. For the two variants, we follow the exact weight initializations that
are described in [23] and follow [65] to set the bottleneck dimension as 8.

Implementation details. Following [65], we use the AdamW optimizer [36] with
cosine learning rate decay and set the batch size, learning rate, and weight
decay as 64, $1\times 10^{-3}$, and $1\times 10^{-4}$, respectively. We also
follow [65] for the standard data augmentation pipeline. We set the number of
training samples $C$ used to calculate our parameter sensitivities in
Algorithm 1 universally to be 800 for the main experiments.

ViT-B/16 | Total | VTAB-1k MAE | VTAB-1k MoCo v3
---|---|---|---
(85.8M) | Params | Tuned / Total | Natural | Specialized | Structured | Mean Acc. | Tuned / Total | Natural | Specialized | Structured | Mean Acc.
Full | 38.02$\times$ | 100% | 59.3 | 79.7 | 53.8 | 64.3 | 100% | 72.0 | 84.7 | 42.0 | 69.6
Addition-based methods
Adapter-8 | 1.08$\times$ | 0.23% | 57.2 | 78.4 | 54.7 | 63.4 | 0.23% | 27.6 | 70.9 | 48.4 | 49.0
Adapter-32 | 1.28$\times$ | 0.95% | 55.3 | 78.8 | 53.3 | 62.5 | 0.99% | 74.2 | 82.7 | 47.7 | 68.2
Prompt-shallow | 1.02$\times$ | 0.12% | 40.0 | 69.7 | 27.5 | 45.7 | 0.12% | 67.3 | 82.3 | 37.6 | 62.4
Prompt-deep | 1.05$\times$ | 0.23% | 36.0 | 60.6 | 26.6 | 41.1 | 0.07% | 70.3 | 83.0 | 42.4 | 65.2
SPT-Adapter (Ours) | 1.07$\times$ | 0.26% | 64.8 | 82.4 | 60.4 |  69.2 | 0.08% | 76.1 | 84.9 | 60.1 | 73.7
SPT-Adapter (Ours) | 1.13$\times$ | 0.41% | 65.6 | 82.7 | 60.7 |  69.7 | 0.30% | 76.6 | 85.0 | 61.7 | 74.4
Reparameterization-based methods
Linear | 1.02$\times$ | 0.04% | 18.9 | 52.7 | 23.7 | 32.1 | 0.04% | 67.5 | 81.1 | 30.3 | 59.6
Partial-1 | 4.16$\times$ | 8.30% | 58.4 | 78.3 | 47.6 | 61.5 | 8.30% | 72.3 | 84.6 | 47.9 | 68.3
Bias | 1.06$\times$ | 0.13% | 54.6 | 75.7 | 47.7 | 59.3 | 0.13% | 72.9 | 81.1 | 53.4 | 69.2
LoRA-8 | 1.08$\times$ | 0.23% | 57.5 | 77.7 | 57.7 | 64.3 | 0.23% | 21.2 | 66.7 | 45.1 | 44.3
LoRA-16 | 1.28$\times$ | 0.69% | 57.3 | 77.1 | 59.9 | 64.8 | 0.69% | 16.0 | 64.0 | 48.7 | 42.9
SPT-LoRA (Ours) | 1.11$\times$ | 0.29% | 63.8 | 81.6 | 60.0 |  68.5 | 0.30% | 76.5 | 85.4 | 63.0 | 75.0
SPT-LoRA (Ours) | 1.23$\times$ | 0.69% | 65.4 | 82.4 | 61.5 |  69.8 | 0.50% | 76.5 | 86.0 | 63.6 | 75.3

Table 2: Comparisons on VTAB-1k [62] benchmark using self-supervised ViT-B/16
backbone pre-trained by MAE [20] and MoCo v3 [11]. “Total params” denotes the
ratio of the total number of parameters needed for all downstream tasks
relative to the one for the pre-trained backbone, and “Tuned/Total” denotes
the fraction of trainable parameters. Top-1 accuracy (%) is reported. The best
result is in bold, and the second-best result is underlined.

Method |  | Tuned /
---
Total
Natural | Specialized | Structured |  | Mean /
---
Acc.
Full | 100% | 79.1 | 86.2 | 59.7 | 75.0
Addition-based methods
Mlp-3 | 1.60% | 73.6 | 75.2 | 35.7 | 61.5
Prompt-shallow | 0.04% | 79.9 | 82.5 | 37.8 | 66.7
Prompt-deep | 0.23% | 76.8 | 84.5 | 53.4 | 71.6
Adapter-8 | 1.18% | 81.7 | 87.3 | 61.2 | 76.7
SPT-Adapter (ours) | 0.33% | 83.0 | 87.3 | 62.1 | 77.5
Reparameterization-based methods
Linear | 0.04% | 73.5 | 80.8 | 33.5 | 62.6
Partial-1 | 2.15% | 73.1 | 81.7 | 35.0 | 63.3
LoRA-8 | 1.18% | 81.7 | 87.2 | 60.1 | 76.3
SPT-LoRA (ours) | 0.49% | 83.1 | 87.4 | 60.4 | 77.2

Table 3: Comparisons on VTAB-1k [62] benchmark with supervised pre-trained
Swin-B [34]. “Tuned/Total” denotes the fraction of trainable parameters. Top-1
accuracy (%) is reported. The best result is in bold.

###  4.2 Main Results

We compare our method with the baseline methods under different backbones,
pre-training strategies, and tasks.

Visual recognition with ViT backbone. First, _our proposed SPT-Adapter and
SPT-LoRA achieve the best performance under different trainable parameter
budgets_ with supervised pre-trained ViT-B/16 backbone, as shown in Table 1
and Figure 3 (a). For instance, SPT-Adapter outperforms the SOTA method NOAH
by a clear margin of 0.9% mean top-1 accuracy over the 19 VTAB-1k datasets
with fewer trainable parameters. We speculate that our SPT variants allocate
trainable parameters at task-specific positions compared to the heuristically
selected positions in the baseline methods, which contributes to our superior
performance. We also observe that our SPT-Adapter and SPT-LoRA achieve large
performance gains over Adapter and LoRA variants, respectively. For example,
SPT-Adapter and SPT-LoRA with 0.41% trainable parameters respectively improve
Adapter-8 and LoRA-8 significantly by 4.0% and 3.3% mean accuracy on the FGVC
benchmark. This suggests that identifying task-specific important positions
and combining both unstructured and structured tuning granularities with SPT
are complementary to the existing PEFT methods and boost their performance.

Second, _SPT variants outperform baseline methods and full fine-tuning by
significant margins with the self-supervised pre-trained ViT-B/16 backbones._
As shown in Table 2, existing PEFT approaches exhibit inferior results than
full fine-tuning with the self-supervised pre-trained backbones MAE and MoCo
v3. It is worth noting that previous PEFT methods yield inconsistent results
with the backbones of different pre-training strategies. In contrast, SPT
variants consistently outperform full fine-tuning. In particular, SPT-Adapter
achieves remarkable 5.8% and 5.5% mean top-1 accuracy gains over the best-
performing baseline method on VTAB-1k benchmark with only 0.26% and 0.08%
trainable parameters for MAE and MoCo v3 pre-trained backbones, respectively.
Moreover, our observation in supplementary material suggests that self-
supervised pre-trained ViT backbones have more diverse sensitivity
distributions and a higher variance in sensitivity across different tasks than
the supervised pre-trained one. This leads to the conjecture that baseline
methods which assign trainable parameters to the same positions for all tasks
may fail to mitigate the distinct domain gaps in individual downstream
datasets, whereas our SPT allocates trainable parameters to task-specific
positions accurately.

Visual recognition with Swin and ConvNeXt backbones. From Table 3, we observe
that our SPT-LoRA and SPT-Adapter also achieve SOTA performance with Swin-B
backbone on all dataset groups. We also follow VPT [24] to apply SPT variants
to ResNet-alike architecture ConvNeXt-Base [35] and report the results in
Table 4. We observe that SPT variants achieve better trade-offs between
accuracy and parameter efficiency than the baseline methods. The results
further demonstrate the versatility and effectiveness of our SPT.

Semantic segmentation. We follow VPT [24] to conduct semantic segmentation on
ADE20k dataset. Following the settings of [24], we apply our SPT to SETR-PUP
[67] framework with ViT-L backbone and only allocate trainable parameters to
the backbone with the head fully fine-tuned. We report the mIoU results in
Table 5. Notably, SPT-LoRA and SPT-Adapter outperform the baseline methods by
large margins, indicating our SPT can be generalized to the semantic
segmentation task. We report the results for conducting only structured tuning
for both SPT-LoRA and SPT-Adapter as it yields higher mIoU.

Method | Tuned/Total | Natural | Specialized | Structured | Mean Acc.
---|---|---|---|---|---
FULL | 100% | 78.0 | 83.7 | 60.4 | 74.0
LoRA | 0.79% | 82.2 | 84.7 | 64.1 | 77.0
Adapter | 0.47% | 83.1 | 84.9 | 64.6 | 77.5
SPT-LoRA (ours) | 0.57% | 83.4 | 86.7 | 65.9 | 78.7
SPT-Adapter (ours) | 0.36% | 83.7 | 86.2 | 65.3 | 78.4

Table 4: Comparisons on VTAB-1k [62] benchmark with supervised pre-trained
ConvNeXt-Base [35] backbone. “Tuned/Total” denotes the fraction of trainable
parameters. Top-1 accuracy (%) is reported. The best result is in bold.

Method | mIoU-s.s. (%) | mIoU-m.s. (%) | Trainable Param.
---|---|---|---
VPT | 44.0 | 45.6 | 15.8M
LoRA | 43.9 | 45.9 | 14.7M
Adapter | 44.4 | 46.6 | 14.6M
SPT-LoRA (ours) | 45.4 | 47.5 | 14.6M
SPT-Adapter (ours) | 45.2 | 47.2 | 14.6M

Table 5: Semantic Segmentation: Comparisons on ADE20k [68] val with SETR [67]
on ViT-L backbone. We report both single-scale (s.s.) and multi-scale (m.s.)
mIoU results. The best result is in bold.  Figure 3:  (a) Accuracy vs.
parameter efficiency with supervised pre-trained ViT-B/16 backbone on VTAB-1k
[62]. SPT variants perform favorably against the other PEFT approaches and are
more scalable. (b) Applying other PEFT structured tuning methods [24, 22, 23]
to the task-specific sensitive weight matrices (denoted by TSM) identified by
our criterion with supervised pre-trained ViT-B/16 backbone on VTAB-1k. Our
criterion brings consistent performance gain. (c) Domain vs. performance gaps
for different dataset groups in VTAB-1k [62]. The blue bars show the domain
gaps between the source domain (ImageNet [27]) and target domains, which are
measured by Maximum Mean Discrepancy (MMD) distance [51]. The red line
represents the performance gaps between SPT-Adapter w/o unstructured and w/o
structured, using supervised pre-trained ViT-B/16 backbone. The dataset groups
are Natural, Specialized, and Structured. Structured tuning is important for
achieving good performance on Structured datasets with larger domain gaps.

###  4.3 Ablation Study

Effect of the sensitivity criterion. We investigate the effectiveness of our
sensitivity criterion on VTAB-1k by employing structured tuning methods from
[24, 22, 23] to the task-specific sensitive weight matrices. Note that we do
not conduct unstructured tuning to ensure fair comparisons. The results are
presented in Figure 3 (b). Our criterion brings consistent 1.1%, 1.6%, and
0.8% performance gains for Prompt-deep, Adapter-32, and LoRA-16, respectively,
which demonstrates the effectiveness and versatility of our sensitivity
criterion to identify accurate task-specific important positions.

Effect of structured and unstructured tuning. We investigate the effectiveness
of unstructured and structured tuning individually on VTAB-1k. The results are
presented in Table 6. We start by applying Adapter-8 to the sensitive weight
matrices identified by our sensitivity criterion (SPT-Adapter w/o
unstructured). We observe that our sensitivity criterion boosts the
performance of all the dataset groups by clear margins, which again
demonstrates the importance of our sensitivity criterion. Next, we observe
that allocating the trainable parameters to the unstructured sensitive weight
connections also brings accuracy improvement to the Natural and Specialized
datasets from Adapter-8. However, we find that structured tuning is especially
important for achieving good performance on Structured datasets. To further
investigate this phenomenon, we observe that Structured datasets have larger
domain gaps from the pre-training source domain [27] compared to Natural and
Specialized datasets as visualized in Figure 3 (c). We hence conjecture that
structured tuning has a higher representational capability than unstructured
tuning which facilitates mitigating the large domain gaps during fine-tuning
(see the supplementary material for visual examples). Finally, we observe that
incorporating both structured and unstructured tuning at task-specific
important positions achieves the highest performance on all dataset groups.

Effect of number of training samples $C$ to get parameter sensitivity. We
investigate the effect of the number of training images $C$ for calculating
our parameter sensitivity (Algorithm 1 of the main paper). We randomly sample
training samples and report the mean results over three runs in Table 8. We
find that our SPT is robust to the number of training samples $C$ and randomly
sampling 400 out of a total of 800 training samples is sufficient to obtain
accurate task-specific important positions, _e.g_., calculating the
sensitivity for ViT-B/16 backbone takes only 5.5 seconds with a single GPU on
any of the VTAB-1k datasets and this computation is required only once.

Figure 4: Parameter sensitivity patterns under 0.4M trainable parameter budget
for ViT-B/16 backbone with different pre-training strategies on three sample
tasks from VTAB-1k [63]. The proportions exhibit task-specific varying
patterns in terms of network depth (upper figures) and task-agnostic similar
patterns in terms of operations (lower figures).

Method |  | Tuned /
---
Total
Natural | Specialized | Structured |  | Mean /
---
Acc.
Adapter-8 | 0.23% | 79.0 | 84.1 | 58.5 | 73.9
SPT-Adapter w/o unstructured | 0.29% | 81.2 | 85.1 | 60.3 | 75.5
SPT-Adapter w/o structured | 0.34% | 81.2 | 85.0 | 59.6 | 75.3
SPT-Adapter | 0.30% | 81.3 | 85.3 | 60.8 | 75.8

Table 6: Ablation study on structured and unstructured tuning only with
supervised pre-trained ViT-B/16 backbone. Top-1 accuracy (%) is reported. We
set different parameter constraints to align the fractions of trainable
parameters for these cases. The best result is in bold

Method |  | Inference Latency
---
(ms/img)
| Inference Memory
---
(GB)
| Fine-tuning Memory
---
(GB)
Full | 2.8 | 1.3 | 11.9
Prompt-deep | 3.8 | 1.9 | 13.2
LoRA-16 | 2.8 | 1.3 | 8.2
SPT-LoRA w/o unstructured | 2.8 | 1.3 | 8.3
SPT-LoRA | 2.8 | 1.3 | 9.8

Table 7: Cost comparisons with ViT-B/16 backbone on the VTAB-1k [62] benchmark
with around 0.70% fractions of the trainable parameters. We report the latency
(ms/img) and the peak memory usage (GB) with image resolution 224$\times$224
on a single GeForce 3090 GPU. The best result is in bold.

$C$ | 240 | 400 | 560 | 800
---|---|---|---|---
Mean Acc. | 76.3 | 76.4 | 76.4 | 76.4

Table 8: Effect of the number of training samples used to get the sensitivity
for SPT-LoRA with supervised pre-trained ViT-B/16 backbone on VTAB-1k [62].
Top-1 accuracy (%) is reported. The best result is in bold.

Computational cost analysis. We investigate the computational cost of SPT-LoRA
by comparing with full fine-tuning, addition-based method Prompt-deep, and
reparameterization-based method LoRA-16. The results are presented in Table 7.
We observe that Prompt-deep has higher inference latency and inference GPU
memory due to the additional prompts. In contrast, since the updated
parameters after fine-tuning can be reparameterized and merged into the pre-
trained model, our SPT-LoRA, SPT-LoRA w/o unstructured tuning, and LoRA-16 are
more efficient than Prompt-deep during inference. However, we observe that our
SPT-LoRA has slightly higher fine-tuning memory than the full fine-tuning and
LoRA-16 which is taken up by updating the unstructually-tuned parameters with
sparse gradients in Eq. (5). Additionally, for memory-intense scenarios, one
can employ SPT-LoRA w/o unstructured tuning for improved performance (0.8%
Top-1 accuracy on VTAB-1k) and similar fine-tuning memory as LoRA, as shown in
Figure 3 (b) and Table 6.

###  4.4 Observations on Sensitivity Patterns

Our sensitivity criterion identifies task-specific important positions, which
can reveal the contributions of the pre-trained weights to different
downstream tasks during transfer learning. We visualize the proportions of the
sensitive parameters for the supervised pre-trained ViT-B/16 backbone under
0.4M trainable parameter budget in Figure 4. First, we investigate the most
sensitive blocks, whose numbers of sensitive parameters are summed and
normalized over the 12 ViT-B/16 blocks. We observe that the patterns of the
sensitive parameter proportions vary markedly across different tasks, which
echoes the observations made in [17]. This suggests that we should not
introduce trainable parameters to the same positions for each individual task
but allocate trainable parameters at task-specific ones as we proposed. Next,
we investigate the most insensitive weight matrices within a block. A ViT
block consists of a query ${\bm{W}}_{q}$, a key ${\bm{W}}_{k}$, a value
${\bm{W}}_{v}$, and an output ${\bm{W}}_{o}$ weight matrices in the multi-head
self-attention layer and two weight matrices ${\bm{W}}_{fc1}$ and
${\bm{W}}_{fc2}$ in the feed-forward network as elaborated in [53, 13]. We
observe that the query ${\bm{W}}_{q}$ and key ${\bm{W}}_{k}$ weight matrices
have the lowest proportions of sensitive parameters for all three sample
tasks. Since ${\bm{W}}_{q}$ and ${\bm{W}}_{k}$ are responsible for learning
the attention scores which indicate the pairwise similarity among the patches,
we speculate that although domain changes, the patch relationships learned
during pre-training can be efficiently reused when transferred to downstream
classification tasks.

##  5 Conclusion

In this paper, we have explored identifying and allocating trainable
parameters to task-specific important positions for visual parameter-efficient
tuning. Specifically, we have proposed a novel criterion to quickly measure
the sensitivity of the pre-trained parameters for each specific task before
fine-tuning. Based on the parameter sensitivity, we have proposed a trainable
parameter allocation strategy that adaptively combines both unstructured and
structured tuning under a desired trainable parameter budget, enabling high
representational capability and flexibility. Finally, we have conducted
extensive experiments on a total of 24 downstream recognition tasks with both
plain and hierarchical vision Transformer backbones under different pre-
training strategies to demonstrate the versatility and effectiveness of our
proposed SPT. Notably, we have shown that our approach is complementary to the
existing PEFT methods and improves their performance significantly. In the
future, we will explore adapting large vision models to more downstream tasks
with SPT, _e.g_., dense prediction and vision-and-language tasks, and improve
the training efficiency of SPT for on-device training [7, 32].

Acknowledgement. We thank Jing liu and Ziyi Liu for their helpful discussions.
This research is partially supported by Monash FIT Start-up Grant. Dr. Jing
Zhang is supported by the Australian Research Council project FL-170100117.

## References

  * [1] A. F. Agarap.  Deep learning using rectified linear units (relu).  arXiv preprint arXiv:1803.08375, 2018.
  * [2] Y. Bao, Y. Li, S.-L. Huang, L. Zhang, L. Zheng, A. Zamir, and L. Guibas.  An information-theoretic approach to transferability in task transfer learning.  In ICIP, pages 2309–2313. IEEE, 2019.
  * [3] G. Bender, P.-J. Kindermans, B. Zoph, V. Vasudevan, and Q. Le.  Understanding and simplifying one-shot architecture search.  In ICML, pages 550–559. PMLR, 2018.
  * [4] A. Brock, T. Lim, J. M. Ritchie, and N. Weston.  Smash: one-shot model architecture search through hypernetworks.  In ICLR, 2018.
  * [5] S. Caelles, K.-K. Maninis, J. Pont-Tuset, L. Leal-Taixé, D. Cremers, and L. Van Gool.  One-shot video object segmentation.  In CVPR, pages 221–230, 2017.
  * [6] H. Cai, C. Gan, T. Wang, Z. Zhang, and S. Han.  Once-for-all: Train one network and specialize it for efficient deployment.  In ICLR, 2020.
  * [7] H. Cai, C. Gan, L. Zhu, and S. Han.  Tinytl: Reduce memory, not parameters for efficient on-device learning.  NeurIPS, 33:11285–11297, 2020.
  * [8] H. Cai, L. Zhu, and S. Han.  ProxylessNAS: Direct neural architecture search on target task and hardware.  In ICLR, 2019.
  * [9] N. S. Chatterji, B. Neyshabur, and H. Sedghi.  The intriguing role of module criticality in the generalization of deep networks.  In ICLR, 2020.
  * [10] S. Chen, C. Ge, Z. Tong, J. Wang, Y. Song, J. Wang, and P. Luo.  Adaptformer: Adapting vision transformers for scalable visual recognition.  NeurIPS, 2022.
  * [11] X. Chen, S. Xie, and K. He.  An empirical study of training self-supervised vision transformers.  In ICCV, pages 9640–9649, 2021.
  * [12] Y. Cui, Y. Song, C. Sun, A. Howard, and S. Belongie.  Large scale fine-grained categorization and domain-specific transfer learning.  In CVPR, pages 4109–4118, 2018.
  * [13] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In ICLR, 2021.
  * [14] T. Gebru, J. Krause, Y. Wang, D. Chen, J. Deng, and L. Fei-Fei.  Fine-grained car detection for visual census estimation.  In AAAI, 2017.
  * [15] D. Guo, A. Rush, and Y. Kim.  Parameter-efficient transfer learning with diff pruning.  In ACL-IJCNLP, 2021.
  * [16] Y. Guo, Y. Li, L. Wang, and T. Rosing.  Adafilter: Adaptive filter fine-tuning for deep transfer learning.  In AAAI, pages 4060–4066, 2020.
  * [17] Y. Guo, H. Shi, A. Kumar, K. Grauman, T. Rosing, and R. Feris.  Spottune: transfer learning through adaptive fine-tuning.  In CVPR, pages 4805–4814, 2019.
  * [18] T. Hao, H. Chen, Y. Guo, and G. Ding.  Consolidator: Mergable adapter with group connections for vision transformer.  In ICLR, 2023.
  * [19] J. He, C. Zhou, X. Ma, T. Berg-Kirkpatrick, and G. Neubig.  Towards a unified view of parameter-efficient transfer learning.  In ICLR, 2022.
  * [20] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick.  Masked autoencoders are scalable vision learners.  In CVPR, pages 16000–16009, 2022.
  * [21] D. Hendrycks and K. Gimpel.  Gaussian error linear units (gelus).  arXiv preprint arXiv:1606.08415, 2016.
  * [22] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly.  Parameter-efficient transfer learning for nlp.  In ICML, pages 2790–2799, 2019.
  * [23] E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen.  LoRA: Low-rank adaptation of large language models.  In ICLR, 2022.
  * [24] M. Jia, L. Tang, B.-C. Chen, C. Cardie, S. Belongie, B. Hariharan, and S.-N. Lim.  Visual prompt tuning.  In ECCV, 2022.
  * [25] S. Jie and Z.-H. Deng.  Convolutional bypasses are better vision transformer adapters.  arXiv preprint arXiv:2207.07039, 2022.
  * [26] A. Khosla, N. Jayadevaprakash, B. Yao, and L. Fei-Fei.  Novel dataset for fine-grained image categorization.  In CVPRW, 2011.
  * [27] A. Krizhevsky, I. Sutskever, and G. E. Hinton.  Imagenet classification with deep convolutional neural networks.  NeurIPS, 25, 2012.
  * [28] A. Kumar, A. Raghunathan, R. M. Jones, T. Ma, and P. Liang.  Fine-tuning can distort pretrained features and underperform out-of-distribution.  In ICLR, 2022.
  * [29] B. Lester, R. Al-Rfou, and N. Constant.  The power of scale for parameter-efficient prompt tuning.  In EMNLP, 2021.
  * [30] X. L. Li and P. Liang.  Prefix-tuning: Optimizing continuous prompts for generation.  In IJCNLP, pages 4582–4597. Association for Computational Linguistics, 2021.
  * [31] D. Lian, D. Zhou, J. Feng, and X. Wang.  Scaling & shifting your features: A new baseline for efficient model tuning.  NeurIPS, 2022.
  * [32] J. Lin, L. Zhu, W.-M. Chen, W.-C. Wang, C. Gan, and S. Han.  On-device training under 256kb memory.  In NeurIPS, 2022.
  * [33] H. Liu, K. Simonyan, and Y. Yang.  DARTS: Differentiable architecture search.  In ICLR, 2019.
  * [34] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In ICCV, 2021.
  * [35] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie.  A convnet for the 2020s.  In CVPR, pages 11976–11986, 2022.
  * [36] I. Loshchilov and F. Hutter.  Fixing weight decay regularization in adam, 2018.
  * [37] P. Molchanov, A. Mallya, S. Tyree, I. Frosio, and J. Kautz.  Importance estimation for neural network pruning.  In CVPR, pages 11264–11272, 2019.
  * [38] M. M. Naseer, K. Ranasinghe, S. H. Khan, M. Hayat, F. Shahbaz Khan, and M.-H. Yang.  Intriguing properties of vision transformers.  NeurIPS, 34:23296–23308, 2021.
  * [39] B. Neyshabur, H. Sedghi, and C. Zhang.  What is being transferred in transfer learning?  NeurIPS, 33:512–523, 2020.
  * [40] C. Nguyen, T. Hassner, M. Seeger, and C. Archambeau.  Leep: A new measure to evaluate transferability of learned representations.  In ICML, pages 7294–7305. PMLR, 2020.
  * [41] M.-E. Nilsback and A. Zisserman.  Automated flower classification over a large number of classes.  In ICVGIP, pages 722–729. IEEE, 2008.
  * [42] J. Plested and T. Gedeon.  Deep transfer learning for image classification: a survey.  arXiv preprint arXiv:2205.09904, 2022.
  * [43] J. Puigcerver, C. Riquelme, B. Mustafa, C. Renggli, A. S. Pinto, S. Gelly, D. Keysers, and N. Houlsby.  Scalable transfer learning with expert models.  arXiv preprint arXiv:2009.13239, 2020.
  * [44] M. Raghu, T. Unterthiner, S. Kornblith, C. Zhang, and A. Dosovitskiy.  Do vision transformers see like convolutional neural networks?  NeurIPS, 34:12116–12128, 2021.
  * [45] M. T. Rosenstein, Z. Marx, L. P. Kaelbling, and T. G. Dietterich.  To transfer or not to transfer.  In NeurIPSW, 2005.
  * [46] R. K. Srivastava, K. Greff, and J. Schmidhuber.  Training very deep networks.  NeurIPS, 28, 2015.
  * [47] X. Sun, R. Panda, R. Feris, and K. Saenko.  Adashare: Learning what to share for efficient deep multi-task learning.  NeurIPS, 33:8728–8740, 2020.
  * [48] Y.-L. Sung, J. Cho, and M. Bansal.  Lst: Ladder side-tuning for parameter and memory efficient transfer learning.  In NeurIPS, 2022.
  * [49] A. T. Tran, C. V. Nguyen, and T. Hassner.  Transferability and hardness of supervised classification tasks.  In ICCV, pages 1395–1405, 2019.
  * [50] C.-H. Tu, Z. Mai, and W.-L. Chao.  Visual query tuning: Towards effective usage of intermediate representations for parameter and memory efficient transfer learning.  In CVPR, 2023.
  * [51] E. Tzeng, J. Hoffman, N. Zhang, K. Saenko, and T. Darrell.  Deep domain confusion: Maximizing for domain invariance.  arXiv preprint arXiv:1412.3474, 2014.
  * [52] G. Van Horn, S. Branson, R. Farrell, S. Haber, J. Barry, P. Ipeirotis, P. Perona, and S. Belongie.  Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection.  In CVPR, pages 595–604, 2015.
  * [53] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin.  Attention is all you need.  NeurIPS, pages 5998–6008, 2017.
  * [54] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie.  The caltech-ucsd birds-200-2011 dataset.  Tech. Rep. CNS-TR-2011-001, California Institute of Technology, 2011\.
  * [55] Z. Wang, Z. Dai, B. Póczos, and J. Carbonell.  Characterizing and avoiding negative transfer.  In CVPR, pages 11293–11302, 2019.
  * [56] H. Wu, J. Xu, J. Wang, and M. Long.  Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting.  NeurIPS, 34:22419–22430, 2021.
  * [57] R. Xu, F. Luo, Z. Zhang, C. Tan, B. Chang, S. Huang, and F. Huang.  Raise a child in large language model: Towards effective and generalizable fine-tuning.  In EMNLP, 2021.
  * [58] Y. Xu, Q. Zhang, J. Zhang, and D. Tao.  Vitae: Vision transformer advanced by exploring intrinsic inductive bias.  In NeurIPS, 2021.
  * [59] J. Yoon, S. Arik, and T. Pfister.  Data valuation using reinforcement learning.  In ICML, pages 10842–10851. PMLR, 2020.
  * [60] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson.  How transferable are features in deep neural networks?  NeurIPS, 27, 2014.
  * [61] E. B. Zaken, Y. Goldberg, and S. Ravfogel.  Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models.  In ACL, pages 1–9, 2022.
  * [62] X. Zhai, J. Puigcerver, A. Kolesnikov, P. Ruyssen, C. Riquelme, M. Lucic, J. Djolonga, A. S. Pinto, M. Neumann, A. Dosovitskiy, et al.  A large-scale study of representation learning with the visual task adaptation benchmark.  arXiv preprint arXiv:1910.04867, 2019.
  * [63] X. Zhai, J. Puigcerver, A. Kolesnikov, P. Ruyssen, C. Riquelme, M. Lucic, J. Djolonga, A. S. Pinto, M. Neumann, A. Dosovitskiy, et al.  A large-scale study of representation learning with the visual task adaptation benchmark.  arXiv preprint arXiv:1910.04867, 2019.
  * [64] C. Zhang, S. Bengio, and Y. Singer.  Are all layers created equal?  arXiv preprint arXiv:1902.01996, 2019.
  * [65] Y. Zhang, K. Zhou, and Z. Liu.  Neural prompt search.  arXiv preprint arXiv:2206.04673, 2022.
  * [66] M. Zhao, T. Lin, F. Mi, M. Jaggi, and H. Schütze.  Masking as an efficient alternative to finetuning for pretrained language models.  In EMNLP, 2020.
  * [67] S. Zheng, J. Lu, H. Zhao, X. Zhu, Z. Luo, Y. Wang, Y. Fu, J. Feng, T. Xiang, P. H. Torr, et al.  Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers.  In CVPR, pages 6881–6890, 2021.
  * [68] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba.  Scene parsing through ade20k dataset.  In CVPR, pages 633–641, 2017.

## Appendix

We organize our supplementary material as follows.

  * •

In Section 6, we introduce more details about the contenders.

  * •

In Section 7, we show more sensitivity patterns for ViT-B/16 with various pre-
training strategies.

  * •

In Section 8, we show some dataset samples from ImageNet [27] and VTAB-1k
[62].

  * •

In Tables 9 and 10, we show per-task results for our SPT variants on FGVC and
VTAB-1k benchmarks, respectively.

##  6 More Details of Contenders

  * •

Full: fully tunes all the backbone and classification head parameters.

  * •

Linear: freezes all the backbone parameters and only tunes a linear
classification head.

  * •

Bias [61]: freezes all the backbone parameters except for the bias terms and
also tunes the linear classification head.

  * •

Partial-$k$: freezes all the backbone parameters except for the last $k$
layers and also tunes the linear classification head as described in [24].

  * •

Mlp-$k$: freezes all the backbone parameters and tunes the classification head
which is implemented by a trainable $k$-layer multi-layer perceptron as
described in [24].

  * •

Prompt-shallow [24]: freezes all the backbone parameters while introducing
additional trainable prompts to the input space of the pretrained ViT.

  * •

Prompt-deep [24]: freezes all the backbone parameters while appending
additional trainable prompts to the sequence in the multi-head self-attention
layer of each ViT block.

  * •

Adapter-$k$ [22]: freezes all the backbone parameters while adding a down
projection, a ReLU [21] non-linearity, and an up projection layer sequentially
in the feed-forward network (FFN) of each visual Transformer block. We follow
the training details of [65] to achieve better performance.

  * •

LoRA-$k$ [23]: freezes all the backbone parameters while adding a concurrent
branch including two low-rank matrices to the weight matrices in the multi-
head self-attention layers to approximate efficiently updating them. The low-
rank matrices can be merged into the backbone weights after fine-tuning. We
follow the training details of [65] to achieve better performance.

  * •

AdaptFormer [10]: freezes all the backbone parameters while adding a
concurrent branch including a down projection, a ReLU [1] non-linearity, an up
projection layer, and a pre-defined scaling factor to the FFN layer of each
ViT block.

  * •

NOAH [65]: searches for an optimal configuration with a once-for-all [6]
network that includes trainable prompts, adapter modules, and LoRA modules,
which requires a longer training schedule than the other VPET methods.

##  7 More Parameter Sensitivity Patterns

We show more parameter sensitivity patterns for ViT-B/16 with various pre-
training strategies (i.e., MAE [20] and MoCo V3 [11]) and datasets sampled
from FGVC benchmark [24]. We visualize the proportions of the sensitive
parameters under 0.4M trainable parameter budget. Visualizations of sampled
VTAB-1k datasets with MAE and MoCo V3 pre-trained ViT-B/16 are shown in
Figures 5, 6, 7. Visualizations of sampled FGVC datasets with supervised pre-
trained ViT-B/16 are shown in Figure 8. We find our observations in the main
paper are general: the proportions of the sensitive parameter exhibit: 1)
dataset-specific varying patterns in terms of network depth; and 2) dataset-
agnostic similar patterns in terms of operations. We empirically find that the
self-supervised pre-trained backbones have higher sensitivity variances than
the supervised pre-trained one across the 19 downstream tasks. In particular,
the variance of ViT-B/16 pre-trained with MAE [20] is twice as large as that
of the supervised pre-trained ViT-B/16. We speculate that our SPT variants can
better handle the large variances for self-supervised pre-trained backbones
(Table 2 of the main paper) by identifying task-specific positions to
introduce the trainable parameters.

Figure 5: The distribution of sensitive parameters by blocks under 0.4M
trainable parameter budget with supervised pre-trained ViT-B/16 backbone. We
sample six tasks from VTAB-1k [62].  Figure 6: The distribution of sensitive
parameters by blocks under 0.4M trainable parameter budget with MAE [20] pre-
trained ViT-B/16 backbone. We sample six tasks from VTAB-1k [62]. Figure 7:
The distribution of sensitive parameters by blocks under 0.4M trainable
parameter budget for MoCo v3 [11] pre-trained ViT-B/16 backbone. We sample six
tasks from VTAB-1k [62]. Figure 8: Sensitivity patterns under 0.4M trainable
parameters for Oxford Flowers [41], Stanford Cars [14], and Stanford Dogs
[26]. We show the proportions of the sensitive parameters for the query
${\bm{W}}_{q}$, key ${\bm{W}}_{k}$, value ${\bm{W}}_{v}$, and ${\bm{W}}_{o}$
weight matrices in the multi-head self-attention layer and two weight matrices
${\bm{W}}_{fc1}$ and ${\bm{W}}_{fc2}$ in the feed-forward network.  Figure 9:
Comparisons of sensitivity variances across backbones with different pre-
training strategies on VTAB-1k. Figure 10: Dataset samples from ImageNet [27]
and VTAB-1k [62]. Samples from Natural tasks of VTAB-1k ((a), (b), and (c))
are relatively more similar to the source ImageNet samples compared to the
ones from Structured tasks of VTAB-1k ((d), (e), and (f)).

##  8 Dataset Samples for the Source and Target Domains

We visualize some sampled images from the source domain (ImageNet [27]) and
the target domains (VTAB-1k [62]) in Figure 10. We observe that the images
from the Natural tasks of VTAB-1k are relatively more similar to the source
domain compared to those from the Structured tasks of VTAB-1k, which aligns
with our observation that Structured tasks have large domain gaps. As
structured tuning improves the performance of Structured datasets (Section 4.3
of the main paper), we speculate that structured tuning facilitates mitigating
such large domain gaps.

| Tuned / Total | CUB-200-2011 | NABirds | Oxford Flowers | Stanford Dogs | Stanford Cars | Mean Acc.
---|---|---|---|---|---|---|---
Full | 100% | 87.3 | 82.7 | 98.8 | 89.4 | 84.5 | 88.5
Addition-based methods
Mlp-3 | 1.50% | 85.1 | 77.3 | 97.9 | 84.9 | 53.8 | 79.8
Prompt-shallow | 0.31% | 86.7 | 78.8 | 98.4 | 90.7 | 68.7 | 84.6
Prompt-deep | 0.98% | 88.5 | 84.2 | 99.0 | 90.2 | 83.6 | 89.1
Adapter-8 | 0.39% | 87.3 | 84.3 | 98.4 | 88.8 | 68.4 | 85.5
Adapter-32 | 0.95% | 87.2 | 84.3 | 98.5 | 89.6 | 68.4 | 85.6
AdaptFormer | 0.44% | 84.7 | 75.2 | 97.9 | 84.7 | 83.1 | 85.1
SPT-Adapter | 0.41% | 89.1 | 83.3 | 99.2 | 90.5 | 85.6 | 89.5
SPT-Adapter | 0.47% | 89.1 | 83.3 | 99.2 | 91.1 | 86.2 | 89.8
Reparameterization-based methods
Linear | 0.12% | 85.3 | 75.9 | 97.9 | 86.2 | 51.3 | 79.3
Partial-1 | 8.38% | 85.6 | 77.8 | 98.2 | 85.5 | 66.2 | 82.6
Bias | 0.13% | 88.4 | 84.2 | 98.8 | 91.2 | 79.4 | 88.4
LoRA-8 | 0.55% | 84.9 | 79.0 | 98.1 | 88.1 | 79.8 | 86.0
LoRA-16 | 0.90% | 85.6 | 79.8 | 98.9 | 87.6 | 72.0 | 84.8
SPT-LoRA | 0.41% | 88.6 | 82.8 | 99.4 | 91.4 | 84.5 | 89.3
SPT-LoRA | 0.60% | 88.6 | 83.4 | 99.5 | 91.4 | 87.3 | 90.1

Table 9:  Per-task results on the FGVC benchmark from Table 1 of the main
paper. “Tuned / Total” denotes the fraction of the trainable parameters. Top-1
accuracy (%) is reported. The best result is in bold, and the second-best
result is underlined.

|  | Natural | Specialized | Structured
---|---|---|---|---
|  Tuned / Total |  Cifar100 |  Caltech101 |  DTD |  Flower102 |  Pets |  SVHN |  Sun397 |  Mean Acc. |  Camelyon |  EuroSAT |  Resisc45 |  Retinopathy |  Mean Acc. |  Clevr-Count |  Clevr-Dist |  DMLab |  KITTI-Dist |  dSpr-Loc |  dSpr-Ori |  sNORB-Azim |  sNORB-Ele |  Mean Acc.
Full | 100% | 68.9 | 87.7 | 64.3 | 97.2 | 86.9 | 87.4 | 38.8 | 75.9 | 79.7 | 95.7 | 84.2 | 73.9 | 83.4 | 56.3 | 58.6 | 41.7 | 65.5 | 57.5 | 46.7 | 25.7 | 29.1 | 47.6
Addition-based methods |  |
Mlp-3 | 1.50% | 63.8 | 84.7 | 62.3 | 97.4 | 84.7 | 32.5 | 49.2 | 67.8 | 77.0 | 88.0 | 70.2 | 56.1 | 72.8 | 47.8 | 32.8 | 32.3 | 58.1 | 12.9 | 21.2 | 15.2 | 24.8 | 30.6
Prompt-shallow | 0.31% | 77.7 | 86.9 | 62.6 | 97.5 | 87.3 | 74.5 | 51.2 | 76.8 | 78.2 | 92.0 | 75.6 | 72.9 | 79.7 | 50.5 | 58.6 | 40.5 | 67.1 | 68.7 | 36.1 | 20.2 | 34.1 | 47.0
Prompt-deep | 0.98% | 78.8 | 90.8 | 65.8 | 98.0 | 88.3 | 78.1 | 49.6 | 78.5 | 81.8 | 96.1 | 83.4 | 68.4 | 82.4 | 68.5 | 60.0 | 46.5 | 72.8 | 73.6 | 47.9 | 32.9 | 37.8 | 55.0
Adapter-8 | 0.39% | 69.2 | 90.1 | 68.0 | 98.8 | 89.9 | 82.8 | 54.3 | 79.0 | 84.0 | 94.9 | 81.9 | 75.5 | 84.1 | 80.9 | 65.3 | 48.6 | 78.3 | 74.8 | 48.5 | 29.9 | 41.6 | 58.5
Adapter-32 | 0.71% | 68.7 | 92.2 | 69.8 | 98.9 | 90.3 | 84.2 | 53.0 | 79.6 | 83.2 | 95.4 | 83.2 | 74.3 | 84.0 | 81.9 | 63.9 | 48.7 | 80.6 | 76.2 | 47.6 | 30.8 | 36.4 | 58.3
NOAH | 0.50% | 69.6 | 92.7 | 70.2 | 99.1 | 90.4 | 86.1 | 53.7 | 80.2 | 84.4 | 95.4 | 83.9 | 75.8 | 84.9 | 82.8 | 68.9 | 49.9 | 81.7 | 81.8 | 48.3 | 32.8 | 44.2 | 61.3
SPT-Adapter | 0.30% | 72.9 | 93.2 | 72.5 | 99.3 | 91.4 | 84.6 | 55.2 | 81.3 | 85.3 | 96.0 | 84.3 | 75.5 | 85.3 | 82.2 | 68.0 | 49.3 | 80.0 | 82.4 | 51.9 | 31.7 | 41.2 | 60.8
SPT-Adapter | 0.44% | 72.9 | 93.2 | 72.5 | 99.3 | 91.4 | 88.8 | 55.8 | 82.0 | 86.2 | 96.1 | 85.5 | 75.5 | 85.8 | 83.0 | 68.0 | 51.9 | 81.2 | 82.4 | 51.9 | 31.7 | 41.2 | 61.4
Reparameterization-based methods |  |
Linear | 0.12% | 63.4 | 85.0 | 63.2 | 97.0 | 86.3 | 36.6 | 51.0 | 68.9 | 78.5 | 87.5 | 68.6 | 74.0 | 77.2 | 34.3 | 30.6 | 33.2 | 55.4 | 12.5 | 20.0 | 9.6 | 19.2 | 26.8
Partial-1 | 8.38% | 66.8 | 85.9 | 62.5 | 97.3 | 85.5 | 37.6 | 50.6 | 69.4 | 78.6 | 89.8 | 72.5 | 73.3 | 78.5 | 41.5 | 34.3 | 33.9 | 61.0 | 31.3 | 32.8 | 16.3 | 22.4 | 34.2
Bias | 0.13% | 72.8 | 87.0 | 59.2 | 97.5 | 85.3 | 59.9 | 51.4 | 73.3 | 78.7 | 91.6 | 72.9 | 69.8 | 78.3 | 61.5 | 55.6 | 32.4 | 55.9 | 66.6 | 40.0 | 15.7 | 25.1 | 44.1
LoRA-8 | 0.55% | 67.1 | 91.4 | 69.4 | 98.8 | 90.4 | 85.3 | 54.0 | 79.5 | 84.9 | 95.3 | 84.4 | 73.6 | 84.6 | 82.9 | 69.2 | 49.8 | 78.5 | 75.7 | 47.1 | 31.0 | 44.0 | 60.5
LoRA-16 | 0.90% | 68.1 | 91.4 | 69.8 | 99.0 | 90.5 | 86.4 | 53.1 | 79.8 | 85.1 | 95.8 | 84.7 | 74.2 | 84.9 | 83.0 | 66.9 | 50.4 | 81.4 | 80.2 | 46.6 | 32.2 | 41.1 | 60.2
SPT-LoRA | 0.31% | 72.3 | 93.0 | 72.5 | 99.3 | 91.5 | 86.2 | 55.5 | 81.5 | 85.0 | 96.2 | 85.1 | 75.9 | 85.6 | 83.7 | 66.4 | 52.5 | 80.2 | 80.1 | 51.1 | 30.1 | 41.3 | 60.7
SPT-LoRA | 0.63% | 73.5 | 93.3 | 72.5 | 99.3 | 91.5 | 87.9 | 55.5 | 81.9 | 85.7 | 96.2 | 85.9 | 75.9 | 85.9 | 84.4 | 67.6 | 52.5 | 82.0 | 81.0 | 51.1 | 30.2 | 41.3 | 61.3

Table 10:  Per-task results on the VTAB-1k benchmark from Table 1 of the main
paper. “Tuned / Total” denotes the fraction of the trainable parameters. Top-1
accuracy (%) is reported.