Large Convolutional Model Tuning via Filter Subspace

Large Convolutional Model Tuning via Filter Subspace

Wei Chen, Zichen Miao, Qiang Qiu

W.Chen et al.

Purdue University, IN, USA

{chen2732, miaoz<EMAIL_ADDRESS>
Efficient fine-tuning methods are critical to address the high computational and parameter complexity while adapting large pre-trained models to downstream tasks.
Our study is inspired by prior research that represents each convolution filter as a linear combination of a small set of filter subspace elements, referred to as filter atoms. In this paper, we propose to fine-tune pre-trained models by adjusting only filter atoms, which are responsible for spatial-only convolution, while preserving spatially-invariant channel combination knowledge in atom coefficients.
In this way, we bring a new filter subspace view for model tuning.
Furthermore, each filter atom can be recursively decomposed as a combination of another set of atoms, which naturally expands the number of tunable parameters in the filter subspace.
By only adapting filter atoms constructed by a small number of parameters, while maintaining the rest of model parameters constant, the proposed approach is highly parameter-efficient. It effectively preserves the capabilities of pre-trained models and prevents overfitting to downstream tasks.
Extensive experiments show that such a simple scheme surpasses previous tuning baselines for both discriminate and generative tasks.

§ INTRODUCTION

Large models have demonstrated exceptional performance across diverse domains and tasks [2, 5, 13, 24, 45, 49, 56, 62, 66], attributing to their capability to effectively represent complex patterns and relationships [21] by pre-training on massive datasets [51, 46, 79].
A common strategy to adapt these large models for specific downstream tasks is fine-tuning them with full parameters. But this method presents two main challenges: (1) Adjusting a vast number of parameters for particular target tasks is computationally intensive; (2) The limited availability of target data increases the risk of overfitting [30].

To address these challenges, researchers have developed parameter-efficient methods [3, 16, 18, 55, 71, 74] by fine-tuning
the pre-trained models with only a minimal number of parameters.
Among these methods, LoRA [16] fine-tunes models without altering the model architecture, becoming notably popular for its simplicity and efficacy.
However, LoRA still risks overfitting when fine-tuned on limited data and compromising the generalization capability of large models. For instance, Figure <ref> illustrates that with only 5 training samples, LoRA tends to produce images that closely resemble the training data, compromising the ability for diverse image generation, compared with pre-trained models.

Compared with LoRA [16], our method updates a small set of parameters and mitigates the risk of overfitting to the target concept. In this example task of learning the concept <castle> from 5 input images, we only require fine-tuning 0.75M parameters, a stark reduction from the 22.67M parameters required by LoRA. Furthermore, the model fine-tuned by our method captures the target concept while ensuring a rich diversity and strong alignment with input text prompts. It demonstrates that our approach preserves the generalization capability of large models. The text prompts used to generate images from left to right are: “The <castle> stands against a backdrop of snow-capped mountains”, “A <castle> surrounded by a lush, vibrant forest”, “The <castle> overlooks a serene lake”, and “The <castle> in the autumn season with colorful foliage”.

§.§.§ Motivation.

To preserve the capabilities of pre-trained models when fine-tuning them on the downstream tasks, one prominent approach in continual learning [41, 52, 72] is to formulate convolution filters in ConvNets as a linear combination of filter atoms [29, 40, 43] and fine-tuning only filter atoms [36, 75].
Specifically, filters in each convolutional layer are decomposed over a small set of filter subspace elements, referred to as filter atoms, responsible for spatial-only convolution. Each convolutional layer is now constructed as linear combinations of filter atoms using decomposition coefficients, referred to as atom coefficients, responsible for the spatially invariant channel combination.
Hypothesizing variations across tasks can be reduced by bridging spatial discrepancies in the images, we propose to calibrate the pre-trained model by solely fine-tuning the spatial-only filter atoms while preserving the spatially-invariant channel weights, i.e., atom coefficients.

In our work, we demonstrate that fine-tuning a large model via filter atoms is substantially effective and parameter-efficient, as filter atoms are responsible for spatial-only convolution and usually comprise only a few hundred parameters.
This strategy is in harmony with task subspace modeling principles, which suggest that task parameters occupy a low-dimensional subspace, allowing tasks to be represented as combinations of latent basis tasks [7, 26, 35, 50, 77].
We also discover that maintaining fixed atom coefficients, i.e., spatially-invariant channel mixing weights, plays a crucial role in preserving the generalization capability of pre-trained large models.

A filter subspace view of model tuning.
(a) Each convolutional layer $\mathcal{F}$ is constructed as linear combinations of filter subspace elements, i.e., filter atoms $\atom$, using decomposition coefficients, i.e., atom coefficients $\coeff$.
Hypothesizing variations across tasks can be reduced by bridging spatial discrepancies in the images, we propose to calibrate the pre-trained model by solely fine-tuning the spatial-only $\atom$ while keeping the spatially-invariant channel weights $\coeff$ fixed.
(b) The convolution operation is represented in two stages: At the spatial-only convolution stage, each filter atom $\atom$ is adapting to the target task, and then at the cross-channel mixing stage, $\feat'$ are combined into $\feat$ using fixed atom coefficients $\coeff$, which is obtained from the pre-trained model.
More details are provided in Section <ref>.

With a large number of parameters fixed, fine-tuning only a tiny set of parameters in filter atoms is potentially challenging to adapt to more complex tasks.
We further demonstrate a simple yet effective way to expand the tunable parameters in filter subspace, without any modification on atom coefficients, by decomposing each filter atom over another set of filter atoms. This process provides an overcomplete set of filter atoms and expands the tunable parameter space, all while still requiring fewer parameters than LoRA.
Additionally, we provide a simple technique to extend this method to linear layers, ensuring alignment with the characteristics in prior literature [29, 36, 40, 43].
The illustration of our method is displayed in Figure <ref>.

We demonstrate the effectiveness of our approach on both discriminative and generative tasks with ResNet50 [13], ConvNeXt [32] and Stable Diffusion [49].
We summarize our contributions as follows,

* We propose a method by adapting only filter subspace elements (filter atoms), with a few hundred parameters, to achieve significantly parameter-efficient fine-tuning.

* We observe that maintaining fixed atom coefficients plays a crucial role in preserving the generalization capability of large models.

* We further demonstrate a simple way to expand the number of tunable parameters in filter subspace by recursively decomposing each filter atom over another set of filter atoms, which extends the parameter space for tuning.

* We conduct extensive experiments demonstrating the efficacy of our approach on discriminative and generative tasks for fine-tuning large models.

§ PRELIMINARY

§.§ Low-rank Adaptation for Fine-tuning

LoRA [16] introduces trainable low-rank matrices into layers to approximate weight updates. For a pre-trained weight matrix $\filter \in \mathbb{R}^{d' \times d}$, LoRA represents the weight update with a low-rank decomposition

\begin{equation*}
\filter + \Delta \filter = \filter + \filter_{down} \filter_{up},
\end{equation*}

where $\filter_{down} \in \mathbb{R}^{d' \times r}$ and $\filter_{up} \in \mathbb{R}^{r \times d}$ are tunable parameters, $r$ is the intrinsic rank of weight updates, $d'$ and $d$ are the size of inputs and outputs. $r$ is usually much smaller than $d'$ and $d$. A learnable scalar hyperparameter $\alpha$ scales the weight updates, which leads to $\filter + \alpha \filter_{down} \filter_{up}$.

§.§ Sparse Coding and Matrix Factorization

Sparse coding attempts to find the representation of input $\mathbf{w}$ with respect to a dictionary of $m$ atoms $\{\mathbf{d}_l\}_{l=1}^{m}$ with the fewest number of coefficients $\alpha^l$, i.e., $\mathbf{w} = \sum_{l=1}^{m} \alpha^l \mathbf{d}_l$. The objective of a sparse coding problem can be written as

\begin{equation*}
\min_{\alpha^{i, l}, \mathbf{d}_l} \sum_j \lVert \mathbf{w}^{i} - \sum_i \alpha^{i, l} \mathbf{d}_l \rVert^2_2 + \lambda \sum_i \lVert \alpha^{i, l} \rVert_1,
\end{equation*}

where $\lVert \cdot \rVert_1$ is the $L_1$ norm, and $\lambda$ is a Lagrange multiplier.

It can be further expressed in a tensor form,

\begin{equation}
\label{eq:sparse_code}
\min_{\coeff, \atom} \lVert \mathcal{F} - \coeff \times \atom \rVert^2_F + \lambda \lVert \coeff \rVert_{1,1},
\end{equation}

where $\mathcal{F} \in \mathbb{R}^{c' \times c \times k' \times k}$ is the input tensor, $\coeff \in \mathbb{R}^{c' \times c \times m}$ is the tensor of coefficients and $\atom \in \mathbb{R}^{m \times k' \times k}$ is a tensor of basis.
A number of algorithms have been developed to solve (<ref>), such as fast iterative soft-thresholding algorithm (FISTA) [1], matching pursuit [34], and least absolute shrinkage and selection operator (LASSO) [53].

We can interpret the first term in (<ref>) as finding a suitable tensor factorization for $\mathcal{F}$, while the second term serves as a penalty enforcing sparsity in the representation of $\mathcal{F}$.

§ METHODS

In this section, we decompose convolution filters over a small set of filter subspace elements, referred to as fitler atoms. This formulation enables a new model tuning method via filter subspace by solely adjusting filter atoms.

§.§ Formulation of Filter Decomposition

Our approach involves decomposing each convolutional layer $\mathcal{F}$ into two standard convolutional layers: a filter atom layer $\atom$ that models filter subspace[The filter subspace is a span of $m$ filter atoms $\atom$.], and an atom coefficient layer $\coeff$ with $1 \times 1$ filters that represent combination rules of filter atoms, as displayed in Figure <ref> (a).

This formulation is written as

\begin{equation}
\mathcal{F}=\coeff \times \atom,
\end{equation}

where $\times$ is the tensor product.

$\mathcal{F} \in \mathbb{R}^{c' \times c \times k \times k}$ contains $c' \times c$ filters $\{\mathcal{F}^{i,j}\}_{i=1, j=1}^{i=c', j=c}$, where $c'$ and $c$ are the numbers of input and output channels, $k$ is the kernel size. $\atom \in \mathbb{R}^{m \times k\times k }$ is a set of $m$ filter atoms, i.e., $\atom = \{\mathbf{d}_l\}_{l=1}^{m}$, where $\mathbf{d}_l\in \mathbb{R}^{k \times k}$. Each filter $\mathcal{F}^{i,j}$ is a linear combination of $m$ filter atoms, calculated by $\mathcal{F}^{i,j}=\sum_{l=1}^{m} \alpha^{i, j, l} \mathbf{d}_l$, $\alpha^{i, j, l}$ represents each element in $\coeff \in \mathbb{R}^{c'\times c\times m}$.

Considering input features $\inp \in \mathbb{R}^{c'\times h' \times w'}$ and output features  $\feat \in \mathbb{R}^{c\times h \times w}$, where $h' \times w'$ and $h \times w$ represent the dimension of input features and output features. The convolution process now can be comprehended as the following two steps:

§.§.§ $\bullet$ Spatial-only Convolution with $\atom$.

Each channel of the input features $\inp$ convolves with each filter atom separately to produce intermediate features

\begin{equation*}
\feat' = \atom * \inp,
\end{equation*}

where $\feat' \in \mathbb{R}^{(c'\times m) \times h \times w}$, $*$ is the convolution operation.

For each input feature channel $\inp^i$ and filter atom $\mathbf{d}_l\in \mathbb{R}^{k \times k}$, we have $\feat'^{i, l} = \inp^i * \mathbf{d}_l$, where $\feat'^{i, l}\in \mathbb{R}^{h \times w}, \inp^i \in \mathbb{R}^{h' \times w'}$ are one feature channel in $\feat', \inp$, and $\mathbf{d}_l$ is one filter atom in $\atom = \{\mathbf{d}_l\}_{l=1}^{m}$.

This process leads to the generation of $m$ distinct intermediate output channels for each input channel, which is illustrated in Figure <ref> (b). In this step, filter atoms focus only on handling the spatial information of input features, and cross-channel mixing is postponed to the next step.

§.§.§ $\bullet$ Cross-channel Mixing with $\coeff$.

Subsequently, atom coefficients weigh and linearly combine the intermediate features to produce output features

\begin{equation*}
\feat = \coeff \times \feat'.
\end{equation*}

Each output feature channel $\feat^j \in \mathbb{R}^{h \times w}$ is linearly combined from $c'\times m$ intermediate feature channels $\feat'^{i,l} \in \mathbb{R}^{h \times w}$ with the coefficient $\{\alpha^{i, j, l}\}_{i=1,l=1}^{i=c',l=m}$,

which is, $\feat^j = \sum_{i=1}^{c'}\sum_{l=1}^{m} \alpha^{i,j,l} \cdot \feat'^{i,l}$. Here, $\alpha^{i, j, l}$ represents each element in atom coefficients $\coeff \in \mathbb{R}^{c'\times c\times m}$.

The spatially invariant channel weights, atom coefficients $\coeff$, serve as operators for channel mixing, functioning as distinct combination rules that construct the output features from the elemental feature maps generated by the filter atoms. During the model tuning, $\coeff$ is obtained from the pre-trained model and remains unchanged, while only filter atoms $\atom$ adapt to the target task.

§.§.§ Summary.
The two-step convolution operation explains different functionalities of filter atoms $\atom$ and atom coefficients $\coeff$, which is, $\atom$ only contribute to spatial convolution and $\coeff$ only perform cross-channel mixing.

In practice, the convolution operation is still performed as one layer, without generating intermediate features, to avoid memory cost. In the fine-tuning process, we solely adjust $\atom$, which contains a small set of number of parameters, $k \times k \ll c'\times c$, thereby facilitating parameter-efficient fine-tuning.

§.§ Overcomplete Filter Atoms

The parameters of filter atoms are extremely small compared with overall model parameters. For instance, the filter atoms constitute a mere $0.004\%$ of the total parameters in ResNet50 [13]. To fully explore the potential of filter subspace fine-tuning, we show next a simple way to construct a set of overcomplete[
As the size of filters is $k \times k$, $m=k^2$ independent filter atoms is complete since every filter can be linearly combined by these filter atoms. As the number of filter atoms is larger than $k^2$, it becomes overcomplete. Overcompleteness can potentially bring a more stable fine-tuning, and in our case, expand the number of parameters for adapting to downstream tasks.
] filter atoms by recursively applying the above decomposition to each filter atom, to expand the parameter space for fine-tuning as needed.

Specifically, each filter atom $\mathbf{d}_l$ can be further decomposed over $m_{1}$ number of filter atoms in $\atom_{1,l}=\{\mathbf{d}_{1,l}^{j}\}_{j=1}^{m_{1}}$ with its corresponding coefficients $\coeffs^{i} = \begin{bmatrix} \beta^{i,1}, \cdots, \beta^{i,m_{1}}\end{bmatrix} \in \mathbb{R}^{m_{1}}$,

i.e., $\mathbf{d}_l = \sum_{j=1}^{m_{1}} \beta^{i,j} \mathbf{d}_{1,l}^{j}$.

Therefore, $\atom_1 \in \mathbb{R}^{(m\cdot m_1) \times k \times k}$ contains $m\cdot m_1$ filter atoms, leading to a overcomplete set. The recursive decomposition process is illustrated in Figure <ref> (a).

Considering input $\inp \in \mathbb{R}^{c'\times h' \times w'}$ and output $\feat \in \mathbb{R}^{c\times h \times w}$, convolution using the overcomplete filter atoms can be comprehended as three steps. While the spatial-only convolution with $\atom$ and cross-channel mixing with $\coeff$ steps are the same as before, it linearly combines the features channels corresponding to $m_1$ decomposed filter atoms in intra-channel mixing step.

[formulation of atom decomposition]
(a) The process of constructing overcomplete filter atoms: One filter atom in $\atom$ is presented as a linear combination of $m_1$ filter atoms in $\atom_1$.
(b) The convolution operation is represented in three stages: At the spatial-only convolution stage, a set of overcomplete filter atoms $\atom_{1}$ is adapting to the target task and generates $m\cdot m_1$ feature channels by convolving with each input channel.

At the intra-channel mixing stage, every group of $m_1$ features is linearly combined by coefficients $\coeffs$ with no cross-channel mixing.
At the cross-channel mixing stage, $\feat'$ are combined into $\feat$ using fixed atom coefficients $\coeff$, which is obtained from the pre-trained model.

§.§.§ $\bullet$ Intra-channel Mixing with $\coeffs$.
After convolving $\inp$ and $\atom_{1}$ to get $\feat'_{1} \in \mathbb{R}^{(c'\cdot m \times m_1)\times h \times w}$, the intermediate feature map $\feat' \in \mathbb{R}^{(c'\cdot m)\times h \times w}$ is obtained by linearly combining feature channels within the corresponding input channel via coefficients $\coeffs$,

\begin{equation*}
\feat' = \coeffs \times \feat'_{1}.
\end{equation*}

Each feature channel $\feat^{'i}$ is calculated by $\feat^{'i} = \sum_{j=1}^{m_1} \beta^{i,j} \feat_{1}^{'i, j}$, where $\beta^{i,j}$ is one element in $\coeffs$, and $\feat_{1}^{'i, j} \in \mathbb{R}^{h \times w}$ is one feature channel in $\feat'_{1}$.
This phase exclusively blends the feature channels within the same input channel, avoiding any mixing across different channels.
The whole process is illustrated in Figure <ref> (b).
To adapt the model to downstream tasks, we fine-tune both $\coeffs$ and $\atom_1$, which contain more tunable parameters than $\atom$.

§.§ Decomposition of Linear Layers

Formulate the linear layer $\mathbf{W}$ as a combination of atoms, $\mathbf{W} = \coeff_c \times \atom_c$.

This method can be readily adapted to $1\times 1$ convolutional and linear layers as well.
We start by applying Kronecker decomposition on a linear layer $\mathbf{W} = \mathbf{A} \otimes \mathbf{B}$, where $\mathbf{A} \in \mathbb{R}^{\frac{c'}{k'_c} \times \frac{c}{k_c}}$, $\mathbf{B} \in \mathbb{R}^{k'_c  \times k_c}$, $\mathbf{W} \in \mathbb{R}^{c'\times c}$, $\otimes$ is the Kronecker product. This approach has been explored by recent literature on fine-tuning [6, 71]. The idea is to represent a matrix as multiple blocks, where each block comes from the same basis $\mathbf{B}$ but varied by different constant $A_{ij}$.
We then introduce $m_c$ pairs of $\mathbf{A}$ and $\mathbf{B}$ to reconstruct $\mathbf{W}$, $\mathbf{W} = \sum_{i=1}^{m_c} \mathbf{A}_i \otimes \mathbf{B}_i$, such that each block in $\mathbf{W}$ is represented as a linear combination of all $\mathbf{B}_i$.

We re-write the above formulation as the notation with coefficients and atoms such that $\coeff_c = \{\mathbf{A}_i\}_{i=1}^{m_c}$ and $\atom_c = \{\mathbf{B}_i\}_{i=1}^{m_c}$. Therefore, $\mathbf{W}$ is constructed from $m_c$ atoms in $\atom_c \in \mathbb{R}^{m_c\times k'_c  \times k_c}$ by the combination using coefficients $\coeff_c \in \mathbb{R}^{\frac{c'}{k'_c} \times \frac{c}{k_c} \times m_c}$, which is, $\mathbf{W} = \coeff_c \times \atom_c$. This formulation is mathematically equal to the decomposition filters as filter atoms and atom coefficients. Similarly, the $1\times 1$ convolutional layer $\mathcal{F}_c \in \mathbb{R}^{c'\times c \times 1 \times 1}$ can be formulated as $\coeff_c$ and $\atom_c$ with the same process,

\begin{equation}
\label{eq:linear}
\mathcal{F}_c = \coeff_c \times \atom_c.
\end{equation}

The procedure of formulating linear layers as coefficients and atoms is illustrated in Figure <ref>.
During the model tuning, $\coeff_c$ is obtained from the pre-trained model and remains unchanged, while only filter atoms $\atom_c$ adapt to the target task.

§.§ Parameter Efficient Fine-tuning

Building on our filter subspace formulation, we investigate various parameter-efficient fine-tuning strategies for tuning the pre-trained model.
For convolutional filters $\mathcal{F}$, we directly use sparse coding (<ref>) to decompose them as atom coefficients and filter atoms $\coeff \times \atom$. For linear layers or $1\times 1$ convolution layers, we first formulate them as $\mathcal{F}_c$ (<ref>), and then decompose them as coefficients $\coeff_c$ and atoms $\atom_c$ using (<ref>). For fine-tuning the models, we only adapt $\atom$ or $\atom_c$, while keeping $\coeff$ or $\coeff_c$ fixed.

When more parameters are needed, we represent $\atom$ as $\coeffs$ and $\atom_{1}$ to incorporate overcomplete filter atoms. We initialize $\atom_{1}$ by simply repeating each filter atom in $\atom$ for $m_1$ times and initialize every element in $\coeffs$ with the value $1/m_1$. This enables us to fine-tune both $\coeffs$ and $\atom_{1}$.

In this study, we investigate three strategies that offer a range of tunable parameters from a small set to a slightly larger number: fine-tuning (1) only $\atom$, (2) $\atom$ and $\atom_c$, and (3) $\coeffs$, $\atom_{1}$ and $\atom_c$. Downstream tasks are modeled by their updates $\Delta \atom$, $\Delta \coeffs$, $\Delta \atom_{1}$ and $\Delta \atom_c$.

§ EXPERIMENTS

In this section, we begin with studying the effectiveness of our method across various configurations to determine the most suitable application scenario for each configuration.
Subsequently, we demonstrate that fine-tuning only filter atoms requires far fewer parameters while preserving the capacity of pre-trained models, compared with baseline methods in the contexts of discriminative and generative tasks.

§.§ Experimental Settings

§.§.§ Datasets.

Our experimental evaluations are mainly conducted on the Visual Task Adaptation Benchmark (VTAB) [76], which contains 19 distinct visual recognition tasks sourced from 16 datasets. As a subset of VTAB, VTAB-1k comprises a mere $1,000$ labeled training examples in each dataset.

§.§.§ Models.

For the validation experiment, we choose ResNet50 [13] pre-trained on ImageNet-1K. For discriminative tasks, we choose the convolution-based model, ConvNeXt-B [32] pre-trained on ImageNet-21K as the initialization for fine-tuning.
For generative tasks, we choose Stable Diffusion [49] which contains a downsampling-factor 8 autoencoder with an 860M UNet and CLIP ViT-L/14 as text encoder for the diffusion model. The model is pre-trained on the LAION dataset [54], which contains over 5 billion image-text pairs. More details of pre-trained models are listed in the Appendix <ref>.

§.§ Validation Experiments

In this section, we study the performance of our approach across various configurations.

§.§.§ Implementation details.
We employ ResNet50 [13] pre-trained on ImageNet-1K [51] and fine-tune it on CIFAR-100 [25] for 50 epochs. We utilize the Adam optimizer [23] with a learning rate of 0.001 and a weight decay of $1\times 10^{-4}$ and a batch size of 256.

We compare different configures of our methods with the following experimental settings and $\coeff$, $\coeff_c$ are always fixed: (1) When fine-tuning $\atom$, we experiment with different numbers of filter atoms $m$ in the range of $[6, 9, 12]$. (2) When fine-tuning $\atom$ and $\atom_c$, we choose $(k'_c, k_c) \in \{(2, 2),(4, 4)\}$, and the number of filter atoms is $m=9$. (3) When fine-tuning $\atom_1$ and $\coeffs$, we choose $m=9$, and $m_{1}$ in the range of $[3, 4, 5]$. (4) When fine-tuning $\atom_c$, $\atom_1$ and $\coeffs$, we choose $m=9$, $m_{1}=3$ and $(k'_c, k_c)=(2, 2)$.

Comparison among different configurations of our method.

3cFine-tune $\atom$
Linear Probe     LoRA     Full Finetuning     $m=6$     $m=9$      $m=12$
Accuracy $\uparrow$      55.4             78.6       83.3                66.25             66.86        68.68

Param. (M) $\downarrow$     0.2              2       25.6                0.21              0.21       0.21
3c|Fine-tune $\coeffs$ and $\atom_1$      2cFine-tune $\atom$ and $\atom_c$     FT $\coeffs$, $\atom_1$ and $\atom_c$
$m_1=3$     $m_1=4$      $m_1=5$     $k_c=2$     $k_c=4$     $m_1=4$, $k_c=4$
Accuracy $\uparrow$      78.69             78.7         78.9               75.03              79.83          81.8

Param. (M) $\downarrow$     0.82              1       1.2               0.23              0.62       2.1

§.§.§ The effectiveness of filter subspace.
The results of all configurations are displayed in Table <ref>, with $m=9$ as an example, and $m=6,12$ are detailed in the Appendix <ref>.
(1) While fine-tuning $\atom$ with fixed $\coeff$, our approach requires a negligible increase in the number of parameters, amounting to only $0.5\%$ additional parameters, but it achieves an almost $20\%$ improvement in accuracy when compared to linear probing.
(2) Fine-tuning $\coeffs$ and $\atom_{1}$ further improves accuracy from $66.8\%$ to $78.7\%$ compared to fine-tuning $\atom$ alone. It validates that overcomplete filter atoms provide more capacities for model tuning.
(3) Fine-tuning $\atom$ and $\atom_c$ results in significant improvements compared to fine-tuning $\atom$ alone. For example, with $k_c=2$, fine-tuning $0.02$M additional parameters improves accuracy from $66.86\%$ to $75.03\%$.
(4) Fine-tuning $\atom_c$, $\atom_{1}$, and $\coeffs$ achieves the highest accuracy but it also requires the greatest number of parameters among all of our configurations.

§.§.§ Discussion.

Fine-tuning the filter atoms $\atom$ or atoms $\atom_c$ enhances accuracy while keeping the parameter increase minimal, making it highly appropriate for scenarios where the number of parameters is a critical consideration. However, increasing the number of atoms does not easily yield further accuracy improvements. To achieve additional accuracy gains, incorporating the set of overcomplete filter atoms $\atom_1$ and their coefficients $\coeffs$ can be effective, but at the cost of an increase in parameters.

§.§ Generative Tasks

In this section, we apply our method to the generative task by evaluating the generative samples from VTAB dataset [76].

§.§.§ Baselines.

We compare our method to 6 baseline fine-tuning approaches: (i) Full fine-tuning, which entails updating all model parameters during the fine-tuning process; (ii) LoRA [16], involving the introduction of a low-rank structure of accumulated gradient update by decomposing it as up-projection and down-projection. (iii) LoHa [71] utilizes the Hadamard product across two sets of low-rank decompositions to elevate the rank of the resultant matrix and reduce the approximation error. (iv) LoKr [71] introduces the Kronecker product for matrix decomposition to reduce the tunable parameters. (v) BitFit [74] fine-tunes the bias term of each layer. (vi) DiffFit [69] fine-tunes the bias term, as well as the layer norm and the scale factor of each layer.

§.§.§ Implementation details.

We fine-tune Stable Diffusion using the AdamW optimizer [33] with a learning rate of $5\times 10^{-6}$ for full parameter fine-tuning, and $1\times 10^{-3}$ for all other parameter-efficient fine-tuning methods.

For few-shot learning, we fine-tune Stable Diffusion to learn the concept “castle” from the CustomConcept101 dataset [27] with the image shape of $512 \times 512$. We follow the method in Yeh et al. [71] and fine-tune the model for 1000 steps, and evaluate the fidelity, diversity, and text-to-image alignment of generated images.

We assess the fidelity as the average cosine similarity between DINOv2 embeddings [38] of the generated and dataset images. The diversity of generated images is measured by the Vendi score [8] calculated with the DINOv2 embeddings. The alignment between generated images and corresponding prompts is measured via average cosine similarity in the CLIP feature space [44].

For the VTAB dataset, we fine-tune $\coeffs$, $\atom_1$, and $\atom_c$ of the model for 5000 steps with the image shape of $256 \times 256$ and utilize Frechet Inception Distance (FID) [14] as a quantitative metric for evaluation. To calculate FID, we generate 20,000 images from each checkpoint and compare them with images from the corresponding dataset. In cases where the dataset contains more than 20,000 images, we sample 20,000 images for comparison. See the Appendix <ref> for details of experimental settings.

Input Images

No Fine-tuning

BitFit [74]

Ours, $\atom$ only

LoRA [16]

LoHa [71]

Full Fine-tuning

Ours, $\atom$ and $\coeff$

Fine-tune Stable Diffusion [49] to learn the concept <castle> from (a) and generate images using text prompt: “A peacock in front of the <castle>”.
(b) Images generated by the pre-trained model, without any fine-tuning, align well with the text prompt but fail to match the target concept. (d) When only fine-tuning the filter atoms $\atom$, our approach achieves good alignment with the text prompt and the target concept. (h) However, fine-tuning the spatially-invariant channel weights, i.e., atom coefficients $\coeff$, results in generated images only align with the target concept, compromising the capability of the pre-trained model of producing diverse images that align with the text prompt.
This issue is also observed in (e-g).

Evaluate different approaches in learning the concept <castle>.

Pre-trained     Full FT     LoRA [16]      LoHa [71]     LoKr [71]         DiffFit [69]     BitFit [74]                $\mathbf{C}_1$      $\mathbf{C}_2$     $\mathbf{C}_3$     $\mathbf{C}_4$
Fidelity $\uparrow$      0.15            0.74        0.71      0.73     0.73     0.57        0.44                  0.44     0.62     0.72     0.77

Diversity $\uparrow$     32.56           4.29                 4.85      3.51                 4.27                 10.38       16.8      16.82     8.97     8.53     2.56

T2I Alignment $\uparrow$    0.32             0.24      0.23      0.21                 0.23                 0.26        0.25                  0.31     0.28     0.27     0.19
Param. (M) $\downarrow$     -               860               22.67     8.47                 1.06                 0.58        0.34      0.05     0.75     2.39     587

§.§.§ Few-shot generative transfer learning.

We report the evaluation of all methods in Table <ref>. Our method with different configurations is denoted as $C_1 - C_4$.
(1) $C_1$ and $C_2$ represent fine-tuning $\atom$ and $\atom_c$ of the model with $(m, k_c) \in \{(6,4), (9, 16)\}$.
(2) $C_3$ fine-tunes $\coeffs$, $\atom_1$ and $\atom_c$, with $(m,k_c, m_1)=(9,16,3)$.
(3) $C_4$ is similar to $C_2$ but it fine-tunes $\coeff$ together with $\atom$ and $\atom_c$.

Compared with other baseline methods, our method with configuration $C_{1}-C_3$ generate diverse images that are well-aligned with the text prompt. It means they preserve the capability of the pre-trained model while learning the new concept.

However, $C_4$ fine-tunes the atom coefficients $\coeff$, resulting in the model overfitting to the target concept, which is reflected in a notably high fidelity score, while compromising the model's ability to generate images aligned with the text prompt. This observation suggests that maintaining the spatially invariant channel weights $\coeff$ helps prevent overfitting when fine-tuning pre-trained models to downstream tasks. The visual examples can be found in Figure <ref> and <ref>.

Methods like LoRA [16] or full fine-tuning potentially update these $\coeff$, thus, they lead to lower diversity and text-to-image alignment in generated images.
In contrast, BitFit [74] and DiffFit [69] mostly fine-tune the bias, leaving $\coeff$ fixed, thus, they have a higher diversity and text-to-image alignment than LoRA. However, they also keep the spatial operation $\atom$ unchanged, resulting in a lower fidelity score compared with $C_2$.
More results can be found in Appendix <ref>.

FIDs (lower the better) of image generation models on VTAB benchmark with Stable Diffusion pre-trained on LAION.

7c|Natural                                        4c|Specialized                      5c|Structured
90Caltech101     90CIFAR100     90DTD     90Flowers102     90Pets     90SVHN     90Sun397
90Patch Camelyon     90EuroSAT     90Resisc45     90Retinopathy
90Clevr     90DMLab     90KITTI     90dSprites     90SmallNORB
90Mean     90Params. (M)
No fine-tuning          52.0            113.3       74.5           45.5              117.1        212.2           28.8
258.8            186.2              144.2              307.7
271.3                 225.0             288.5            373.9                266.7
179.1         -
Full fine-tuning      39.0      33.8             46          42.9           32.6          93.2          17.0
107.6               144.5             54.3             69.0
25.8               30.3               92.9          75.8             75.6
61.3        860

LoRA [16]                   36.8           42.4             40.6         45.3         30.8         120.9          17.7
101.2                 81.9             61.6             69.5
33.6            32.1           64.4              69.4               71.6
57.5           22.67

LoHa [71]                   33.8           40.5             41.6         41.5         33         140.9          17.3
111.4                 73.7             60.4             66.5
32.1            37.9           64.5              73.3               65.7
58.4           33.86

LoKr [71]                   37.7           43.5             43.2         50.6         41.1         144.2          21.4
95.4                 70.5             65.5             79.3
33.9            44.8           64.1              79.4               74.4
61.8           2.12
Ours                     38.6             32.4             43.9        38.5          30.6          96.8         17.2
109.7                64.1             62.4             60.6
18.7           41.9          69.5              73.2             76.5
54.7           1.11

§.§.§ Performance comparisons on generative transfer learning.
We report FIDs of models trained and evaluated on VTAB tasks in Table <ref>. In contrast to full parameter fine-tuning and LoRA, our approach attains the lowest FID scores ($54.7$ v.s. $57.5$) while employing the least number of fine-tuning parameters ($1.11$M v.s. $22.67$M). Despite fine-tuning only $0.13\%$ of the total model parameters, our method effectively tailors pre-trained Stable Diffusion to align it with the desired target distribution.

§.§ Discriminative Tasks

In this section, we apply our method to the discriminative task, namely the classification on VTAB-1k [76].

We compare our method to 4 baseline fine-tuning approaches: (i) Full fine-tuning, (ii) Linear probing, (iii) BitFit [74], and (iv) LoRA [16].

§.§.§ Implementation details.

Images are resized to $224 \times 224$, following the default settings in VTAB [76]. We employ the AdamW [33] optimizer to fine-tune models for 100 epochs. The cosine decay strategy is adopted for the learning rate schedule, and the linear warm-up is used in the first 10 epochs.

In this experiment, we fine-tune $\atom$ and $\atom_c$ while keeping $\coeff$ and $\coeff_c$ fixed, as this configuration delivers adequate accuracy without increasing parameters.

Performance comparisons on the VTAB-1k benchmark with ConvNeXt models pre-trained on ImageNet-21K.

7c|Natural                                        4c|Specialized                      8c|Structured
90Caltech101     90CIFAR100     90DTD     90Flowers102     90Pets     90SVHN     90Sun397
90Patch Camelyon     90EuroSAT     90Resisc45     90Retinopathy
90Clevr/count     90Clevr/distance     90DMLab     90KITTI     90dSprites/loc     90dSprites/ori     90SmallNORB/azi     90SmallNORB/ele
90Mean     90Params. (M)
Full fine-tuning     94.9    64.2              73.6           99.5             90.8         89.6          37.7
86.6          85.1             85.9             73.6
73.3                  61.3                52.1         83.1             86.8                 61.1                 32.7                   38.8
72.14            87.67

Linear Probing           92.3              65.8            76.8           99.3            92.7        50.5           55.8
84.0                92.7             82.5             74.7
46.1                 38.5                41.1         66.3             24.2                 35.4                   18.4               26.0
61.21         0.11
BitFit [74]        94.5              71.6            76.7            99.4            93.0         85.7         57.2                86.4                94.0             86.4              74.3                67.8                 57.2                49.8          80.5              77.7                59.1                  30.4                    22.0            71.77            0.24

LoRA [16]                      94.3              51.7          61.4           88.1             69.8         91.2          38.1
74.5                 91.9            81.4              73.6
60.8                  62              50.3         80.1             96.3                   56.3               39.3                  21.9
67.53          17.4
Ours                     94.8              71.7            76.9         99.6              93.1         87.1          57.5
85.1                  94.6            86.4              74.8
70.9                  62.8                50.3         82.7             89.4                   60.4                 31.2                29
73.59          0.45

19c|Generalization Error $\downarrow$

BitFit    0.29    1.04    0.82    0.062    0.21    0.47    2.52    0.39    0.12    0.38    0.83    0.46    0.61    1.29    1.02    0.34    0.57    4.08    1.99    0.98    0.24

LoRA      0.23    1.22    0.93    0.028    0.24    0.4    2.53    0.72    0.16    0.41    2.35    0.94    1.09    1.56    0.93    0.25    1.17    2.56    2.39    1.06    17.4

Ours      0.24    0.95    0.96    0.029    0.22    0.43    2.52    0.54    0.14    0.42    1.64    0.5    1.08    1.84    1.01    0.16    0.99    2.37    2.01    0.94    0.45

§.§.§ Performance comparisons on few-shot transfer learning.

We compare the performance of our approach and other baseline methods, and the results on VTAB-1k are shown in Table <ref>. In these tables, the bold font shows the best accuracy of all methods and the underlined font shows the second best accuracy. Our method outperforms other parameter-efficient fine-tuning methods and even outperforms full fine-tuning. Specifically, our method obtains $6\%$ improvement in accuracy compared to LoRA on the VTAB-1k benchmark while utilizing significantly fewer trainable parameters ($0.45$M v.s. $17.4$M).

The generalization error is measured by the discrepancy between the training loss and the test loss, with the findings illustrated in Table <ref> and Figure <ref> from Appendix <ref>.

Our technique, which employs fixed atom coefficients, leads to a comparatively lower generalization error. It means our tuning method better preserves the generalization ability of the pre-trained model.

§ RELATED WORKS

§.§ Pre-training and Fine-tuning

The standard practice of pre-training and fine-tuning [13, 17, 60, 70] entails models initially undergoing pre-training on datasets such as ImageNet-21K, BookCorpus, and Common Crawl [51, 46, 79]. Subsequently, these models are fine-tuned to enhance their convergence and performance on specific tasks [12].

In the realm of parameter-efficient fine-tuning [78], various approaches have been proposed. LoRA [16] fine-tunes lower-rank matrices at each layer to represent weight updates.
The adapter [15] approach inserts small modules between layers and reduces parameters by only tuning these adapters [3, 19, 28, 74].
Visual prompt tuning (VPT) [18, 58] has introduced a limited number of learnable parameters for optimization while keeping the backbone frozen.
SSF [30] proposes scaling and shifting deep features extracted by a pre-trained model.

§.§ Model Architectures

Compared with transformer-based models [5, 31, 61, 73], convolution has been used for a long time as the main module to extract the image features in computer vision tasks. With an inductive prior, convolution-based models require fewer training images and computation resources to achieve good generalization. Convolution-based architectures have been largely studied [13, 32, 57] and have found multiple applications, such as feature extracting [48], image generation [20, 59], super-resoluton [68], and et cetera. Numerous studies explore the integration of convolutional techniques with vision transformers to enhance their performance [10, 47].
Parameter-efficient fine-tuning in downstream tasks is crucial and requires further examinations when utilizing pre-trained large-scale convolution-based models.

§.§ Discriminative and Generative Tasks

Discriminative and generative tasks represent fundamental concepts in machine learning. Discriminative models [11, 13, 39, 80] are designed to distinguish between various data instances, while generative models [20, 48, 59, 68] are employed to create new data instances. Discriminative models have been applied to image classifications [13, 32, 57], object detection [39, 80], and semantic segmentation [11]. Generative models have been extensively studied for image synthesis, including variational autoencoder [22, 48, 63, 65], diffusion [4, 49, 59], and autoregressive models [42, 37, 64].

In this study, our primary focus is on implementing parameter-efficient fine-tuning techniques for two tasks: image classification using ConvNeXt [32] and image synthesis employing Stable Diffusion [49].

§ CONCLUSION

In this work, we proposed the parameter-efficient fine-tuning method for large convolutional models by formulating the convolutional layers over the filter subspace.
Fine-tuning filter atoms composed of a small number of parameters and keeping the atom coefficients unchanged, is notably efficient in terms of parameters. It successfully maintains the capabilities of pre-trained models while avoiding overfitting to downstream tasks.
We then formulate a simple yet effective way to achieve an overcomplete filter subspace by decomposing each filter atom over another set of filter atoms, thereby expanding the parameter space available for fine-tuning as needed. Our approach has demonstrated effectiveness in different configurations on both discriminate and generative tasks.

§.§.§ Limitations.
Our method, which concentrates on tuning models within the filter subspace, is particularly advantageous for ConvNets. While it can be naturally extended to linear layers through appropriate mathematical formulations, the full potential of our approach when applied to linear layers remains under-explored.

[1]
Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM journal on imaging sciences pp. 183–202 (2009)

[2]
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in neural information processing systems  (2020)

[3]
Chen, S., Ge, C., Tong, Z., Wang, J., Song, Y., Wang, J., Luo, P.: Adaptformer: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems  (2022)

[4]
Dhariwal, P., Nichol, A.: Diffusion models beat gans on image synthesis. Advances in neural information processing systems  (2021)

[5]
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. In: International Conference on Learning Representations (2020)

[6]
Edalati, A., Tahaei, M., Kobyzev, I., Nia, V.P., Clark, J.J., Rezagholizadeh, M.: Krona: Parameter efficient tuning with kronecker adapter. arXiv preprint arXiv:2212.10650  (2022)

[7]
Evgeniou, A., Pontil, M.: Multi-task feature learning. Advances in neural information processing systems  (2007)

[8]
Friedman, D., Dieng, A.B.: The vendi score: A diversity evaluation metric for machine learning. arXiv preprint arXiv:2210.02410  (2022)

[9]
Gildenblat, J., contributors: Pytorch library for cam methods. <https://github.com/jacobgil/pytorch-grad-cam> (2021)

[10]
Guo, J., Han, K., Wu, H., Tang, Y., Chen, X., Wang, Y., Xu, C.: Cmt: Convolutional neural networks meet vision transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022)

[11]
Hao, S., Zhou, Y., Guo, Y.: A brief survey on semantic segmentation with deep learning. Neurocomputing pp. 302–321 (2020)

[12]
He, K., Girshick, R., Dollár, P.: Rethinking imagenet pre-training. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2019)

[13]
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition (2016)

[14]
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems  (2017)

[15]
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., Gelly, S.: Parameter-efficient transfer learning for nlp. In: International Conference on Machine Learning (2019)

[16]
Hu, E.J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al.: Lora: Low-rank adaptation of large language models. In: International Conference on Learning Representations (2021)

[17]
Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition (2017)

[18]
Jia, M., Tang, L., Chen, B.C., Cardie, C., Belongie, S., Hariharan, B., Lim, S.N.: Visual prompt tuning. In: European Conference on Computer Vision (2022)

[19]
Karimi Mahabadi, R., Henderson, J., Ruder, S.: Compacter: Efficient low-rank hypercomplex adapter layers. Advances in Neural Information Processing Systems  (2021)

[20]
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of stylegan. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (2020)

[21]
Khan, A., Sohail, A., Zahoora, U., Qureshi, A.S.: A survey of the recent architectures of deep convolutional neural networks. Artificial intelligence review  (2020)

[22]
Kingma, D., Salimans, T., Poole, B., Ho, J.: Variational diffusion models. Advances in neural information processing systems  (2021)

[23]
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: International Conference on Learning Representations (2015)

[24]
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., Dollár, P., Girshick, R.: Segment anything. arXiv:2304.02643  (2023)

[25]
Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. Ph.D. thesis, University of Toronto (2009)

[26]
Kumar, A., Daume III, H.: Learning task grouping and overlap in multi-task learning. International Conference on Machine Learning  (2012)

[27]
Kumari, N., Zhang, B., Zhang, R., Shechtman, E., Zhu, J.Y.: Multi-concept customization of text-to-image diffusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023)

[28]
Li, X.L., Liang, P.: Prefix-tuning: Optimizing continuous prompts for generation. In: Proceedings of the Association for Computational Linguistics (2021)

[29]
Li, Y., Gu, S., Gool, L.V., Timofte, R.: Learning filter basis for convolutional neural network compression. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2019)

[30]
Lian, D., Zhou, D., Feng, J., Wang, X.: Scaling & shifting your features: A new baseline for efficient model tuning. Advances in Neural Information Processing Systems  (2022)

[31]
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF international conference on computer vision (2021)

[32]
Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., Xie, S.: A convnet for the 2020s. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (2022)

[33]
Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: International Conference on Learning Representations (2018)

[34]
Mallat, S.G., Zhang, Z.: Matching pursuits with time-frequency dictionaries. IEEE Transactions on signal processing pp. 3397–3415 (1993)

[35]
Maurer, A., Pontil, M., Romera-Paredes, B.: Sparse coding for multitask and transfer learning. In: International conference on machine learning (2013)

[36]
Miao, Z., Wang, Z., Chen, W., Qiu, Q.: Continual learning with filter atom swapping. In: International Conference on Learning Representations (2021)

[37]
Van den Oord, A., Kalchbrenner, N., Espeholt, L., Vinyals, O., Graves, A., et al.: Conditional image generation with pixelcnn decoders. Advances in neural information processing systems  (2016)

[38]
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.: Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193  (2023)

[39]
Padilla, R., Netto, S.L., Da Silva, E.A.: A survey on performance metrics for object-detection algorithms. In: 2020 international conference on systems, signals and image processing (IWSSIP) (2020)

[40]
Papyan, V., Romano, Y., Elad, M.: Convolutional neural networks analyzed via convolutional sparse coding. The Journal of Machine Learning Research  18,  2887–2938 (2017)

[41]
Parisi, G.I., Kemker, R., Part, J.L., Kanan, C., Wermter, S.: Continual lifelong learning with neural networks: A review. Neural Networks  (2019)

[42]
Parmar, N., Vaswani, A., Uszkoreit, J., Kaiser, L., Shazeer, N., Ku, A., Tran, D.: Image transformer. In: International conference on machine learning (2018)

[43]
Qiu, Q., Cheng, X., Sapiro, G., Calderbank, R.: DCFNet: Deep neural network with decomposed convolutional filters. In: International Conference on Machine Learning (2018)

[44]
Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning (2021)

[45]
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.: Language models are unsupervised multitask learners

[46]
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research  (2020)

[47]
Raghu, M., Unterthiner, T., Kornblith, S., Zhang, C., Dosovitskiy, A.: Do vision transformers see like convolutional neural networks? Advances in Neural Information Processing Systems  (2021)

[48]
Razavi, A., Van den Oord, A., Vinyals, O.: Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems  (2019)

[49]
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (2022)

[50]
Romera-Paredes, B., Aung, H., Bianchi-Berthouze, N., Pontil, M.: Multilinear multitask learning. In: International Conference on Machine Learning (2013)

[51]
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV)  (2015)

[52]
Rusu, A.A., Rabinowitz, N.C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., Hadsell, R.: Progressive neural networks. NIPS Deep Learning Symposium  (2016)

[53]
Santosa, F., Symes, W.W.: Linear inversion of band-limited reflection seismograms. SIAM journal on scientific and statistical computing pp. 1307–1330 (1986)

[54]
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al.: Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems  (2022)

[55]
Shen, Z., Liu, Z., Qin, J., Savvides, M., Cheng, K.T.: Partial is better than all: revisiting fine-tuning strategy for few-shot learning. In: Proceedings of the AAAI Conference on Artificial Intelligence (2021)

[56]
Silver, D., Huang, A., Maddison, C.J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al.: Mastering the game of go with deep neural networks and tree search. nature  (2016)

[57]
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: International Conference on Learning Representations (2015)

[58]
Sohn, K., Chang, H., Lezama, J., Polania, L., Zhang, H., Hao, Y., Essa, I., Jiang, L.: Visual prompt tuning for generative transfer learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023)

[59]
Song, Y., Durkan, C., Murray, I., Ermon, S.: Maximum likelihood training of score-based diffusion models. Advances in Neural Information Processing Systems  (2021)

[60]
Tan, M., Le, Q.: Efficientnet: Rethinking model scaling for convolutional neural networks. In: International conference on machine learning (2019)

[61]
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., Jégou, H.: Training data-efficient image transformers & distillation through attention. In: International conference on machine learning (2021)

[62]
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971  (2023)

[63]
Vahdat, A., Kautz, J.: Nvae: A deep hierarchical variational autoencoder. Advances in neural information processing systems  (2020)

[64]
Van Den Oord, A., Kalchbrenner, N., Kavukcuoglu, K.: Pixel recurrent neural networks. In: International conference on machine learning (2016)

[65]
Van Den Oord, A., Vinyals, O., et al.: Neural discrete representation learning. Advances in neural information processing systems  (2017)

[66]
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems  (2017)

[67]
Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S.: Cub-200-2011. Tech. Rep. CNS-TR-2011-001, California Institute of Technology (2011)

[68]
Wang, Z., Chen, J., Hoi, S.C.: Deep learning for image super-resolution: A survey. IEEE transactions on pattern analysis and machine intelligence  (2020)

[69]
Xie, E., Yao, L., Shi, H., Liu, Z., Zhou, D., Liu, Z., Li, J., Li, Z.: Difffit: Unlocking transferability of large diffusion models via simple parameter-efficient fine-tuning. arXiv preprint arXiv:2304.06648  (2023)

[70]
Xie, S., Girshick, R., Dollár, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition (2017)

[71]
YEH, S.Y., Hsieh, Y.G., Gao, Z., Yang, B.B., Oh, G., Gong, Y.: Navigating text-to-image customization: From lycoris fine-tuning to model evaluation. In: International Conference on Learning Representations (2023)

[72]
Yoon, J., Kim, S., Yang, E., Hwang, S.J.: Scalable and order-robust continual learning with additive parameter decomposition. In: International Conference on Learning Representations (2019)

[73]
Yu, W., Luo, M., Zhou, P., Si, C., Zhou, Y., Wang, X., Feng, J., Yan, S.: Metaformer is actually what you need for vision. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (2022)

[74]
Zaken, E.B., Goldberg, Y., Ravfogel, S.: Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (2022)

[75]
Zhai, M., Chen, L., Mori, G.: Hyper-lifelonggan: Scalable lifelong learning for image conditioned generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021)

[76]
Zhai, X., Puigcerver, J., Kolesnikov, A., Ruyssen, P., Riquelme, C., Lucic, M., Djolonga, J., Pinto, A.S., Neumann, M., Dosovitskiy, A., et al.: A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910.04867  (2019)

[77]
Zhang, Y., Yang, Q.: A survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering  (2021)

[78]
Zhou, K., Yang, J., Loy, C.C., Liu, Z.: Learning to prompt for vision-language models. International Journal of Computer Vision  (2022)

[79]
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., Fidler, S.: Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In: Proceedings of the IEEE international conference on computer vision (2015)

[80]
Zou, Z., Chen, K., Shi, Z., Guo, Y., Ye, J.: Object detection in 20 years: A survey. Proceedings of the IEEE  (2023)

§ DETAILS OF EXPERIMENTS

§.§ Details of Datasets

VTAB dataset is uniquely challenging and well-suited for the evaluation of parameter-efficient tuning methods in the context of few-shot knowledge transfer. VTAB-1k encompasses a diverse range of image domains, including natural, structured, and specialized categories such as medical or satellite imagery. The tasks span various objectives, comprising object and scene recognition, distance classification, and counting. Consequently, VTAB-1k emerges as a highly valuable resource catering to the needs of both discriminative and generative transfer learning tasks.

In Table <ref>, we provide information on 19 tasks of the VTAB dataset, including the number of classes and the number of images in each data split of VTAB. Images in the VTAB benchmark encompass three distinct domains: (1) Natural images captured using standard cameras, (2) Specialized images captured using non-standard cameras like those in remote sensing and medical applications, and (3) Structured images generated through simulation environments.

VTAB-1k is a subset of VTAB. It contains only 1000 training and validation samples, which are designed for few-shot transfer learning.

Information of VTAB dataset.

Dataset                     classes     train      val       test      all
Caltech-101                 102         2754       306       6084      9144

CIFAR-100                   100         45000      5000      10000     60000

Clevr (object distance)     6           63000      7000      15000     85000

Clevr (count)               8           63000      7000      15000     85000

Diabetic Retinopathy        5           35126      10906     42670     88702

DMLab                       6           65550      22628     22735     110913

Dsprites (x position)       16          589824     73728     73728     737280

Dsprites (orientation)      16          589824     73728     73728     737280

DTD                         47          1880       1880      1880      5640

EuroSAT                     10          16200      5400      5400      27000

Flowers102                  102         1020       1020      6149      8189

Kitti                       4           6347       423       711       7481

Patch Camelyon              2           262144     32768     32768     327680

Pet                         37          2944       736       3669      7349

Resisc45                    45          18900      6300      6300      31500

Smallnorb (azimuth)         18          24300      12150     12150     48600

Smallnorb (elevation)       9           24300      12150     12150     48600

SUN397                      397         76128      10875     21750     108753

SVHN                        10          65931      7326      26032     99289

§.§ Experimental Settings

§.§.§ LoRA Implementation

We adopt the LoRA implementation from <https://github.com/microsoft/LoRA>.

§.§.§ LoHa and LoKr Implementation

We adopt the LoHa and LoKr implementation from <https://github.com/KohakuBlueleaf/LyCORIS>.

§.§.§ DiffFit and BitFit Implementation

We adopt the DiffFit and BitFit implementation from <https://github.com/mkshing/DiffFit-pytorch>.

§.§.§ Generative Tasks

Stable diffusion checkpoints. The pre-trained checkpoint we choose for Stable Diffusion is stable-diffusion-v1-4, which can be found at <https://huggingface.co/CompVis/stable-diffusion>.

Text prompts for the few-shot generative task. We use specific text prompts to train the Stable Diffusion or generate the images. We list the example prompts for each dataset as follows:

* photo of a <castle>.

* The <castle> stands against a backdrop of snow-capped mountains.

* A <castle> surrounded by a lush, vibrant forest.

* The <castle> overlooks a serene lake.

* The <castle> in the autumn season with colorful foliage.

* The <castle> on a rocky cliff, with crashing waves below.

* The <castle> guarded by mythical elves.

* A <castle> surrounded by a field of grazing sheep.

* A peacock in front of the <castle>.

* The <castle> overlooks a serene lake, where a family of geese swims.

* <castle>, oil painting ghibli inspired.

* <castle> painting by artist claude monet.

* <castle> digital painting 3d render geometric style.

* Georgia O'Keeffe style <castle> painting.

* a watercolor painting of the <castle>.

* The <castle> is surrounded by an otherworldly landscape, with glowing mushrooms and mystical creatures.

* The <castle>, made of crystal, shimmers in the sunlight.

* The <castle>, steampunk aesthetic, adorned with gears and metallic accents.

* The <castle> atop a mystical floating island.

* Top view of the <castle>.

Text prompts for the full generative task. We use specific text prompts to train the Stable Diffusion or generate the images. We list the example prompts for each dataset as follows:

* Caltech-101: This is a picture of accordion.

* CIFAR-100: This is a picture of apple.

* Clevr: This is a picture from CLEVR dataset.

* Diabetic Retinopathy: This is a retina image with no diabetic retinopathy.

* DMLab: This is a picture from DMLab dataset.

* Dsprites: This is a picture from dSprites dataset.

* DTD: This is a picture of banded texture.

* EuroSAT: This is a satellite picture of annual crop.

* Flowers102: This is a picture of pink primrose.

* Kitti: This is a picture from KITTI dataset.

* Patch Camelyon: This is a histopathologic scans without tumor.

* Pet: This is a picture of Abyssinian cat.

* Resisc45: This is a remote sensing picture of airplane.

* Smallnorb: This is a picture from SmallNORB dataset.

* SUN397: This is a picture of abbey.

* SVHN: This is a picture of street view house number 0.

§ ADDITIONAL EXPERIMENTAL RESULTS

§.§ Validation Experiments

We provide additional experiments with $m=6,12$ in Figure <ref>. As we increase $m$ from 6 to 12, the accuracy improves from $66.86\%$ to $68.68\%$.

The relations between accuracy and number of fine-tuning parameters, with different numbers of filter atoms ($m=6$ and $m=12$).

§.§ Additional Experiments of Discriminative Tasks

§.§.§ Performance Comparisons on Full Dataset Fine-tuning.

Implementation details. For CIFAR-100 and ImageNet-1K, we follow the fine-tuning setting of ConvNeXt in [30]. We employ the AdamW [33] optimizer to fine-tune models for 100 epochs for CIFAR-100, and 30 epochs for ImageNet-1K. The cosine decay strategy is adopted for the learning rate schedule, and the linear warm-up is used in the first 10 epochs for CIFAR-100 and 5 epochs for ImageNet-1K.

We compare the performance of our approach with other baseline methods, and the results on CIFAR-100 and ImageNet-1K are shown in Table <ref>. With full dataset fine-tuning, the full fine-tuning achieves the highest accuracy, outperforming the parameter-efficient fine-tuning methods. One possible reason is both datasets have sufficient data to prevent over-fitting of the model. Our method achieves a higher accuracy than LoRA while requiring only a small number of parameters ($1.2$M v.s. $21$M).
In contrast, in the VTAB-1k benchmark, the amount of data is not very large (e.g., only 1,000 training images), which might cause over-fitting of the model for the full fine-tuning.

Performance comparisons on the VTAB-1k benchmark with ConvNeXT models pre-trained on ImageNet-21K.

CIFAR-100     Params. (M)     ImageNet-1k     Params. (M)
Full Fine-tuning         94.1                87.7                85.8               88.9

Linear Probe             88.6                0.1                 84.7               1.0

LoRA [16]                       89.2             20.1                 84.8             21.0
Ours                       91.8               0.3                  84.9             1.2

§.§.§ Visualization of Generalization Error.

To delve deeper into how various fine-tuning methods impact the generalization capabilities of pre-trained models, we illustrate in Figure <ref> the generalization error for a discriminative task trained on the CIFAR-100 and Diabetic Retinopathy datasets, in relation to the number of fine-tuned parameters.

Generalization error of (a) CIFAR-100 and (b) Diabetic Retinopathy.

§.§ Results of Few-shot Generative Tasks

We provide more experimental results of few-shot generative learning in Table. <ref> and <ref>. In this experiment, we also include LoRA, LoHa, and LoKr with different configurations.

The generated images of different fine-tuning methods are shown in Figure <ref> and <ref>.

Full Fine-tuning

Images sampled from Stable Diffusion [49] checkpoints fine-tuned with different approaches.
The text prompts used to generate images from top to bottom are: “The <castle> stands against a backdrop of snow-capped mountains”, “A <castle> surrounded by a lush, vibrant forest”, “A peacock in front of the <castle>”, and `The <castle> overlooks a serene lake, where a family of geese swims”.

Images sampled from Stable Diffusion [49] checkpoints fine-tuned with different approaches.
The text prompts used to generate images from top to bottom are: “The <castle> stands against a backdrop of snow-capped mountains”, “A <castle> surrounded by a lush, vibrant forest”, “A peacock in front of the <castle>”, and `The <castle> overlooks a serene lake, where a family of geese swims”.

Evaluate different approaches in learning the concept <castle>.

2c|LoHa         3c|LoRA     2c|LoKr     DiffFit     BitFit         3cOurs

$d=16$            $d=4$             $r=16$      $r=4$      $r=1$      $f=16$              $f=4$                                  $m=6,k_c=4$        $m=9,k_c=8$     $m=9,k_c=8,m_1=3$
Fidelity          0.73     0.73     0.71        0.67       0.7        0.73       0.65        0.57        0.44           0.44               0.62            0.72

Diversity         3.51              3.51              4.85        5.39       4.96       4.27                6.98        10.38       16.8     16.82     8.97            8.53

T2I Alignment     0.21              0.21              0.23        0.23       0.23       0.23                0.25        0.26        0.25           0.31      0.28            0.27
Param. (M)        33.86             8.47              22.67       5.67       1.42       1.12                1.06        0.58        0.34     0.05      0.75            2.39

Evaluate different approaches in learning the concept <canal>.

2c|LoHa      3c|LoRA     2c|LoKr     DiffFit     BitFit         3cOurs

$d=16$            $d=4$          $r=16$      $r=4$      $r=1$      $f=16$           $f=4$                                     $m=6,k_c=4$        $m=9,k_c=8$     $m=9,k_c=8,m_1=3$
Fidelity          0.52     0.47     0.39        0.38       0.37       0.36             0.38           0.31        0.33           0.16               0.29            0.39

Diversity         6.29              12.49          15.03       15.71      16.18      18.47            19.53          26.48       21.11          38.63     24.72           24.92

T2I Alignment     0.15               0.18           0.19        0.20       0.20       0.22             0.21           0.24        0.23           0.29      0.25            0.26
Param. (M)        33.86             8.47           22.67       5.67       1.42       1.12             1.06           0.58        0.34     0.05      0.75            2.39

§.§ Visualization of Generated Images

We visualize images generated by the models trained on each of VTAB tasks from Figure <ref> to Figure <ref>.

Full tuning

Images sampled from Stable Diffusion checkpoints fine-tuned on the Caltech-101.

Full tuning

Images sampled from Stable Diffusion checkpoints fine-tuned on the CIFAR-100.

Full tuning

Images sampled from Stable Diffusion checkpoints fine-tuned on the SUN397.

Full tuning

Images sampled from Stable Diffusion checkpoints fine-tuned on the SVHN.

Full tuning

Images sampled from Stable Diffusion checkpoints fine-tuned on the Flowers102.

Full tuning

Images sampled from Stable Diffusion checkpoints fine-tuned on the Pets.

Full tuning

Images sampled from Stable Diffusion checkpoints fine-tuned on the DTD.

Full tuning

Images sampled from Stable Diffusion checkpoints fine-tuned on the EuroSAT.

Full tuning

Images sampled from Stable Diffusion checkpoints fine-tuned on the Resisc45.

Full tuning

Images sampled from Stable Diffusion checkpoints fine-tuned on the Patch Camelyon.

Full tuning

Images sampled from Stable Diffusion checkpoints fine-tuned on the Diabetic Retinopathy.

Full tuning

Images sampled from Stable Diffusion checkpoints fine-tuned on the Kitti.

Full tuning

Images sampled from Stable Diffusion checkpoints fine-tuned on the Smallnorb.

Full tuning

Images sampled from Stable Diffusion checkpoints fine-tuned on the Dsprites.

Full tuning

Images sampled from Stable Diffusion checkpoints fine-tuned on the CLEVR.

Full tuning

Images sampled from Stable Diffusion checkpoints fine-tuned on the DMLab.

The Grad-CAM heatmap comparisons between our method and LoRA reveal that our approach exhibits larger active regions. The heatmap is generated from the first block of ResNet50 [13] utilizing the CUB dataset [67]. Fine-tuning the model with $\Delta \atom_{1}$ involves additional filter atoms, which leads to larger active regions in the heatmap compared to fine-tuning $\Delta \atom$ only. (a) The Grad-CAM from the first block of ResNet50. (b-d) The Grad-CAM from the 2-4 blocks of ResNet50.

§.§ Grad-CAM

To understand the underlying reason for the effectiveness of our approach on convolution-based models, we employ Grad-CAM [9] on the first block of ResNet50, which are fine-tuned on the CUB dataset [67] using the same experimental setting as above. For our method, we compare the experiment setting with $m=9$, which means 9 filter atoms $\Delta \atom$ and the setting with $(m, m_1)=(9,4)$, which means $36$ $\Delta \atom_{1}$.

Based on the Grad-CAM visualization in Figure <ref>, our method exhibits larger active regions compared with LoRA. This observation indicates that our approach benefits from preserving the spatial structure of convolutional layers. When utilizing $\Delta \atom_{1}$, which expands the number of filter atoms, we observe more active regions in the Grad-CAM heatmap. This suggests that the introduction of extra filter atoms potentially captures a wider range of feature maps.

We provide more heatmap visualizations of Grad-CAM from the first block of ResNet50 in Figure <ref>.

Additional Grad-CAM heatmap comparisons between our method and LoRA from the first block of ResNet50.