# Mini but Mighty: Finetuning ViTs with Mini Adapters

Imad Eddine Marouf Enzo Tartaglione Stéphane Lathuilière
LTCI, Télécom-Paris, Institut Polytechnique de Paris, France
<EMAIL_ADDRESS>
###### Abstract

Vision Transformers (ViTs) have become one of the dominant architectures in
computer vision, and pre-trained ViT models are commonly adapted to new tasks
via finetuning. Recent works proposed several parameter-efficient transfer
learning methods, such as adapters, to avoid the prohibitive training and
storage cost of finetuning.

In this work, we observe that adapters perform poorly when the dimension of
adapters is small, and we propose MiMi, a training framework that addresses
this issue. We start with large adapters which can reach high performance, and
iteratively reduce their size. To enable automatic estimation of the hidden
dimension of every adapter, we also introduce a new scoring function,
specifically designed for adapters, that compares the neuron importance across
layers. Our method outperforms existing methods in finding the best trade-off
between accuracy and trained parameters across the three dataset benchmarks
DomainNet, VTAB, and Multi-task, for a total of 29 datasets.111Code is
available: https://github.com/IemProg/MiMi

##  1 Introduction

Transformers have gained increasing attention owing to their outstanding
performance [9, 36, 56, 58]: Vision Transformers (ViTs) trained on large-scale
datasets have demonstrated a remarkable ability to learn new tasks [9]. The
most commonly adopted strategy to learn new tasks consists of fully or
partially fine-tuning a pre-trained network; however, when dealing with
multiple tasks, this approach necessitates training multiple separate models,
which results in large storage costs.

Figure 1: Layer-wise small blocks are injected into ViTs to efficiently adapt
to new domains. MiMi estimates the best rank for each adapter weight, it
reduces the number of parameters and removes completely injected adapters for
some layers if necessary.

Recently, Parameter-Efficient Training (PET) approaches have been developed to
help large pre-trained models adapt to new tasks, with minimal added
parameters [16, 21, 23]. Among these, adapters [20] and its variants [16, 27,
38] are frequently employed for Natural Language Processing (NLP) tasks.
Adapters are small modules inserted into transformer blocks, which enable
efficient adjustment of the data representation to the downstream task: they
offer similar performance to full fine-tuning (_i.e._ updating all parameters)
while requiring a very low number of trainable parameters [20, 50].

When it comes to vision tasks, PET approaches are mostly explored for
convolutional neural networks [3, 39, 51, 52]. In contrast, several PET
approaches have been proposed in NLP tasks: here, adapters are Multi-Layer-
Perceptrons (MLPs) equipped with residual connections [20]. These multi-layer
adapters can fit new tasks with enough representation power and the size of
their hidden layers provides a simple trade-off between performance and
parameter efficiency [20]. Nevertheless, they suffer from two weaknesses.
First, the performance drops when the size of multi-layer adapters is too
small [5] (as confirmed by our experiments -see Se. 4.1, and supplementary
material-). Second, the optimal hyper-parametrization of adapters is complex:
the hidden layer dimensions must be specified for every adapter in every
layer, and its optimal size depends on the downstream task. Thus, these
adapters cannot be employed where the available storage is limited.

In this work, we propose a training scheme named MiMi (Fig. 1) which addresses
these two limitations. Our approach facilitates efficient parameter allocation
by predominantly assigning additional parameters to layers that genuinely
necessitate adaptation to the new task (Fig. 5). More specifically, we start
by training adapters with high-dimensional hidden spaces and gradually
decrease their dimensionality by identifying neurons that can be omitted in
each adapter. Additionally, we introduce a novel scoring criterion to
determine the layers where more adaptation is needed, which enables the
comparison of a “neuron importance” among adapters in various layers.

Our work makes the following key contributions:

  * •

We propose a novel iterative training scheme for learning small adapters for
ViTs.

  * •

We present a new scoring function that can effectively compare the
significance of neurons across adapters. This approach enables us to estimate
the optimal hidden dimension of adapters for ViTs automatically, which leads
to a more efficient parameter allocation.

  * •

Finally, we compare the proposed approach with multiple PET methods designed
for both NLP and vision tasks using a total of 29 datasets. From these
experiments, we draw several conclusions: (i) we demonstrate that our approach
obtains the best performance in terms of accuracy among methods with similar
numbers of parameters; (ii) our ablation study validates the positive impact
of our adaptive strategy to automatically estimate the hidden dimension of
adapters.

##  2 Related Work

#### Vision Transformers.

Originally designed for NLP tasks, Transformers [59] have recently been
adapted for vision tasks, such as image classification. Vision Transformers
(ViTs) divide the image into patches, process them as token embeddings, and
employ transformer encoders with self-attention to learn image representations
[9]. ViTs have shown impressive performance, outperforming ConvNets in some
cases [13]. However, their large parameter count leads to significant storage
costs, limiting complete finetuning for each new task. This motivates our
study. While Swin [36] is a widely adopted ViT due to its excellent
performance across vision tasks, our approach of using tiny adapters can be
applied to any ViT architecture (see Sec. 3).

Network Pruning. When referred to deep neural networks, pruning consists of
reducing the number of parameters of a pre-trained model [8, 14]. It can be
roughly categorized into two groups: (i) unstructured pruning, which removes
the least significant weights (according to certain criteria like weight
magnitude [15] or gradient magnitude [42]) without a specific structure to be
followed; (ii) structured pruning, which focuses in removing model sub-
structures, like channels [55, 18] or attention heads [41]. Pruning techniques
usually reduce the number of parameters in a network trained for a specific
task, while MiMi decreases the number of parameters added through adapters
that fit the model to a new task without altering the original model’s
parameters. SparseAdapters (SA) [17], show that applying unstructured pruning
to adapters [20] achieves comparable performance. In comparison to SA, our
method incorporates a look-ahead strategy that considers the effects of
upsampling layers, while SA does not. Furthermore, MiMi employs structured
pruning, whereas SA utilizes unstructured pruning techniques to reduce the
size of adapters and remove them if necessary. GraSP [62] utilizes Hessian-
gradient products for each layer, discarding weights with elevated scores in a
single move, emphasizing those that improve gradient flow. Conversely, SNIP
[32] determines layer gradients using sampled mini-batch data, assigning
scores and eliminating weights with the highest scores in one step. However,
both these approaches are not apt for pruning Adapters. Our experiments show
that they do not perform well when applied to adapters.

Efficient Transformers Finetuning. ViTs’ lack of typical CNN inductive biases
makes their finetuning on new tasks susceptible to overfitting [13, 35].
Additionally, the need to update all the parameters and store a separate model
copy per task hinders scalability and real-world applicability. To tackle
this, three types of approaches have emerged: (i) updating only newly added
parameters [5, 20, 21, 23, 50]; (ii) sparse parameter updates [2, 21]; and
(iii) low-rank factorization for weight updates [27]. While prompt techniques
like VPT [23] achieve excellent performance, they lack flexibility for
downstream tasks that differ significantly from pre-training [5].

Our work falls into the first category, building on adapters [20] for NLP
tasks. However, we introduce a specific training algorithm enabling high
performance, with small-size adapters for downstream tasks. Unlike previous
adapter approaches [5, 20] with fixed-size adapters across layers [5], MiMi
dynamically assesses adapter sizes and even removes them if necessary. By
minimizing trainable parameters, MiMi enhances performance and reduces storage
footprint in multi-task scenarios. Our preliminary results demonstrate that
different layers require different adapter sizes, as highlighted in the
supplementary material.
In contrast, [68, 22] utilize Neural Architecture Search (NAS) to identify
optimal PET configurations, facilitating an expansive configuration search
space populated by various representative PET methods. However, this approach
is notably computation-intensive and uses different PET modules. Our work is
primarily concentrated on determining the appropriate adapter size for each
layer and eliminating certain adapters when deemed essential.

##  3 Proposed Method

In this section, we start with the description of adapters [20] and discuss
their practical benefits. Then, we introduce MiMi, our method to estimate the
hidden dimension for each adapter that can effectively maintain high
performance with fewer parameters for memory efficiency.

###  3.1 Preliminaries

Our objective is to adapt a pre-trained ViT network for a new task by
incorporating small modules called “adapters” into the existing layers. This
adaptation process involves training the linear classifier (referred to as the
“head”) and the adapter parameters while keeping the weights of the original
model frozen. In our training procedure, we focus on describing the adapter
parameters, and he linear classifier parameters are also learned
simultaneously.

ViT architectures, such as the original ViT [9] or Swin [36], consist of
layers with two main sub-layers: a multi-head self-attention (MSA) layer and a
fully-connected layer (MLP). Layer normalization is applied before each of
these sub-layers, and residual connections are employed to skip MSA and MLP.
In our approach, we introduce two adapters after each sub-layer. The adapter
is directly applied to the output of the corresponding sub-layer, as depicted
in Fig. 2a. The internal structure of the adapters is illustrated in Fig. 2b.

Considering the $i$-th adapter added to our pre-trained ViT, let
$\bm{h}_{i}\\!\in\\!\mathbb{R}^{M_{i}}$ denote its input, of size $M_{i}$.
Following [20], adapters employ a first fully-connected layer that down-
projects $\bm{h}_{i}$ into $\bm{z}_{i}\in\mathbb{R}^{N_{i}}$ with some non-
linear activation $\phi(\cdot)$. This layer is parametrized by a linear
projection matrix $\bm{W}^{\text{down}}_{i}\in\mathbb{R}^{M_{i}\times N_{i}}$.
Then, a second fully connected layer with parameters
$\bm{W}^{\text{up}}_{i}\in\mathbb{R}^{N_{i}\times M_{i}}$ up-samples
$\bm{z}_{i}$, producing as output $\bm{r}_{i}\in\mathbb{R}^{M_{i}}$. Finally,
a residual skip-connection is employed inside the adapter module such that, if
$\bm{r}_{i}$ is close to zero, the whole adapter module degenerates to an
identity function. To summarize, given the input vector $\bm{h}_{i}$, the
output vector $\bm{h}_{i}^{\prime}$ is calculated as:

$\bm{h^{\prime}}_{i}=\bm{W}^{\text{up}}_{i}\cdot\phi\left(\bm{W}^{\text{down}}_{i}\cdot\bm{h}_{i}\right)+\bm{h}_{i}.$
(1)

The total number of parameters in the adapter is equal to
$2\\!\cdot\\!N_{i}\\!\cdot\\!M_{i}$: since $M_{i}$ is fixed, we generally
choose $N_{i}$ such that $N_{i}\ll M_{i}$ to obtain a low number of
parameters. We define the compression rate $\sigma_{i}$ of an adapter as
$\sigma_{i}\\!=\\!\frac{M_{i}}{N_{i}}$.

Previous works [20, 50, 54] have employed adapters with a uniform hidden
dimension $N_{i}$ for all the adapters. However, this approach may not be
optimal as early and late layers to the input of the model may focus on
different types of patterns [5, 67] (see supplementary material). If we enable
dynamic adjustment of the adapter’s hidden dimension $N_{i}$ (or equivalently,
$\sigma_{i}$) and determine their injection points, we enhance adaptation to
downstream tasks effectively.

Figure 2: The adapter structure injected into ViT model (a), and our approach
to adjust the adapter’s size (b). MSA and MLP are multi-head self-attention
and feed-forward blocks, respectively.

###  3.2 Overview of MiMi

Let $\bm{W}^{\text{ViT}}$ be the initial parameters of the ViT model which are
frozen through the whole adaptation process. With MiMi our goal is to learn
$\bm{W}^{\text{ada}}$, the set containing the adapter parameters
$\bm{W}_{i}^{\text{ada}}=\bm{W}_{i}^{\text{up}}\cup\bm{W}_{i}^{\text{down}}$
of every $i$-th ViT sub-layer. In previous works [20, 52]
$\bm{W}_{i}^{\text{ada}}$ is straightforwardly learned with stochastic
gradient descent-based optimization; however, in our experiments (see Sec. 4)
we show that this approach does not perform well in the case of tiny adapters
(small $N_{i}$ values). We start from the observation that, with the existing
optimizers, sequentially training and pruning a large network is a successful
strategy to find small networks with good performance, while directly training
small networks usually suffers from optimization issues [10]. Therefore, we
propose to start from large adapters and adopt an iterative pruning strategy
that iteratively reduces their dimensions as detailed in Alg. 1.

Algorithm 1 MiMi

1:procedure MiMi($\bm{W}^{ViT}$, $\bm{W}^{ada}$, $\rho$, $\sigma^{target}$)

2: Learn $\bm{W}^{ada}$ $\triangleright$ $\bm{W}^{\text{ViT}}$ is frozen

3: while $\sigma<\sigma^{\text{target}}$ do

4: Sort $\bm{W}^{\text{ada}}$ according to $\mathcal{I}^{ij}$ (Sec. 3.3)

5: $\bm{W}^{\text{ada}}\leftarrow$ top$(1\\!-\\!\rho)$ in
$\bm{W}^{\text{ada}}$ $\triangleright$ Selection

6: Fine-tune $\bm{W}^{\text{ada}}$ $\triangleright$ $\bm{W}^{\text{ViT}}$ is
frozen

7: end while

8: return $\bm{W}^{\text{ada}}$

9:end procedure

We initialize every adapter with a hidden dimension proportional to its input
dimension. We start from compression rates $\sigma_{i}\\!=\\!\sigma_{0}$ for
every layer, where $\sigma_{0}$ is the initial compression rate. In our first
training stage (line 2), we learn the adapter parameters $\bm{W}^{\text{ada}}$
via cross-entropy loss minimization using stochastic gradient descent. Then,
we estimate a score that measures the importance of each adapter’s neurons
(more details will be provided in Sec. 3.3). This score is used to select the
neurons that have the smallest impact on the adapter outputs; more precisely,
we remove the bottom fraction $\rho$ of neurons from $\bm{W}^{\text{ada}}$
(line 5). The remaining ones will constitute the new adapter configuration,
and the hidden space sizes $N_{i}$ are updated accordingly. If the achieved
average compression rate $\sigma$ is still lower than the target
$\sigma^{\text{target}}$, another compression iteration follows; otherwise,
the achieved configuration will be returned and the method stops. Note that
the total number of training cycles $C$ is given by:

$C=\left\lceil\frac{\log\left(\sigma^{0}\right)-\log\left(\sigma^{\text{target}}\right)}{\log\left(\rho\right)}-1\right\rceil,$
(2)

where $\lceil\cdot\rceil$ denotes the ceiling function. Therefore, our
training scheme stops after a deterministic number of iterations that can be
computed in advance. While we employ a stopping criterion based on a specific
target compression rate, a target performance on a validation set could also
be used.

###  3.3 Importance Score in MiMi

In this section, we present the importance score function that we use in our
training algorithm. Our design of the scoring function is motivated by the
observation that, if an entire row in $\bm{W}_{i}^{\text{down}}$ and an entire
column in $\bm{W}_{i}^{\text{up}}$ are equal to zero, then our adapter is
strictly equivalent to one with a smaller dimension $M_{i}$. Therefore, we
propose a novel scoring function to employ the sum of the $L^{1}$ norm of the
corresponding row in $\bm{W}_{i}^{\text{down}}$ and the corresponding column
in $\bm{W}_{i}^{\text{up}}$. More precisely, our importance score is
formulated as follows:

$\mathcal{I}^{ij}=\frac{1}{N_{i}+M_{i}}\left(\sum_{k=1}^{M_{i}}\Big{|}W_{i}^{\text{down}}[j,k]\Big{|}+\sum_{k=1}^{N_{i}}\Big{|}W_{i}^{\text{up}}[k,j]\Big{|}\right),$
(3)

where $[\cdot,\cdot]$ denotes the matrix indexing operator. This importance
score can be interpreted as a “look-ahead” strategy, where we observe, besides
the output of a specific $j$-th neuron in the hidden space, also the impact of
such an output in the next layer. Note that this formulation is based only on
the magnitude of parameters belonging to the same neuron of down-sampling, and
its corresponding up-sampling one, and not on the magnitude of activations.
This makes the importance score more computationally efficient since
activation-based scoring depends on the input images, and consequently,
statistics should be gathered at the batch or the dataset level, inducing non-
negligible computation overhead. Furthermore, this choice is empirically
supported by many works in the literature, like [4, 15, 43, 53]. A noteworthy
element is that $\mathcal{I}^{ij}$ is normalized by the total number of
parameters associated with a specific dimension of the adapter: this enables
fair comparison across adapters, despite different input and hidden layer
sizes. More details behind the motivation of our choice for equation 3 are
provided in the supplementary material.

##  4 Experiments

We provide here the details about the datasets and our experimental setup.

Datasets. We evaluate our methods using the protocol adopted by [35], which
consists of ten datasets for image classification tasks divided into two
benchmarks. The first benchmark is known as _DomainNet_ [48]. It contains six
different visual domains, which makes the finetuning experiments non-trivial.
Since _DomainNet_ does not have a labeled testing set, we use the validation
dataset for testing, as in [48]. The second benchmark contains
CIFAR-10/CIFAR-100 [30], Oxford Flowers102 [46], and SVHN [44], which are
widely used as low-regime training datasets. Contrarily to DomainNet, these
datasets are not single-task oriented but contain a larger variety of
domains/tasks. We refer to them as belonging to the _Multi-task_ benchmark.
Additionally, we provide an evaluation of the _VTAB_ benchmark [65],
consisting of 19 diverse visual tasks (see supplementary).

Implementation Details. We follow the training protocol adopted by [35]. We
conduct our experiments with the official pre-trained Swin-T [36] ($\sim$27M
parameters) trained on ImageNet-1K. In all our experiments, we use the AdamW
[37] optimizer with a cosine decay learning-rate scheduler for 80 epochs,
preceded by 20 epochs of linear warm-up. In all the experiments, the images
are resized to the same fixed resolution (224 × 224). With MiMi, $\rho$ is set
to $50\%$, namely we half the number of neurons in the adapters, at every 100
epochs.

###  4.1 Main results

We compare our proposed method MiMi with multiple PETs methods. We remark that
all the baselines are obtained with a $C\\!=\\!5$ cycles training, while MiMi
will always have a lower or equal number of training cycles in Tab. 1. We
include the following methods:

  * •

Full finetuning: finetune all parameters of the model.

  * •

Att/MLP finetune: we only tune the Attention/MLP layers and the classification
head.

  * •

Linear-probe: all parameters are frozen except for the task-specific
classification layer.

  * •

Adapters [20]: we add adapters with $\sigma=32$ to have adapters with hidden
dimensionality proportional to the input dimension $M_{i}$. We also include
variants where the size of every adapter is fixed over all the layers:
$N_{i}\\!=\\!47$, and $N_{i}\\!=\\!23$. These baselines are considered to
emphasize the effect of parameter allocation throughout the layers on the
final performance.

  * •

BitFit [2]: only the biases are finetuned.

  * •

PHM-Adapter [66]: the weights of the adapters are learned using parameterized
hyper-complex multiplication layers (PHM) layers.

  * •

Compacter [27]: adapter weights are learned using shared PHM layers.

  * •

AdaptFormer [5]: introduces Adapters, but only after MLP block with a scaling
parameter $s$ applied to the output of the injected modules.

  * •

VPT [23]: finetuning learnable parameters (i.e. prompts) injected into the
embedding space for each layer in ViT.

  * •

SSF [34]: aims to adjust the feature activation scaling and shifting its
output.

  * •

Fact-TK [24]: a tensorization-decomposition method to store the weight updates
into a single 3D tensor.

Figure 3: Evaluation of PET baselines mean top-1 accuracy on Multi-task
benchmark. We observe MiMi (◆) maintains good performance when reducing the
number of parameters, compared to other PET baselines.

Method | Params | Trained | C100 | C10 | F. | S. | Mean
---|---|---|---|---|---|---|---
|  (M) $\downarrow$ |  (%) $\downarrow$ |  |  |  |  |
Full finetuning | 27.8 | 100 | 88.13 | 98.50 | 97.35 | 96.59 | 95.14
Att-blocks | 8.93 | 32.14 | 88.03 | 98.41 | 97.79 | 95.99 | 95.05
MLP-blocks | 17.54 | 63.12 | 88.44 | 98.47 | 96.50 | 96.14 | 94.89
MiMi (0 cycle)† | 4.35 | 15.81 | 88.27 | 98.53 | 97.59 | 97.28 | 95.41
AdaptFormer-64 | 0.66 | 2.38 | 83.79 | 96.93 | 90.50 | 92.45 | 90.91
AdaptFormer-256 | 2.98 | 8.55 | 84.74 | 97.23 | 92.13 | 94.97 | 92.27
Adapters $N_{i}\\!=\\!47$ | 1.37 | 4.90 | 85.04 | 97.52 | 92.72 | 96.35 | 92.91
Adapters $N_{i}\\!=\\!23$ | 0.68 | 2.47 | 85.18 | 97.57 | 92.16 | 95.81 | 92.68
Adapters $\sigma\\!=\\!32$ | 1.37 | 4.90 | 85.59 | 97.49 | 94.80 | 96.27 | 93.53
Linear prob | 0.27 | 0.95 | 75.58 | 91.84 | 76.80 | 55.26 | 74.87
PHM-Adapter | 0.47 | 1.72 | 84.17 | 96.48 | 89.18 | 93.32 | 90.78
Compacter | 0.41 | 1.44 | 83.95 | 96.26 | 88.43 | 92.67 | 90.32
BitFit | 0.34 | 1.22 | 83.56 | 96.14 | 87.85 | 90.29 | 89.46
VPT-deep (10 tokens) | 0.33 | 1.20 | 67.69 | 90.99 | 22.77 | 85.11 | 66.64
VPT-deep (100 tokens) | 0.52 | 1.88 | 72.53 | 93.03 | 34.88 | 86.70 | 71.78
Adapters $N_{i}\\!=\\!1$ | 0.30 | 1.07 | 82.60 | 96.03 | 89.77 | 88.80 | 89.30
SSF | 0.28 | 0.96 | 83.02 | 96.46 | 95.59 | 95.11 | 92.54
Fact-TK32 | 0.33 | 1.18 | 82.91 | 96.59 | 87.46 | 90.84 | 89.45
MiMi (1 cycle) | 0.80 | 2.89 | 87.12 | 97.98 | 96.59 | 96.98 | 94.67
MiMi (2 cycles) | 0.53 | 1.92 | 86.33 | 97.49 | 96.73 | 96.48 | 94.26
MiMi (3 cycles) | 0.40 | 1.43 | 85.22 | 97.11 | 96.81 | 95.60 | 93.69
MiMi (4 cycles) | 0.30 | 1.07 | 84.07 | 97.11 | 96.81 | 93.94 | 92.98

Table 1: Results on the Multi-task benchmark. C100, C10, F and S stand for
CIFAR100, CIFAR10, Flowers, and SVHN. † is equivalent to Adapters with
$\sigma=8$. Methods are grouped according to the relative number of trainable
parameters ($\leq 2\%$, $\in]2,10[\%$,$\geq 10\%$)

Discussion. Fig. 3 visualizes the average accuracy versus the number of
trainable parameters achieved for the Multi-task benchmark, while Table 1
reports the number of trained parameters and the average accuracy across
datasets in the MultiTask benchmark. The detailed Tables for DomainNet and
VTAB benchmarks are in the supplementary material. For all the benchmarks, the
number of trained parameters is reported in millions, and the average top-1
accuracy on the datasets is reported in the rightest column.

We observe that _full finetuning_ achieves commendable performance, albeit
demanding an extensive parameter-tuning for each dataset. In comparison,
finetuning solely the _attention/MLP_ layer proves remarkably effective among
the vanilla finetuning baselines. However, this approach still necessitates a
substantial number of task-specific parameters, unlike other PET approaches.
Notably, the underwhelming performance of _linear probing_ emphasizes the
significance of altering the feature representations within the model when
adapting to new tasks.

Notably, both _PHM_ and _Compacter_ demonstrate their effectiveness by
achieving impressive performance while adjusting less than 2% of the
parameters. Unlike NLP tasks where PETs have shown success with a small number
of trainable parameters [20], visual tasks do not attain _full finetuning_
performance with such limited parameter adjustments. Additionally, the subpar
performance of _VPT_ indicates that injecting tokens into the embedding space
offers minimal benefit when the pre-training dataset differs from the
downstream task. Remarkably, all PET methods consistently maintain similar
performance rankings across all tasks, suggesting that the optimal adaptation
strategy is independent on the specific downstream task.

_Adapters_ achieve impressive results with a slightly higher number of
trainable parameters (1.37M, 4.90% of the total) for $\sigma\\!=\\!32$.
Remarkably, _Adapters_ outperform _AdaptFormer_ [5] while utilizing fewer
parameters (92.91% with 1.37M parameters compared to 92.27% with 2.98M
parameters). This outcome highlights the superiority of adapting
representations after both MSA and MLP blocks, as demonstrated in the
architecture of _Adapters_ (Fig. 2), over solely acting on the MLP block, as
in done _AdaptFormer_.

We observe that _MiMi_ significantly reduces the number of parameters by 4
times the initial size (0.40M, $1.43\%$) while outperforming all PET methods
in the Multi-task benchmark. In particular, MiMi outperforms _adapters-
$n_{i}\\!=\\!47$_ despite having fewer parameters, demonstrating that our
iterative training procedure improves the parameter efficiency. To further
emphasize the performance gap between the two approaches, we introduce Fig. 4
illustrating the performance as a function of the number of trainable
parameters for VGG-Flowers (for CIFAR-100 dataset in supplementary). We
observe the significant performance gap between vanilla adapters compared to
adapters trained with the MiMi approach.

Furthermore, MiMi outperforms methods with similar trained parameters, in all
the compression ranges. In particular, in the most challenging one (with 0.30M
parameters), MiMi outperforms the closest approach, BitFit, which trains 0.34M
parameters, showing a gain in average accuracy larger than $3\%$ and $2\%$,
for Multi-Task and DomainNet benchmarks, respectively.

Figure 4: Comparison of top-1 accuracy versus compression rate $\sigma$ on
VGG-Flowers. All MiMi results originate from the same MiMi run. Adapters are
trained for the exact same number of epochs as their MiMi counterparts. The
size of blob markers represents the number of trainable parameters.

Upon comparing _adapter_ with uniform and proportional parameter distribution
($N_{i}$ vs $\sigma$), results are in favor of allocating parameters
proportionally to the layer dimension. Notably, adapters with
$\sigma\\!=\\!32$ outperform adapters with $N_{i}\\!=\\!47~{}\forall i$ in
both the Multi-task (93.53% vs 92.91%) and DomainNet (70.65% vs 69.39%)
benchmarks. This observation suggests that the task-specific nature of the
last layers, with higher dimensionality, necessitates greater adaptation.
Furthermore, we demonstrate that reducing the size of adapters
($N_{i}\\!=\\!23$) negatively affects performance, with a marginal drop in
Multi-task (0.23%) but a more consistent decrease in DomainNet (1.01%). These
findings underscore the unsatisfactory performance obtained from training
adapters in a vanilla fashion and serve as motivation for our specific
training procedure.
MiMi versus Vanilla training. Looking at the Multi-task benchmark (Fig. 3,
Table 1), we observe that _MiMi_ significantly reduces the number of
parameters by 4$\times$ (0.40M, $1.43\%$) while outperforming all PET methods
in the Multi-task benchmark. In particular, _MiMi_ outperforms _adapters-
$N_{i}=47$_ despite having fewer parameters, demonstrating that our iterative
training procedure improves the parameter efficiency. To further emphasize the
performance gap between the two approaches, we introduce Fig. 4, we observe
the significant performance gap between vanilla adapters compared to adapters
trained with MiMi approach.

###  4.2 Ablation study

#### Importance score for MiMi.

Next, we move on to our design choice of dimensionality reduction inside
adapters throughout the adaptation cycles. We report the contribution of
various components of MiMi with different setups.

  * •

_Vanilla Adapters_ : corresponds to injecting adapters with a compression rate
$\sigma$.

  * •

Random: we select randomly a percentage of neurons for each adapter to be
removed.

  * •

Gradient-Based $L^{1}(\nabla)$: We determine the neurons to be removed based
on the $L^{1}$ norm of the gradients.

  * •

Local neuron selection: We uniformly choose a percentage of neurons to be
removed, independently applied to both down-sampling and up-sampling layers.

  * •

Global neuron selection: The number of neurons to be removed per adapter is
determined using equation 3 given $\rho$, considering the scaling factor if
applicable. Additionally, we assess our scoring function without the inclusion
of the scaling factor $n_{i}+m_{i}$. This modified version of our score is
referred to as $\mathcal{I}_{0}$.

  * •

MiMi: our method as in Alg. 1.

To compare the different methods we proceed as follows. When using an
iterative method, we always start from the baseline model where
$\sigma\\!=\\!32$. When using a non-iterative method: we start with adapters
of $\sigma_{0}\\!=\\!\sigma_{target}/(1-\rho)$ and prune once only after the
first cycle. Training continues for $C\\!-\\!1$ cycles to guarantee fair
comparison with iterative methods. Results are reported in Table 2.

Method | Selection | Score | Iter. | Scale | $\bm{\sigma}$
---|---|---|---|---|---
$32$ | $64$ | $128$ | $256$ | $512$
Vanilla | - | - | - | - | 94.80 | 90.12 | 89.42 | 88.85 | 86.03
Baseline | Random | $-$ | ✓ |  | - | 95.43 | 95.80 | 95.11 | 95.12
Local (DW) | $L^{1}(w)$ |  |  | - | 95.46 | 95.06 | 94.28 | 93.79
Local | $L^{1}(\nabla)$ | ✓ |  | - | 95.46 | 96.17 | 96.11 | 96.42
Local | SA [17] |  |  | - | 96.10 | 95.57 | 96.15 | 96.23
Local | SA [17] | ✓ |  | - | 96.41 | 96.65 | 96.72 | 96.72
Local | GraSP [62] |  |  |  | 90.62 | 89.71 | 87.22 | 86.66
Local | SNIP [32] |  |  |  | 93.53 | 92.39 | 91.36 | 90.62
Global | $L^{1}(a)$ |  | ✓ | - | 96.10 | 94.28 | 93.80 | 93.25
Global | $L^{1}(a)$ | ✓ | ✓ | - | 96.13 | 95.15 | 95.77 | 95.72
Global | $\mathcal{I}_{0}$ |  |  | - | 94.88 | 95.28 | 95.66 | 95.45
Global | $\mathcal{I}$ |  | ✓ | - | 96.10 | 95.82 | 96.34 | 96.50
MiMi | Global | $\mathcal{I}$ | ✓ | ✓ | - | 96.59 | 96.92 | 96.73 | 96.81

Table 2: Performance analysis for neuron selection on VGG-Flowers. $L^{1}(w)$
and $L^{1}(a)$ denote the magnitude pruning of the parameters and the
activations respectively. Local (DW) represents local pruning applied to down-
sampling layers only.

Discussion. Table 2 summarizes the performance of local and global neuron
selection for adapters. Firstly, we observe that reducing the number of
parameters in vanilla adapters (higher values of $\sigma$ in Fig. 4) leads to
a drop in performance. Additionally, we find that using the magnitude of
parameters instead of activations is advantageous. Activation-based scoring
methods rely on input images, requiring batch or dataset-level statistics,
which are computationally less efficient.

Secondly, global neuron selection proves to be superior to local neuron
selection. The former method focuses on finetuning adapters in specific layers
while completely removing other adapters, while the latter removes the same
amount of neurons from each adapter’s layer. Moreover, MiMi surpasses SA by
employing structured pruning (neuron removal) instead of unstructured pruning
(weight removal), to reduce the size of adapters. Additionally, MiMi
incorporates a look-ahead strategy that accounts for the impact of up-sampling
layers, ensuring consistently high performance. Notably, with MiMi, adapter
size can be reduced for efficient computation, unlike SA.

In the final cycles, MiMi identifies crucial adapters for adaptation,
prioritizing their finetuning. This approach improves model performance by
adjusting specific latent representations tailored to the downstream task
while using fewer parameters.  MiMi consistently outperforms both GraSP and
SNIP across all $\sigma$ values due to its iterative pruning approach. Pruning
at initialization as done in SNIP/GraSP, before the adapters have been trained
are less effective. Since the initialization is random, they are missing out
on retaining potentially important weights.

Table 2 reveals that MiMi achieves superior performance compared to vanilla
adapter training on the VGG-Flowers dataset, with a performance gap of
$6.14\%$, when using $\sigma=64$ (regardless of local/global neuron
selection). Notably, this performance gap increases as we reduce the adapter
size to $\sigma=256,512$. Furthermore, when comparing to a vanilla $L^{1}$
importance scoring, we observe the benefits of considering both down-sampling
and up-sampling parameters for the adapters. This approach consistently
improves performance across compression rates ranging from $0.5\%$ to over
$3\%$. Notably, the performance gap becomes more prominent at higher
compression rates.

Finally, scaling the importance score according to equation 3 enhances the
performance of the _Global_ method by approximately $1\%$ across all $\sigma$
values.

Figure 5: Layer-wise analysis of adapter’s neurons distribution at $4^{th}$
cycle. Bar plots represent the number of neurons $n_{i}$ at each adapter $i$
for VGG-Flowers and CIFAR-10, respectively. Global neuron selection leads to
different neuron distributions depending on the dataset. Compared to VGG-
Flowers, fewer adapters are completely removed on CIFAR-10.

Parameter allocation analysis with MiMi. In Fig. 5, we visualize the
distribution of removed and remaining neurons achieved through the application
of MiMi on VGG-Flowers and CIFAR-10. Notably, this illustration highlights the
contrasting outcomes between local neuron selection, which uniformly removes
neurons from adapters, and the utilization of global neuron selection.
Remarkably, we observe that the latter approach completely removes certain
adapters from the model (evident in layers $4,5,7,$ and $8$ of VGG-Flowers),
while redistributing a significant number of parameters to other adapters.

Moreover, _global neuron selection_ exhibits distinct adaptations for each
dataset, as evidenced in Fig. 5. Notably, the distribution of removed neurons
varies between CIFAR-10 and VGG-Flowers. In the case of CIFAR-10, fewer
adapters are completely removed compared to VGG-Flowers. Conversely, for VGG-
Flowers, only adapters at later stages are retained, suggesting that early
layer representations are well-suited for this particular dataset. However,
for CIFAR-10, the remaining adapters are dispersed throughout all layers of
the ViT model, indicating that almost all the layers’ representations need to
be finetuned. These observations highlight the adaptability and dataset-
specific optimizations achieved through _global neuron selection_. To provide
a more comprehensive analysis, we also present normalized plots in
supplementary material.

ViT variants with MiMi. We evaluate the performance of MiMi on different ViT
backbones, including Swin-S/L ($\sim$50M/$\sim$197M parameters), ViT [9]
($\sim$86M parameters), and CVT [63] ($\sim$20M parameters).

For three training cycles, we compare the three baselines: _finetuning_ ,
_adapters_ , and _MiMi_. Table 3 presents the best scores achieved in the
final cycle. Remarkably, MiMi achieves comparable performance to full model
finetuning, with a margin of 1.2%, 1.2%, and 1.4% for ViT-B/16, Swin-S, and
CvT, respectively. This is accomplished by finetuning less than 1.5% of the
parameters, including the head classifier. MiMi surpasses vanilla adapters’
performance with four times fewer parameters across all ViT backbones (ViT,
Swin-T, and CvT). These experiments demonstrate the generalizability of MiMi
to various ViT backbones.

| Method | # Params | Trained | C100 | C10 | V | S | Mean
---|---|---|---|---|---|---|---|---
|  |  (M) $\downarrow$ |  (%) $\downarrow$ |  |  |  |  | $\uparrow$
ViT-B-16 | Finetune | 85.90 | 100 | 91.22 | 99.01 | 99.32 | 97.68 | 96.81
Adapters | 0.96 | 0.89 | 89.39 | 98.02 | 97.69 | 94.17 | 94.82
MiMi | 0.62 | 0.54 | 89.86 | 98.09 | 98.75 | 94.94 | 95.41
MiMi | 0.37 | 0.32 | 89.84 | 98.17 | 98.85 | 95.32 | 95.55
Swin-S | Finetune | 48.80 | 100 | 90.12 | 98.88 | 98.37 | 98.16 | 96.38
Adapters | 0.41 | 4.88 | 89.05 | 98.48 | 94.60 | 97.25 | 94.84
MiMi | 0.23 | 2.75 | 88.86 | 98.53 | 96.16 | 97.22 | 95.19
MiMi | 0.11 | 1.32 | 88.62 | 98.50 | 96.68 | 96.94 | 95.18
Swin-L | Finetune | 197M | 100% | 95.12 | 99.34 | 99.67 | 98.22 | 98.08
Adapters | 20.1M | 10.2% | 94.31 | 99.46 | 99.76 | 97.98 | 97.88
MiMi | 10.9M | 5.53% | 94.78 | 99.44 | 99.51 | 99.77 | 98.38
MiMi | 6M | 3.04% | 92.92 | 99.30 | 99.74 | 97.96 | 97.48
CvT | Finetune | 19.65 | 100 | 90.01 | 98.68 | 97.98 | 98.09 | 96.19
Adapters | 0.78 | 4.00 | 86.68 | 97.91 | 88.93 | 96.96 | 92.62
MiMi | 0.47 | 2.40 | 86.47 | 97.98 | 93.28 | 97.17 | 93.73
MiMi | 0.28 | 1.44 | 85.87 | 97.77 | 94.31 | 96.67 | 93.66

Table 3: Performance of our method using different ViT backbones on the Multi-
task benchmark. The highest score is in bold and the second best score is
underlined. C100, C10, F, and S stand for CIFAR100, CIFAR10, Flowers, and SVHN
datasets.

Evaluating Inference Cost/Storage Footprint. In this section, we conduct a
comprehensive analysis of the GFLOPS at inference time and storage footprint
for PETs methods in the context of multi-task learning. Table 4 presents the
findings, including the number of trainable parameters and the storage
requirement in MegaBytes (MB) for saving the Swin-T model after finetuning per
task $T$.

Storing a complete ViT for each task can impose significant burdens on storage
space and computational resources. With ViTs containing millions of
parameters, these storage requirements quickly accumulate in multi-task
settings. However, by storing only a subset of the model’s parameters, both
storage costs and computational resources for training can be significantly
reduced.

Method | # Params (M)$\downarrow$ | Storage (MB)$\downarrow$ | GFLOPS $\downarrow$ | Accuracy(%) $\uparrow$
---|---|---|---|---
Full-finetuning | 27.8 | 111 | 8.72 | 97.35
Att-block | 8.93 | 34.7 | 8.72 | 97.79
MLP-blocks | 17.54 | 73.4 | 8.72 | 96.50
Full-model W/ Adapter($\sigma=32$) | 1.37 | 115.3 | 9.06 | 96.27
BitFit | 0.34 | 0.34 | 8.72 | 87.85
VPT-Deep (100 tokens) | 0.32 | 160.1 | 18.40 | 34.88
AdaptFormer-64 | 0.84 | 1.63 | 9.08 | 90.50
SSF | 0.28 | 0.96 | 8.72 | 95.59
Fact-TK32 | 0.33 | 1.18 | 10.6 | 87.46
Adapters($\sigma=32$) | 1.37 | 4.30 | 9.06 | 96.27
Adapters($n_{i}=47$) | 1.37 | 4.40 | 9.26 | 92.72
Adapters($n_{i}=1$) | 0.30 | 0.47 | 8.74 | 89.77
Linear-prob | 0.27 | 0.31 | 8.72 | 76.80
MiMi (3 cycles) | 0.40 | 0.63 | 8.92 | 96.81
MiMi (4 cycles) | 0.30 | 0.47 | 8.82 | 96.81

Table 4: Comparison with different PET methods on VGG-Flowers with respect to
inference cost (GFLOPs).

Larger model components, like attention and MLP blocks, demand significant
storage due to their extensive trainable parameters. On the other hand,
finetuning only the head (Linear-prob) is lightweight but comes at the cost of
performance. Notably, MiMi achieves a +6% performance enhancement compared to
Adapters $n_{i}\\!=\\!1$ while maintaining similar storage needs. However,
MiMi offers both robust performance and reduced memory demands, positioning it
as a superior alternative.

In terms of GFLOPS during inference, full fine-tuning, and similar variants,
such as Att-block and MLP-blocks, achieve the lowest GFLOP values at 8.72.
However, they come at the expense of a high memory footprint. On the other
hand, VPT-Deep (100 tokens) stands out with the highest GFLOPS at 18.40,
thanks to an increase in the embedding space for each layer to 100 tokens.
This emphasizes that fewer parameters do not necessarily guarantee
computational efficiency. MiMi in its 3-cycle and 4-cycle variants, achieves
GFLOPS values of 8.92 and 8.82, respectively. This efficiency is attributed to
our method’s ability to completely remove some adapters, effectively reducing
the computational cost during inference.

##  5 Conclusion

In this work, we propose MiMi, a training algorithm to learn small adapters
for the problem of ViT efficient finetuning. Rather than directly training
adapters with few parameters, we propose to start with large adapters, and
then iteratively select the more important neurons in every adapter. Our
training procedure estimates the hidden dimension for each adapter, reducing
the number of trainable parameters and even removing adapters if unnecessary.
We empirically demonstrate the greater performance of MiMi to adapters and
show that our method achieves excellent performance with low numbers of
trainable parameters. Our ablation study validates the positive impact of our
novel importance score to estimate the hidden dimension of each adapter.
Acknowledgements. This paper has been supported by the French National
Research Agency (ANR) in the framework of its JCJC. Furthermore, this research
was partially funded by Hi!PARIS Center on Data Analytics and Artificial
Intelligence.

## References

  * [1] Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, et al.  Deepmind lab.  arXiv preprint arXiv:1612.03801, 2016.
  * [2] Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel.  BitFit: Simple parameter-efficient fine-tuning for transformer-based masked language-models.  In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Dublin, Ireland, May 2022\. Association for Computational Linguistics.
  * [3] Rodrigo Berriel, Stéphane Lathuillère, Moin Nabi, Tassilo Klein, Thiago Oliveira-Santos, Nicu Sebe, and Elisa Ricci.  Budget-aware adapters for multi-domain learning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019.
  * [4] Yves Chauvin.  A back-propagation algorithm with optimal use of hidden units.  Advances in neural information processing systems, 1, 1988.
  * [5] Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo.  Adaptformer: Adapting vision transformers for scalable visual recognition, 2022.
  * [6] Gong Cheng, Junwei Han, and Xiaoqiang Lu.  Remote sensing image scene classification: Benchmark and state of the art.  Proceedings of the IEEE, 2017.
  * [7] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, , and A. Vedaldi.  Describing textures in the wild.  In CVPR, 2014.
  * [8] Yann Le Cun, John S. Denker, and Sara A. Solla.  Optimal brain damage.  In Advances in Neural Information Processing Systems. Morgan Kaufmann, 1990.
  * [9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations, 2020.
  * [10] Jonathan Frankle and Michael Carbin.  The lottery ticket hypothesis: Finding sparse, trainable neural networks.  In International Conference on Learning Representations, 2018.
  * [11] Timnit Gebru, Jonathan Krause, Yilun Wang, Duyun Chen, Jia Deng, and Li Fei-Fei.  Fine-grained car detection for visual census estimation.  In AAAI, 2017.
  * [12] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun.  Vision meets robotics: The kitti dataset.  International Journal of Robotics Research, 2013.
  * [13] Jianyuan Guo, Kai Han, Han Wu, Yehui Tang, Xinghao Chen, Yunhe Wang, and Chang Xu.  Cmt: Convolutional neural networks meet vision transformers.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
  * [14] Song Han, Huizi Mao, and William J. Dally.  Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding.  In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
  * [15] Song Han, Jeff Pool, John Tran, and William Dally.  Learning both weights and connections for efficient neural network.  Advances in neural information processing systems, 28, 2015.
  * [16] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig.  Towards a unified view of parameter-efficient transfer learning.  In International Conference on Learning Representations, 2021.
  * [17] Shwai He, Liang Ding, Daize Dong, Miao Zhang, and Dacheng Tao.  Sparseadapter: An easy approach for improving the parameter-efficiency of adapters, 2022.
  * [18] Yihui He, Xiangyu Zhang, and Jian Sun.  Channel pruning for accelerating very deep neural networks.  In Proceedings of the IEEE international conference on computer vision, 2017.
  * [19] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth.  Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
  * [20] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.  Parameter-efficient transfer learning for nlp.  In International Conference on Machine Learning. PMLR, 2019.
  * [21] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al.  Lora: Low-rank adaptation of large language models.  In International Conference on Learning Representations, 2021.
  * [22] Shengding Hu, Zhen Zhang, Ning Ding, Yadao Wang, Yasheng Wang, Zhiyuan Liu, and Maosong Sun.  Sparse structure search for parameter-efficient tuning, 2022.
  * [23] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim.  Visual prompt tuning, 2022.
  * [24] Shibo Jie and Zhi-Hong Deng.  Fact: Factor-tuning for lightweight adaptation on vision transformer, 2023\.
  * [25] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick.  Clevr: A diagnostic dataset for compositional language and elementary visual reasoning.  In CVPR, 2017.
  * [26] Kaggle and EyePacs.  Kaggle diabetic retinopathy detection, July 2015.
  * [27] Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder.  Compacter: Efficient low-rank hypercomplex adapter layers.  Advances in Neural Information Processing Systems, 34, 2021.
  * [28] Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei.  Novel dataset for fine-grained image categorization.  In First Workshop on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, CO, June 2011.
  * [29] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images, 2009.
  * [30] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton.  Cifar-100 (canadian institute for advanced research).
  * [31] Yann LeCun, Fu Jie Huang, and Leon Bottou.  Learning methods for generic object recognition with invariance to pose and lighting.  In CVPR, 2004.
  * [32] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip H. S. Torr.  Snip: Single-shot network pruning based on connection sensitivity, 2018\.
  * [33] Fei-Fei Li, Rob Fergus, and Pietro Perona.  One-shot learning of object categories.  IEEE TPAMI, 2006.
  * [34] Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang.  Scaling and shifting your features: A new baseline for efficient model tuning, 2023.
  * [35] Yahui Liu, Enver Sangineto, Wei Bi, Nicu Sebe, Bruno Lepri, and Marco De Nadai.  Efficient training of visual transformers with small datasets.  In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
  * [36] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
  * [37] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In International Conference on Learning Representations, 2019.
  * [38] Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson.  Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks.  In ACL/IJCNLP, 2021.
  * [39] Arun Mallya, Dillon Davis, and Svetlana Lazebnik.  Piggyback: Adapting a single network to multiple tasks by learning to mask weights.  In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
  * [40] Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner.  dsprites: Disentanglement testing sprites dataset.  https://github.com/deepmind/dsprites-dataset/, 2017.
  * [41] Paul Michel, Omer Levy, and Graham Neubig.  Are sixteen heads really better than one?  Advances in neural information processing systems, 32, 2019.
  * [42] Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz.  Importance estimation for neural network pruning.  2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [43] Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz.  Pruning convolutional neural networks for resource efficient inference.  In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, volume abs/1608.08710. OpenReview.net, 2017.
  * [44] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng.  Reading digits in natural images with unsupervised feature learning.  In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011.
  * [45] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng.  Reading digits in natural images with unsupervised feature learning.  In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011.
  * [46] Maria-Elena Nilsback and Andrew Zisserman.  Automated flower classification over a large number of classes.  In Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008.
  * [47] Maria-Elena Nilsback and Andrew Zisserman.  Automated flower classification over a large number of classes.  In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing. IEEE, 2008.
  * [48] Yingwei Pan, Yehao Li, Qi Cai, Yang Chen, and Ting Yao.  Multi-source domain adaptation and semi-supervised domain adaptation with focus on visual domain adaptation challenge 2019, 2019.
  * [49] O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. V. Jawahar.  Cats and dogs.  In CVPR, 2012.
  * [50] Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych.  AdapterFusion: Non-destructive task composition for transfer learning.  In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, Online, Apr. 2021\. Association for Computational Linguistics.
  * [51] Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi.  Learning multiple visual domains with residual adapters.  Advances in neural information processing systems, 30, 2017.
  * [52] Sylvestre-Alvise Rebuffi, Andrea Vedaldi, and Hakan Bilen.  Efficient parametrization of multi-domain deep neural networks.  In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.
  * [53] Alex Renda, Jonathan Frankle, and Michael Carbin.  Comparing rewinding and fine-tuning in neural network pruning.  In International Conference on Learning Representations, 2020.
  * [54] Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych.  Adapterdrop: On the efficiency of adapters in transformers.  In EMNLP (1), 2021.
  * [55] Enzo Tartaglione, Skjalg Lepsøy, Attilio Fiandrotti, and Gianluca Francini.  Learning sparse neural networks via sensitivity-driven regularization.  Advances in neural information processing systems, 31, 2018.
  * [56] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In International Conference on Machine Learning. PMLR, 2021.
  * [57] Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge Belongie.  Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection.  In CVPR, 2015.
  * [58] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  NeurIPS, 30, 2017.
  * [59] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [60] Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling.  Rotation equivariant cnns for digital pathology.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, 2018.
  * [61] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie.  The caltech-ucsd birds-200-2011 dataset.  Technical Report CNS-TR-2011-001, California Institute of Technology, 2011\.
  * [62] Chaoqi Wang, Guodong Zhang, and Roger Grosse.  Picking winning tickets before training by preserving gradient flow, 2020\.
  * [63] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang.  Cvt: Introducing convolutions to vision transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
  * [64] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba.  Sun database: Large-scale scene recognition from abbey to zoo.  In CVPR, 2010.
  * [65] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al.  A large-scale study of representation learning with the visual task adaptation benchmark.  arXiv preprint arXiv:1910.04867, 2019.
  * [66] Aston Zhang, Yi Tay, SHUAI Zhang, Alvin Chan, Anh Tuan Luu, Siu Hui, and Jie Fu.  Beyond fully-connected layers with quaternions: Parameterization of hypercomplex multiplications with $1/n$ parameters.  In International Conference on Learning Representations, 2021.
  * [67] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Michael C. Mozer, and Yoram Singer.  Identity crisis: Memorization and generalization under extreme overparameterization.  In International Conference on Learning Representations, 2020.
  * [68] Han Zhou, Xingchen Wan, Ivan Vulić, and Anna Korhonen.  Autopeft: Automatic configuration search for parameter-efficient fine-tuning, 2023.

## Supplementary Material

In this supplementary material, we provide (1) a visual representation of our
method MiMi for clarity (2) additional experimental results to further analyze
the proposed MiMi approach (Multi-task benchmark, and VTAB benchmark). (3)
justify in more detail our choice for the design of the importance score (4)
the effect of the parameters allocation in Adapters for ViTs (5) provide
details regarding the datasets used in our experiments.

##  6 Illustration of MiMi Design

In our work, we augment a pre-existing ViT model with an ’adapter’ module. For
each adapter, the input is denoted as $\bm{h}_{i}$ with dimension $M_{i}$. The
adapter undergoes two primary transformations.

Firstly, it maps $\bm{h}_{i}$ to $\bm{z}_{i}\in\mathbb{R}^{N_{i}}$ through a
fully-connected layer characterized by a parameter matrix
$\bm{W}_{i}^{\text{down}}$. A non-linear activation function $\phi(\cdot)$ is
also applied during this step. Subsequently, $\bm{z}_{i}$ is transformed back
to an output $\bm{r}_{i}\in\mathbb{R}^{M_{i}}$ by another fully-connected
layer, parameterized by $\bm{W}_{i}^{\text{up}}$.

A special feature of the adapter is its residual skip-connection. If
$\bm{r}_{i}$ is near zero, the adapter essentially acts as an identity
function, making minimal changes to the input.

Our design of MiMi is influenced by an observation: if an entire row in
$\bm{W}_{i}^{\text{down}}$ and an entire column in $\bm{W}_{i}^{\text{up}}$
are zero, the adapter behaves as though its complexity is reduced, effectively
acting as if it has a smaller dimension $M_{i}$, as illustrated in Fig.6.

Figure 6: Illustration of the design of adapter after using MiMi.

##  7 MiMi versus Vanilla training on CIFAR-100

Looking at Fig. 7), we observe the significant performance gap between vanilla
adapters compared to adapters trained with MiMi approach. Our method
outperforms vanilla adapters with a more than 10% accuracy gap at small sizes.

Figure 7: Comparison of top-1 accuracy of vanilla adapters, and MiMi with
respect to compression rate $\sigma$ on CIFAR-100 dataset. All MiMi results
originate from the same MiMi run. Adapters are trained for the exact same
number of epochs as their MiMi counterparts. The size of blob markers
represents the number of trainable parameters. We notice that at
$\sigma=2,4,8$, MiMi outperforms full finetuning.

##  8 Local versus Global Neurons Selection

Table 5: A comparative performance analysis of local and global neuron
selection on VGG-Flowers.

Method | Neurons Removal | Selection | Iterative | Scaling | $\bm{\sigma}$
---|---|---|---|---|---
$32$ | $64$ | $128$ | $256$ | $512$ | $1024$ | $2048$ | $4096$
Vanilla adapters |  |  |  |  | 94.80 | 90.12 | 89.42 | 88.85 | 86.03 | 86.09 | 85.14 | 85.14
Baselines | Local | ✓ |  |  | - | 96.10 | 95.57 | 96.15 | 96.23 | 96.76 | 95.83 | 95.83
Local | ✓ | ✓ |  | - | 96.41 | 96.65 | 96.72 | 96.72 | 96.81 | 96.83 | 94.54
Global | ✓ |  |  | - | 94.88 | 95.28 | 95.66 | 95.45 | 95.56 | 96.03 | 96.03
Global | ✓ |  | ✓ | - | 96.10 | 95.82 | 96.34 | 96.50 | 96.15 | 96.03 | 96.03
MiMi | Global | ✓ | ✓ | ✓ | - | 96.59 | 96.92 | 96.73 | 96.81 | 96.55 | 96.47 | 96.17

Table 6: A comparative performance analysis of local and global neuron
selection on CIFAR-10 and CIFAR-100.

Method | Neurons Removal | Selection | Iterative | Scaling | $\bm{\sigma}$
---|---|---|---|---|---
$32$ | $64$ | $128$ | $256$ | $512$ | $1024$ | $2048$ | $4096$
|  |  |  |  | CIFAR-10 |  |  |  |  |  |  |
Vanilla adapters | - |  |  |  | 97.89 | 97.39 | 97.06 | 96.60 | 96.53 | 96.27 | 82.06 | 82.06
Baseline | Local | ✓ | ✓ |  | - | 98.06 | 97.92 | 97.64 | 97.11 | 96.91 | 96.44 | 93.49
| Random | ✓ | ✓ |  | - | 97.79 | 97.68 | 97.42 | 96.98 | 96.56 | 96.15 | 93.84
| Gradient | ✓ | ✓ |  | - | 97.76 | 97.73 | 97.56 | 97.14 | 96.71 | 96.29 | 93.92
MiMi | Global | ✓ | ✓ | ✓ | - | 97.98 | 97.92 | 97.49 | 97.15 | 96.71 | 96.04 | 95.57
|  |  |  |  | CIFAR-100 |  |  |  |  |  |  |
Vanilla adapters | - |  |  |  | 86.22 | 85.02 | 84.33 | 83.54 | 82.26 | 82.19 | 83.17 | 82.19
Baseline | Local | ✓ | ✓ |  | - | 86.88 | 86.15 | 85.30 | 84.06 | 83.71 | 83.35 | 78.44
| Random | ✓ | ✓ |  | - | 85.97 | 85.71 | 84.82 | 84.23 | 83.72 | 83.08 | 78.24
| Gradient | ✓ | ✓ |  | - | 86.11 | 85.67 | 85.16 | 84.57 | 84.16 | 83.35 | 78.35
MiMi | Global | ✓ | ✓ | ✓ | - | 87.12 | 86.22 | 85.42 | 84.67 | 83.25 | 82.67 | 82.10

Here below, we provide extra experiments for adapters with different sizes for
VGG-Flowers, CIFAR-10, and CIFAR-100.

Tables 5 and 6 report the performance of MiMi algorithm with respect to
vanilla training -_Baseline_ \- on VGG-Flowers, CIFAR-10 and CIFAR-100. MiMi
outperforms vanilla adapters on all the adapters with a significant
performance gap. Interestingly, this gap in performance expands, as we compare
smaller adapters $\sigma=256,...,4096$ (higher compression ratio).

Furthermore, these results emphasize the usefulness of each component of the
importance score for MiMi. We notice that applying global neuron selection,
normalization, and iterative training outperforms local neuron selection on
almost all adapter sizes for VGG-Flowers (Table 5) and CIFAR-100 (Table 6).
This indicates that each component of the importance score of MiMi is
important to boost performance and reduce the parameters.

##  9 Vanilla versus MiMi Training for Adapters

To validate the greater optimization performance of MiMi, in Figs. 8, 9, we
show the training loss curves of vanilla, and MiMi training of adapters for
CIFAR-100, and SVHN, respectively. At the end of the training, the two models
(_i.e._ vanilla training and MiMi) have similar numbers of training
parameters.

Figure 8: Training loss curves of finetuning adapters with vanilla, and MiMi
training on CIFAR-100 dataset. Figure 9: Training loss curves of finetuning
adapters with vanilla, and MiMi training on SVHN dataset.

We notice that the loss of the training using MiMi algorithm is much smoother
than vanilla training resulting in adapters that generalize well on the
downstream task as previously shown in Fig. 4. Furthermore, we notice spikes
in the training loss of MiMi, due to the removal of neurons after each cycle.
Eventually, sequential training and neuron selection is a successful strategy
to find small networks while maintaining good performance, since directly
training small networks does not provide similar results [10].

##  10 Impact of the parameter allocation

In this ablation study, we validate the idea that every layer of a ViT needs
to be adapted differently in order to learn a new task. The Swin-B vision
transformer model that we use in all our experiments consists of 4 stages.
Therefore, we propose to evaluate the performance when we vary the adapter
size in each stage. The results are reported in table 7.

Table 7: Effect of adapters compression rate $\sigma_{i}$ on adapters
performance in terms for top-1 accuracy (%). Compression rate $\sigma=\infty$
is equivalent to not adding an adapter. We vary the $\sigma_{i}$ value for
each ViT stage (I, II, III, IV).

$\bm{\sigma}_{i}$ in each Swin Stage | Dataset
---|---
I | II | III | IV | CIFAR-10 | VGG-Flowers
128 |  128 | 32 | 32 | 97.91 | 89.90
32 | 32 |  128 |  128 | 97.64 | 87.05
128 | 32 | 128 | 32 | 97.84 | 89.45
32 | 128 | 32 | 128 | 97.72 | 88.49
128 | 128 | 128 | 32 | 97.79 | 89.22
256 | 128 | 64 | 32 | 97.91 | 89.84
32 | 64 | 128 | 256 | 97.43 | 86.90
128 | 128 | $\infty$ | $\infty$ | 95.29 | 73.38
$\infty$ | $\infty$ | 128 | 128 | 97.40 | 87.04
$\infty$ | 128 | 128 | $\infty$ | 96.97 | 84.65
128 | $\infty$ | $\infty$ | 128 | 96.53 | 80.84

First, it shows that the size of adapters has a strong effect on the model
performance. In general, the best performances are achieved when using
adapters with a higher number of parameters. Furthermore, bigger sizes of
adapters are not sufficient for better performance, but subject to which stage
they are injected into.

We observe that adding adapters to late stages (_i.e._ III and IV) boosts the
performance better than injecting them into early stages: adapters with
$\sigma_{i}=128$ added to (III, IV) stages rather than (I, II) improve the
performance from $95.29\%$, $73.38\%$ to $97.40\%$, $87.04\%$ on CIFAR-10 and
VGG-Flowers, respectively.

##  11 Illustrations of Local versus Global Neuron Removal

Figures 10 and 11 show additional illustrations of the distribution of the
removed and remaining neurons using MiMi on VGG-Flowers, and CIFAR-10. We show
the learned adapters at different cycles for both local and global neuron
selection methods. We also complete these visualizations (Figures 12 and 13)
with histograms where we show the percentages of remaining neurons. Overall,
these experiments show that our method is capable of obtaining different
parameter allocations that are specific to every task.

(a) Local pruning: all datasets.

(b) Global pruning: VGG-Flowers .

(c) Global pruning: CIFAR-10.

Figure 10: Layer-wise analysis of adapter’s neurons distribution at $3^{rd}$
cycle. Bar plots represent a number of neurons $n_{i}$ at each adapter $i$
using local, and global neuron selection for VGG-Flowers and CIFAR-10,
respectively.

(a) Local pruning: all datasets.

(b) Global pruning: VGG-Flowers.

(c) Global pruning: CIFAR-10.

Figure 11: Layer-wise analysis of adapter’s neurons distribution at $5^{th}$
cycle. Bar plots represent the number of neurons $n_{i}$ at each adapter $i$
using local and global neuron selection for VGG-Flowers and CIFAR-10,
respectively.

(a) Local pruning: all datasets.

(b) Global pruning: VGG-Flowers .

(c) Global pruning: CIFAR-10.

Figure 12: Layer-wise analysis of adapter’s neurons distribution at $3^{rd}$
cycle. Normalized bar plots represent percentage (%) of remaining neurons
$n_{i}$ at each adapter $i$ using local, global neuron selection for VGG-
Flowers and CIFAR-10, respectively.

(a) Local pruning: all datasets.

(b) Global pruning: VGG-Flowers .

(c) Global pruning: CIFAR-10.

Figure 13: Layer-wise analysis of adapter’s neurons distribution at $5^{th}$
cycle. Normalized bar plots represent percentage (%) of remaining neurons
$n_{i}$ at each adapter $i$ using local, global neuron selection for VGG-
Flowers and CIFAR-10, respectively.

##  12 Magnitude assumption as importance score

In this section, we provide more insights on the importance score employed
within MiMi. In particular, under Gaussian input assumption for adapters and
imposing weight decay at training time, we will see that, towards a better
choice of parameters to be removed, considering just $\bm{W}^{down}_{i}$ is
sub-optimal, and $\bm{W}^{up}_{i}$ should be accounted as well. We drop the
adapter index $i$ for abuse of notation, as we will always refer to the same
adapter.
Let us have an $m$-dimensional input $\bm{h}$, whose elements are distributed
according to a Gaussian $\mathcal{N}(\mu_{k},\Sigma_{k})$. We assume the
adapter has already been trained; hence we consider, in the down-sampling
phase all the $w_{jk}^{down}$ as constants. From the property of linear
expectations, we know that, before reaching the non-linear activation, the
post-synaptic potential is still a Gaussian random variable, having an average

$\mu_{j}^{down}=\sum_{k=1}^{m}W_{jk}^{down}\cdot\mu_{k}$ (4)

and variance

$\Sigma_{j}^{down}=\sum_{k=1}^{m}W_{jk}^{down}\cdot\left[W_{jk}^{down}\Sigma_{kk}+2\sum_{k^{\prime}<k}W_{jk^{\prime}}^{down}\Sigma_{kk^{\prime}}\right],$
(5)

where $\Sigma_{ab}$ indicates an element of the covariance matrix for the
input of the adapter. For the sake of tractability, if we assume
$\Sigma_{kk^{\prime}}=0~{}\forall k\neq k^{\prime}$, equation 5 simply reduces
to

$\Sigma_{j}^{down}=\sum_{k=1}^{m}\left(W_{jk}^{down}\right)^{2}\Sigma_{kk}.$
(6)

In transformers, the commonly-used activation function is the Gaussian error
linear unit (GELU), whose analytical expression is

$\phi(x)=x\cdot\frac{1}{2}\left[1+\text{erf}\left(\frac{x}{\sqrt{2}}\right)\right]$
(7)

where erf$(\cdot)$ is the error function. For values close to zero, or larger
than zero, it can be approximated to the identity function, while for values
much lower than zero, it asymptotically tends to zero. Let us focus on the
first scenario: we can approximate the post-synaptic potential to the output
of the non-linearity, saying that the output

$z_{j}\approx\mathcal{N}(\mu_{j}^{down},\Sigma_{j}^{down}).$ (8)

At this point, the signal undergoes an up-sampling: following up on the same
approach adopted for the down-sampling, we find that the output $\bm{r}$ still
follows a Gaussian distribution having an average

$\mu_{l}^{up}=\sum_{j=1}^{n}W_{jl}^{up},\mu_{j}^{down}=\sum_{j=1}^{n}W_{jl}^{up}\sum_{k=1}^{m}W_{jk}^{down}\cdot\mu_{k}$
(9)

and variance

$\Sigma_{l}^{up}=\sum_{j=1}^{n}\left(W_{jl}^{up}\right)^{2}$ (10)
$\Sigma_{j}^{down}=\sum_{j=1}^{n}\left(W_{jl}^{up}\right)^{2}\sum_{k=1}^{m}\left(W_{jk}^{down}\right)^{2}\Sigma_{kk}$
(11)

$\displaystyle\mu_{l,\bar{a}}^{up}$
$\displaystyle=\mu_{l}^{up}-W_{al}^{up}\sum_{k=1}^{m}W_{ak}^{down}\cdot\mu_{k}$
$\displaystyle\Sigma_{l,\bar{a}}^{up}$
$\displaystyle=\Sigma_{l}^{up}-\left(W_{al}^{up}\right)^{2}\sum_{k=1}^{m}\left(W_{ak}^{down}\right)^{2}\Sigma_{kk}$
(12)

In order to assess the impact of removing a whole neuron in the embedding
space, we can write the KL-divergence of the distribution for $r_{l}$ with and
without the $a$-th neuron in the embedding space:

$D_{\mathrm{KL}}(r_{l},r_{l,\bar{a}})=\\\
\log\left(\frac{\Sigma_{l}^{up}\\!-\\!\left(W_{al}^{up}\right)^{2}\sum_{k=1}^{m}\left(W_{ak}^{down}\right)^{2}\Sigma_{kk}}{\Sigma_{l}^{up}}\right)\\\
+\frac{\left(\Sigma_{l}^{up}\right)^{2}+\left(\mu_{l}^{up}-\mu_{l}^{up}+W_{al}^{up}\sum_{k=1}^{m}W_{ak}^{down}\cdot\mu_{k}\right)}{2\cdot\left[\Sigma_{l}^{up}-\left(W_{al}^{up}\right)^{2}\sum_{k=1}^{m}\left(W_{ak}^{down}\right)^{2}\Sigma_{kk}\right]^{2}}-\frac{1}{2}.$
(13)

According to equation 12, we rewrite equation 13 as 14.

$D_{\mathrm{KL}}(r_{l},r_{l,\bar{a}})=\\\
\log\left(1\\!-\\!\frac{\left(W_{al}^{up}\right)^{2}\sum_{k=1}^{m}\left(W_{ak}^{down}\right)^{2}\Sigma_{kk}}{\Sigma_{l}^{up}}\right)\\\
+\frac{\left(\Sigma_{l}^{up}\right)^{2}+\left(W_{al}^{up}\sum_{k=1}^{m}W_{ak}^{down}\cdot\mu_{k}\right)}{2\cdot\left[\Sigma_{l}^{up}-\left(W_{al}^{up}\right)^{2}\sum_{k=1}^{m}\left(W_{ak}^{down}\right)^{2}\Sigma_{kk}\right]^{2}}-\frac{1}{2}.$
(14)

Let us now investigate which is the $a$-th neuron which, when removed, causes
the least perturbation at the output $r_{l}$ (or in other words, such that
$D_{\mathrm{KL}}(r_{l},r_{l,\bar{a}})$ is as low as possible). Looking at the
argument of the logarithm, we ask
$\frac{\left(W_{al}^{up}\right)^{2}\sum_{k=1}^{m}\left(W_{ak}^{down}\right)^{2}\Sigma_{kk}}{\Sigma_{l}^{up}}=0$
and, since we can safely assume $\Sigma_{l}^{up}$, we need to select $a$ such
that
$\left(W_{al}^{up}\right)^{2}\sum_{k=1}^{m}\left(W_{ak}^{down}\right)^{2}\Sigma_{kk}=0$.
Considering that also $\Sigma_{kk}>0~{}\forall k$, we satisfy the condition if
either:

  * •

$W_{ak}^{down}=0~{}\forall k$, namely the $L^{1}$ norm for
$\bm{W}_{-a}^{down}$ is zero;

  * •

$W_{al}^{up}=0$. Considering though that this condition needs to be satisfied
for all the $l$ outputs of the adapters, we ask $W_{al}^{up}=0~{}\forall l$
or, in other words, the $L^{1}$ norm for $\bm{W}_{a-}^{up}$ is also zero.

We observe that, when either of the two conditions is met, the KL divergence
is zero as

$D_{\mathrm{KL}}(r_{l},r_{l,\bar{a}})=\log(1)+\frac{\left(\Sigma_{l}^{up}\right)^{2}}{2\cdot\left(\Sigma_{l}^{up}\right)^{2}}-\frac{1}{2}=0$

We can also assume that if either $W_{ak}^{down}=0~{}\forall k$ or
$W_{al}^{up}=0~{}\forall l$, the norm of the non-zero parameters associated
with some neuron $a$ are small when training with any weight penalty
regularizer (as the contribution to the output is zero, the signal is either
not forward or back-propagated, leaving the weight penalty term the only
update for these parameters).

##  13 Detailed evaluation of MiMi on MultiTask/DomainNet benchmarks

Table 8 reports the number of trained parameters and the average accuracy
across datasets in the DomainNet benchmark. For both, the number of trained
parameters is reported in millions, and the average top-1 accuracy on the
datasets is reported in the rightest column.

We observe that _full fine-tuning_ has generally the highest accuracy, but it
requires a huge number of parameters to be finetuned for each dataset. Among
the vanilla fine-tuning baselines, we observe that tuning the parameters of
the _attention/MLP_ layer turns out to be surprisingly effective.
Nevertheless, it still requires a high number of task-specific parameters,
compared to other PET approaches. _Linear probing_ does not perform well
illustrating the need to change the feature representations of the model when
adapting to new tasks.

_PHM_ , and _Compacter_ are effective methods to get on-par performance with
_full-model finetune_ while adjusting less than 2% of the parameters.
Contrarily to what is observed for NLP tasks [20], PETs on visual tasks do not
reach _full fine-tuning_ performance on any dataset with a low number of
trainable parameters (smaller than 2%). VPT does not perform well, indicating
that injecting tokens into the embedding space does not help much if the pre-
training dataset is different from the downstream task. Generally speaking,
all PET methods maintain similar performance rankings on all tasks. This
suggests that the choice of the best adaptation strategy does not depend on
the downstream task.

_Adapters_ outperform all PET methods in terms of accuracy ($69.39\%$ for
DomainNet, $92.91\%$ for Multi-task) but just with a higher number of
trainable parameters (1.37M, $4.90\%$ of the total) for $\sigma=32$.

_Adapters_ outperform _AdaptFormer_ with fewer parameters (92.91% with 1.37M
parameters, versus 92.27% with 2.98M parameters). This result indicates that
adapting the representations after both MSA and MLP blocks, as done in
_Adapters_ (see Fig. 14), allows better adaptation than acting only on the MLP
block via a parallel branch (as done in _AdaptFormer_ [5]).

When comparing _adapter_ with uniform and proportional parameter distribution,
we observe that allocating parameters proportionally to the layer dimension
performs better. Indeed, adapters with $\sigma=32$ outperform adapters with
$n_{i}=47\forall i$ ($70.65\%$ vs $69.39\%$ in DomainNet, $93.53\%$ vs
$92.91\%$ in Multi-task). This suggests that the last layers, which have
higher dimensionality, are more task-specific, and consequently require more
adaptation. We also show that reducing the size of adapters ($n_{i}=23$) hurts
the performance with a drop which, despite being marginal for Multi-task
($0.23\%$) is more consistent in DomainNet ($1.01\%$). This emphasizes that
training tiny adapters in a vanilla fashion leads to unsatisfying performance
and motivates our specific training procedure.

Figure 14: Evaluation of PET baselines mean top-1 accuracy on DomainNet
benchmark. Reducing MiMi (◆) parameters does not significantly reduce
performance compared to other PET baselines. Table 8: Results on the DomainNet
benchmark [48]. C, I, P, Q, R, and S stand for Clipart, Infograph, Painting,
Quickdraw, Real, and Sketch respectively. † is equivalent to Adapters with
$\sigma=8$. Methods are grouped according to the relative number of trainable
parameters ($\leq 2\%$, $\in]2,10[\%$,$\geq 10\%$)

Method | # Params | Trained | C. | I. | P. | Q. | R. | S. | Mean
---|---|---|---|---|---|---|---|---|---
|  (M) $\downarrow$ |  (%) $\downarrow$ |  |  |  |  |  |  | $\uparrow$
Full fine-tuning | 27.8 | 100 | 79.16 | 48.29 | 74.64 | 75.88 | 86.21 | 73.26 | 72.90
Att-blocks | 8.93 | 32.14 | 48.36 | 75.38 | 73.28 | 86.13 | 72.81 | 72.57 | 72.57
MLP-blocks | 17.54 | 63.12 | 79.23 | 48.11 | 75.02 | 74.82 | 86.35 | 73.29 | 72.80
MiMi (0 cycle)† | 4.44 | 16.15 | 78.11 | 46.93 | 74.38 | 72.19 | 86.09 | 71.97 | 71.61
AdaptFormer-256 | 2.54 | 9.24 | 75.32 | 43.74 | 72.16 | 66.00 | 83.88 | 67.68 | 68.13
AdaptFormer-64 | 0.84 | 3.06 | 73.76 | 42.38 | 71.11 | 63.41 | 83.23 | 66.14 | 66.67
VPT (100 tokens) | 0.71 | 2.57 | 64.33 | 18.65 | 63.20 | 59.40 | 79.65 | 56.41 | 56.94
Adapters ($\sigma=32$) | 1.37 | 4.90 | 77.42 | 46.51 | 74.06 | 69.81 | 85.30 | 70.84 | 70.65
Adapters ($n_{i}=47$) | 1.37 | 4.90 | 76.15 | 45.28 | 73.04 | 67.86 | 84.83 | 69.17 | 69.39
Adapters ($n_{i}=23$) | 0.68 | 2.47 | 75.28 | 44.17 | 72.41 | 66.44 | 83.98 | 68.02 | 68.38
MiMi (1 cycle) | 0.80 | 2.89 | 76.83 | 46.00 | 73.76 | 67.93 | 85.05 | 69.61 | 69.86
Linear prob | 0.27 | 0.95 | 62.89 | 33.96 | 64.93 | 42.95 | 81.69 | 54.24 | 56.77
PHM-Adapter | 0.47 | 1.72 | 75.79 | 44.62 | 72.49 | 66.62 | 83.73 | 68.51 | 68.62
Compacter | 0.41 | 1.44 | 75.25 | 44.19 | 72.09 | 66.01 | 83.42 | 67.99 | 68.16
BitFit | 0.34 | 1.22 | 72.14 | 41.07 | 70.00 | 60.39 | 82.43 | 64.43 | 65.08
VPT (10 tokens) | 0.32 | 1.15 | 61.91 | 24.87 | 57.05 | 57.09 | 76.94 | 55.12 | 55.49
SSF | 0.28 | 0.96 | 74.45 | 42.64 | 71.72 | 64.70 | 82.95 | 65.90 | 67.06
Fact-TK32 | 0.33 | 1.18 | 72.46 | 41.14 | 70.32 | 61.26 | 80.23 | 63.87 | 64.88
Adapters ($n_{i}=1$) | 0.30 | 1.07 | 72.12 | 41.30 | 69.93 | 59.95 | 82.49 | 64.18 | 65.00
MiMi (2 cycles) | 0.53 | 1.92 | 76.83 | 45.45 | 73.11 | 66.67 | 84.42 | 69.05 | 69.26
MiMi (3 cycles) | 0.40 | 1.43 | 75.44 | 44.60 | 72.59 | 64.73 | 83.87 | 68.05 | 68.21
MiMi (4 cycles) | 0.30 | 1.07 | 74.38 | 43.52 | 71.50 | 63.41 | 83.12 | 67.46 | 67.23

###  13.1 Impact of $\rho$

In this section, we investigate the effect of the hyper-parameter $\rho$
(amount of neuron removal) in our method MiMi.

Figure 15: Analysis of MiMi performance on VGG-Flowers dataset with different
values of $\rho$. If $\rho$ is very high, the drop in performance is
significant, but it requires less $C$ training cycles to reach
$\sigma_{traget}$.

In Fig. 15. We notice that higher values of $\rho$ hurt the performance
because we remove many parameters after each cycle, but we reduce the size of
adapters significantly. On the other hand, if $\rho$ is small (i.e 25%), we
maintain good performance on the VGG-Flowers dataset, but it requires higher
training cycles $C$ to reach the target compression rate $\sigma_{target}$.

We have a trade-off between the performance, and training budget in order to
reach the $\sigma_{target}$. Removing too many parameters at each cycle hurts
performance. Maintaining good performance requires a higher number of training
cycles $C$.

###  13.2 Evaluation on VTAB Benchmark

We experiment on VTAB [65] is a collection of 19 diverse visual classification
tasks, which are organized into three groups: Natural \- tasks that contain
natural images captured using standard cameras; Specialized \- tasks that
contain images captured via specialized equipment, such as medical and
satellite imagery; and Structured \- tasks that require geometric
comprehension like object counting. Each task of VTAB contains 1000 training
examples. Following [65], we use the provided 800-200 split of the train set
to determine hyperparameters and run the final evaluation using the full
training data. We report the average accuracy score on the test set within
three runs.

Table 9: Per-task fine-tuning results for VTAB with a pre-trained ViT-B/16.
Results for other baselines are reported from [23].

|  |  CIFAR-100 |  Caltech101 |  DTD |  Flowers102 |  Pets |  SVHN |  Sun397 |  Mean |  Patch Camelyon |  EuroSAT |  Resisc45 |  Retinopathy |  Mean |  Clevr/count |  Clevr/distance |  DMLab |  KITTI/distance |  dSprites/location |  dSprites/orientation |  SmallNORB/azimuth |  SmallNORB/elevation |  Mean
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
(a) | Full | 68.9 | 87.7 | 64.3 | 97.2 | 86.9 | 87.4 | 38.8 | 75.88 | 79.7 | 95.7 | 84.2 | 73.9 | 83.36 | 56.3 | 58.6 | 41.7 | 65.5 | 57.5 | 46.7 | 25.7 | 29.1 | 47.64
_Head-oriented_ |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
(a) | Linear | 63.4 | 85.0 | 63.2 | 97.0 | 86.3 | 36.6 | 51.0 | 68.93 (1) | 78.5 | 87.5 | 68.6 | 74.0 | 77.16 (1) | 34.3 | 30.6 | 33.2 | 55.4 | 12.5 | 20.0 | 9.6 | 19.2 | 26.84 (0)
Partial-1 | 66.8 | 85.9 | 62.5 | 97.3 | 85.5 | 37.6 | 50.6 | 69.44 (2) | 78.6 | 89.8 | 72.5 | 73.3 | 78.53 (0) | 41.5 | 34.3 | 33.9 | 61.0 | 31.3 | 32.8 | 16.3 | 22.4 | 34.17 (0)
Mlp-2 | 63.2 | 84.8 | 60.5 | 97.6 | 85.9 | 34.1 | 47.8 | 67.70 (2) | 74.3 | 88.8 | 67.1 | 73.2 | 75.86 (0) | 45.2 | 31.6 | 31.8 | 55.7 | 30.9 | 24.6 | 16.6 | 23.3 | 32.47 (0)
Mlp-3 | 63.8 | 84.7 | 62.3 | 97.4 | 84.7 | 32.5 | 49.2 | 67.80 (2) | 77.0 | 88.0 | 70.2 | 56.1 | 72.83 (0) | 47.8 | 32.8 | 32.3 | 58.1 | 12.9 | 21.2 | 15.2 | 24.8 | 30.62 (0)
Mlp-5 | 59.3 | 84.4 | 59.9 | 96.1 | 84.4 | 30.9 | 46.8 | 65.98 (1) | 73.7 | 87.2 | 64.8 | 71.5 | 74.31 (0) | 50.8 | 32.3 | 31.5 | 56.4 | 7.5 | 20.8 | 14.4 | 20.4 | 29.23 (0)
Mlp-9 | 53.1 | 80.5 | 53.9 | 95.1 | 82.6 | 24.4 | 43.7 | 61.90 (1) | 78.5 | 83.0 | 60.2 | 72.3 | 73.49 (0) | 47.5 | 27.9 | 28.9 | 54.0 | 6.2 | 17.7 | 10.8 | 16.2 | 26.15 (0)
_Backbone-oriented_ |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
(b) | Sidetune | 60.7 | 60.8 | 53.6 | 95.5 | 66.7 | 34.9 | 35.3 | 58.21 (0) | 58.5 | 87.7 | 65.2 | 61.0 | 68.12 (0) | 27.6 | 22.6 | 31.3 | 51.7 | 8.2 | 14.4 | 9.8 | 21.8 | 23.41 (0)
Bias | 72.8 | 87.0 | 59.2 | 97.5 | 85.3 | 59.9 | 51.4 | 73.30 (3) | 78.7 | 91.6 | 72.9 | 69.8 | 78.25 (0) | 61.5 | 55.6 | 32.4 | 55.9 | 66.6 | 40.0 | 15.7 | 25.1 | 44.09 (2)
Adapter-256 | 74.1 | 86.1 | 63.2 | 97.7 | 87.0 | 34.6 | 50.8 | 70.50 (4) | 76.3 | 88.0 | 73.1 | 70.5 | 76.98 (0) | 45.7 | 37.4 | 31.2 | 53.2 | 30.3 | 25.4 | 13.8 | 22.1 | 32.39 (0)
Adapter-64 | 74.2 | 85.8 | 62.7 | 97.6 | 87.2 | 36.3 | 50.9 | 70.65 (4) | 76.3 | 87.5 | 73.7 | 70.9 | 77.10 (0) | 42.9 | 39.9 | 30.4 | 54.5 | 31.9 | 25.6 | 13.5 | 21.4 | 32.51 (0)
Adapter-8 | 74.2 | 85.7 | 62.7 | 97.8 | 87.2 | 36.4 | 50.7 | 70.67 (4) | 76.9 | 89.2 | 73.5 | 71.6 | 77.80 (0) | 45.2 | 41.8 | 31.1 | 56.4 | 30.4 | 24.6 | 13.2 | 22.0 | 33.09 (0)
_Visual-Prompt Tuning_ |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
(c) | shallow-VPT | 77.7 | 86.9 | 62.6 | 97.5 | 87.3 | 74.5 | 51.2 | 76.81 (4) | 78.2 | 92.0 | 75.6 | 72.9 | 79.66 (0) | 50.5 | 58.6 | 40.5 | 67.1 | 68.7 | 36.1 | 20.2 | 34.1 | 46.98 (4)
Prompt length ($p$) | 100 | 5 | 1 | 200 | 50 | 200 | 1 | 79.4 | 5 | 50 | 50 | 10 | 28.7 | 100 | 200 | 100 | 100 | 100 | 100 | 200 | 200 | 137.5
Tuned / Total (%) | 0.18 | 0.10 | 0.04 | 0.27 | 0.08 | 0.19 | 0.36 | 0.17 | 0.01 | 0.05 | 0.09 | 0.01 | 0.04 | 0.10 | 0.18 | 0.09 | 0.09 | 0.10 | 0.10 | 0.19 | 0.19 | 0.13
deep-VPT | 78.8 | 90.8 | 65.8 | 98.0 | 88.3 | 78.1 | 49.6 | 78.48 | 81.8 | 96.1 | 83.4 | 68.4 | 82.43 | 68.5 | 60.0 | 46.5 | 72.8 | 73.6 | 47.9 | 32.9 | 37.8 | 54.98
Prompt length ($p$) | 10 | 10 | 10 | 1 | 1 | 50 | 5 | 12.4 | 100 | 100 | 10 | 1 | 52.8 | 50 | 200 | 100 | 50 | 10 | 50 | 200 | 200 | 107.5
Tuned / Total (%) | 0.20 | 0.20 | 0.15 | 0.10 | 0.04 | 0.54 | 0.41 | 0.23 | 1.06 | 1.07 | 0.15 | 0.02 | 0.57 | 0.54 | 2.11 | 1.07 | 0.54 | 0.12 | 0.55 | 2.12 | 2.11 | 1.14
(Ours) | MiMi | 61.39 | 86.77 | 66.65 | 96.13 | 90.98 | 79.3 | 53.4 | 76.37 | 83.06 | 95.77 | 85.9 | 75.51 | 85.06 | 62.57 | 65.77 | 46.43 | 74.91 | 76.58 | 53.57 | 24.59 | 35.91 | 55.04

##  14 Details about Datasets

In Table 10, we report different statistics that capture the diversity of the
datasets we use in our experiments.

| Dataset | Train Size | Test Size | Classes
---|---|---|---|---
Multi-task | CIFAR-10 | 50000 | 10000 | 10
CIFAR-100 | 50000 | 10000 | 100
Oxford Flowers | 2040 | 6149 | 102
SVHN | 73257 | 26032 | 10
DomainNet | Clipart | 33525 | 14604 | 345
Infograph | 36023 | 15582 | 345
Painting | 50416 | 21850 | 345
Quickdraw | 120750 | 51750 | 345
Real | 120906 | 52041 | 345
Sketch | 48212 | 20916 | 345
Table 10: Datasets used in our empirical analysis

Table 11: Specifications of the various datasets evaluated. ⋆: we randomly
sampled the train and val sets since there are no public splits available

Dataset | Description | # Classes | Train | Val | Test
---|---|---|---|---|---
Fine-grained visual recognition tasks (FGVC) |  |  |
CUB-200-2011 [61] | Fine-grained bird species recognition | 200 |  5,394⋆ |  600⋆ | 5,794
NABirds [57] | Fine-grained bird species recognition | 55 |  21,536⋆ |  2,393⋆ | 24,633
Oxford Flowers [47] | Fine-grained flower species recognition | 102 | 1,020 | 1,020 | 6,149
Stanford Dogs [28] | Fine-grained dog species recognition | 120 |  10,800⋆ |  1,200⋆ | 8,580
Stanford Cars [11] | Fine-grained car classification | 196 |  7,329⋆ |  815⋆ | 8,041
Visual Task Adaptation Benchmark (VTAB-1k) [65] |  |  |
CIFAR-100 [29] | Natural | 100 | 800/1000 | 200 | 10,000
Caltech101 [33] | 102 | 6,084
DTD [7] | 47 | 1,880
Flowers102 [47] | 102 | 6,149
Pets [49] | 37 | 3,669
SVHN [45] | 10 | 26,032
Sun397 [64] | 397 | 21,750
Patch Camelyon [60] | Specialized | 2 | 800/1000 | 200 | 32,768
EuroSAT [19] | 10 | 5,400
Resisc45 [6] | 45 | 6,300
Retinopathy [26] | 5 | 42,670
Clevr/count [25] | Structured | 8 | 800/1000 | 200 | 15,000
Clevr/distance [25] | 6 | 15,000
DMLab [1] | 6 | 22,735
KITTI/distance [12] | 4 | 711
dSprites/location [40] | 16 | 73,728
dSprites/orientation [40] | 16 | 73,728
SmallNORB/azimuth [31] | 18 | 12,150
SmallNORB/elevation [31] | 9 | 12,150