# SEMI-SUPERVISED CONVOLUTIVE NMF FOR AUTOMATIC PIANO TRANSCRIPTION

###### Abstract

Automatic Music Transcription, which consists in transforming an audio
recording of a musical performance into symbolic format, remains a difficult
Music Information Retrieval task. In this work, which focuses on piano
transcription, we propose a semi-supervised approach using low-rank matrix
factorization techniques, in particular Convolutive Nonnegative Matrix
Factorization. In the semi-supervised setting, only a single recording of each
individual notes is required. We show on the MAPS dataset that the proposed
semi-supervised CNMF method performs better than state-of-the-art low-rank
factorization techniques and a little worse than supervised deep learning
state-of-the-art methods, while however suffering from generalization issues.

##  1 Introduction

Automatic Music Transcription (AMT) is the task of transforming music
recordings into symbolic format, such as scores or MIDI. It is a fundamental
musical skill to acquire, taught from early age up to professional level in
music schools and, given enough training, humans can be extremely accurate at
transcription. Automatic music transcription aims at accelerating and
improving time-consuming manual transcription and has applications in music
tutoring and rehearsing, musicology analysis or in other music information
retrieval tasks [1].

However, while audio generation from MIDI is rather mature, its counterpart
AMT is still a very challenging task, even in scenarios involving a single
multipitch instrument like a piano, which is our case study. As reported in
the 2018 survey by Benetos et. al. [1], there are mainly two families of
methods to perform AMT: 1) Methods based on low-rank factorizations of
spectrograms, and in particular Nonnegative Matrix Factorization (NMF). These
methods are mostly unsupervised [2, 3, 4]. 2) Deep Neural Networks (DNN) which
are heavily supervised. They require registered symbolic-audio training data
in a large amount, which can be hard to acquire [5, 6, 7, 8, 9].

A recent outbreak in the task of piano transcription (as well as other related
tasks) is due to the release of the MAESTRO dataset [7], a large dataset of
tightly matched MIDI and audio piano recordings of professional quality which
improved the training quality of deep learning techniques. However, the
supervised methods require extensive amounts of training data which may not be
available for most instruments. The quality of the MAESTRO dataset comes from
the existence of the Yamaha Disklavier™, which enables co-recording of audio
and MIDI. This high level technology does not exist for most instruments, and
building large training dataset for most polyphonic instruments would be
extremely challenging on the practical side.

In contrast, since unsupervised factorization-based approaches do not require
training data, they obviously solve the data frugality and generalization
problems at the cost of being far less accurate than deep supervised
approaches.

The goal of this paper is two-fold. On a first hand, leveraging training data
available only in limited quantity. On another hand, deploying a variant of
NMF, coined Convolutive NMF, in the context of transcription, to improve the
transcription performance with respect to NMF. The most closely related work
is surely the Attack Decay model [3], which also performs semi-supervision,
and proposes a model reminiscent of CNMF. The major differences between the
proposed CNMF framework and this work of Cheng et. al. are discussed in
Section 2.2. Moreover, in Section 4, we show that the performance of the
proposed approach are generally much higher and can reach the performance
levels observed with Deep Learning at the cost of poor generalization
properties. In [4], authors also consider CNMF for piano transcription but
CNMF is not the main focus of their work.

This paper is organized as follows: in Section 2, we review the basics of NMF
and CNMF for transcription. In Section 3, semi-supervised CNMF is introduced.
In Section 4 we show experimental results on MAPS and MAESTRO. Section 5 is
devoted to discussions and perspectives.

Notations: Matrices and higher-order arrays are denoted by capital letters,
$T_{ijk}$ is the element $(i,j,k)$ in the three-way array $T$. To denote
slices, we use semicolons, so that $T_{i::}$ denotes for instance the slice of
all elements of $T$ on row $i$. Finally, we denote $T_{[a:b]jk}$ elements
$(i,j,k)$ with $i\in[a,b]$.

##  2 CNMF for transcription

###  2.1 NMF and CNMF formalisms

Figure 1: A visual comparison of NMF (left) and CNMF (right). CNMF allows to
model complex time dependance while maintaining the number of templates low.

Given an element-wise nonnegative matrix $M\in\mathbb{R}_{+}^{n\times m}$
indexed as $M_{ft}$ with $f\in[1,n],\;t\in[1,m]$, Nonnegative Matrix
Factorization (NMF) is a low-rank approximation technique that summarizes $M$
as a sum of rank-one parts, such that

$M_{ft}=\sum_{q=1}^{r}{W_{fq}H_{qt}}$ (1)

where $r\leq\min(n,m)$ is a user-defined parameter relating to the number of
patterns underlying $M$, see Figure 1. In practice, when $M$ is an amplitude
spectrogram, such as in this work, NMF is computed approximately and boils
down to solving a bi-level constrained optimization problem

$\underset{W\in\mathbb{R}_{+}^{n\times r},H\in\mathbb{R}_{+}^{r\times
m}}{\mathop{\text{~{}argmin~{}}}}D_{KL}(M,WH)$ (2)

where $D_{KL}(M,WH)$ is the element-wise Kullback-Leibler divergence between
matrix $M$ and its nonnegative low-rank approximation
$WH=\sum_{q=1}^{r}W_{:q}H_{q:}$. In AMT, parameter $r$ often relates to the
number of notes expected in the recording, and therefore is generally set to
(sometimes a multiple of) $r=88$ for piano recordings [2].

Furthermore, factor matrices $W$ and $H$ are respectively related to pitch and
time activation. More specifically, each column of $W$ is expected to contain
a spectral template characteristic of a single pitch on the instrument used in
the recording, while each corresponding row in $H$ is expected to provide the
activation of that note in the recording [10], see Figure 2.

Figure 2: A toy example of transcription using NMF (adapted from [11]).

An immediate critic about applying NMF to AMT is that reducing a note to a
single frequency template, even tailored for a given instrument, is too
restrictive. In practice, frequency templates should evolve with both
amplitude and time. While explicit amplitude dependence would break the
principle of low-rank approximation underlying NMF, it is possible to extend
NMF to include a time-dependence on the templates, which yields Convolutive
NMF [12]:

$\underset{W\in\mathbb{R}_{+}^{n\times\tau\times
r},H\in\mathbb{R}_{+}^{r\times
m}}{\mathop{\text{~{}argmin~{}}}}D_{KL}(M,\sum_{q=1}^{r}W_{::q}\ast H_{q:})$
(3)

where $\left[W_{::q}\ast
H_{q:}\right]_{:t}=\sum_{i=0}^{\tau-1}W_{:iq}H_{q(t-i)}$ is a discrete
convolution and $q\in[1,r]$, see Figure 1 for an illustration. By convention,
we set $H_{q(t-i)}=0$ whenever $t-i\leq 0$. Integer $\tau$ is again a user-
defined hyperparameter that dictates the size of the convolution window. To
provide a different perspective, the element-wise noiseless CNMF also writes

$M_{ft}=\sum_{q=1}^{r}\sum_{i=0}^{\tau-1}W_{fiq}H_{q(t-i)}~{}.$ (4)

In a nutshell, CNMF enriches NMF by allowing each note to have a full STFT
matrix $W_{::q}$ as a frequency template instead of a single column.
Therefore, it may also captures time-dependent events such as echoes or non-
uniform partials attenuation. It can also be interpreted as a constrained NMF
with large rank $r\times\tau$ where each note is represented by $\tau$
templates, and the corresponding $\tau$ rows in $H$ are constrained to be
equal up to a shift. Other works have also considered enriching NMF with
several templates per note albeit not using convolution, typically by fusing
rows of the estimated $H$ matrix a posteriori [13, 14, 15].

###  2.2 Comparing the Attack Decay model with CNMF

A reader familiar with the work of Cheng et. al. [3] will notice that our work
is similar in several aspects with their proposed Attack Decay (AD) framework
for music transcription, but let us properly compare the models. After some
rewriting of the original AD (see additional
material111https://github.com/cohenjer/TransSSCNMF), AD decomposes the data
$M$ into two terms

$\displaystyle
M_{ft}=\sum_{q=1}^{r}\underbrace{\sum_{i=0}^{2\tau}\left(\tilde{W}^{\text{attack}}_{fq}P_{-i}\right)H_{q(t-i)}}_{\text{one
note attack}}$ (5)
$\displaystyle+\sum_{q=1}^{r}\underbrace{\sum_{i=\tau}^{t+\tau-1}\left(\tilde{W}^{\text{decay}}_{fq}e^{-\alpha_{q}(i-\tau)}\right)H_{q(t-i)}}_{\text{one
note decay}}~{}.$ (6)

It thus appears that the attack term is a CNMF with rank-one templates
$W^{(\text{CNMF})}_{fiq}=W^{\text{a}}_{fq}P_{i}$ which is therefore less
general than the CNMF model. The decay term is also a CNMF with rank-one
templates.

With some further manipulations, one can see that it is possible to entirely
recast the AD model as a CNMF model with rank-two templates, which may explain
the performance gap between the two models observed in Section 4. Indeed in
the semi-supervised setting, we seem to have enough data to learn
unconstrained templates $W^{\text{train}}$, and the Attack-Decay structure on
the templates may not be beneficial.

##  3 Template learning and CNMF

###  3.1 Challenges in unsupervised CNMF

In the context of music transcription, it is rarely discussed why NMF performs
extremely well on simple dataset, but rather poorly on more complex ones.
Saying that NMF, or CNMF, is a part-based representation with no destructive
interferences between components does not explain this behavior. In fact,
supposing the data indeed is generated reasonably well with a “ground-truth”
NMF $M=AB$ for some true frequency templates $A\in\mathbb{R}_{+}^{n\times r}$
and activations $B\in\mathbb{R}_{+}^{r\times m}$, we need to ensure that
computing an exact NMF $M=WH$ will indeed yield $A=W$ and $B=H$. In other
words, the data $M$ must admit a unique NMF.

Theoretically speaking, it is known that NMF will only enjoy this uniqueness
property in particular cases, such as when sources are sufficiently scattered
or when the data is very sparse [16, 17]. While this may hold for simple songs
where notes do not overlap a lot, in the general case one should not expect
that $W$ and $H$ behave as expected without restricting the set of solutions.
Even worse, CNMF being a generalized NMF model, it is bound to have even
weaker uniqueness properties than NMF (but nothing is known on CNMF
identifiability to the best of our knowledge). Blind CNMF has been used with
additional sparsity constraints for drums transcription, but dealing with
drums typically yields much sparser and lower-rank data than pitched audio due
to the temporal localization of percussive sounds.

Therefore, in general, unsupervised CNMF is not regularized enough to perform
transcription. While some works focus on further regularization of NMF [18],
we instead turn towards semi-supervision.

###  3.2 Learning note-wise templates

Our working hypothesis is that audio recordings of isolated pitches are
available, similarly to what is used for virtual instruments, except that we
only make use of one template per note. Each recording is processed as the
module of its complex STFT, denoted $V_{::q}\in\mathbb{R}_{+}^{n\times m_{q}}$
where $m_{q}$ is the number of STFT frames for that recording. For a regular
piano one needs $88$ such templates. Apart from pitch knowledge, no registered
MIDI information is required.

The goal of the learning phase here is to estimate $W_{::q}$ for each $q$
using each individual recording $V_{::q}$. We propose to compute an
approximate rank-one CNMF of each $V_{::q}$ to estimate $W_{::q}$ and
$h^{\text{train}}_{q}$, the latter being discarded after the training phase.
From a theoretical perspective, rank-one CNMF is a constrained version of NMF
of rank $\tau$, furthermore computed on a very simple dataset. Therefore it
fulfills the qualitative NMF uniqueness criteria discussed above, and we
expect the recovered $W$ to contain adequate note frequency templates.

Practically, we solve for each $q\in[1,r]$ the following optimization problem

$W^{\text{train}}_{::q},h^{\text{train}}_{q:}\in\underset{W\in\mathbb{R}_{+}^{n\times\tau\times
1},\;h\in\mathbb{R}_{+}^{1\times
m}}{\mathop{\text{~{}argmin~{}}}}D_{KL}(V_{q},W\ast h)$ (7)

using a recently proposed multiplicative algorithm [19] which alternates
between $W$ and $h$ updates while preserving nonnegativity and ensuring cost
decrease.

In spite of the rank-one approximation and the simple data, the optimization
problem still proves challenging with many local minima. Therefore
initialization plays an important role in the learning phase. Because it is
reasonable to look for $W_{::q}$ in the $V_{::q}$ data itself, we set

$W^{\text{init}}_{::q}=V_{:[t^{\ast}:t^{\ast}+\tau-1]q}\;\text{and}\;t^{\ast}=\underset{t\leq
m_{q}}{\mathop{\text{~{}argmax~{}}}}\|V_{:[t:t+\tau-1]q}\|_{1}$ (8)

which amounts to finding the $\tau$ consecutive columns with most energy for
initialization. Then we fill $h^{\text{init}}_{q:}$ with zeros and place a one
at $t^{\ast}$. Note that this initialization procedure mimics a recently
proposed algorithm for separable CNMF222Separable CNMF is a computationally
simpler variant of CNMF which looks for all matrices $W_{::q}$ in the data
itself. [20] but is less computationally intensive. A total of 500 outer
iterations are performed to learn a single note template.

Figure 3: Three trained templates from the AkPnCGdD synthetic piano in MAPS,
using $\tau=10$ convolution size. Templates $W_{::q}$ have been square rooted
to better highlight higher frequencies.

Once the training phase is over, for a single multipitch instrument, we have
at our disposal the whole dictionary $W^{\text{train}}$, see Figure 3.

###  3.3 CNMF transcription with templates

Testing in the semi-supervised framework only consists of computing the time
activations $H$ for a given music excerpt $M$ to transcribe, since $W$ has
been pre-trained. This makes the transcription task much easier since the
problem

$H^{\text{test}}\in\underset{H\in\mathbb{R}_{+}^{r\times
m}}{\mathop{\text{~{}argmin~{}}}}D_{KL}(M,\sum_{q=1}^{r}W^{\text{train}}_{::q}\ast
H_{q:})$ (9)

is convex and therefore can be solved up to arbitrary precision with the
algorithm proposed in [19]. In practice 100 iterations are used, which is
generally enough to reach convergence. Initialization was carried out using a
few iterations of NMF with $W$ fixed as the first column of each trained
template $W^{\text{train}}_{::q}$. An example output $H^{\text{test}}$ is
provided in Figure 4.

Figure 4: An example of $H^{\text{test}}$ computed through rank-one CNMF.

###  3.4 Post-processing of activations

The post-processing of $H^{\text{test}}$ that produces a MIDI file matters a
lot. Hopefully, prior works have already proposed quite efficient post-
processing using an adaptive threshold [3]. We essentially use the same
technique but simplified.

In short, activation values in each row of $H^{\text{test}}$, averaged over
several consecutive frames, are added to a user-defined threshold $\delta$,
defining an adaptative threshold. An onset is detected at the position where
the signal is above this adaptive threshold, see Figure 5 for an illustration.
Formally, an onset is detected at frame $t$ for note $q$ when

$h_{qt}>\frac{1}{21}\sum_{j=-10}^{10}h_{q(t+j)}+\delta,$ (10)

using zero-padding when necessary. The activations are typically very sparse,
so we generally did not observe spurious double peaks using the adaptive
threshold contrarily to what was observed in [3].

Figure 5: The CNMF activation and adaptive peak-picking method shown for note
F4 using a song from MAPS.

##  4 Experiments on MAPS and MAESTRO

Although the proposed semi-supervised CNMF framework works in principle for
transcribing any multipitch instrument, we only evaluate the performance for
piano transcription as a proof on concept. Among the few existing open piano
recordings dataset with registered audio and MIDI, in Section 4.2 we focused
especially on MAPS [21] which has several kinds of individual notes recordings
for several pianos, both virtual and acoustic. We also used MAESTRO [7] to
evaluate generalization performance in Section 4.3. In our tests, we only
considered the first 30 seconds of each song, as in [3]. Results are discussed
in Section 5.

###  4.1 Experimental Setup

Let us briefly state the various experimental parameters required to reproduce
the experiments333Python code to compute CNMF and reproduce all the
experiments is available at https://github.com/cohenjer/TransSSCNMF. All time
signals are sampled at 44100Hz, the STFT is computed with windows of 80ms
(3528 samples) with a hop-length of 20ms (882 samples). This results in
$n=4097$ frequency bins and $m=1501$ time frames in the STFT for $30s$ of raw
audio signal. No smoothing is applied to the STFT, and we set $M$ as the
amplitude spectrogram.

The $\tau$ values are chosen among $\tau=5,10,20$. We used one template for
each piano note such that $r=88$. Finally to fix the peak-picking threshold
for $H^{\text{test}}$, two oracle strategies are used: 1) use the same
threshold for all songs, and report results for the best value on the grid
$[0.01:0.01:0.4]$ 2) perform transcription with a song-dependent threshold,
optimized on the same grid. The first case corresponds to a scenario where the
threshold is pre-trained for a category of recording (music genre, recording
conditions) while the second case corresponds to a hand-tuned threshold for a
specific song to transcribe.

We compare our method with the Attack Decay (AD) model presented in Section 3
which, to the best of our knowledge, is the current state-of-the-art for
unsupervised/semi-supervised piano transcription. The results reported in
Table 1 are the exact results from [3] (AD [3]), and the results of AD when
applying our post-processing (AD*). In both cases transcription is performed
on $H^{\text{attack}}$ as defined in [3]. Despite our efforts we were unable
to exactly reproduce the original AD scores. In particular the original AD
paper introduces smoothness in several aspects: the data spectrograms are
locally averaged, and peak-fusion is performed in the post-processing.
Consequently, the AD* results enable comparison between the proposed CNMF and
AD in the same pre/postprocessing conditions, while AD[3] are the best results
achieved by Cheng et. al.. For completeness, we also report the transcription
score from the state-of-the-art piano transcription network introduced in [8]
which was trained on MAESTRO [7].

To measure performance, we compute a notewise score using the mir_eval [22]
toolbox with a tolerance of 50ms. The offset detection problem is not tackled.
Results are shown using only F-measure (F) and Accuracy (A) metrics (reported
in percent), but full results including Precision and Recalls for all pianos
are available in the complementary materials online.

###  4.2 Transcription performance on MAPS

The MAPS dataset contains classical piano music pieces recorded with different
pianos and conditions: a Yamaha Disklavier™in two settings ’ENSTDkCl’ (EN1)
and ’ENSTDkAm’ (EN2), and six synthetic pianos ’AkPnBcht’, ’AkPnBsdf’
(AkB1-2), ’AkPnCGdD’ (AkC), ’AkPnStgb’ (AkS), ’SptkBGAm’ (Sp), ’StbgTGd2’
(St). For each piano/setting listed above, we train a template
$W^{\text{train}}$ using a rank-one CNMF as presented in Equation 7. Since
there are many available single notes recordings in MAPS, we chose based on
performance to use the Isolated notes (ISOL / NO) recorded at Medium intensity
(M).

In a first experiment, we study the sensitivity of the proposed method to the
selection of the convolution window size $\tau$ and the choice of a threshold
$\delta$ tuned on the whole corpus versus on each song individually. We also
compare our results to the Attack Decay model and the ByteDance supervised
neural network. Results are shown in Table 1. From this experiment, we see
that generally $\tau=10$ performs best, and that the song-wise threshold gives
better results.

In a second experiment, the templates for all other pianos are used to
transcribe AkPnCGdD and ENSTDkCl to estimate the generalization capacities of
the trained CNMF templates, see results in Table 2. Only CNMF with song-tuned
threshold is shown and we set $\tau=10$, to show the best results only.

Additionally, Table 3 reports the average running times when training the
templates and performing transcription on the AkPnCGdD recordings. This test
was run on a personal computer with AMD Ryzen 5 2600™ processor and 16GB RAM.

|  |  | EN1 | EN2 | AkB1 | AkB2 | AkC | AkS | Sp | St
---|---|---|---|---|---|---|---|---|---|---
thresh |  | $\tau$ | F | A | F | A | F | A | F | A | F | A | F | A | F | A | F | A
|  | 5 | 78 | 65 | 70 | 55 | 88 | 80 | 75 | 62 | 83 | 72 | 80 | 69 | 81 | 70 | 75 | 61
|  | 10 | 85 | 75 | 77 | 64 | 93 | 88 | 87 | 78 | 91 | 84 | 88 | 79 | 89 | 82 | 84 | 74
| CNMF | 20 | 83 | 72 | 76 | 63 | 94 | 89 | 87 | 79 | 92 | 86 | 87 | 79 | 90 | 83 | 86 | 77
global | AD* | 81 | 69 | 68 | 53 | 66 | 50 | 71 | 56 | 60 | 43 | 67 | 51 | 64 | 47 | 67 | 50
|  | 5 | 82 | 70 | 74 | 59 | 90 | 82 | 80 | 69 | 87 | 78 | 84 | 74 | 86 | 77 | 81 | 69
|  | 10 | 88 | 79 | 80 | 68 | 95 | 91 | 90 | 83 | 94 | 89 | 90 | 82 | 93 | 87 | 89 | 80
| CNMF | 20 | 85 | 75 | 78 | 66 | 95 | 91 | 90 | 83 | 94 | 90 | 89 | 81 | 92 | 87 | 89 | 81
song | AD* | 82 | 70 | 69 | 54 | 68 | 52 | 73 | 59 | 61 | 45 | 69 | 54 | 66 | 50 | 70 | 54
AD [3] | 82 | 70 | - | - | - | - | - | - | 85 | 74 | - | - | - | - | - | -
ByteDance DNN[8] | 89 | 81 | 77 | 65 | 98 | 97 | 95 | 90 | 98 | 96 | 87 | 77 | 97 | 95 | 95 | 90
Table 1: CNMF, AD, and the ByteDance supervised network performance with respect to the choice of hyperparameter $\tau$ and the choice of the peak-picking threshold, without training/testing mismatch for CNMF and AD. Only the first 30s of each songs were used. AD* uses the same pre/post-processing as CNMF. Tolerance is 50ms. |  | EN2 | AkB1 | AkB2 | AkS | Sp | St
---|---|---|---|---|---|---|---
| F | 74 | 77 | 77 | 70 | 74 | 77
AkC | A | 59 | 64 | 63 | 56 | 59 | 63
| F | 76 | 67 | 68 | 69 | 67 | 69
EN1 | A | 62 | 50 | 52 | 53 | 52 | 53
Table 2: Transcription scores for CNMF, with training/testing mismatch. | Training | Transcription
---|---|---
$\tau$ | 5 | 10 | 20 | 10
Av. time | 56s | 193s | 634s | 239s
Table 3: Average computation time for a learning pattern (Training) or
transcribing 30s of a song (Transcription) for semi-supervised CNMF. Results
are reported for the AkPnCGdD piano in MAPS.

###  4.3 Generalization on MAESTRO

A natural question regarding CNMF templates is how well they can be used
outside their training context without any domain adaptation. While results
shown in Table 2 already provide a partial answer, we also tried to apply CNMF
to the MAESTRO dataset. However, since no individual notes recordings are
publicly available for MAESTRO, we used the templates learnt from MAPS. We
transcribed 20 songs from the MAESTRO test set randomly chosen.

The results are quite poor: even when choosing song-wise thresholds, for all
templates, CNMF does not reach above 59% in F-measure (test results are
available in the supplementary materials). For comparison, the state-of-the-
art with supervised deep learning techniques reaches above 95% F-measure on
MAESTRO. Its performance on MAPS with data augmentation are also state-of-the-
art, around 89% F-measure on EN1, despite the training/testing mismatch.

##  5 Discussion

In light of the experiments conducted in Section 4, let us discuss the
strengths of the proposed CNMF. It exhibits a significant improvement with
respect to the Attack Decay model, which as far as we know is state-of-the-art
for semi-supervised piano transcription. This is even more true when using the
same pre-processing and post-processing for AD and CNMF, the former being in
particular prone to unstable activations which were not observed in the
latter. We may therefore affirm that the improvement in performance is indeed
due to the CNMF model design. In other words, CNMF with a semi-supervised
setting is an efficient piano transcription method. From numerical results, it
seems that a convolution window size $\tau=10$ is a good compromise between
quality of transcription and transcription computation time.

The CNMF method does not perform better than the supervised state-of-the-art
method we denoted as ByteDance DNN, which is expected given that this neural-
network competitor is trained on MAESTRO which contains more than two hundred
hours of perfectly aligned MIDI and audio piano recording of professional
level. We still reach similar performances on some pianos such as EN1, EN2 and
AkS. Nevertheless, the ByteDance DNN is not trained on MAPS contrarily to the
proposed semi-supervised CNMF.

Moreover, the proposed semi-supervised setting only requires a handful of
training dataset which are relatively easy to acquire. Indeed, only individual
notes recordings are necessary, without any audio and MIDI registration.
Compared to the very large amount of data currently required by state-of-the-
art deep learning approaches, this is a huge advantage of the proposed
approach applicable to any acoustic instrument with well-defined onsets
readily available. Sadly our study is limited to piano transcription. A
perspective of this work is to apply it to transcribe polyphonic instruments
for which recording registered MIDI and audio is challenging.

Finally, while the performance does depend on the choice of a good activation
threshold, CNMF still performs well using a global threshold over all songs in
MAPS for each piano. Therefore extensively tuning the threshold hyperparameter
is not essential to the success of CNMF here.

Despite these encouraging results, CNMF has a few issues which open
interesting perspectives. First, it clearly has a significant generalization
problem, or in other words, the learning stage overfits the training data.
From Table 2, it appears that a mismatch between training and testing inside
MAPS, while detrimental to transcription performance, is not as severe as a
learning on MAPS and testing on MAESTRO. A tentative explanation is that the
MAESTRO recordings are live performances with quite loud reverberation, while
the MAPS recordings are drier. Looking for an audio transformation of the
templates that minimizes recording conditions mismatch would therefore
probably prove beneficial to generalize pre-recorded CNMF templates.
Retraining a template library given few annotated data in the testing set
could also be a possible solution. Whether this domain adaptation can be done
fully blindly is still unclear however.

Second, despite performance not relying too much on the threshold level, the
threshold selection method on the other hand is extremely important. Using a
fixed threshold instead of the adaptive peak-picking drastically decreased
performances in our early tests. But this also means that the post-processing
of activations can be further improved using more involved technique than
thresholding each note individually, and this research direction should not be
overlooked if transcription performances of CNMF are to be further improved.

Third, for simplicity only one template for each note was used for the
transcription phase. However, most instruments sound quite differently
depending on how they are played. The proposed semi-supervised framework
currently does not account for this timbre variation with amplitude or
technique, and adapting the current method to make use of several templates
per notes is an interesting research direction.

Finally according to the results shown in Table 3, computation time is rather
large even in the testing phase. With the current implementation, real-time
processing is therefore prohibited. Using a CNMF solver dedicated to Kullback-
Leibler divergence or working on a more efficient rank-one CNMF solver than
[19] could nevertheless drastically reduce computation time.

##  6 Conclusion

The state-of-the-art for automatic piano transcription is undeniably nowadays
detained by deep learning techniques. However these methods rely on very large
audio and symbolic registered dataset which are potentially very hard to
obtain. In this work, we propose a competitive semi-supervised matrix
factorization model which only requires labeled recordings of each individual
notes. We show that when there is no mismatch between the training data and
the test data, our approach performs significantly better than semi-supervised
state-of-the-art approaches, approaching supervised deep learning performance.
Therefore, we believe that using CNMF instead of NMF is an important step
towards learning more reasonable frequency templates in low-rank approximation
techniques for piano transcription or other similar tasks. Further works
should however be devoted to adapt pre-trained templates to reduce
generalization error. Improving the onset detection method, allowing timbre
variation in templates and reducing computation time are other important
research directions. Finally, the proposed semi-supervised approach should be
tested with other instruments than the piano and in a multi-instrument setup.

Acknowledgments
Jeremy E. Cohen and Axel Marmoret thank ANR JCJC LoRAiA ANR-20-CE23-0010 for
supporting this work.

## References

  * [1] E. Benetos, S. Dixon, Z. Duan, and S. Ewert, “Automatic music transcription: An overview,” _IEEE Signal Processing Magazine_ , vol. 36, no. 1, pp. 20–30, 2018.
  * [2] E. Vincent, N. Bertin, and R. Badeau, “Harmonic and inharmonic nonnegative matrix factorization for polyphonic pitch transcription,” in _2008 IEEE International Conference on Acoustics, Speech and Signal Processing_. IEEE, 2008, pp. 109–112.
  * [3] T. Cheng, M. Mauch, E. Benetos, and S. Dixon, “An attack/decay model for piano transcription,” in _ISMIR 2016-17st International Society for Music Information Retrieval_ , 2016.
  * [4] L. Gao, L. Su, Y.-H. Yang, and T. Lee, “Polyphonic piano note transcription with non-negative matrix factorization of differential spectrogram,” in _2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2017, pp. 291–295.
  * [5] S. Sigtia, E. Benetos, and S. Dixon, “An end-to-end neural network for polyphonic piano music transcription,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 24, no. 5, pp. 927–939, 2016.
  * [6] C. Hawthorne, E. Elsen, J. Song, A. Roberts, I. Simon, C. Raffel, J. Engel, S. Oore, and D. Eck, “Onsets and frames: Dual-objective piano transcription,” _Proceedings of the 19th International Society for Music Information Retrieval Conference_ , 2018.
  * [7] C. Hawthorne, A. Stasyuk, A. Roberts, I. Simon, C.-Z. A. Huang, S. Dieleman, E. Elsen, J. Engel, and D. Eck, “Enabling factorized piano music modeling and generation with the MAESTRO dataset,” in _International Conference on Learning Representations_ , 2019.
  * [8] Q. Kong, B. Li, X. Song, Y. Wan, and Y. Wang, “High-resolution piano transcription with pedals by regressing onsets and offsets times,” _arXiv preprint arXiv:2010.01815_ , 2020.
  * [9] Y. Yan, F. Cwitkowitz, and Z. Duan, “Skipping the frame-level: Event-based piano transcription with neural semi-crfs,” _Advances in Neural Information Processing Systems_ , vol. 34, 2021.
  * [10] P. Smaragdis and J. C. Brown, “Non-negative matrix factorization for polyphonic music transcription,” in _2003 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics_. IEEE, 2003, pp. 177–180.
  * [11] N. Bertin, “Les factorisations en matrices non-négatives : approches contraintes et probabilistes, application à la transcription automatique de musique polyphonique,” Ph.D. dissertation, 2009. [Online]. Available: http://www.theses.fr/2009ENST0051
  * [12] P. Smaragdis, “Convolutive speech bases and their application to supervised speech separation,” _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 15, no. 1, pp. 1–12, 2006.
  * [13] T.-M. Wang, P.-Y. Tsai, and A. W. Su, “Score-informed pitch-wise alignment using score-driven non-negative matrix factorization,” in _2012 International Conference on Audio, Language and Image Processing_. IEEE, 2012, pp. 206–211.
  * [14] E. Benetos, A. Klapuri, and S. Dixon, “Score-informed transcription for automatic piano tutoring,” in _2012 Proceedings of the 20th European Signal Processing Conference (EUSIPCO)_. IEEE, 2012, pp. 2153–2157.
  * [15] D. Jeong and J. Nam, “Note intensity estimation of piano recordings by score-informed nmf,” in _Audio Engineering Society Conference: 2017 AES International Conference on Semantic Audio_. Audio Engineering Society, 2017.
  * [16] D. Donoho and V. Stodden, “When does non-negative matrix factorization give a correct decomposition into parts?” in _In Advances in Neural Information Processing 16_ , 2003.
  * [17] X. Fu, K. Huang, and N. D. Sidiropoulos, “On identifiability of nonnegative matrix factorization,” _IEEE Signal Processing Letters_ , vol. 25, no. 3, pp. 328–332, 2018.
  * [18] V. Leplat, A. M. Ang, and N. Gillis, “Minimum-volume rank-deficient nonnegative matrix factorizations,” in _2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2019, pp. 3402–3406.
  * [19] D. Fagot, H. Wendt, C. Févotte, and P. Smaragdis, “Majorization-minimization algorithms for convolutive NMF with the beta-divergence,” in _2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2019, pp. 8202–8206.
  * [20] A. Degleris and N. Gillis, “A provably correct and robust algorithm for convolutive nonnegative matrix factorization,” _IEEE Transactions on Signal Processing_ , vol. 68, pp. 2499–2512, 2020.
  * [21] V. Emiya, N. Bertin, B. David, and R. Badeau, “MAPS-a piano database for multipitch estimation and automatic transcription of music,” 2010.
  * [22] C. Raffel, B. McFee, E. J. Humphrey, J. Salamon, O. Nieto, D. Liang, D. P. Ellis, and C. C. Raffel, “mir_eval: A transparent implementation of common mir metrics,” in _In Proceedings of the 15th International Society for Music Information Retrieval Conference, ISMIR_ , 2014.


# Minimal Interaction Edge Tuning: A New Paradigm for Visual Adaptation

Ningyuan Tang, Minghao Fu, Jianxin Wu*
National Key Laboratory for Novel Software Technology
School of Artificial Intelligence, Nanjing University, China
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

The rapid scaling of large vision pretrained models makes fine-tuning tasks
more and more difficult on edge devices with low computational resources. We
explore a new visual adaptation paradigm called edge tuning, which treats
large pretrained models as standalone feature extractors that run on powerful
cloud servers. The fine-tuning carries out on edge devices with small networks
which require low computational resources. Existing methods that are
potentially suitable for our edge tuning paradigm are discussed. But, three
major drawbacks hinder their application in edge tuning: low adaptation
capability, large adapter network, and high information transfer overhead. To
address these issues, we propose Minimal Interaction Edge Tuning, or MIET,
which reveals that the sum of intermediate features from pretrained models not
only has minimal information transfer but also has high adaptation capability.
With a lightweight attention-based adaptor network, MIET achieves information
transfer efficiency, parameter efficiency, computational and memory
efficiency, and at the same time demonstrates competitive results on various
visual adaptation benchmarks.

11footnotetext: J. Wu is the corresponding author.

##  1 Introduction

Pretraining a large model [10, 9, 3, 22] on general data and fine-tune it on
downstream tasks is emerging as the mainstream methodology for computer vision
tasks. Knowledge or representation learned from the pretraining makes
adaptation to a target domain much easier than training from scratch.

However, some paradoxes begin to emerge within this process or learning
paradigm. One natural issue is: if the downstream dataset is small, we cannot
afford to and should not store one new fine-tuned model for each downstream
visual task. Parameter-efficient fine-tuning (PEFT) methods [14, 32, 15, 28,
19] are proposed to fine-tune large pretrained models with only a small set of
trainable parameters, by either inserting small learnable blocks, adding few
learnable tokens, or learning low rank decomposition of extra parameters.
These methods achieve competitive results on various downstream adaptation
tasks with less than 1% trainable parameters compared to full fine-tuning.

While PEFT methods partially solved the storage cost of fine-tuned models,
other problems remain. As the scale of pretrained models grows with a blazing
fast speed, fine-tuning or even inference itself is becoming unaffordable,
especially on devices with limited computational resources. On the other hand,
the need for fine-tuning and inference on edge devices (e.g., personal
computers, mobile phones or even cameras) rather than on the cloud AI
infrastructure is increasing rapidly.

In this paper, we propose a new visual adaptation paradigm: _edge tuning_ ,
which enables edge devices to fine-tune downstream visual tasks with a small
edge network. The edge network merely receives intermediate
features/activations from a pretrained model, such that the huge pretrained
network is not required to be (either forward or backward) computed on edge
devices. Upon the proposed paradigm, we design a new visual adaptation method:
Minimal Interaction Edge Tuning, which is based on the side-tuning idea [27,
29]. Methods in this style potentially enables edge-tuning, but the
_interaction_ (i.e., features transferred) from the pretrained network to the
edge device is too much for an edge device to handle. Our contributions can be
summarized as follows:

  * •

We propose a new visual adaptation paradigm: edge tuning, where the pretrained
model works as a service in a cloud server, and edge devices fine-tune small
edge networks to transfer the pretrained large model towards specific
downstream tasks.

  * •

To reduce the interaction between server and edge devices, we reveal that the
sum of intermediate features effectively capture essence of the pretrained
network, which leads to both minimal interaction and high accuracy.

  * •

We propose a new visual adaptation method: MIET, as an effective means for
edge tuning. MIET is parameter-efficient, GPU memory-efficient, computation
efficient, and enjoys a low communication overhead between a cloud server and
edge devices. Experimental results on various datasets show that MIET is
comparable with state-of-the art PEFT methods in visual adaptation accuracy,
but enjoys significantly smaller resource consumptions.

##  2 New Pipeline: Edge Tuning

Figure 1: Illustration of Edge Tuning. Visual samples for different tasks
($x_{1},x_{2},x_{3},\dots$) are collected and transferred to the cloud server.
Within the cloud server, a large pretrained model $B$ acts as a _standalone
feature extractor_ , producing intermediate feature sets $(B,x_{i})$. A gather
function compresses this set to $\mathrm{Gather}(B,x_{i})$ to achieve minimal
interaction but still keeps essential information in $(B,x_{i})$ for
downstream task learning. The gathered features for task $i$ is sent to an
edge device $E_{i}$, where the fine-tuning is performed _solely_ on $E_{i}$
with input $\mathrm{Gather}(B,x_{i})$. During inference, the pretrained model
extracts features, gather them to a minimal level, transfers only a small
chunk of bytes to the edge device, which then makes decisions with small
storage, computational, and memory costs.

As illustrated and described in Figure 1, the idea and structure of edge
tuning are pretty simple. Computations are split between a pretrained model
$B$ (residing in a cloud server) and edge devices. The pretrained model $B$ is
general-purpose and pretrained with abundant data and computing resources,
which is too big to infer on the edge device. The edge device is mostly likely
not powerful enough to store the model $B$ itself, nor does it has the compute
or GPU memory resources to forward compute $B$. The solution is to let $B$
extract features and transfer them to the edge device.

Very important but less discussed, the edge device often lacks communication
resources. That is, in real world it is often difficult or even impossible to
transfer large chunks of data between the edge and the server, e.g., due to
bandwidth or energy constraints. Given an input image $x$, $B$ extracts
features $(B,x)$ and sends it to the edge device. For example, $(B,x)$ can be
the set of activations of all intermediate Transformer blocks’ activations,
which contains tens of times more bytes to transfer than that in the input $x$
itself. In such case communication is too heavy a burden for the edge device.

Practical applications, however, are often keen to fine-tune and infer with a
model for downstream tasks _solely on the edge device_ with _limited storage,
compute, memory, and communication capabilities_.

###  2.1 Recipes and Obstacles

Methods that allow to split computations between a large backbone and a small
side-network has appeared in the community [27, 29], where the side-network
receives features from the backbone but does _not_ send its features back to
the computation graph of the backbone. Thus, during fine-tuning, the backbone
does not need to compute gradients. This split greatly reduces the compute and
(GPU) memory costs of fine-tuning.

An even simpler recipe is linear evaluation, in which final features of the
pretrained model $B$ is coupled with a simple fully connected (FC) layer.
Fine-tuning is then learning the FC layer, where the transferred features are
tiny, the storage, compute and memory costs for edge devices are small, too.
Partial-$k$ tuning unfreezes the last $k$ layers of $B$ during fine-tuning.
When $k$ is small (e.g., $k=1$), it might be able to fit into edge devices.

However, these methods do _not_ yet fit the purpose of edge tuning we propose
in Figure 1. The set of 5 goals in our edge tuning (small storage/parameters,
small compute small memory footprint, small communication, high accuracy)
cannot be met by existing methods. For example,

  * •

Linear evaluation and partial-$k$ tuning leads to low accuracy in recognition,
because they overlook the domain gap between the general-purpose model $B$ and
the downstream task.

  * •

Parameter-efficient (PEFT) methods [14, 4] often requires large compute and
GPU memory during fine-tuning.

  * •

Side-tuning methods [27] produces and transfers large features $(B,x)$, and
the side-network may be too big for the edge device.

###  2.2 Minimal Interaction Edge Tuning

Our solution, as depicted in Figure 1, meets these requirements by proposing a
gather function $\mathrm{Gather}$, which compresses the information in $(B,x)$
to $\mathrm{Gather}(B,x)$. The gathered output is succinct, hence achieving
_minimal interaction or communication_. So long as $\mathrm{Gather}(B,x)$
distills essential information for the downstream task, the small network on
the edge device, $E(\mathrm{Gather}(B,x))$ will achieve high accuracy. Because
in Figure 1, $E_{i}$ is very small and it does _not_ transfer any features
back to $B$, the fine-tuning and inference processes resemble side-tuning
methods, which are storage (parameter), compute, and memory efficient, too.

One additional benefit of our pipeline is that the pretrained model is
_agnostic to downstream tasks_. The pretrained model $B$ does not need to
store any task-specific parameters or gradients. It simply extract features
for any input images. If an input image $x$ is useful in many downstream tasks
(edge devices), the pretrained model $B$ only needs to compute its features
$(B,x)$ _once_.

Then, the key and most difficult question is: how to design a gather function
that _captures essential information in $(B,x)$_ and at the same time
_achieves minimal interaction_?

##  3 Method

We start by studying existing recipes. For example, side-tuning methods can be
placed in our pipeline so long as we adopt the following gather function:

$\mathrm{Gather}(B,x)=\mathrm{Stack}(z_{0},z_{1},\cdots,z_{N}),$ (1)

where $z_{0}$ is embedding of $x$, and $z_{i}$ is the activations/features of
the $i$-th Transformer block in $B$, as illustrated in Figure 2. The ‘Stack’
operator stacks or concatenates all tensors $z_{i}$ together. For example,
when the pretrained network $B$ is ViT-B, one $3\times 224\times 224$ input
image (0.60MB in size) is required to transfer more than 7.5MB to edge
devices!

Figure 2: Architecture of a typical ladder side tuning network [27], where
intermediate features $z_{i}$ from the pretrained model $B$ are added to side
blocks through side paths. The interaction (features to be transferred)
$(B,x)$ are the concatenation of all $z_{i}$ ($i=0,1,\dots,N$).

Because side tuning methods [27, 29] have achieved state-of-the-art accuracy
in visual adaptation tasks, it is a natural hypothesis that
$\mathrm{Stack}(z_{0},z_{1},\cdots,z_{N})$ contains enough information for
downstream tasks. And, we aim to find a gather function $\mathrm{Gather^{*}}$
that can most effectively reduce its size but keep the essential information
for downstream tasks.

###  3.1 Gather Function: Sum is All You Need

Given a typical side tuning structure as shown in Figure 2, learnable side
blocks are placed beneath the frozen base network blocks. The $i$-th side
block $S_{i}$ takes as input the sum of previous block output $o_{i-1}$ and
corresponding feature map $z_{i-1}$ from the pretrained model $B$. Suppose the
computing function in side block $S_{i}$ is $F_{i}$, then the output from
$S_{i}$ is:

$o_{i}=F_{i}(o_{i-1}+z_{i-1})+o_{i-1}+z_{i-1}\,.$ (2)

By induction, we have

$o_{i}=F_{1}(z_{0})+\Sigma_{l=2}^{i}F_{l}(o_{l-1}+z_{l-1})+\Sigma_{l=0}^{i-1}z_{l}\,.$
(3)

Then the input for the $i+1$-th side block is $o_{i}+z_{i}$, or equivalently,

$in_{i+1}=F_{1}(z_{0})+\Sigma_{l=2}^{i}F_{l}(o_{l-1}+z_{l-1})+\Sigma_{l=0}^{i}z_{l}\,.$
(4)

The first two terms in the right-hand side of Eq. 4 are outputs from learnable
side network blocks, while the third term is the sum of intermediate features
(or activations) from the pretrained model $B$. It is worth noting that the
the first two terms both depend on the side-network, while the third term does
_not_ —it is determined by $B$ alone. We name this term (or more generally a
term that only depends on $B$) as an _external feature term_.

To fit into our edge tuning pipeline, $B$ can _only send external feature
terms_ to edge devices, a requirement only the third term
$\Sigma_{l=0}^{i}z_{l}$ satisfies. This term is developed through repeatedly
applying the skip connections.

Furthermore, because side network computations ($F_{i}$) are low rank
transformation to achieve parameter efficiency [29], a natural _conjecture_ is
that this third term (which is determined by the complex model $B$) contains
more complicated features compared to the first two terms (which are
determined by the simple computations $F_{i}$). In other words, it is
reasonable to guess that the third term, $\Sigma_{l=0}^{i}z_{l}$, contains
essential information in $in_{i+1}$. Please note that this conjecture is made
with respect to _any_ $i$ ($1\leq i\leq N$), not only the final block (i.e.,
$i=N$).

Our analysis in the above clearly suggests a simple gather function:

$\mathrm{Gather^{*}}(B,x)=\mathrm{Sum}(z_{0},z_{1},\cdots,z_{N})\,.$ (5)

This gather function leads to only $1/(N+1)$ communication overhead compared
to the one in Eq. 1. Smaller input to the edge devices also means that the
storage, compute, and memory costs will be reduced accordingly. As our
experimental results will show in later sections, $\mathrm{Gather^{*}}(B,x)$
does contains essential information for downstream tasks, which leads to high
accuracy for them.

$\mathrm{Gather^{*}}(B,x)$ is very simple in its formation, and the idea to
sum up intermediate features may seem naive from a first peak. We want to
suggest that a simple gather function is the most effective in practice: this
gather function incurs almost zero extra computation or memory cost.
Furthermore, this simple and natural form is derived from careful analyses of
existing methods, which is endorsed by both existing experimental results and
supportive conjectures.

###  3.2 Gather Function: Migrating to Downstream Tasks

One difficulty in handling different downstream tasks are the domain gaps. In
some simple tasks with only low-level image patterns, high-level features
extracted by a pretrained model may be harmful to the adaptation. In such
cases, gathering all the intermediate features altogether is suboptimal.

So we migrate the extracted features to specific downstream tasks by only
gathering features from the first $k$ layers, which leads to the final form of
our minimal interaction gather function:

$\mathrm{Gather^{*}}(B,x,k)=\mathrm{Sum}(z_{0},z_{1},\cdots,z_{k})\,,$ (6)

in which $0\leq k\leq N$. As we emphasized earlier, our hypothesis that the
third term in Eq. 4 contains essential information inside $in_{i+1}$ is made
for all $i$ ($1\leq i\leq N$). Hence, this final form is also supported by our
hypothesis.

We also want to point out that previous side-tuning methods such as [27, 29]
correspond to $k=N$ (i.e., use the feature set $\\{z_{0},z_{1},\dots,z_{N}\\}$
all the way up to the final layer’s output $z_{N}$). However, as our analysis
and experiments show, it is both reasonable and beneficial to use $i<N$ for
tasks with large domain gaps.

This final form introduces a hyperparameter, $k$, which needs to be set before
edge tuning. We select the best $k$ from a candidate set by using the
validation set. The candidate set is formed by $N$, $3N/4$ and $N/2$. For
example, for ViT-B with 12 layers, we pick $k$ from the set $[6,9,12]$.

Figure 3: Illustration of MIET and other fine-tuning methods. Adapter fine-
tunes a model by inserting trainable adapter blocks. LoRA adds trainable low-
rank decomposed matrices to parameters. Side-tuning trains a decoupled side
network by passing intermediate features to the side network block-wise. MIET
adds intermediate features together, and pass the sum to a trainable edge
network to adapt to specific task.

###  3.3 The Proposed MIET Method

With $\mathrm{Gather^{*}}$, our Minimal Interaction Edge Tuning (MIET) method
becomes clear and easy to describe. It contains a pretrained base vision
transformer network $B$ and a low-rank attention edge network $\mathrm{LAE}$
(which will be introduced latter).

First we gather the intermediate features with $\mathrm{Gather^{*}}$. For base
network $B$ with $N$ layers, it receives an input image $x$ and a $k$ for
gather function. The forward path of $x$ produces a set of feature maps, Then
we gather the feature maps with $\mathrm{Gather^{*}}$ to form a mixed feature
map $Z_{\mathrm{mix}}$:

$Z_{\mathrm{mix}}=\mathrm{Gather^{*}}(B,x,k)\,.$ (7)

Then the low-rank attention edge network $\mathrm{LAE}$ takes
$Z_{\mathrm{mix}}$ as input and output the final tuning result:

$Z_{\mathrm{out}}=\mathrm{LAE}(Z_{\mathrm{mix}})\,.$ (8)

From the result in [29], attention block with low-dimensional QKV projection
is an effective and efficient structure for visual adaptation. Our
experimental results also show that it is very effective for edge tuning. So
we adopt this structure for edge network.

As shown in Figure 3, $\mathrm{LAE}$ contains $L$ blocks. Each block contains
a layer norm [1] and a low-rank attention, where low-rank attention refers to
self-attention with an extremely low QKV hidden dimension $r$ (e.g., hidden
dimension equals 16 for feature dimension equals 768). Skip connection is also
applied in each block. In practice, we take $L=4$ and $r=32$ for all
downstream tasks, and find that such a small edge network scale is already
enough for most visual adaptation tasks.

##  4 Experiments

Table 1: Results on VTAB-1K. “Params”: number of trainable parameters. “MACs Cloud”, “MACs Edge”: multiply–accumulate operations for cloud and edge model during fine-tuning. “GPU Mem Cloud”, “GPU Mem Edge”: GPU memory footprint within cloud and edge devices, respectively during fine-tuning with batch size 32. “Nat.”, “Spe.”, “Str.”: average accuracy for “Natural”, “Specialized”, “Structured” task groups. “Avg.”: group-wise average accuracy. Best results are in boldface. ViT-B/16 | Support | Transfer | Trainable | MACs | GPU Mem | VTAB
---|---|---|---|---|---|---
| Edge | Overhead | Params | Cloud | Edge | Cloud | Edge |
(85.8M) | Tuning | (MB/img) | (M) | $\bm{(\times{10}^{9})}$ | $\bm{(\times{10}^{9})}$ | (GB) | (GB) | Nat. | Spe. | Str. | Avg.
# of tasks |  |  |  |  |  |  |  | 7 | 4 | 8 | 19
Full fine-tuning |  | - | 85.8 |  0000 | 17.57 |  000 | 6.09 | 75.9 | 83.4 | 47.6 | 68.9
BitFit |  | - | 0.10 |  0000 | 17.57 |  000 | 3.80 | 73.3 | 78.2 | 44.1 | 65.2
VPT |  | - | 0.56 |  0000 | 23.18 |  000 | 5.63 | 78.5 | 82.4 | 54.9 | 72.0
LoRA |  | - | 0.29 |  0000 | 21.87 |  000 | 3.40 | 79.5 | 84.6 | 59.8 | 74.5
AdaptFormer |  | - | 0.16 |  0000 | 17.60 |  000 | 4.11 | 80.6 | 84.9 | 59.0 | 74.7
FacT |  | - | 0.07 |  0000 | 18.67 |  000 | 4.81 | 80.6 | 85.3 | 60.7 | 75.6
SPT-LoRA |  | - | 0.54 |  0000 | 20.44 |  000 | 5.13 | 81.9 | 85.9 | 61.3 | 76.4
Linear probing | ✓ | 0.003 |  000 | 17.57 |  0000 | 0.57 | 0.07 | 69.1 | 77.1 | 26.8 | 57.6
Partial-1 | ✓ | 0.577 | 6.75 | 17.57 |  01.42 | 0.57 | 0.52 | 69.4 | 78.5 | 34.2 | 60.7
LST | ✓ | 7.503 | 2.38 | 17.57 |  00.51 | 0.57 | 2.08 | 77.6 | 85.6 | 59.7 | 74.3
LAST | ✓ | 4.040 | 0.66 | 17.57 |  00.15 | 0.57 | 0.77 | 80.9 | 86.2 | 62.3 | 76.5
MIET (ours) | ✓ | 0.577 | 0.38 | 17.57 |  00.09 | 0.57 | 0.45 | 81.4 | 86.4 | 62.4 | 76.7

###  4.1 Experiments on VTAB-1K

We conducted experiments on VTAB with a ViT-B/16 pretrained with ImageNet-21k.
Visual Tasks Adaptation Benchmark (VTAB) [34] is a collection of visual
adaptation tasks widely used in assessing transferability of pretrained vision
models and fine-tuning methods. It is a benchmark composed of 19 tasks, which
can be categorized into three groups: Natural, Specialized, and Structured.
Each task includes 800 training images and 200 validation images.

#### Baseline methods.

We compare MIET with various fine-tuning methods. Basically, we divide them
into two categories: those potentially support edge tuning and those do not.
For the first category, we have linear probing, Partial-1 (which unfreeze the
final layer of ViT-B), LST [27], LAST [29] and our method MIET. For the second
category, we have BitFit [33], VPT [15], LoRA [14], AdaptFormer [4], FacT
[16], SPT-LoRA [8].

#### Results.

The results on VTAB-1K is shown in Table 1. Methods that support edge tuning
are marked with $\checkmark$ in corresponding cell. In the “Transfer Overhead”
column, we list the communication cost per image between cloud and edge
devices. MIET only requires to transfer 0.58MB of information per image (which
roughly equals the size of an original image), lower than other compared
methods (except linear probing). This overhead is only about $1/7$ of LAST and
$1/13$ of LST. Hence, MIET has met our “minimal interaction” goal.

With respect to trainable parameters, MIET only introduces 0.38M trainable
parameters, lower than other methods that support edge tuning. Though MIET
requires slightly more trainable parameters than some non-edge tuning methods,
this difference is in fact negligible with respect to parameter size of the
pretrained model (e.g., MIET:0.38/85.8=0.44% vs. FacT:0.07/85.8=0.08%, both
$<1\%$).

We also list the MACs and GPU memory footprint for different methods. For non-
edge tuning methods, the whole model are enforced to be fine-tuned on edge
devices. For methods that support edge tuning, pretrained model is deployed on
a cloud server and edge networks are deployed on edge devices, leading to very
low FLOPs and memory footprint on edge devices. MIET requires the lowest MACs
and GPU memory (again except linear probing) within edge devices, which are
respectively about 1% and 10% of normal PEFT methods.

MIET achieves the highest group-wise accuracy compared to other methods. It
has the highest group accuracy for specialized datasets and structured
datasets. This result proves that MIET has strong adaptation ability and our
$\mathrm{Gather^{*}}$ indeed capture essential information in intermediate
features.

###  4.2 Experiments on Fine-Grained Few-Shot Learning

To further assess the few-shot adaptation ability of MIET, following [16] we
run MIET on fine-grained few-shot (FGFS) benchmark, which contains 5 fine-
grained datasets: Aircraft [20], Food101 [2], Pets [23], Flowers102 [21] and
Cars [17]. We fine-tune the edge network with train set containing 1, 2, 4, 8,
16-shot per class. Experiments are repeated three times with three different
random seeds, and we report the average test set accuracy.

Figure 4: Top-1 accuracy on fine-grained few-shot datasets with train set
containing 1, 2, 4, 8, 16-shot per class.

#### Results.

As shown in Figure 4, the average few-shot accuracy of MIET consistently
outperforms other 5 baseline methods in all cases. The average improvement of
MIET compared to previous state-of-the-art FacT gets to 3.2% in 16-shot, which
is the greatest improvement among all settings.

###  4.3 Experiments on Domain Generalization

Following the setting in [35], we test the robustness of MIET to domain shifts
[36] when they are inevitable. We randomly sample 16 images from each class in
ImageNet-1K to form train set. Apart from the validation set from ImageNet-1K
[26], we also test the fine-tuned model on four out-of-domain datasets: 1)
ImageNet-V2 [25] is collected from different sources than ImageNet but
following the same collection protocol, 2) ImageNet-Sketch [31] is composed of
sketch images of the same 1,000 classes in ImageNet, 3) ImageNet-A [12]
contains adversarially-filtered images, 4) ImageNet-R [11] contains various
artistic renditions of ImageNet-1K. We repeat the experiments for three times
with different seeds and report average result over three seeds.

Table 2: Results on domain generalization with ViT-B/16 as the pretrained model $B$. Following [35], we randomly sample 16 images from each class in ImageNet-1K, and test the fine-tuned model on test sets with different domain shifts. We report average test accuracy over 3 different random seeds. Best Results are shown in boldface. | Source | Target
---|---|---
| ImageNet | -Sketch | -V2 | -A | -R
Adapter | 70.5 | 16.4 | 59.1 | 5.5 | 22.1
VPT | 70.5 | 18.3 | 58.0 | 4.6 | 23.2
LoRA | 70.8 | 20.0 | 59.3 | 6.9 | 23.3
NOAH | 71.5 | 24.8 | 66.1 | 11.9 | 28.5
MIET (ours) | 76.5 | 37.9 | 66.5 | 21.4 | 37.5

#### Results.

As shown in Table 2, MIET outperforms previous methods consistently in both
the source domain and in generalization to other domains. Compared to the
previous the state-of-the-art method NOAH, the accuracy gain of MIET is very
significant in ImageNet, ImageNet-Sketch,ImagnetNet-A and ImageNet-R, which
are 5.0, 13.1, 9.5 and 9.0 percentage points, respectively. These result
verifies the robustness of MIET against test set domain shift.

Table 3: Ablations on removing important components of MIET. “Avg.” denotes the group wise average accuracy on VTAB. $\mathrm{Gather}^{*}$ for (add intermediate features together / use output feature alone). LAE for (use low-rank attention edge network / linear probing). $\mathrm{Gather}^{*}$ | LAE | Avg.
---|---|---
|  | 57.6
✓ |  | 59.3
| ✓ | 66.7
✓ | ✓ | 76.7
Table 4: Ablations on different block functions with ViT-B/16 as the pretrained model. # Layers denotes the number of layers in edge network. Avg. denotes the group wise average accuracy on VTAB. Block Func | # Layers | Params (M) | Avg.
---|---|---|---
LAE | 4 | 0.38 | 76.7
LAE | 8 | 0.76 | 76.9
MLP | 4 | 0.38 | 75.8
MLP | 8 | 0.76 | 76.1
DWConv | 4 | 0.03 | 69.8
DWConv | 8 | 0.06 | 70.2

###  4.4 Ablation studies

Now we conduct ablation studies on different components of MIET.

#### Ablations on feature gathering and LAE.

we test the effectiveness of minimal interaction feature gathering
($\mathrm{Gather}^{*}$) and low-rank edge network in our method. Results are
shown in Table 4.3. When both of them are not adopted, the method degrade to
naive linear probing. When only minimal interaction feature gathering is used,
then intermediate features are added for linear probing, the result in the
second row reveal a 1.7% gain over naive linear probing, but this improvement
is limited. When only LAE is used, the accuracy gain over naive linear probing
is 9.1%, which indicates that adding an edge network directly after a
pretrained model is useful for visual adaptation, but still lags far behind
other fine-tuning methods. When both $\mathrm{Gather}^{*}$ and LAE are
adopted, the average accuracy increases to 76.7% (on par with or better than
other fine-tuning methods). This ablation shows that both components are
important for MIET in visual adaptation, and only when two components are used
together can we get the best results.

#### Ablations on different edge network structure.

We substitute the block function of edge network with MLP and Depthwise
convolution (DWConv) [5]. Results are shown in Table 4.3. We can see that
while low-rank attention (LAE) and MLP have the same number of parameters, LAE
outperforms MLP with about 1% average accuracy lead. When the block function
is changed to depthwise convolution, the number of parameters decreases, but
the average accuracy also drops by a large margin. Overall, low-rank attention
leads to the best performance and parameter efficiency tradeoff.

#### Ablations on domain migration.

To verify the effect of domain migration with different $k$, we test the
respective accuracy on all 19 VTAB datasets for $k=6$, $k=9$ and $k=12$.
Results are shown in Figure 5. For Natural and Specialized tasks, $k=12$
(adding all intermediate features) usually lead to the best result. But for
structured datasets which have more low-level features, $k=9$ outperforms
$k=12$ in 6 out of 9 datasets. These results indicate that in datasets with
rich low-level features, intermediate outputs from early layers of the
pretrained model are more important.

Figure 5: Ablation study on gather function with different $k$ on VTAB.
Figures from left to right respectively refer to “Natural”, “Specialized”, and
“Structured” task group.

##  5 Related Work

Parameter-efficient fine-tuning (PEFT) aims at fine-tuning a large pretrained
model with only a small amount of trainable parameters. Adapter [13] inserts
learnable MLP layers after each attention [30] layer in a Transformer block.
AdaptFormer [4] inserts MLP layers to FFN in a parallel form. BitFit [33]
fine-tunes the pretrained model to downstream tasks by learning all bias terms
within the pretrained model. SSF [18] learns scale factor $\gamma$ and shift
factor $\beta$, and $X^{\prime}=\gamma\odot X+\beta$, where $\odot$ is
element-wise product. VPT [15] does visual adaptation by learning additional
task-specific tokens. VPT-Shallow only inserts learnable tokens before the
first later, while VPT-Deep inserts learnable tokens before each later.

LoRA [14] fine-tunes low-rank decomposition matrices of a dense layer instead
of fine-tuning dense layers directly. Weights are updated with
$W^{\prime}=W+AB$, where $A\in\mathbb{R}^{h\times h^{\prime}}$,
$B\in\mathbb{R}^{h^{\prime}\times h}$, $h^{\prime}\ll h$. NOAH [35] combines
the structure of Adapter, VPT and LoRA and use evolutionary algorithm to
search for best combination.

Another line for visual adaptation is about memory-efficiency. LST [27]
proposes to fine-tune a side network which is disentangled from pretrained
model, and save memory by freeing pretrained model from back-propagation. DTL
[7] considers adding side information back to pretrained network to make a
tradeoff between adaptation ability and memory consumption. LAST [29] proposes
a low-rank attention side network for effective visual adaptation.

##  6 Conclusion and Discussion

In this paper, we proposed a visual adaptation paradigm and a method which
allows edge devices with extremely low computation resources to finetune large
vision model without access to the pretrained model itself. While a pretrained
model resides in cloud and provides features that are too large to transfer to
the edge device, our minimal interaction edge tuning (MIET) compresses the
features while keeping essential information for downstream tasks. The entire
fine-tuning process is performed only on the edge device, requesting only
limited storage, compute, memory, and communication resources while achieving
high accuracy. In particular, we emphasized the communication factor, which
has been largely ignored in previous research.

Currently, our MIET is restricted to visual recognition tasks. However, side-
tuning has been shown to be able to work for object detection. We will work on
this limitation, by extending MIET to object detection and other visual tasks.
More vision tasks, such as image and video generation, may also enjoy benefits
brought up by MIET. We will also explore other gather functions in the minimal
interaction side tuning framework, which may provide even higher accuracy and
smaller interaction.

## References

  * Ba et al. [2016] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  _arXiv preprint arXiv:1607.06450_ , 2016.
  * Bossard et al. [2014] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool.  Food-101–mining discriminative components with random forests.  In _European Conference on Computer Vision (ECCV)_ , pages 446–461, 2014.
  * Caron et al. [2021] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 9650–9660, 2021.
  * Chen et al. [2022] Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo.  Adaptformer: Adapting vision transformers for scalable visual recognition.  In _Advances in Neural Information Processing Systems (NIPS)_ , pages 16664–16678, 2022.
  * Chollet [2017] François Chollet.  Xception: Deep learning with depthwise separable convolutions.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 1800–1807, 2017.
  * Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _International Conference on Learning Representations (ICLR)_ , pages 1–21, 2021.
  * Fu et al. [2024] Minghao Fu, Ke Zhu, and Jianxin Wu.  Dtl: Disentangled transfer learning for visual recognition.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 12082–12090, 2024.
  * He et al. [2023] Haoyu He, Jianfei Cai, Jing Zhang, Dacheng Tao, and Bohan Zhuang.  Sensitivity-aware visual parameter-efficient fine-tuning.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 11825–11835, 2023.
  * He et al. [2020] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 9726–9735, 2020.
  * He et al. [2022] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 15979–15988, 2022.
  * Hendrycks et al. [2021a] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al.  The many faces of robustness: A critical analysis of out-of-distribution generalization.  In _Proceedings of the IEEE/CVF international conference on computer vision (ICCV)_ , pages 8340–8349, 2021a.
  * Hendrycks et al. [2021b] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song.  Natural adversarial examples.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR)_ , pages 15262–15271, 2021b.
  * Houlsby et al. [2019] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.  Parameter-efficient transfer learning for NLP.  In _International Conference on Machine Learning_ , pages 2790–2799, 2019.
  * Hu et al. [2022] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  LoRA: Low-rank adaptation of large language models.  In _International Conference on Learning Representation (ICLR)_ , pages 1–13, 2022.
  * Jia et al. [2022] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim.  Visual Prompt Tuning.  In _European Conference on Computer Vision (ECCV)_ , pages 709–727. Springer, 2022.
  * Jie and Deng [2023] Shibo Jie and Zhi-Hong Deng.  FacT: Factor-tuning for lightweight adaptation on vision transformer.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 1060–1068, 2023.
  * Krause et al. [2013] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , pages 554–561, 2013.
  * Lian et al. [2022] Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang.  Scaling & Shifting Your Features: A new baseline for efficient model tuning.  In _Advances in Neural Information Processing Systems (NIPS)_ , pages 109–123, 2022.
  * Liu et al. [2024] Ting Liu, Xuyang Liu, Siteng Huang, Honggang Chen, Quanjun Yin, Long Qin, Donglin Wang, and Yue Hu.  DARA: Domain- and relation-aware adapters make parameter-efficient tuning for visual grounding.  In _Proceedings of the IEEE International Conference on Multimedia and Expo (ICME)_ , pages 1–6, 2024.
  * Maji et al. [2013] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi.  Fine-grained visual classification of aircraft.  _arXiv preprint arXiv:1306.5151_ , 2013.
  * Nilsback and Zisserman [2006] M-E Nilsback and Andrew Zisserman.  A visual vocabulary for flower classification.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 1447–1454. IEEE, 2006.
  * Oquab et al. [2024] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski.  Dinov2: Learning robust visual features without supervision, 2024.
  * Parkhi et al. [2012] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar.  Cats and dogs.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 3498–3505, 2012.
  * Paszke et al. [2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  PyTorch: An imperative style, high-performance deep learning library.  In _Advances in Neural Information Processing Systems (NIPS)_ , pages 8026–8037, 2019.
  * Recht et al. [2019] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar.  Do imagenet classifiers generalize to imagenet?  In _International conference on machine learning_ , pages 5389–5400, 2019.
  * Russakovsky et al. [2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei.  ImageNet large scale visual recognition challenge.  _IJCV_ , 115(3):211–252, 2015.
  * Sung et al. [2022a] Yi-Lin Sung, Jaemin Cho, and Mohit Bansal.  LST: Ladder Side-Tuning for parameter and memory efficient transfer learning.  In _Advances in Neural Information Processing Systems (NIPS)_ , pages 12991–13005, 2022a.
  * Sung et al. [2022b] Yi-Lin Sung, Jaemin Cho, and Mohit Bansal.  VL-Adapter: Parameter-efficient transfer learning for vision-and-language tasks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5227–5237, 2022b.
  * Tang et al. [2024] Ningyuan Tang, Minghao Fu, Ke Zhu, and Jianxin Wu.  Low-rank attention side-tuning for parameter-efficient fine-tuning, 2024.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Advances in Neural Information Processing Systems (NIPS)_ , pages 6000–6010, 2017.
  * Wang et al. [2019] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing.  Learning robust global representations by penalizing local predictive power.  _Advances in Neural Information Processing Systems (NIPS)_ , 32, 2019.
  * Yin et al. [2023] Dongshuo Yin, Yiran Yang, Zhechao Wang, Hongfeng Yu, Kaiwen Wei, and Xian Sun.  1% vs 100%: Parameter-efficient low rank adapter for dense predictions.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 20116–20126, 2023.
  * Zaken et al. [2022] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg.  BitFit: Simple parameter-efficient fine-tuning for transformer-based masked language-models.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics_ , pages 1–9, 2022.
  * Zhai et al. [2019] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al.  A large-scale study of representation learning with the visual task adaptation benchmark.  _arXiv preprint arXiv:1910.04867_ , 2019.
  * Zhang et al. [2022] Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu.  Neural prompt search, 2022.
  * Zhou et al. [2023] Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy.  Domain generalization: A survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 45(4):4396–4415, 2023.

##  Appendix A Detailed results on VTAB-1K

Here we present a detailed version of Table 1, where per-task accuracy on
VTAB-1K datasets are listed.

Table 5: Per-task results on the VTAB-1K benchmark with ViT-B/16 pretrained on IN21k. “Mean Acc” denoted groupwise average accuracy on three VTAB task groups. | Natural | Specialized | Structured |
---|---|---|---|---
|  CIFAR-100 |  Caltech101 |  DTD |  Flowers102 |  Pets |  SVHN |  Sun397 |  Patch Camelyon |  EuroSAT |  Resisc45 |  Retinopathy |  Clevr/count |  Clevr/distance |  DMLab |  KITTI/distance |  dSprites/loc |  dSprites/ori |  SmallNORB/azi |  SmallNORB/ele |  Mean Acc
Linear probing | 64.4 | 85.0 | 63.2 | 97.0 | 86.3 | 36.6 | 51.0 | 78.5 | 87.5 | 68.5 | 74.0 | 34.3 | 30.6 | 33.2 | 55.4 | 12.5 | 20.0 |  09.6 | 19.2 | 57.6
Full finetuning | 68.9 | 87.7 | 64.3 | 97.2 | 86.9 | 87.4 | 38.8 | 79.7 | 95.7 | 84.2 | 73.9 | 56.3 | 58.6 | 41.7 | 65.5 | 57.5 | 46.7 | 25.7 | 29.1 | 68.9
BitFit | 72.8 | 87.0 | 59.2 | 97.5 | 85.3 | 59.9 | 51.4 | 78.7 | 91.6 | 72.9 | 69.8 | 61.5 | 55.6 | 32.4 | 55.9 | 66.6 | 40.0 | 15.7 | 25.1 | 65.2
VPT | 78.8 | 90.8 | 65.8 | 98.0 | 88.3 | 78.1 | 49.6 | 81.8 | 96.1 | 83.4 | 68.4 | 68.5 | 60.0 | 46.5 | 72.8 | 73.6 | 47.9 | 32.9 | 37.8 | 72.0
LST | 59.5 | 91.5 | 69.0 | 99.2 | 89.9 | 79.5 | 54.6 | 86.9 | 95.9 | 85.3 | 74.1 | 81.8 | 61.8 | 52.2 | 81.0 | 71.7 | 49.5 | 33.7 | 45.2 | 74.3
LoRA | 67.1 | 91.4 | 69.4 | 98.8 | 90.4 | 85.3 | 54.0 | 84.9 | 95.3 | 84.4 | 73.6 | 82.9 | 69.2 | 49.8 | 78.5 | 75.7 | 47.1 | 31.0 | 44.0 | 74.5
AdaptFormer | 70.8 | 91.2 | 70.5 | 99.1 | 90.9 | 86.6 | 54.8 | 83.0 | 95.8 | 84.4 | 76.3 | 81.9 | 64.3 | 49.3 | 80.3 | 76.3 | 45.7 | 31.7 | 41.1 | 74.7
FacT | 70.6 | 90.6 | 70.8 | 99.1 | 90.7 | 88.6 | 54.1 | 84.8 | 96.2 | 84.5 | 75.7 | 82.6 | 68.2 | 49.8 | 80.7 | 80.8 | 47.4 | 33.2 | 43.0 | 75.6
SPT-LoRA | 73.5 | 93.3 | 72.5 | 99.3 | 91.5 | 87.9 | 55.5 | 85.7 | 96.2 | 75.9 | 85.9 | 84.4 | 67.6 | 52.5 | 82.0 | 81.0 | 51.1 | 30.2 | 41.3 | 76.4
LAST | 66.7 | 93.4 | 76.1 | 99.6 | 89.8 | 86.1 | 54.3 | 86.2 | 96.3 | 86.8 | 75.4 | 81.9 | 65.9 | 49.4 | 82.6 | 87.9 | 46.7 | 32.3 | 51.5 | 76.5
MIET | 67.8 | 93.2 | 76.8 | 99.7 | 91.1 | 87.5 | 54.0 | 87.3 | 95.8 | 87.1 | 75.3 | 84.5 | 66.8 | 51.2 | 83.7 | 84.0 | 45.7 | 33.3 | 50.3 | 76.7

##  Appendix B Imlementation details

We conducted all visual adaptation tasks with a ViT-B/16 pretrained on IN21k
as backbone. For VTAB-1K, FGFS and domain generalization tasks, we used Adam
optimizer with batch size 32, and cosine learning scheduler to adjust learning
rate. Learning rate is selected according to the accuracy on validation set.
Following the setting of [15], images were directly resized to 224 × 224 for
VTAB-1K. For FGFS and domain generalization tasks, following the setting in
[16] and [35], we performed random resize crop and random horizontal flip, and
no extra data augmentation tricks were used.

All the experiments were implemented with PyTorch [24].

##  Appendix C Experiment on varying side input for side tuning

In section 3.1, we conjecture that the third term in Eq. 4
($\Sigma_{l=0}^{i}z_{l}$, or external feature term), contains essential
information in $in_{i+1}$, and thus important to the fine-tune accuracy. So we
vary the side input term, by taking $\hat{z_{i}}$ instead of $z_{i}$ as
auxiliary input for side block $S_{i+1}$, where:

$\hat{z_{i}}=\left\\{\begin{aligned} &z_{i}&,i<g\\\ &z_{i}-z_{i-g}&,i\geq
g\end{aligned}\right.$ (9)

$g$ varies from 1 to N+1. then,
$in_{i}=F_{1}(z_{0})+\Sigma_{l=2}^{i}F_{l}(o_{l-1}+z_{l-1})+\Sigma_{l=0}^{i}\hat{z}_{l}$,
making:

$in_{i}=\left\\{\begin{aligned}
&F_{1}(z_{0})+\Sigma_{k=2}^{i}F_{k}(o_{k-1}+z_{k-1})+\Sigma_{k=0}^{i}z_{k}&,i<g\\\
&F_{1}(z_{0})+\Sigma_{k=2}^{i}F_{k}(o_{k-1}+z_{k-1})+\Sigma_{k=i-g+1}^{i}z_{k}&,i\geq
g\end{aligned}\right.$ (10)

Figure 6: The upper figure shows the architecture of typical ladder side
tuning, where intermediate features from pretrained model are added to side
blocks through side paths. The lower-left figure shows a modified version of
ladder side tuning, where the input from $i$-th side path $z_{i}$ is replaced
with $z_{i}-z_{i-g}$. Experiment result on varying intermediate feature input
to side network is shown in the lower-right figure. Where $g$ is the
hyperparameter which controls the form of intermediate feature input to each
side block. Accuracy stands for group-wise average accuracy on VTAB benchnark.

In the experiment, we use ViT-B [6] as backbone, and adopts the low-rank
attention structure from [29] as block function in side network. We test the
effect of different $g$ on VTAB benchmark, and results are shown in lower-
right of Figure 2 (Notice that when $g=13$, the setting falls backs to normal
side tuning). From this experiment, we can find that, without previous
external feature term from skip connection (the case when $g=1$), overall
accuracy suffers from a big cut (73.1). So $z_{i}$ from the side path alone is
not enough, the accumulated external feature term is the key for side network.
Furthermore, we can discover that increase of $g$ will lead to a higher
accuracy, this indicates that more complicated external feature term can
result in better model accuracy. The overall accuracy reaches a platform when
$g\geq 6$, indicating the benefit of more complicated external feature term is
limited.