#  Adapter-X: A Novel General Parameter-Efficient Fine-Tuning Framework for
Vision

Minglei Li1† Peng Ye1† Yongqi Huang1 Lin Zhang1
Tao Chen1 Tong He2 Jiayuan Fan1 Wanli Ouyang2
<EMAIL_ADDRESS><EMAIL_ADDRESS>
1Fudan University 2 Shanghai AI Laboratory
https://github.com/leoli646/Adapter-X Corresponding Author. †Equal
Contribution.

###### Abstract

Parameter-efficient fine-tuning (PEFT) has become increasingly important as
foundation models continue to grow in both popularity and size. Adapter has
been particularly well-received due to their potential for parameter reduction
and adaptability across diverse tasks. However, striking a balance between
high efficiency and robust generalization across tasks remains a challenge for
adapter-based methods. We analyze existing methods and find that: 1) parameter
sharing is the key to reducing redundancy; 2) more tunable parameters, dynamic
allocation, and block-specific design are keys to improving performance.
Unfortunately, no previous work considers all these factors. Inspired by this
insight, we introduce a novel framework named Adapter-X. First, a Sharing
Mixture of Adapters (SMoA) module is proposed to fulfill token-level dynamic
allocation, increased tunable parameters, and inter-block sharing at the same
time. Second, some block-specific designs like Prompt Generator (PG) are
introduced to further enhance the ability of adaptation. Extensive experiments
across 2D image and 3D point cloud modalities demonstrate that Adapter-X
represents a significant milestone as it is the first to outperform full fine-
tuning in both 2D image and 3D point cloud modalities with significantly fewer
parameters, i.e., only $0.20\%$ and $1.88\%$ of original trainable parameters
for 2D and 3D classification tasks. Our code will be publicly available.

##  1 Introduction

In an era where models are becoming increasingly larger and more complex, the
ability to efficiently adapt them to specific tasks and datasets has become
increasingly crucial. Parameter-efficient fine-tuning (PEFT) has emerged as a
promising solution to this challenge. Among the techniques proposed for PEFT,
adapter-based methods have attracted significant interest due to the few
trainable parameters and the potential to enhance the model’s performance on
different downstream tasks.

Figure 1: Comparison of efficiency and performance between our Adapter-X and
other methods.

Adapter-based methods can primarily be organized into two categories, focusing
on two distinct advantages: the performance capabilities across various tasks
and the efficiency gained from the minimal scale of parameters required for
fine-tuning. Methods focusing on performance improvements are represented by
three distinct approaches. [29][11] aim to create a versatile adapter
structure capable of handling a broad spectrum of tasks. [27] [1] [11] [16]
aim to either combine the advantages of diverse PEFT approaches or seek to
establish a unified perspective by analyzing the similarities among different
adapters and other PEFT methods. [52] [53] utilize neural architecture search
(NAS) to discover superior PEFT combination approaches. For the aspect of
adapter efficiency, specific methods [13] [26] [12] [18] have been devised.
They further improve adapter efficiency from different perspectives of
redundancy, including rank, density, and precision. While these methods strive
to balance the dual advantages of adapters, none have yet achieved both high
efficiency and good generalization across various modalities and tasks, which
invites us to ponder: _How to further enhance generalization capabilities of
Adapters while inspiring their efficiency potential?_

To answer this question, we analyze existing methods and summarize the
following factors influencing efficiency and performance. From an efficiency
standpoint, lots of approaches [26] [13] [19] [37] [42] have employed a
sharing strategy between different blocks, which show that the parameter-
sharing mechanism can indeed reduce redundancy, thus enhancing the parameter
efficiency of PEFT methods. From a performance perspective, we find that
improving the performance of the existing PEFT methods on different tasks can
be roughly categorized into three primary perspectives. Firstly, increasing
the trainable parameters, such as increasing the rank of adapters, can enhance
the adapter’s performance. Secondly, dynamic allocation of tunable parameters
has proven to be highly beneficial in the context of PEFT. For instance, the
NAS-based strategy [52] [53] can be considered as allocating parameters
following the change of data distribution. Thirdly, diverse methods implement
distinct designs for blocks at different positions within the architecture,
such as utilizing different PEFT methods. These block-specific designs enable
the model to better accommodate a range of datasets. Previous works only focus
on one or a few key points, making them unable to handle the balance problem
well. We suppose that if we can design a method that can consider all these
effective factors at the same time, we can achieve concurrent improvements in
both efficiency and performance.

To this end, we propose a new general parameter-efficient fine-tuning
framework named Adapter-X. The core of the framework is that different blocks
recursively select the most suitable adapter from a block sharing adapter
expert library. We refer to this component as the Sharing Mixture of Adapters
(SMoA). This element endows us with the three aforementioned advantages:
sharing strategy, dynamic allocation, and an increased number of tunable
parameters for each block. First, Sharing Mixture of Adapters, is utilized
across all blocks, significantly reducing the parameter count. Secondly, for
any given set of tokens, Adapter-X will choose distinct adapter experts for
different blocks to process them, thereby achieving a token-level dynamic
allocation of fine-tuning parameters. Third, since each block will recursively
access all the adapter experts from the shared expert library, it implies that
each block can access more adapters than the previous traditional approach.
Lastly, in order to enhance the block-specific design, Adapter-X utilizes
different norm layers and Prompt Generator to generate block-specific prompts,
further improving the model’s performance.

In order to validate the effectiveness and generalization capability of the
proposed method Adapter-X, extensive experiments are conducted on different
datasets across 2D image and 3D point cloud modalities. The results
demonstrate the ability of Adapter-X to achieve superior performance with a
significantly reduced number of parameters, as shown in Fig. 1. To the best of
our knowledge, this is the first work that outperforms full fine-tuning in
both 2D image and 3D point cloud modalities. In particular, on the 2D VTAB
dataset, our method achieves better classification results with 7.5 times less
parameters compared to the original Adapter approach. On the 3D ScanObjectNN
dataset, our method performs up to $1.32\%$ better than the SOTA PEFT approach
with less than half of the tunable parameters of it. Our contributions can be
summarized as:

  * •

We first analyze existing PEFT works and summarize four factors related to
improving efficiency and performance: sharing strategy, larger tunable
parameters count, dynamic allocation, and block-specific design.

  * •

Based on the above factors, we design a novel general parameter-efficient
fine-tuning framework named Adapter-X. A Sharing Mixture of Adapters (SMoA) is
proposed to dynamically share adapters, which in turn increases parameters
each block can access. We further adopt the idea of block-specific design and
introduce different norm layer and generated prompt for each block, further
enhancing the performance.

  * •

Our proposed Adapter-X is the first to outperform full fine-tuning in both 2D
image and 3D point cloud modalities, even performing better than the SOTA PEFT
method tailored for point cloud modality with less tunable parameters.

##  2 Related Works

Adapter Series, as introduced in [14], are lightweight modules that enhance
Transformer layers with a structure comprising a linear down-projector, a
nonlinear activation, an up-projector, and a residual connection. Through the
continuous improvement of [29][11], the current mainstream Adapter usage
method[55][20][8] is to add a parallel original Adapter at the FFN position of
each block, and scale the output. Existing methods aim to improve the
adapter’s performance on specific tasks by introducing specialized designs for
individual blocks. For instance, approaches like [27] [1] [11] [16] either
activate various PEFT techniques, such as LoRA [15], prefix-tuning [21],
within distinct blocks or blend their outputs in varying ratios. Furthermore,
methods like [52] [53] utilize NAS to discover a range of PEFT configurations
that are superior for different tasks. Besides, some work focuses on more
compact designs for adapters. [26] [13] regard the weights of adapters as the
Kronecker product of two smaller matrices, one of which is shared among
adapters. [12] prunes the dense weights of adapters before fine-tuning. In the
domain of computer vision, for 2D image processing tasks, [2] employs a
scaling mechanism in conjunction with Adapter to adapt different visual tasks.
[52] leverages search methodologies to identify different configurations
across blocks for adaptation. [18] improves parameter efficiency of Adapters
with quantization from the perspective of precision redundancy. When it comes
to 3D point cloud, [49] extends the Prompt tuning with a DGCNN [30] to extract
instance-aware prompts for model fine-tuning instead of using static prompts.
DAPT [54] proposes Dynamic Adapter and integrates it with Prompt Tuning to
reduce the tunable parameters and achieve remarkable performance. While
various strategies are used to decrease trainable parameters, our method of
sharing adapters across blocks offers a distinct advantage on efficient
adaptation on various tasks. Additionally, it is easy to achieve integration
with other advanced adapter method.

Parameter Sharing, as a method to reduce model size and computational costs,
has been utilized in various studies. For instance, [4][38] [24][36] explores
strategies for sharing parameters between different layers of the Transformer
to reduce the number of trainable parameters and increase model calculation
speed. [10] [31] share a complete module of Transformer to reduce the
parameter count. Recent advancements have taken this a step further by
combining parameter sharing with PEFT methods. VERA [19] shares two frozen
random matrices across all layers, updating disentangled combination vectors
for each layer, while Tied-LoRA [37] enhances efficiency through weight tying
and selective training. PRoLoRA [42] introduces an intra-layer sharing
mechanism which greatly reduces the number of trainable parameters. Instead of
simply sharing the same fine-tuning parameters for different blocks to reduce
trainable parameters, our proposed method, Adapter-X, realizes fine-tuning
parameters allocation from block-general level with sharing mechanism which we
find crucial to Adapters.

Mixture of Experts(MoE) is an ensemble method, typically envisioned as a group
of specialized sub-modules, known as experts, each adept at handling distinct
types of input data. These experts are controlled by a router that dynamically
selects the most relevant expert based on the input. Inspired by the ideas of
MoE and its advanced variants MH-MoE [44] and X-MoE [3], we first redefine the
role of standard adapter as expert within our framework, and introduce a more
efficient and effective procedure of processing input to select a more
compatible adapter for each block.

##  3 Method

The overview of Adapter-X is depicted in Fig. 2. In this following, we first
give a brief review of Adapter and MoE in Sec. 3.1. We then present our
Sharing MultiHead-MoA(SMMoA) and block-specific design, especially Prompt
Generator(PG) in Sec. 3.2 and Sec. 3.3, respectively. Finally, we introduce
the optimization function of the framework briefly in Sec. 3.4.

Figure 2: The overview of the proposed Adapter-X. By comprehensively
considering parameter sharing, more tunable parameters, dynamic allocation,
and block-specific design, the model achieves efficient adaptation to
different tasks with minimal trainable parameters.

###  3.1 Preliminaries

Adapters are implemented by embedding compact, auxiliary layers into the
structure of Transformer architecture. Typically, an adapter layer consists of
a down-projection matrix $W_{\mathrm{down}}\in\mathbb{R}^{r\times d}$,
followed by a non-linear activation function $\sigma(\cdot)$, and an up-
projection matrix $W_{\mathrm{up}}\in\mathbb{R}^{d\times r}$. In this context,
_d_ represents the dimension of the hidden layer, and _r_ serves as the
bottleneck dimension, which is a hyperparameter used in configuring the
adapters. Denote _$x$_ as the input to the adapter, the computation within the
adapter module (with residual) can be summarized as follows:

$\operatorname{Adapter}(x)=W_{\text{up }}\sigma\left(W_{\text{down
}}x\right)+x$ (1)

Mixture-of-Experts (MoE) [46] is a promising approach to scale up the number
of parameters within the same computational bounds. Different from standard
transformer models, each MoE layer consists of $N$ independent feed-forward
networks $\\{\boldsymbol{E}_{i}\\}^{N}_{i=0}$ as the experts, along with a
gating function $\alpha\left(\cdot\right)$ to model a probability distribution
indicating the weights over these experts’ outputs. For the hidden
representation $\boldsymbol{h}\in\mathbb{R}^{d}$ of input token, the gate
value of routing $\boldsymbol{h}$ to expert $\boldsymbol{E}_{i}$ is denoted
as:

$\alpha\left(\boldsymbol{E}_{i}\right)=\exp\left(\boldsymbol{h}\cdot\boldsymbol{e}_{i}\right)/\sum_{j=0}^{N}\exp\left(\boldsymbol{h}\cdot\boldsymbol{e}_{j}\right),$
(2)

where $\boldsymbol{e}_{i}$ denotes the trainable embedding of
$\boldsymbol{E}_{i}$. Then, the corresponding $k$ experts, according to the
top-$k$ gated values, are activated and the output $\boldsymbol{O}$ of the MoE
layer is

$\boldsymbol{O}=\boldsymbol{h}+\sum_{i=0}^{N}\alpha\left(\boldsymbol{E}_{i}\right)\cdot\boldsymbol{E}_{i}\left(\boldsymbol{h}\right).$
(3)

###  3.2 Sharing MoA

Sharing Strategy. As shown in Fig. 2, Adapter-X adopts parameter sharing
across transformer blocks to consider the allocation of trainable parameters
from an inter-layer perspective. Instead of sharing the only same adapter
across all layers like [19], [37], Mixture of Adapters(MoA) is used to improve
model flexibility since the router can select the most suitable adapter expert
for each block. With this, we achieve the parameter allocation rather than
simply sharing. In order to allow routers and experts to learn the features of
all blocks for more accurate and comprehensive parameter allocation, Adapter-X
uses the same router and adapter expert repository across all blocks.
Formally, let $X^{1}=\\{x_{1}^{1},x_{2}^{1},\dots,x_{L}^{1}\\}$ represent the
input hidden representations of the first Transformer block. We define
parameter sharing within our model as $X^{i+1}=\mathrm{MoA}(X^{i})$, which
deviates from the approach used in existing Mixture of Experts (MoE) models,
where $X^{i+1}=\mathrm{MoE}^{i}(X^{i})$ is typical.

It is important to note a key distinction in our approach: while the MoE
layer, including the router, shares trainable parameters across the model, the
representations for the same token are not identical in every transformer
block. This means that the representation of a given token, denoted as
$h_{i}^{j}$, at layer $j$, may be directed to a different expert compared to
its representation, $h_{i}^{j}+1$, at layer $j+1$. Consequently, on blocks in
different locations, even for the same token, due to its different features
and block locations, the router is still very likely to assign different
adapter experts to it. Based on this principle, we implement token-level
allocation. And this approach is validated in Sec. 4.5.

Detailed design of MoA. To handle features in more detail, we first split the
input hidden states into multi-heads, then conduct a projection over the sub
hidden states to the expert embedding space with lower dimension. Formally, we
denote a sequence of inputs tokens by $\mathbf{X}\in\mathbb{R}^{l\times d}$,
where $l$ is the number of tokens and $d$ represents the length of token
dimension. First, every token in $X$ is split into $h$ sub-tokens along the
token dimensions, and these subtokens are arranged in parallel according to
the original token sequence, forming a new feature space
$\hat{\mathbf{X}}\in\mathbb{R}^{(l\times h)\times\frac{d}{h}}$ as:

$\displaystyle\hat{\mathbf{X}}$
$\displaystyle=\digamma_{\text{s}}({\mathbf{X}})$
$\displaystyle=\left[\underbrace{\overbrace{\mathbf{x}^{0}_{0},\ldots,\mathbf{x}^{0}_{h-1}}^{h},\ldots,\mathbf{x}^{i}_{j},\mathbf{x}^{i}_{j+1},\ldots,\mathbf{x}^{l}_{h-1}}_{l\times
h}\right],$ (4)

where function $\digamma_{\text{s}}$ denotes the token splitting operation:
$\mathbb{R}^{l\times d}\rightarrow\mathbb{R}^{\left(l\times
h\right)\times\frac{d}{h}}$, and each sub-token is presented as
$\mathbf{x}^{i}_{j}\in\mathbb{R}^{\frac{d}{h}}$, meaning it is the the
$j^{th}$ sub-token split from the $i^{th}$ token.

Then, we first parameterize the experts with lower-dimensional embeddings
$\mathbf{e}^{i}\in\mathbb{R}^{d_{e}}$ such that $d_{e}$ is much smaller than
the Transformer hidden size $d$, to be simple, $d_{e}$ is defined same as the
rank of adapters. And we normalize it with a frozen $L_{2}$ norm function to
get the result $\mathbf{e}^{in}_{norm}$. For input token $\hat{\mathbf{X}}$,
we use a linear projection $f^{proj}(\hat{\mathbf{X}})$ such that
${\mathbf{W}}\in\mathbb{R}^{d_{e}\times d}$. Thus, the routing scoring
function between the tokens and experts can be written as
$s_{i}=({\mathbf{W}}\hat{\mathbf{X}})\cdot\mathbf{e}^{in}_{norm}$. The gating
value of routing $\mathbf{h}$ to expert $f^{\text{FFN}}_{i}$ is denoted as:

$g\left(f^{\text{FFN}}_{i}\right)=\exp\left(s_{i}\right)/\sum_{j=0}^{N}\exp\left(s_{i}\right),$
(5)

With the multi-head split and lower-dimension projector, the dimension of
input can gradually decrease a lot, which fits the number of experts better
than the larger hidden size of Transformer. As a result, the combination of
these two design can better fits in with the low-rank nature of routing. The
algorithm flow is shown in Algorithm 1.

###  3.3 Block-Specific Design

As outlined in Sec. 1, our methodology leverages the strengths of its
predecessor. While embracing a block-general perspective, we do not entirely
discard block-specific design principles. In fact, we highlight the importance
of providing diverse input representations for each block, which serves to
improve the model’s adaptability to a variety of datasets. To achieve this
diversity, we have implemented two key design strategies.

Different norm layer for each block. We keep the norm layer before FFN
different across blocks. Given that layer normalization operates with just two
small vectors, introducing individual normalization layers adds only a minimal
increase in trainable parameters to our framework. By employing separate
LayerNorm for each block, rather than a shared one, we introduce this separate
normalization scheme that promotes a more varied representation of inputs.
This tailored approach allows each block to capture a broader range of
features.

Different generated prompt for each block. To enhance the distinctiveness of
inputs fed into each transformer block, we propose a Prompt Generator (PG)
block to leverage the output from the MultiHead-MoA layer, as illustrated in
Fig. 2. The process begins with reducing the number of input tokens through an
averaging mechanism. Subsequently, we introduce a scale and shift operation on
the averaged representation to generate a unique prompt for each block. This
prompt is then concatenated with output tokens. This simple yet effective
technique cleverly combines the strengths of MultiHead-MoA and prompt tuning
methods, thereby enriching the diversity of inputs across different
transformer blocks.

###  3.4 Objective Function

Following [9], given the sub-token set $\hat{\mathbf{X}}$ (depicted in Eq. 4)
and the frequency $t_{p}$ of how many sub-tokens are routed to the $p^{th}$
expert, we compute the load balancing loss $\mathcal{L}_{\text{balance}}$ via:

$\mathcal{L}_{\text{balance}}=\frac{N}{|\hat{\mathbf{X}}|}\sum_{p=1}^{N}\sum_{\mathbf{x}^{i}_{j}\in\hat{\mathbf{X}}}t_{p}\cdot
g\left(f^{\text{FFN}}_{p}\right),$ (6)

where $N$ denotes the number of experts, $|\hat{\mathbf{X}}|$ is the number of
sub-tokens contained in $\hat{\mathbf{X}}$. $g\left(f^{\text{FFN}}_{p}\right)$
is the gating function depicted in Eq. 5, denoting the gating value of routing
a certain sub-token $\mathbf{x}^{i}_{j}$ into the $p^{th}$ expert. The overall
training objective is to minimize:

$\mathcal{L}=\mathcal{L}_{\text{task}}+\alpha\mathcal{L}_{\text{balance}},$
(7)

where $\alpha$ is a coefficient for load balancing, which is usually set to
0.01 in this work.

##  4 Experiments

We evaluate the effectiveness of Adapter-X by conducting extensive visual
recognition experiments in both the 2d image and 3d point cloud domains. For
clarity, we define the adapter in Adapter-X as a modular component. This
definition facilitates seamless integration with other methods. For instance,
when integrated with Adaptformer, the resulting approach is called
Adaptformer-X.

We first describe our experimental settings in Sec. 4.1, covering pre-trained
backbones, baseline methods, downstream tasks and training details. We then
compare with baseline methods and provide a thorough analysis in Sec. 4.2 and
Sec. 4.3. In addition, we also conduct ablation studies to explore different
experimental configurations and explain what makes for the superiority of
Adapter-X in Sec 4.4 and visual experiments in Sec 4.5 for better
understanding.

###  4.1 Experimental Settings

Pre-trained backbone. For image modality, we adopt the plain Vision
Transformer (ViT) [7], _i.e._ ,ViT-Base (ViT-B/16) as our backbone model, and
directly use the ImageNet-21k [5] supervised pre-trained model. For point
cloud modality, we take widely used pre-trained models from PointMAE [28].
More details about pre-training approaches and datasets can be found in
Appendix.

Baseline methods. In image domain, we compare Adapter-X with (1)_Standard
Adapter_ [14]: adding the adapter after the FFN module in serialeze, and (2)
_AdaptFormer_ [2]: adding adapter after the FFN module in parallel. For point
cloud, we compare Adapter-X with _DAPT_ [54]: generating a unique scale for
each token to dynamically adjust features by considering the significance
score.

Downstream tasks. We evaluate our Adapter-X on both image and point cloud
recognition tasks to verify its effectiveness. The specific datasets leveraged
in this work are presented in the following.

$\bullet$ Image domain : VTAB-1K [50] contains 19 image classification tasks
from diverse fields, which can be categorized into three groups: Natural,
Specialized, and Structured. These tasks cover a large range of possible
domains where downstream tasks come, so the performance of different methods
on this benchmark largely reflects their ability to transfer learning. Each
dataset contains 800 samples for training and 200 for validation.

$\bullet$ Point Cloud domain : Tasks in point cloud classification can be
divided into two categories, namely Real-World Object Classification and
Synthetic Object Classification. The former is represented by ScanObjectNN
[39], which consists of $\sim$$15$K point clouds of indoor objects obtained by
scanning across $15$ categories. This benchmark is challenging since objects
contain points from cluttered backgrounds and occlusion caused by other
objects. As shown in Tab. 2, we conducted experiments on three variants of
ScanObjectNN (_i.e._ , OBJ_BG, OBJ_ONLY, and PB_T50_RS), each with increasing
complexity. As a representative of the synthesis task, ModelNet40 [45]
contains $12,311$ 3D CAD models across $40$ categories. As a synthetic
dataset, it consists of complete, uniform, and noise-free point clouds, which
is totally different from ScanObjectNN.

Implement Details. To ensure a fair comparison, we employed identical
experimental settings to the default fine-tuning method for each baseline.
This entails freezing the weights of the pre-trained point cloud backbone and
solely updating the newly inserted parameters during training. All experiments
are conducted on a single NVIDIA A100 GPU.

Table 1: Full results on the VTAB-1K benchmark. “Avg. Acc.” denotes the
average results over three groups. “Params” denotes the number of learnable
parameters excluding the final classification layer, as the number of
parameters in this final layer depend on the number of classes, which varies
between 2 and 397.

| Natural | Specialized | Structured
---|---|---|---
|  |  Params. (M)
---
|  Avg. Acc.
---
|  Cifar100
---
|  Caltech101
---
|  DTD
---
|  Flower102
---
|  Pets
---
|  SVHN
---
|  Sun397
---
|  Camelyon
---
|  EuroSAT
---
|  Resisc45
---
|  Retinopathy
---
|  Clevr-Count
---
|  Clevr-Dist
---
|  DMLab
---
|  KITTI-Dist
---
|  dSpr-Loc
---
|  dSpr-Ori
---
|  sNORB-Azim
---
|  sNORB-Ele
---
_Conventional Fine-Tuning_
Full |  85.84 |  68.9 |  68.9 |  87.7 |  64.3 |  97.2 |  86.9 |  87.4 |  38.8 |  79.7 |  95.7 |  84.2 |  73.9 |  56.3 |  58.6 |  41.7 |  65.5 |  57.5 |  46.7 |  25.7 |  29.1
Linear |  0.04 |  57.6 |  64.4 |  85.0 |  63.2 |  97.0 |  86.3 |  36.6 |  51.0 |  78.5 |  87.5 |  68.5 |  74.0 |  34.3 |  30.6 |  33.2 |  55.4 |  12.5 |  20.0 |  9.6 |  19.2
_Vision PEFT methods_
VPT-Deep [17] |  0.53 |  72.0 |  78.8 |  90.8 |  65.8 |  98.0 |  88.3 |  78.1 |  49.6 |  81.8 |  96.1 |  83.4 |  68.4 |  68.5 |  60.0 |  46.5 |  72.8 |  73.6 |  47.9 |  32.9 |  37.8
NOAH [52] |  0.36 |  75.5 |  69.6 |  92.7 |  70.2 |  99.1 |  90.4 |  86.1 |  53.7 |  84.4 |  95.4 |  83.9 |  75.8 |  82.8 |  68.9 |  49.9 |  81.7 |  81.8 |  48.3 |  32.8 |  44.2
LoRA[15] |  0.29 |  74.5 |  67.1 |  91.4 |  69.4 |  98.8 |  90.4 |  85.3 |  54.0 |  84.9 |  95.3 |  84.4 |  73.6 |  82.9 |  69.2 |  49.8 |  78.5 |  75.7 |  47.1 |  31.0 |  44.0
SSF [22] |  0.24 |  75.7 |  69.0 |  92.6 |  75.1 |  99.4 |  91.8 |  90.2 |  52.9 |  87.4 |  95.9 |  87.4 |  75.5 |  75.9 |  62.3 |  53.3 |  80.6 |  77.3 |  54.9 |  29.5 |  37.9
BitFit [48] |  0.10 |  65.2 |  72.8 |  87.0 |  59.2 |  97.5 |  85.3 |  59.9 |  51.4 |  78.7 |  91.6 |  72.9 |  69.8 |  61.5 |  55.6 |  32.4 |  55.9 |  66.6 |  40.0 |  15.7 |  25.1
Adapter[14] (Baseline)
$rank=8$ |  0.15 |  73.9 |  70.6 |  91.7 |  68.2 |  98.9 |  90.5 |  82.7 |  54.1 |  82.9 |  95.3 |  82.2 |  75.9 |  81.6 |  65.3 |  48.1 |  76.2 |  76.2 |  47.4 |  29.3 |  41.0
$rank=64$ |  1.19 |  74.2 |  68.5 |  92.3 |  67.8 |  98.8 |  90.2 |  85.1 |  53.2 |  84.2 |  95.9 |  83.9 |  75.3 |  80.9 |  63.7 |  49.1 |  80.2 |  75.9 |  48.9 |  31.5 |  36.1
Adapter-X (Ours)
$rank=64$ |  0.17 |  74.3 |  69.5 |  92.7 |  70.9 |  99.2 |  89.9 |  85.4 |  53.3 |  85.5 |  95.9 |  84.8 |  74.0 |  82.2 |  64.0 |  48.0 |  78.5 |  75.8 |  44.8 |  29.0 |  40.1
AdaptFormer[2] (Baseline)
$rank=64$ |  1.26 |  76.1 |  73.1 |  92.6 |  71.2 |  99.2 |  91.3 |  89.7 |  55.2 |  86.9 |  95.4 |  86.0 |  75.0 |  82.5 |  62.3 |  51.9 |  78.9 |  86.4 |  52.7 |  33.9 |  36.5
AdaptFormer-X (Ours)
$rank=64$ |  0.17 |  76.2 |  73.4 |  93.3 |  71.8 |  99.3 |  91.4 |  87.8 |  56.7 |  87.5 |  95.5 |  84.8 |  75.8 |  80.7 |  62.0 |  50.5 |  79.0 |  85.8 |  53.7 |  32.7 |  41.2

###  4.2 Experiment on Image

Tab. 1 shows the comparison of our method with full-tuning, linear
probing(_i.e._ , only training the classification head), some representative
vision PEFT methods and particular vision adapter methods. We conduct the
expertiments based on two baselines: NOAH and AdaptFormer. The Former Adapter
is serialized while the latter is parallel with scale.

Our comparative experiments on the NOAH framework demonstrate that when our
method’s hyperparameters are aligned with those of the baseline, specifically
the rank of the adapter, our approach achieves a reduction in the number of
parameters by a factor of 7.4, while sustaining marginally superior
performance. Furthermore, significant enhancements in performance are observed
when our method is compared to the baseline with an equivalent number of
parameters. The same phenomenon can still be observed when experimenting with
another widely used Adaptformer framework. This also shows that our method has
relatively good applicability to different adapters.

Table 2:  Classification on three variants of ScanObjectNN [39] and ModelNet40
[45] datasets, including the number of trainable parameters and overall
accuracy (OA). Point cloud rotation is used for ScanObjectNN and
scale&transform is used for ModelNet40 following previous works, and † means
the model is reproduced using the same fine-tuning configuration as [6]. We
report ScanObjectNN results without voting, ModelNet40 results without and
with voting, referred to (-/-).

Method |  Tunable params.(M) | ScanObjectNN | ModelNet40
---|---|---|---
OBJ_BG | OBJ_ONLY | PB_T50_RS | Points Num. | OA (%)
Supervised Learning Only
PointNet [32] | 3.5 | 73.3 | 79.2 | 68.0 | 1k | \- / 89.2
PointNet++ [33] | 1.5 | 82.3 | 84.3 | 77.9 | 1k | \- / 90.7
DGCNN [43] | 1.8 | 82.8 | 86.2 | 78.1 | 1k | \- / 92.9
PointNeXt [35] | 1.4 | - | - | 87.7 | 1k | \- / 94.0
PointMLP [25] | 13.2 | - | - | 85.4 | 1k | \- / 94.5
Self-Supervised Representation Learning (Full fine-tuning)
OcCo [41] | 22.1 | 84.85 | 85.54 | 78.79 | 1k | \- / 92.1
Point-BERT [47] | 22.1 | 87.43 | 88.12 | 83.07 | 1k | 92.7 / 93.2
MaskPoint [23] | 22.1 | 89.70 | 89.30 | 84.60 | 1k | \- / 93.8
Point-MAE [28] | 22.1 | 90.02 | 88.29 | 85.18 | 1k | 93.2 / 93.8
Point-M2AE [51] | 15.3 | 91.22 | 88.81 | 86.43 | 1k | 93.4 / 94.0
ACT [6] | 22.1 | 93.29 | 91.91 | 88.21 | 1k | \- / 93.7
Recon [34] | 43.6 | 95.18 | 93.12 | 89.73 | 1k | 93.6 / 93.9
Self-Supervised Representation Learning (Efficient fine-tuning)
Point-MAE [28]† | 22.1 (100%) | 91.74 | 90.88 | 88.31 | 1k |  93.2 / 93.8
\+ DAPT [54] | 1.1 (4.97%) |  92.08(+0.34) |  91.22(+0.34) |  87.13(-1.18) | 1k |  93.5(+0.3) / 94.0(+0.2)
\+ DAPT-X(ours) | 0.42 (1.88%) | 92.60(+0.96) | 92.43(+1.55) | 88.45(+0.14) | 1k | 93.5(+0.3) / 94.1(+0.3)

###  4.3 Experiment on Point Cloud

The experimental results of our Adapter-X on Point Cloud classification tasks
are shown in Tab. 2. Point-MAE [28] are adopted as our baseline. We take the
SOTA point cloud efficient fine-tuning method DAPT [54] into comparison and
report results on ScanObjectNN and ModelNet40. Experimental results presented
in the last three rows of the table indicates that our Adapter-X meets or
exceeds the performance of full fine-tuning and DAPT in all cases with less
tunable parameters. Specially, Adapter-X achieves accuracy increase of
$1.55\%$ over Point-MAE on OBJ_ONLY and $1.32\%$ over DAPT on PB_T50_RS with
only less than half of tunable parameters of DAPT. The results demonstrates
the effectiveness and generalization ability of the mixture of adapters and
block-specific operations in our Adapter-X.

###  4.4 Ablation Studies

Figure 3: Partial t-SNE visualization results of our ablation study on the 2D
VTAB-1K dataset. Please refer to the appendix for more results.

We conducted ablation studies on Adapter-X to validate our core design
assumptions stated in Sec. 1, with 2D experiments on VATB and 3D on ScanObjNN-
PB_T50_RS datasets.

The 2D ablation reveals that using only the Sharing Module of Adapters (SMoA)
in Adapter-X reduces parameters, linked to the specific second norm layer and
Prompt Generator in each block, which is little. But removing these leads to
decreased adaptability. However, eliminating sharing increases parameters and
improves performance, consistent with the reasons outlined in Sec. 1. Compared
to methods relying heavily on block-specific parameters, our approach has over
ten times fewer parameters with only little performance drop, indicating a
balanced efficiency and performance.

In the domain of 3D PEFT, the performance improvement brought by the larger
number of parameters does not always translate to benefits due to the
susceptibility of point cloud datasets to overfitting, as shown in Tab.3(b).
Although more parameters enhance performance, considering all four advantages,
Adapter-X achieves significant performance with fewer parameters.

Table 3: Effect of different perspective, namely block-general and block-
specific. Ablation experiments are conducted using the VTAB dataset on the 2d
image modality and the xxx dataset on the 3d point cloud modality.

(a) 2d test accuracy

| Different
---
design
| Acc.
---
| Total
---
Params.(M)
Both | 76.2 | 0.17
Only Sharing MoA | 74.9 | 0.15
Only Block-specific | 76.4 | 1.8

(b) 3d test accuracy

| Different
---
design
| Acc.
---
| Total
---
Params.
Both | 88.45 | 0.56
Only Sharing MoA | 87.82 | 0.46
Only Block-specific | 87.99 | 1.40

###  4.5 Visualization

To further illustrate how our method achieves surprising results, we conduct a
two-part visualization experiment. Fig. 3 is conducted on Pets validation set
from VTAB-1K dataset, and Fig.4 is conducted on the CIFAR100 dataset of
VTAB-1K.

In order to better evaluate the necessity of model design, we further conduct
t-SNE [40] visualizations on the complete Adapter-X and only retain part of
the design based on the ablation experiments. The features are harvested from
the Pets validation subset of the VTAB-1K dataset, processed through the ViT-
Base backbone. Fig. 3 shows that both the SMoA alone and the block-specific
modules alone result in somewhat blended feature outputs. Compared with the
above two methods, the full strategy performs well in projecting features.

Secondly, as shown in Fig. 4, the left picture represents the ratio of the
number of tokens accepted by four experts at different block positions for the
whole dataset. The right figure refers to the probability that a randomly
selected token is assigned to four different experts in different blocks
during its forward progress. Different colors are used in both figures to
represent the strategies of different experts. The picture on the left shows
that at the dataset level, adapter experts are selected with roughly
equivalent probability for every block, thereby implicitly increasing the
parameter count that can be trained per block. The right figure shows that the
identical token will be allocated to different experts in different blocks for
processing, substantiating that Adapter-X can indeed achieve token-level
dynamics.

Figure 4: The distribution of token allocation and routing score of 4 shared
experts in different blocks on the CIFAR100 dataset of VTAB-1K.

##  5 Conclusion

Our analysis of key in PEFT has highlighted the importance of parameter
sharing for redundancy reduction and the role of tunable parameters, dynamic
allocation, and block-specific design in enhancing performance. Inspired by
these findings, we have introduced Adapter-X, featuring a Sharing Mixture of
Adapters (Sharing MoA) module that achieves token-level dynamic allocation,
implicitly increases tunable parameters per block, and facilitates inter-block
sharing. The incorporation of block-specific designs like the Prompt Generator
(PG) further refines the model’s adaptability. Extensive experiments on 2D
images and 3D point clouds have shown that Adapter-X is a significant
advancement, being the first to exceed the performance of full fine-tuning
with a substantially lower parameter count.

## References

  * [1] J. Chen, A. Zhang, X. Shi, M. Li, A. Smola, and D. Yang.  Parameter-efficient fine-tuning design spaces.  arXiv preprint arXiv:2301.01821, 2023.
  * [2] S. Chen, C. Ge, Z. Tong, J. Wang, Y. Song, J. Wang, and P. Luo.  Adaptformer: Adapting vision transformers for scalable visual recognition.  Advances in Neural Information Processing Systems, 35:16664–16678, 2022.
  * [3] Z. Chi, L. Dong, S. Huang, D. Dai, S. Ma, B. Patra, S. Singhal, P. Bajaj, X. Song, X.-L. Mao, et al.  On the representation collapse of sparse mixture of experts.  Advances in Neural Information Processing Systems, 35:34600–34613, 2022.
  * [4] M. Dehghani, S. Gouws, O. Vinyals, J. Uszkoreit, and Ł. Kaiser.  Universal transformers.  arXiv preprint arXiv:1807.03819, 2018.
  * [5] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2009.
  * [6] R. Dong, Z. Qi, L. Zhang, J. Zhang, J. Sun, Z. Ge, L. Yi, and K. Ma.  Autoencoders as cross-modal teachers: Can pretrained 2d image transformers help 3d representation learning?  In ICLR, 2022.
  * [7] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [8] A. Edalati, M. Tahaei, I. Kobyzev, V. P. Nia, J. J. Clark, and M. Rezagholizadeh.  Krona: Parameter efficient tuning with kronecker adapter.  arXiv preprint arXiv:2212.10650, 2022.
  * [9] W. Fedus, B. Zoph, and N. Shazeer.  Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.  The Journal of Machine Learning Research, 23(1):5232–5270, 2022.
  * [10] T. Ge, S.-Q. Chen, and F. Wei.  Edgeformer: A parameter-efficient transformer for on-device seq2seq generation.  arXiv preprint arXiv:2202.07959, 2022.
  * [11] J. He, C. Zhou, X. Ma, T. Berg-Kirkpatrick, and G. Neubig.  Towards a unified view of parameter-efficient transfer learning.  In Proc. of Intl. Conf. on Learning Representations, 2021.
  * [12] S. He, L. Ding, D. Dong, J. Zhang, and D. Tao.  Sparseadapter: An easy approach for improving the parameter-efficiency of adapters.  In Findings of EMNLP, 2022.
  * [13] X. He, C. Li, P. Zhang, J. Yang, and X. E. Wang.  Parameter-efficient model adaptation for vision transformers.  In Proceedings of AAAI, 2023.
  * [14] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly.  Parameter-efficient transfer learning for nlp.  In International conference on machine learning, pages 2790–2799. PMLR, 2019.
  * [15] E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen.  LoRA: Low-rank adaptation of large language models.  In Proceedings of ICLR, 2022.
  * [16] Z. Hu, L. Wang, Y. Lan, W. Xu, E.-P. Lim, L. Bing, X. Xu, S. Poria, and R. K.-W. Lee.  Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models.  arXiv preprint arXiv:2304.01933, 2023.
  * [17] M. Jia, L. Tang, B. Chen, C. Cardie, S. J. Belongie, B. Hariharan, and S. Lim.  Visual prompt tuning.  In Proceedings of ECCV, 2022.
  * [18] S. Jie, H. Wang, and Z.-H. Deng.  Revisiting the parameter efficiency of adapters from the perspective of precision redundancy.  In Porc. of IEEE Intl. Conf. on Computer Vision, 2023.
  * [19] D. J. Kopiczko, T. Blankevoort, and Y. M. Asano.  Vera: Vector-based random matrix adaptation.  arXiv preprint arXiv:2310.11454, 2023.
  * [20] T. Lei, J. Bai, S. Brahma, J. Ainslie, K. Lee, Y. Zhou, N. Du, V. Zhao, Y. Wu, B. Li, et al.  Conditional adapters: Parameter-efficient transfer learning with fast inference.  Advances in Neural Information Processing Systems, 36, 2024.
  * [21] X. L. Li and P. Liang.  Prefix-tuning: Optimizing continuous prompts for generation.  In Annual Meeting of the Association for Computational Linguistics, 2021.
  * [22] D. Lian, D. Zhou, J. Feng, and X. Wang.  Scaling & shifting your features: A new baseline for efficient model tuning.  In Proceedings of NeurIPS, 2022.
  * [23] H. Liu, M. Cai, and Y. J. Lee.  Masked discrimination for self-supervised learning on point clouds.  In ECCV, 2022.
  * [24] Q. Lou, T. Hua, Y.-C. Hsu, Y. Shen, and H. Jin.  Dictformer: Tiny transformer with shared dictionary.  In International Conference on Learning Representations, 2021.
  * [25] X. Ma, C. Qin, H. You, H. Ran, and Y. Fu.  Rethinking network design and local geometry in point cloud: A simple residual mlp framework.  In ICLR, 2022.
  * [26] R. K. Mahabadi, J. Henderson, and S. Ruder.  Compacter: Efficient low-rank hypercomplex adapter layers.  In Proceedings of NeurIPS, 2021.
  * [27] Y. Mao, L. Mathias, R. Hou, A. Almahairi, H. Ma, J. Han, S. Yih, and M. Khabsa.  Unipelt: A unified framework for parameter-efficient language model tuning.  In Annual Meeting of the Association for Computational Linguistics, 2022.
  * [28] Y. Pang, W. Wang, F. E. Tay, W. Liu, Y. Tian, and L. Yuan.  Masked autoencoders for point cloud self-supervised learning.  In ECCV, 2022.
  * [29] J. Pfeiffer, A. Kamath, A. Rücklé, K. Cho, and I. Gurevych.  Adapterfusion: Non-destructive task composition for transfer learning.  arXiv preprint arXiv:2005.00247, 2020.
  * [30] A. V. Phan, M. Le Nguyen, Y. L. H. Nguyen, and L. T. Bui.  Dgcnn: A convolutional neural network over large-scale labeled graphs.  Neural Networks, 2018.
  * [31] T. P. Pires, A. V. Lopes, Y. Assogba, and H. Setiawan.  One wide feedforward is all you need.  arXiv preprint arXiv:2309.01826, 2023.
  * [32] C. R. Qi, H. Su, K. Mo, and L. J. Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652–660, 2017.
  * [33] C. R. Qi, L. Yi, H. Su, and L. J. Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  In NeurIPS, 2017.
  * [34] Z. Qi, R. Dong, G. Fan, Z. Ge, X. Zhang, K. Ma, and L. Yi.  Contrast with reconstruct: Contrastive 3d representation learning guided by generative pretraining.  In ICML, 2023.
  * [35] G. Qian, Y. Li, H. Peng, J. Mai, H. Hammoud, M. Elhoseiny, and B. Ghanem.  Pointnext: Revisiting pointnet++ with improved training and scaling strategies.  In NeurIPS, 2022.
  * [36] M. Reid, E. Marrese-Taylor, and Y. Matsuo.  Subformer: Exploring weight sharing for parameter efficiency in generative transformers.  arXiv preprint arXiv:2101.00234, 2021.
  * [37] A. Renduchintala, T. Konuk, and O. Kuchaiev.  Tied-lora: Enhacing parameter efficiency of lora with weight tying.  arXiv preprint arXiv:2311.09578, 2023.
  * [38] S. Takase and S. Kiyono.  Lessons on parameter sharing across layers in transformers.  In N. Sadat Moosavi, I. Gurevych, Y. Hou, G. Kim, Y. J. Kim, T. Schuster, and A. Agrawal, editors, Proceedings of The Fourth Workshop on Simple and Efficient Natural Language Processing (SustaiNLP), pages 78–90, Toronto, Canada (Hybrid), July 2023. Association for Computational Linguistics.
  * [39] M. A. Uy, Q.-H. Pham, B.-S. Hua, T. Nguyen, and S.-K. Yeung.  Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data.  In Porc. of IEEE Intl. Conf. on Computer Vision, 2019.
  * [40] L. Van der Maaten and G. Hinton.  Visualizing data using t-sne.  Journal of machine learning research, 9(11), 2008.
  * [41] H. Wang, Q. Liu, X. Yue, J. Lasenby, and M. J. Kusner.  Unsupervised point cloud pre-training via occlusion completion.  In ICCV, 2021.
  * [42] S. Wang, B. Xue, J. Ye, J. Jiang, L. Chen, L. Kong, and C. Wu.  Prolora: Partial rotation empowers more parameter-efficient lora.  arXiv preprint arXiv:2402.16902, 2024.
  * [43] Y. Wang, Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, and J. M. Solomon.  Dynamic graph cnn for learning on point clouds.  ACM Trans. Gr., 2019.
  * [44] X. Wu, S. Huang, W. Wang, and F. Wei.  Multi-head mixture-of-experts.  arXiv preprint arXiv:2404.15045, 2024.
  * [45] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao.  3d shapenets: A deep representation for volumetric shapes.  In Proc. of IEEE Intl. Conf. on Computer Vision and Pattern Recognition, 2015.
  * [46] Y. Xie, S. Huang, T. Chen, and F. Wei.  Moec: Mixture of expert clusters.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 13807–13815, 2023.
  * [47] X. Yu, L. Tang, Y. Rao, T. Huang, J. Zhou, and J. Lu.  Point-bert: Pre-training 3d point cloud transformers with masked point modeling.  In CVPR, 2022.
  * [48] E. B. Zaken, Y. Goldberg, and S. Ravfogel.  Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models.  In Proceedings of ACL, 2022.
  * [49] Y. Zha, J. Wang, T. Dai, B. Chen, Z. Wang, and S.-T. Xia.  Instance-aware dynamic prompt tuning for pre-trained point cloud models.  In Porc. of IEEE Intl. Conf. on Computer Vision, 2023.
  * [50] X. Zhai, J. Puigcerver, A. Kolesnikov, P. Ruyssen, C. Riquelme, M. Lucic, J. Djolonga, A. S. Pinto, M. Neumann, A. Dosovitskiy, L. Beyer, O. Bachem, M. Tschannen, M. Michalski, O. Bousquet, S. Gelly, and N. Houlsby.  The visual task adaptation benchmark.  arXiv preprint, arXiv:1910.04867, 2019.
  * [51] R. Zhang, Z. Guo, P. Gao, R. Fang, B. Zhao, D. Wang, Y. Qiao, and H. Li.  Point-m2ae: multi-scale masked autoencoders for hierarchical point cloud pre-training.  In NeurIPS, 2022.
  * [52] Y. Zhang, K. Zhou, and Z. Liu.  Neural prompt search.  arXiv preprint, arXiv:2206.04673, 2022.
  * [53] H. Zhou, X. Wan, I. Vulić, and A. Korhonen.  Autopeft: Automatic configuration search for parameter-efficient fine-tuning.  Transactions of the Association for Computational Linguistics, 12:525–542, 2024.
  * [54] X. Zhou, D. Liang, W. Xu, X. Zhu, Y. Xu, Z. Zou, and X. Bai.  Dynamic adapter meets prompt tuning: Parameter-efficient transfer learning for point cloud analysis.  arXiv preprint arXiv:2403.01439, 2024.
  * [55] Y. Zhu, J. Feng, C. Zhao, M. Wang, and L. Li.  Counter-interference adapter for multilingual machine translation.  In Proc. of Conf. on Empirical Methods in Natural Language Processing, 2021.

Appendix for Adapter-X

##  Appendix A Pytorch-like code of Sharing Multi-head-MoA

    import torch
    import torch.nn as nn
    class Sharing_MoA(nn.Module):
        def __init__(self, n_experts, rank, expert, d_model):
            super().__init__()
            self.experts = nn.ModuleList([expert() for _ in range(n_experts)])
            self.switch = nn.Parameter(torch.randn(n_experts, rank))
            self.switch_reduction = nn.Linear(d_model // self.num_heads,
                                        rank, bias=False)
        def forward(self, x):
            b, l, d = x.shape()
            # Split input to multi-head
            x = x.view(-1, d_model // self.num_heads).contiguous()
            # Dimension Reduction
            input = self.switch_reduction(x)
            # Compute routing probabilities
            route_prob = torch.softmax(self.switch @ input_reduced.T, dim=0)
            # Route input through experts based on probabilities
            expert_output = torch.stack([expert(x) for expert in self.experts]).T
            # Same as Switch Transformer
            final_output = torch.bmm(route_prob.unsqueeze(1), expert_output)
                                                                    .squeeze(1)
            final_output = final_output.reshape(b, l, d)
            return final_output
    class VisionTransformer(nn.Module):
        def __init__(self, num_experts, expert, d_model, depth):
            super().__init__()
            self.adapter = Sharing_MoA(n_experts=num_experts,
            rank=d_model,
            expert=expert,
            d_model=d_model)
            self.layers = nn.ModuleList([Block(adapter=self.adapter, d_model=d_model)
                                        for _ in range(depth)])
        def forward(self, x):
            for layer in self.layers:
                x = layer(x)
            return x

Algorithm 1 PyTorch-like code of Sharing MultiHead-MoA.

##  Appendix B Full 2d ablation results on VTAB-1K

Table 4: Full results on the VTAB-1K benchmark.

| Natural | Specialized | Structured
---|---|---|---
|  |  Params. (M)
---
|  Avg. Acc.
---
|  Cifar100
---
|  Caltech101
---
|  DTD
---
|  Flower102
---
|  Pets
---
|  SVHN
---
|  Sun397
---
|  Camelyon
---
|  EuroSAT
---
|  Resisc45
---
|  Retinopathy
---
|  Clevr-Count
---
|  Clevr-Dist
---
|  DMLab
---
|  KITTI-Dist
---
|  dSpr-Loc
---
|  dSpr-Ori
---
|  sNORB-Azim
---
|  sNORB-Ele
---
Both |  0.17 |  76.2 |  73.4 |  93.3 |  71.8 |  99.3 |  91.4 |  87.8 |  56.7 |  87.5 |  95.5 |  84.8 |  75.8 |  80.7 |  62.0 |  50.5 |  79.0 |  85.8 |  53.7 |  32.7 |  41.2
Only |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
Sharing-MoA |  1.15 |  74.9 |  69.3 |  93.5 |  70.1 |  99.2 |  91.3 |  87.1 |  55.4 |  86.9 |  95.2 |  84.5 |  75.2 |  76.8 |  61.6 |  47.1 |  77.9 |  82.2 |  50.1 |  32.4 |  38.4
Only |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
Block-specific |  1.8 |  76.4 |  73.6 |  93.1 |  71.8 |  99.3 |  91.0 |  90.1 |  56.4 |  88.2 |  95.6 |  86.4 |  73.7 |  79.8 |  62.6 |  50.2 |  81.0 |  84.4 |  53.5 |  38.1 |  38.9

##  Appendix C Visualization

Figure 5: More t-SNE visualization results of our ablation study on the 2D
VTAB-1K dataset.

##  Appendix D Experimental Setting

We follow the implement of the original codebase:
NOAH111https://github.com/ZhangYuanhan-AI/NOAH, Bi-
adapter222https://github.com/JieShibo/PETL-ViT/tree/main/binary_adapter and
DAPT333https://github.com/LMD0311/DAPT. We draw inspiration from Switch
Transformer444https://nn.labml.ai/transformers/switch/index.html to implement
MoA. The hyperparameters involved in the Sharing MoA module are: num of heads
is 3, and num of experts is 4. Reduction dimension keeps the same as the rank
of each adapter expert.

##  Appendix E Limitations

In this work, we only focus on the image and point cloud modalities of visual
modalities, because adapters have made more progress in the visual field due
to their flexibility. In the future, we will explore how to apply our
Adapter-X method in NLP and even generation tasks.