# Polyhistor: Parameter-Efficient Multi-Task Adaptation for Dense Vision Tasks

Yen-Cheng Liu
Georgia Tech
<EMAIL_ADDRESS>
&Chih-Yao Ma
Meta
<EMAIL_ADDRESS>
&Junjiao Tian
Georgia Tech
<EMAIL_ADDRESS>
&Zijian He
Meta
<EMAIL_ADDRESS>
&Zsolt Kira
Georgia Tech
<EMAIL_ADDRESS>

###### Abstract

Adapting large-scale pretrained models to various downstream tasks via fine-
tuning is a standard method in machine learning. Recently, parameter-efficient
fine-tuning methods show promise in adapting a pretrained model to different
tasks while training only a few parameters. Despite their success, most
existing methods are proposed in Natural Language Processing tasks with
language Transformers, and adaptation to Computer Vision tasks with Vision
Transformers remains under-explored, especially for dense vision tasks.
Further, in multi-task settings, individually fine-tuning and storing separate
models for different tasks is inefficient. In this work, we provide an
extensive multi-task parameter-efficient benchmark and examine existing
parameter-efficient fine-tuning NLP methods for vision tasks. Our results on
four different dense vision tasks showed that existing methods cannot be
efficiently integrated due to the hierarchical nature of the Hierarchical
Vision Transformers. To overcome this issue, we propose Polyhistor and
Polyhistor-Lite, consisting of Decomposed HyperNetworks and Layer-wise Scaling
Kernels, to share information across different tasks with a few trainable
parameters. This leads to favorable performance improvements against existing
parameter-efficient methods while using fewer trainable parameters.
Specifically, Polyhistor achieves competitive accuracy compared to the state-
of-the-art while only use $\sim 10\%$ of their trainable parameters.
Furthermore, our methods show larger performance gains when large networks and
more pretraining data are used.

##  1 Introduction

Foundation models trained with large-scale datasets have shown the success of
adapting to a variety of downstream NLP and vision tasks [1]. As the state-of-
the-art foundation models grow to billion or even trillion parameter models
[2, 3, 4, 5, 6], individually fine-tuning all parameters of the model wastes
significant computational resources. Further, for multi-task models, both
fine-tuning and storing separate models for multiple tasks become infeasible
on devices with low computation resources. ††Polyhistor: someone gifted or
learned in multiple disciplines.

To alleviate this issue, several works [7, 8, 9] have proposed parameter-
efficient fine-tuning methods to derive a better trade-off between trainable
parameters and accuracy on downstream tasks. By only training a small amount
of parameters, these existing methods can substantially narrow the accuracy
gap compared to the baseline that fine-tunes all parameters. However, these
existing approaches mainly focus on NLP tasks [10, 11, 12] or single-task
adaptation on image classification [9], and their applicability to more
complicated vision tasks is unclear. On the other hand, the single-task
adaptation methods [7, 10, 11, 12, 8] still need to learn and store task-wise
parameters, and the number of trainable parameters increase with respect to
the number of tasks.

Therefore, in this paper, we first conduct a thorough study on how the
existing successful parameter-efficient methods on NLP tasks perform on vision
tasks, particularly on more challenging dense vision tasks (e.g., semantic
segmentation, normals estimation). Second, based on our findings, we then
design a novel parameter-efficient method for adaptation to multiple dense
vision tasks. Our method leverage shared modules across tasks and encourage
the model to use shared information in a more parameter-efficient manner.

To start with, we first evaluate the existing parameter-efficient methods in
NLP on dense vision problems. We chose to apply these methods to hierarchical
vision transformers (HVTs) considering their state-of-the-art results on many
per-pixel vision tasks [13, 14]. Through our extensive studies, we find two
limitations in these works. First, adapter-based methods [11, 15], which have
shown strong performance on NLP parameter-efficient adaptation benchmarks,
cannot be efficiently integrated with HVTs. This is because the parameter
usage of adapters in later transformer blocks grows quadratically with respect
to the layer scale (see Fig. 1c). Second, the state-of-the-art multi-task
parameter-efficient method [16] applies a hyper-network to produce the weights
of adapters and shares information across different NLP tasks, while we find
it inherently requires a large number of trainable parameters in the hyper-
network (see Sec. 4.1 for further discussion).

To address the above limitations, we propose Polyhistor-Lite, which consist of
two main components, Decomposed Lite-HyperNetworks and Layer-wise Scaling
Kernels. These two methods reduce the trainable parameters in two aspects
respectively, including parameter reduction for hyper-networks in multi-
tasking architecture and parameter reduction for adapters used in HVTs.

Specifically, to reduce the parameter usage of the multi-task architecture, we
decompose a hyper-network into a pair of separate hyper-networks. Unlike the
existing approach, where a relatively large hyper-network is used to produce
long vector that can be reshaped into the weight matrix of adapter, our
decomposed hyper-networks individually produce two low-rank matrices that are
multiplied to construct the adapter weights. As a result, we can rely on this
low-rank approximation to reduce the parameter usage in the hyper-network yet
maintain its performance on downstream tasks. In addition, to enable the
hypernetworks shared across layers in HVTs, we factorize an adapter weight
matrix into two kernels, including Template Kernels and Scaling Kernels. These
two kernels are multiplied via Kronecker Product to fit in with different
sizes of adapters, and this is achieved by controlling the sizes of Scaling
Kernels based on the scaling of the layer/adapter (and using the same size of
Template Kernels across layers). In this way, the parameters of adapter
weights can be effectively reduced with a minimal sacrifice on the accuracy of
downstream tasks.

To benchmark the problem, we construct a unified framework with the same
implementation details and provide a comprehensive and fair comparison between
existing parameter-efficient adaptation works in NLP on our multi-tasking
dense vision problems. We also demonstrate that, with the integration of our
proposed Decomposed HyperNetworks and Layer-wise Scaling Kernels, we can
achieve a much better trade-off between trainable parameters and accuracies
compared to the existing methods. Specifically, most of existing methods
struggled to match the performance of the simple baseline, which individually
fine-tunes the entire network for each task, while our method achieves better
results than the simple baseline while only training less than $10\%$ of the
parameters in a model. Compared with the state-of-the-art multi-tasking
parameter-efficient adaptation method, Hyperformer [16], our method achieves
competitive performance improvement with $\sim 90\%$ reduction in the
trainable parameters of their method. Interestingly, we also observed that our
proposed method brings high-performance improvement when applied to the
network pre-trained on the larger dataset (ImageNet-22k). We will publicly
release our code to facilitate future research.

To sum up, we list our contributions as follows:

  * •

To the best of our knowledge, we are the first to address parameter-efficient
multi-task adaptation for vision tasks. We develop a unified framework to
benchmark several parameter-efficient fine-tuning NLP methods on dense vision
tasks.

  * •

We propose a novel method — Polyhistor-Lite that achieves significant
performance improvement with very low training parameters compared to existing
methods.

  * •

We observe that our method can bring further performance improvements when
applied to models with larger pre-trained dataset or with larger backbones.

##  2 Related Works

Parameter-efficient Learning aims to adapt a pre-trained model to a new task
by only training a small number of parameters. The most straightforward method
is to freeze the pre-trained encoder and only fine-tune the last layer, while,
in terms of the accuracy of downstream tasks, it is still far from full fine-
tuning. Thus, to achieve a better trade-off between accuracy and the number of
tunable parameters, several works [7, 10, 11, 12, 16, 9, 8] have proposed more
parameter-efficient methods, and we summarize these works in the following
paragraphs.

Single-Task Parameter-efficient Adaptation. Several works build upon the
Adapter [11], which is a bottleneck-like module that is placed across the
architecture and trained while the rest of the original model is frozen. By
changing the dimension of hidden vectors, one can easily control the trade-off
between trainable parameters and accuracy. For example, Houlsby et al. [11]
proposes to apply two adapter modules placed after the attention layers and
the MLP layers respectively, while Pfeiffer et al. [15] only use adapters
after MLP layers and show more parameter-efficiency. Furthermore, PHM-Layer
[12] learns two types of matrices, one "slow" matrix shared across layers and
the other "fast" matrix learned individually in different layers, to produce
the adapter weight via Kronecker Product [17]. Compacter [12] further reduces
the parameters by decomposing the slow matrix into two rank-one vectors.
Different from their goal of sharing slow matrix across layers, we apply
Kronecker Product to efficiently scale up adapters to different layer scales.

In addition, there are other parameter-efficient learning works. BitFit [7]
shows simply tuning biases in all layers improves against the linear probing.
Some other works fine-tune learnable vectors, such as learnable vectors in
input word embeddings [18] and learnable vectors integrated with keys/values
in each layer of transformers [19]. LoRA [10] produces two low-rank matrices,
which are multiplied and served as a residual of attention weight matrices.
While the above methods show favorable results with using fewer trainable
parameters, the goal of these works is single-task adaptation.

Multi-Task Parameter-efficient Adaptation. When multiple tasks are learned
jointly, one can share some homogeneous information across different tasks and
save the parameter usage by removing the duplicated learned features. To this
end, Hyperformer [16] introduces a hyper-network, which takes as input task
embeddings and produces the weights of adapters in different tasks. Since only
the parameters in the hyper-network need to be trained, the number of
trainable parameters in task-wise components can thus be reduced in the multi-
tasking setup. On the other hand, Sung et al. [20] shows that simply adding a
single adapter on language transformer and sharing the adapter across tasks
can achieve promising results in vision-and-language cross-modal tasks (e.g.,
image captioning).

Parameter-efficient Adaptation for Vision. Despite the promising results, most
parameter-efficient learning methods are evaluated on language transformers
and NLP benchmarks, and parameter-efficient learning on Vision Transformer [9]
is still an under-explored topic. A recent work, Visual Prompt Tuning (VPT)
[9], initiates the study of parameter-efficient learning on Vision
Transformers, and it follows the idea of prompt tuning in language tasks and
prepends and fine-tunes some extra learnable vectors in the input space of
pre-trained Vision Transformers. VPT focuses on single-task adaptation, while
our work focuses on multi-task adaptation.

To fairly compare different parameter-efficient learning methods, He et al.
[21] present an empirical study and re-evaluate parameter-efficient learning
methods (BitFit, Adapters, Prefix Tuning, and LoRA) under the same experiment
configurations for NLP tasks. Inspired by their work, we implement the
aforementioned parameter-efficient NLP methods (and include more latest works
[12, 16, 9]) on our dense vision tasks, conduct comparative experiments, and
fairly compare these methods.

##  3 Background

(a)(b)(c) Figure 1:  Illustration of (a) Hierarchical Vision Transformer and
(b) Adapter. (c) When applying adapters in a Hierarchical Vision Transformer,
the number of parameters grows quadratically with the respect to the block
scale. Note that $C$ indicates the dimension of adapter input vectors, $n$ is
the bottleneck size of adapters, and $d$ represents the input size of
adapters.

Hierarchical Vision Transformers. The Vision Transformer [22] is based on
transformer architectures [22] and operates on a set of patch tokens obtained
from an image. As a variant of Vision Transformer, the Hierarchical Vision
Transformer [13, 14, 23, 24, 25, 26] produces multi-scale feature
representations, and its hierarchical structure extracts fine-grained
information and better handles images with scale and size variation. These
properties contribute to the promising results in several per-pixel vision
tasks, including semantic segmentation [13, 14, 26], depth estimation [27],
and saliency detection [28]. As shown in Fig. 1a, a Hierarchy Vision
Transformer (HVT) consists of several transformer layers, and each transformer
layer is mainly composed of an attention layer and an MLP layer. Different
from other transformers (e.g., ViT [22]), a distinct characteristic of HVTs is
its pyramid-like feature maps generated from different transformer blocks as
shown in Fig. 1a.

Adapters. Several parameter-efficient adaptation works [11, 15, 12, 21] build
upon Adapter [11], which is a bottleneck-like module placed in transformer
layers as shown in Fig. 1b. These layers are learnable parameters, while the
rest of the model is frozen during fine-tuning. The Adapter
$\bm{f_{a}}(\cdot)$ consists of a down-projection layer
$\bm{W_{down}}\in\mathbb{R}^{d\times n}$, a non-linearity function
$\delta(\cdot)$, a up-projection layer $\bm{W_{up}}\in\mathbb{R}^{n\times d}$,
and a skip connection from the input of the adapter
$\bm{h_{in}}\in\mathbb{R}^{d}$.

$\displaystyle\bm{h_{out}}=\bm{f_{a}}(\bm{h_{in}};\bm{W})=\delta(\bm{h_{in}}\bm{W_{down}})\bm{W_{up}}+\bm{h_{in}},$
(1)

where $h_{out}\in\mathbb{R}^{d}$ is the output of the adapter and
$\bm{W}=[\bm{W_{down}};\bm{W^{\intercal}_{up}}]\in\mathbb{R}^{d\times 2n}$
represents all learnable parameters in the adapter.

##  4 Method

Problem Setting. Given a Hierarchical Vision Transformer pre-trained on large-
scale image datasets (e.g., ImageNet [29]), our goal is to train a small
number of parameters and adapt the model to the multi-tasking setting, where
training data of $\bm{N}$ tasks are obtained during the training stage.
Following the existing works in NLP, the criteria of parameter-efficient
multi-tasking learning includes the accuracy of downstream tasks and the
numbers of training parameters.

Method Overview. We aim to improve the parameter efficiency in two aspects:
(1) to efficiently share homogeneous information across tasks via lightweight
hyper-networks (Section 4.1) and (2) to efficiently scale up adapter weights
in different transformer blocks of Hierarchical Vision Transformers (Section
4.2). These two components are combined to improve the trade-off between
accuracy and training parameters in multi-tasking per-pixel vision tasks
(Section 4.3).

###  4.1 Polyhistor: Decomposed Lightweight Hyper-networks for Multi-task
Adaptation

With the goal of jointly adapting multiple NLP tasks in a parameter-efficient
manner, a prior work, Hyperformer [16], builds upon a group of adapters in
different tasks and extracts task-sharing information via a hyper-network
shared across different tasks. Specifically, a group of task and layer-wise
adapters with weight parameters
$\\{\bm{W^{t}_{l}}|\bm{t}=1,...,\bm{N};\bm{l}=1,...,\bm{L}\\}$ are
individually inserted into each layer $\bm{l}$ of the model with $\bm{L}$
layers for all $\bm{N}$ tasks. Then, instead of individually learning the
weights of these adapters via backpropagation, Hyperformer constructs a layer-
wise hyper-network $\hat{\bm{W_{l}}}$, which takes as input a learnable task
embedding $\bm{V_{t}}$ and produce the weights of adapters $\bm{W^{t}_{l}}$.

$\displaystyle\bm{W^{t}_{l}}$
$\displaystyle=\Pi(\bm{V_{t}}\hat{\bm{W}_{l}})\in\mathbb{R}^{d\times 2n},$ (2)
$\displaystyle\bm{V_{t}}\in\mathbb{R}^{k},\quad\hat{\bm{W_{l}}}$
$\displaystyle\in\mathbb{R}^{k\times
2dn},\quad\Pi(\cdot):\mathbb{R}^{2dn}\rightarrow\mathbb{R}^{d\times 2n}$

where $\Pi(\cdot)$ maps a vector with size of $2dn$ to a matrix with size of
$d\times 2n$.

While Hyperformer has shown promising results on multi-task NLP benchmarks,
its effectiveness on vision tasks is unclear. In addition, since the hyper-
network produces the vectorization of an adapter weight matrix (i.e.,
$\bm{W^{t}_{l}}\in\mathbb{R}^{d\times 2n}$), the output dimension of the
hyper-network is the order of $\mathcal{O}(dn)$ and the size of the hyper-
network becomes the order of $\mathcal{O}(dnk)$, where $d$ and $n$ are
dimensions of the input and bottleneck vectors and $k$ is the size of task
embeddings. For dense vision tasks, the size of input vectors is usually large
(e.g., $1024$ in SwinTransformer-Base). When the bottleneck dimension is set
to be proportional to the input dimension (e.g., $n=\frac{d}{\alpha}$, where
$\alpha>1$ is a constant), the size of hyper-network is then quadratically
increased with respect to input vectors $\mathcal{O}(kd^{2})$.

To alleviate this issue, we propose to decompose a single hyper-network
$\hat{\bm{W_{l}}}$ into a pair of lightweight hyper-networks
$\\{\hat{\bm{W}}^{\bm{p}}_{\bm{l}},\hat{\bm{W}}^{\bm{q}}_{\bm{l}}\\}$ , each
of which only produces a low-rank matrix. We then multiply the matrices to
obtain the adapter weight as shown in the top of Fig. 2a.

$\displaystyle\bm{W^{t}_{l}}=\sum_{\bm{i}=1}^{\bm{r}}\bm{p_{i}}\bm{q_{i}}^{\intercal}=\bm{\bm{\Pi_{p}}}(\bm{V_{t}}\hat{\bm{W}}^{\bm{p}}_{\bm{l}})\,{\bm{\Pi_{q}}(\bm{V_{t}}\hat{\bm{W}}^{\bm{q}}_{\bm{l}})}^{\intercal},$
(3)
$\displaystyle\bm{\Pi_{p}}(\bm{V_{t}}\hat{\bm{W}}^{\bm{p}}_{\bm{l}})\in\mathbb{R}^{d\times
r},\quad\bm{\Pi_{q}}(\bm{V_{t}}\hat{\bm{W}}^{\bm{q}}_{\bm{l}})\in\mathbb{R}^{2n\times
r},$
$\displaystyle\bm{V_{t}}\in\mathbb{R}^{k},\quad\hat{\bm{W}}^{\bm{p}}_{\bm{l}}\in\mathbb{R}^{k\times
dr},\quad\hat{\bm{W}}^{\bm{q}}_{\bm{l}}\in\mathbb{R}^{k\times 2nr},$
$\displaystyle\bm{\Pi_{p}}(\cdot):\mathbb{R}^{dr}\rightarrow\mathbb{R}^{d\times
r},\quad\bm{\Pi_{q}}(\cdot):\mathbb{R}^{2nr}\rightarrow\mathbb{R}^{2n\times
r},$

where $\bm{\Pi_{p}}(\cdot)$ and $\bm{\Pi_{q}}(\cdot)$ are matrix reshape
functions and $r$ is a matrix rank which is hyper-parameter tuned according
the computational budget. Note that the matrix rank $r$ is usually much
smaller than dimensions of an adapter $n$ or $d$ (i.e.,  $r<<n<d$).

In this way, with the low-rank decomposition and approximation, a heavy hyper-
network $\bm{\hat{W_{l}}}\in\mathbb{R}^{k\times 2dn}$ can be reduced to two
lightweight hyper-networks
$\\{\hat{\bm{W}}^{\bm{p}}_{\bm{l}}\in\mathbb{R}^{k\times
dr},\hat{\bm{W}}^{\bm{q}}_{\bm{l}}\in\mathbb{R}^{k\times 2nr}\\}$, and the
number of trainable parameters can be reduced from $2kdn$ to $k(d+2n)$. The
number of trainable parameters in hyper-networks is reduced from quadratic to
linear increasing with respect to the input size (i.e.,
$\mathcal{O}(kd^{2})\rightarrow\mathcal{O}(kd)$). We will discuss how this
translate into practical usage and demonstrate that our method can
significantly save the number of training parameters while achieving
competitive performance compared to the vanilla Hyperformer in Sec. 5.

###  4.2 Layer-Wise Scaling Kernels for Hierarchical Vision Transformers

Although the number of learnable parameters is reduced by sharing two
lightweight hyper-networks across tasks, each layer of transformer still
requires a pair of layer-wise hyper-networks. To prevent the number of
parameters from growing linearly with respect to the number of layers in a
transformer, one can share a hyper-network across not only different tasks but
also different layers (similar to the recently introduced Hyperformer++ [16]
in NLP). However, since adapters in different blocks of Hierarchical Vision
Transformers have different dimensions, such a property restricts us from
using the same pair of hyper-networks to produce the weights of adapters in
different transformer layers.

To overcome this issue, we introduce Layer-wise Scaling Kernels to enable
sharing a hyper-network across layers.

(a)(b) Figure 2: Illustration of our Polyhistor and Polyhistor-Lite. (a) We
propose Polyhistor, which applies Decomposed HyperNetworks to reduce the
number of training parameters in multi-task adaptation (Section 4.1). We also
introduce Layer-wise Scaling Kernels to efficiently scale up Template Kernels
for different scales of adapters (Section 4.2). (b) By combining Decomposed
HyperNetowrks and Layer-wise Scaling Kernels, our Polyhistor-Lite can
efficiently address multi-task adaptation in per-pixel vision tasks (Section
4.3).

To be more specific, as shown in Fig. 1, Hierarchical Vision Transformers have
four transformers blocks, and each transformers block (indexed by $\bm{b}$)
has multiple transformer layers with a block-wise scale
$\bm{s_{b}}=2^{\bm{b}-1}$ and a output size
$(\frac{H}{4\bm{s_{b}}},\frac{W}{4\bm{s_{b}}},\bm{s_{b}}C)$. However, the
increasing channel sizes in transformer blocks cause two problems. First, as
shown in Fig. 1, the sizes of adapter weights increase quadratically with
respect to the channel size/input size. As a result, the adapters in the later
blocks would require much more trainable parameters than the earlier blocks.
Second, as mentioned above, due to the different sizes of adapters in
different transformer layers, a single hyper-network cannot produce multiple
adapters in different layers and thus cannot be directly shared across layers.

As shown in the bottom of Figure 2a, to address these two problems, we propose
to factorize the weight matrix of a adapter $\bm{W}_{\bm{l}}$ of the layer
$\bm{l}$ into a set of Template Kernels $\tilde{\bm{W}}^{\bm{i}}_{\bm{l}}$ and
Scaling Kernels ${\bm{\kappa}}^{\bm{i}}_{\bm{l}}$. To efficiently scale up
adapter weights in different transformer layers, we resort to Hypercomplex
multiplication [17] and use Kronecker Product to integrate these types of
matrices.111For the ease of understanding, we omit the task index $\bm{t}$.

$\displaystyle\bm{W}_{\bm{l}}$
$\displaystyle=\sum_{\bm{i}=1}^{\bm{s}_{\bm{\psi}(\bm{l})}}\tilde{\bm{W}}^{\bm{i}}_{\bm{l}}\otimes\bm{\kappa}^{\bm{i}}_{\bm{l}},$
(4) $\displaystyle\tilde{\bm{W}}^{\bm{i}}_{\bm{l}}\in\mathbb{R}^{d\times
2n},\quad\bm{\kappa}^{\bm{i}}_{\bm{l}}\in$
$\displaystyle\mathbb{R}^{\bm{s}_{\bm{\psi}(\bm{l})}\times\bm{s}_{\bm{\psi}(\bm{l})}},\quad\bm{W}_{\bm{l}}\in\mathbb{R}^{d\bm{s}_{\bm{\psi}(\bm{l})}\times
2n\bm{s}_{\bm{\psi}(\bm{l})}},$

where $\bm{s}_{\bm{\psi}(\bm{l})}$ and $\bm{\psi}(\bm{l})$ are the scale and
the index of transformer block where transformer layer $\bm{l}$ is located,
and $\otimes$ is Kronecker Product matrix operation. The sizes of Template
Kernels are the same across layers, while the sizes of Scaling Kernels depend
on the block-wise scale $\bm{s_{\bm{\psi}(\bm{l})}}$.

In other words, the purpose of Scaling Kernels is to scale up the Template
Kernels and fit them into the adapters in layers with different scales. This
decomposition not only reduces the parameter usage in a single adapter but
also shows the potential to reduce parameters with a shared hyper-network.

###  4.3 Polyhistor-Lite: Lightweight Hypernetworks for Hierarchical Vision
Transformers

We have described how to reduce the training parameters for multi-tasking
adaptation in Section 4.1 and Hierarchical Vision Transformers in Section 4.2.
To perform parameter-efficient adaptation for multi-tasking per-pixel vision
tasks, we integrate these two components to obtain our final framework.

Specifically, as shown in Fig. 2b, we use a single pair of hyper-networks
$\\{\hat{\bm{W}}^{\bm{p}},\hat{\bm{W}}^{\bm{q}}\\}$ shared across different
layers and different tasks. For the task $\bm{t}$, the hyper-networks take as
input a set of trainable layer embeddings
$\\{\bm{\tilde{V}^{i}_{l}}\\}_{i=1}^{\bm{s_{\bm{\psi(l)}}}}$ and a task
embedding $\bm{\tilde{V}_{t}}$ and produces two low-rank matrices, which are
multiplied and derive a set of Template Kernels of adapters in different
layers and tasks.

$\displaystyle\bm{\tilde{W}^{\bm{t},\bm{i}}_{l}}=\bm{\bm{\Pi_{p}}}([\bm{\tilde{V}_{t}};\bm{\tilde{V}^{i}_{l}}]\hat{\bm{W}}^{\bm{p}})\,{\bm{\Pi_{q}}([\bm{\tilde{V}_{t}};\bm{\tilde{V}^{i}_{l}}]\hat{\bm{W}}^{\bm{q}})}^{\intercal},\forall\bm{i}=1,...,\bm{\psi(l)}$
(5)
$\displaystyle\bm{\bm{\Pi_{p}}}([\bm{\tilde{V}_{t}};\bm{\tilde{V}^{i}_{l}}]\hat{\bm{W}}^{\bm{p}})\in\mathbb{R}^{d\times
r},\quad{\bm{\Pi_{q}}([\bm{\tilde{V}_{t}};\bm{\tilde{V}^{i}_{l}}]\hat{\bm{W}}^{\bm{q}})}\in\mathbb{R}^{2n\times
r},$
$\displaystyle\bm{\tilde{V}_{t}},\bm{\tilde{V}^{i}_{l}}\in\mathbb{R}^{\frac{k}{2}},\quad\hat{\bm{W}}^{\bm{p}}\in\mathbb{R}^{k\times
dr},\quad\hat{\bm{W}}^{\bm{q}}\in\mathbb{R}^{k\times 2nr},$
$\displaystyle\bm{\Pi_{p}}(\cdot):\mathbb{R}^{dr}\rightarrow\mathbb{R}^{d\times
r},\quad\bm{\Pi_{q}}(\cdot):\mathbb{R}^{2nr}\rightarrow\mathbb{R}^{2n\times
r}.$

To derive the parameters of an adapter in each layer, we learn another set of
Scaling Kernels and combine them with the Template Kernels via Kronecker
Product.

$\displaystyle\bm{W}^{\bm{t}}_{\bm{l},}$
$\displaystyle=\sum_{\bm{i}=1}^{\bm{s}_{\bm{\psi}(\bm{l})}}\tilde{\bm{W}}^{\bm{t},\bm{i}}_{\bm{l}}\otimes\bm{\kappa}^{\bm{t},\bm{i}}_{\bm{l}},\forall\bm{t}=1,...,\bm{T}$
(6)
$\displaystyle\tilde{\bm{W}}^{\bm{t},\bm{i}}_{\bm{l}}\in\mathbb{R}^{d\times
2n},$
$\displaystyle\bm{\kappa}^{\bm{t},\bm{i}}_{\bm{l}}\in\mathbb{R}^{\bm{s}_{\bm{\psi}(\bm{l})}\times\bm{s}_{\bm{\psi}(\bm{l})}},\quad\bm{W}^{\bm{t}}_{\bm{l}}\in\mathbb{R}^{d\bm{s}_{\bm{\psi}(\bm{l})}\times
2n\bm{s}_{\bm{\psi}(\bm{l})}}.$

With the integration of Lightweight HyperNetworks and Layer-wise Scaling
Kernels, our framework can efficiently reduce the trainable parameters in
multi-task adaptation for dense vision tasks. We provide two variants of our
method. Polyhistor solely uses the Decomposed HyperNetworks, and Polyhistor-
Lite combines both Decomposed HyperNetworks and Layer-Wise Scaling Kernels.

##  5 Experiments

###  5.1 Implementation Details

Dataset. We follow prior works [30, 31] on multi-task learning for dense
prediction tasks and consider PASCAL-Context [32] to construct our multi-task
efficient adaptation for per-pixel benchmark. We evaluate all methods on four
per-pixel tasks, $21$-class semantic segmentation, $7$-class human part
segmentation, surface normals estimation, and saliency detection. Our
evaluation metrics include the mean intersection-over-union (mIoU) for
semantic segmentation, human part segmentation, and saliency detection and the
mean error (mErr) for surface normals estimation.

Model Architecture. For the encoder, we use Swin-Transformer [13] due to its
strong performance in different vision tasks and the popularity in the vision
community. Our decoders for different dense tasks are based on the All-MLP
decoder of Segformer [14], which uses simple linear layers and bilinear
upsampling layer to efficiently perform dense vision tasks, and we adapt the
number of output dimension to different tasks.

Training. To train our model, we use the commonly-used losses for each task.
Specifically, we use the standard per-pixel cross-entropy for semantic
segmentation and human part segmentation, L1 loss for surface normals
estimation, and balanced cross-entropy for saliency detection. For a fair
comparison, we experiment on a unified codebase implementation with the same
loss functions and training iterations for all baselines and our method.

###  5.2 Baselines

Single-task full fine-tuning uses an individual pretrained model for each
task, and Fine-tuning decoders freezes the feature backbone and only fine-
tunes task-wise decoders for different tasks.

For single-task adaptation methods (Bitfit, VPT, PHM-Layer, Compacter,
Compacter++, Adapter, Low-rank Adapter, and LoRA), we place task-wise modules
for each task.

Bitfit [7] tune the biases in all layers, and, specifically for Swin-
Transformer, and we also tune biases in patch merging layers and patch
projection layers.

VPT [9] inserts tunable embeddings in the first input layer (VPT-shallow) and
all layers (VPT-deep), and we select the best hyper-parameter (i.e., $50$
embeddings per layer) for all results.

PHM layer [12] shares a slow matrix for all layers and learns a fast matrix
for each layer and place modules after attention and MLP layers, Compacter
[12] further decomposes the fast matrix into two low-rank vectors, and
Compacter++ [12] only places modules after MLP layers.

LoRA [10] applied the low-rank decomposition on attention layers, and we
select rank $r=4$ and the adapter output scale (i.e., 4), which performs the
best.

Adapter [11, 21] placed task-wise bottleneck-like modules into transformer
layers, and Shared-Adapter [20] share an adapter across different tasks.

Hyperformer [16] applied a hyper-network and produce the weights for adapter,
and we present the results with different adapter bottleneck dimensions. Since
adapters in different layers have different dimensions, Hyperfomer++ cannot be
simply adapted to Hierarchical Vision Transformers.

###  5.3 Experiment results on Multi-Task Adaptation

(a)(b) Figure 3: (a) Our Polyhistor uses less than one-tenth of the state-of-
the-art multi-task adaptation method (i.e., Hyperformer [16]) in terms of
trainable parameters in the encoder. (b) Tuning hyper-parameters on the
baseline methods leads to limited improvements, and we achieve the best trade-
off between the trainable parameters and accuracy on downstream tasks. Details
are listed in Appendix.

We evaluate all methods by computing relative improvements against Single-Task
Full Fine-tuning and averaging across four tasks. Since all methods use and
train all parameters of task-wise decoders, we provide two values of trainable
parameters, one for the encoder and the other for the whole model.

As presented in Fig. 3 and Table 1, among all experimental methods,
Hyperformer performs the best and achieves $+2.64\%$ on average for the four
downstream tasks, but it requires $72$M trainable parameters in the encoder.
On the other hand, our Polyhistor achieves competitive results ($+2.34\%$),
while we only need $6.41$M trainable parameters in the encoder, which is less
than one-tenth of the Hyperformer. Our Polyhistor-Lite can further reduce the
trainable parameters to $0.41$M by integrating the Layer-wise Scaling Kernels
and sharing the hypernetwork across layers, and it achieves $+1.74\%$ and is
higher than all other methods using a similar amount of trainable parameters
(e.g., BitFit, VPT, Shared Adapter, PHM layer, Compacter, LoRA, and Low-rank
Adapter).

We also found that, while the prior parameter-efficient adaptation vision
method, VPT, presented promising results on single-task image classification
[9], it does not show significant improvements against the baseline that only
fine-tuning decoders in our multi-task dense vision benchmark. A potential
reason is that, compared to the image classification benchmarks which focus
more on the same task with input shifts, our benchmark focuses more on
different task outputs. This makes VPT, which adds the learnable parameters in
the input space, unable to address the difference in the output space and
adapt to the new tasks.

While Hyperformer achieves the best performance improvement against Single-
Task Full Fine-tuning, it requires a larger number of parameters than other
methods. This leads to a natural question: Does reducing the parameters in
Hyperformer derive a better trade-off between the number of tunable parameters
and performance improvements? Therefore, we decrease its size and examine
whether it can maintain its performance with fewer trainable parameters. We
also did a similar experiment on the other existing methods, (e.g., Adapter,
Shared Adapter, Compacter, and Compacter++), in which we increase the number
of tuning parameters by increasing the dimension of the hidden vector in the
adapter modules (increasing the down-projection ratio). As shown in Fig. 3b,
we find that simply tuning the hyper-parameters in baseline methods cannot
obtain a better tradeoff between the number of trainable parameters and
performance improvement. For example, when the number of tuning parameters in
the encoder of Hyperformer is reduced to $20.15$M, its averaged relative
improvement shrinks to only $-0.14\%$. These experiments show that our methods
achieve a better performance improvement with a fewer number of parameters
than the baseline methods.

Table 1:  Experimental results on Multi-Task Adaptation. We use
SwinTransformer-Tiny as the feature backbone. $\Delta_{up}$ represents
relative improvement or gap to the Single-task Full Fine-tuning. Results with
the symbol $\uparrow/\downarrow$ indicate higher/lower is better.

| Number of Trainable Parameters |  | Performance of Each Downstream Task |  | Averaged Results
---|---|---|---|---|---
| Encoder/All |  | Seg. $\uparrow$ | H.Part $\uparrow$ | Sal. $\uparrow$ | Normals $\downarrow$ |  | $\Delta_{up}$
Single-task Full Fine-tuning | 110.07 / 112.62 |  | 67.21 | 61.93 | 62.35 | 17.97 |  | 0.00$\%$
Fine-tuning Decoders | 0.00 / 2.55 |  | 63.14 | 52.37 | 58.39 | 20.89 |  | -11.02$\%$
Multi-task Full Fine-tuning | 27.51 / 30.06 |  | 68.71 | 62.13 | 64.18 | 17.35 |  | 2.23$\%$
Bitfit [7] | 0.30 / 2.85 |  | 68.57 | 55.99 | 60.64 | 19.42 |  | -4.60$\%$
Relative bias [13] | 0.09 / 2.64 |  | 63.51 | 52.35 | 57.74 | 21.07 |  | -11.40$\%$
VPT-shallow [9] | 0.02 / 2.57 |  | 62.96 | 52.27 | 58.31 | 20.90 |  | -11.18$\%$
VPT-deep [9] | 0.88 / 3.43 |  | 64.35 | 52.54 | 58.15 | 21.07 |  | -10.85$\%$
PHM layer [12] | 0.59 / 3.14 |  | 68.55 | 56.28 | 60.35 | 19.23 |  | -4.34$\%$
Compacter [12] | 0.23 / 2.78 |  | 68.08 | 56.41 | 60.08 | 19.22 |  | -4.55$\%$
Compacter++ [12] | 0.11 / 2.66 |  | 67.26 | 55.69 | 59.47 | 19.54 |  | -5.84$\%$
LoRA [10] | 0.32 / 2.87 |  | 70.12 | 57.73 | 61.90 | 18.96 |  | -2.17$\%$
Adapter [21] | 8.69 / 11.24 |  | 69.21 | 57.38 | 61.28 | 18.83 |  | -2.71$\%$
Low-rank adapter | 0.34 / 2.89 |  | 68.31 | 56.53 | 60.29 | 19.36 |  | -4.54$\%$
Shared Adapter [20] | 2.20 / 4.74 |  | 70.21 | 59.15 | 62.29 | 19.26 |  | -1.83$\%$
Hyperformer [16] |  72.77 / 75.32 |  | 71.43 | 60.73 | 65.54 | 17.77 |  | 2.64$\%$
Polyhistor (Ours) | 6.41 / 8.96 |  | 70.87 | 59.54 | 65.47 | 17.47 |  | 2.34$\%$
Polyhistor-Lite (Ours) | 0.41 / 2.96 |  | 70.24 | 59.12 | 64.75 | 17.40 |  | 1.74$\%$

(a)(b) Figure 4: Using the feature backbone with (a) larger size or (b) more
pretraining data leads to larger improvements against Single-task Full Fine-
tuning. All results are produced by our Polyhistor-Lite. For a fair
comparison, both the backbones shown in (a) are pretrained on the ImageNet-1k,
and both the backbones shown in (b) are based on the SwinTrasformer-Base.

###  5.4 Ablation studies and analyses

Different feature backbones. To verify that our proposed method is also
applicable to other larger-size model architecture, we also experiment
SwinTransformer-Base. For a fair comparison, we use the same pretraining
dataset, ImageNet-1k [29], on both SwinTransformer-Tiny and SwinTransformer-
Base. As shown in Fig. 4a, we find that our method achieves a larger
improvement against Single-task Full Fine-tuning when a larger feature
backbone is used, and this shows a potential of obtaining more improvements
when applying our method to a larger model architecture.

Different Pretraining data. We also examine how our proposed method performs
when different pretraining data are used. Specifically, we apply our method to
SwinTransformer-Base pretrained with ImageNet-1k and ImageNet-22k. As
demonstrated in Fig. 4b, we find that our method obtains a larger performance
improvement when more pretraining data was used, and this shows a potential of
deriving more improvements by using more pretraining data.

Figure 5: Our Polyhistor-Lite using more pretraining data can lead to higher
relative improvements to the Single-task Fine-tuning baseline, and this trend
is consistent across varying amounts of trainable parameters. Note that
Single-task Fine-tuning baselines for Polyhistor-Lite(1k) and Polyhistor-
Lite(22k) are trained on ImageNet-1k and ImageNet-22k models respectively.

Varying trainable parameters with different pretraining data. In Fig. 4a, we
showed that using the model with more pretraining data (i.e., ImageNet-22k)
data can lead to a higher performance gain compared to the model with less
pretraining data (i.e., ImageNet-1k). To investigate this phenomenon, we vary
the size of our Polyhistor-Lite and measure the performance gain of the models
with different pretraining data.

As shown in Fig. 5, we find that the model pretrained on ImageNet-22k can use
less trainable parameters to achieve similar performance gains compared to the
model pretrained on ImageNet-1k. In addition, under the same amount of
trainable parameters, the model pretrained on ImageNet-22k can consistently
outperform the model pretrained on ImageNet-1k. This suggests that, with more
pretraining data, feature extractors can learn more diverse representations,
so that we can use less trainable parameters and adapt better to different
downstream tasks.

Dimensions of task embeddings. In addition, we also conduct analysis to
examine how our method performs with different sizes of task embeddings. As
presented in Table 2, when large sizes of task embeddings are used, the
averaged improvement against single-task fine-tuning becomes larger.

Table 2:  Ablation study on the sizes of task embeddings. We vary the sizes of
task embeddings $k$ from $16$ to $64$ on our Polyhistor-Lite. All results in
this table are based on SwinTransformer-Tiny pretrained on ImageNet-1k.

| Size of | Num. of Trainable Parameters |  | Performance of Each Downstream Task |  | Averaged Results
---|---|---|---|---|---|---
| Task Embeddings $k$ | Encoder / All |  | Seg. $\uparrow$ | H.Seg. $\uparrow$ | Sal. $\uparrow$ | Normals $\downarrow$ |  | $\Delta_{up}$
Single-task Full Fine-tuning | - | 110.07 / 112.62 |  | 67.21 | 61.93 | 62.35 | 17.97 |  | 0.00$\%$
Polyhistor-Lite | 16 | 0.23 / 2.78 |  | 69.67 | 58.38 | 63.55 | 18.05 |  | -0.15$\%$
Polyhistor-Lite | 32 | 0.29 / 2.84 |  | 69.80 | 58.72 | 64.14 | 17.73 |  | 0.72$\%$
Polyhistor-Lite | 64 | 0.41 / 2.96 |  | 70.24 | 59.12 | 64.75 | 17.40 |  | 1.74$\%$

We present more ablation studies, analyses, and implementation details in
Appendix.

##  6 Conclusion

We proposed Polyhistor and Polyhistor-Lite — parameter-efficient fine-tuning
methods for dense vision tasks. We showed that most of the existing parameter-
efficient single-task adaptation methods achieved lower performance compared
with Single-task Full Fine-tuning, and the state-of-the-art multi-task
adaptation method achieve favorable results while using a large number of
tunable parameters. Compared to these existing approaches, our proposed
methods do not only achieve a competitive performance gain to the state-of-
the-art but also only use a very limited amount of tunable parameters. The
potential limitation of our method is searching for suitable hyper-parameters,
which is a common limitation among all parameter-efficient learning methods.

Acknowledgments. Yen-Cheng Liu and Zsolt Kira were partly supported by DARPA’s
Learning with Less Labels (LwLL) program under agreement HR0011-18-S-0044, as
part of their affiliation with Georgia Tech.

## References

  * [1] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al.  On the opportunities and risks of foundation models.  arXiv preprint arXiv:2108.07258, 2021.
  * [2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [3] Priya Goyal, Quentin Duval, Isaac Seessel, Mathilde Caron, Mannat Singh, Ishan Misra, Levent Sagun, Armand Joulin, and Piotr Bojanowski.  Vision models are more robust and fair when pretrained on uncurated images without supervision.  arXiv preprint arXiv:2202.08360, 2022.
  * [4] Anuroop Sriram, Abhishek Das, Brandon M Wood, Siddharth Goyal, and C Lawrence Zitnick.  Towards training billion parameter graph neural networks for atomic simulations.  arXiv preprint arXiv:2203.09697, 2022.
  * [5] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  Palm: Scaling language modeling with pathways.  arXiv preprint arXiv:2204.02311, 2022.
  * [6] William Fedus, Barret Zoph, and Noam Shazeer.  Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.  arXiv preprint arXiv:2101.03961, 2021.
  * [7] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg.  Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models.  In In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL), 2022.
  * [8] Demi Guo, Alexander Rush, and Yoon Kim.  Parameter-efficient transfer learning with diff pruning.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2021.
  * [9] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim.  Visual prompt tuning.  arXiv preprint arXiv:2203.12119, 2022.
  * [10] Edward Hu, Yelong Shen, Phil Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  In International Conference on Learning Representations, 2022.
  * [11] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.  Parameter-efficient transfer learning for nlp.  In Proceedings of the International Conference on Machine Learning (ICML), 2019.
  * [12] Rabeeh Karimi mahabadi, James Henderson, and Sebastian Ruder.  Compacter: Efficient low-rank hypercomplex adapter layers.  In Advances in Neural Information Processing Systems (NeurIPS), 2021\.
  * [13] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2021.
  * [14] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo.  Segformer: Simple and efficient design for semantic segmentation with transformers.  In Advances in Neural Information Processing Systems (NeurIPS), 2021\.
  * [15] Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych.  Adapterhub: A framework for adapting transformers.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2020.
  * [16] Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson.  Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks.  In Annual Meeting of the Association for Computational Linguistics, 2021.
  * [17] Aston Zhang, Yi Tay, Shuai Zhang, Alvin Chan, Anh Tuan Luu, Siu Cheung Hui, and Jie Fu.  Beyond fully-connected layers with quaternions: Parameterization of hypercomplex multiplications with $1/n$ parameters.  In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
  * [18] Brian Lester, Rami Al-Rfou, and Noah Constant.  The power of scale for parameter-efficient prompt tuning.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059, 2021.
  * [19] Xiang Lisa Li and Percy Liang.  Prefix-tuning: Optimizing continuous prompts for generation.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2021.
  * [20] Mohit Bansal Yi-Lin Sung, Jaemin Cho.  Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [21] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig.  Towards a unified view of parameter-efficient transfer learning.  In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
  * [22] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [23] Weijian Xu, Yifan Xu, Tyler Chang, and Zhuowen Tu.  Co-scale conv-attentional image transformers.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2021.
  * [24] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang.  Cvt: Introducing convolutions to vision transformers.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2021.
  * [25] Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze.  Levit: a vision transformer in convnet’s clothing for faster inference.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2021.
  * [26] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao.  Pyramid vision transformer: A versatile backbone for dense prediction without convolutions.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2021.
  * [27] Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens van der Maaten, Armand Joulin, and Ishan Misra.  Omnivore: A Single Model for Many Visual Modalities.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [28] Nian Liu, Ni Zhang, Kaiyuan Wan, Ling Shao, and Junwei Han.  Visual saliency transformer.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2021.
  * [29] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  International journal of computer vision, 115(3):211–252, 2015\.
  * [30] Simon Vandenhende, Stamatios Georgoulis, and Luc Van Gool.  Mti-net: Multi-scale task interaction networks for multi-task learning.  In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
  * [31] S. Vandenhende, S. Georgoulis, W. Van Gansbeke, M. Proesmans, D. Dai, and L. Van Gool.  Multi-task learning for dense prediction tasks: A survey.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021\.
  * [32] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  International Journal of Computer Vision (IJCV), 2010.
  * [33] Zhenda Xie, Yutong Lin, Zhuliang Yao, Zheng Zhang, Qi Dai, Yue Cao, and Han Hu.  Self-supervised learning with swin transformers.  arXiv preprint arXiv:2105.04553, 2021.
  * [34] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.

## Appendix

###  A Feature backbones with different pretraining data.

Comparison to baselines. We also experiment with other baseline methods on
SwinTransformer-Base pretrained on ImageNet-1k and ImageNet-22k. As shown in
Table 3 and 4, we find that the trend of all methods are similar to the
results on SwinTransformer-Tiny. Specifically, most baseline methods have a
performance gap with Hyperformer and our Polyhistor-Lite. Compared to
Hyperformer, our Polyhistor-Lite use less than $5\%$ of their trainable
parameters in the encoder to achieve similar or even better results. By tuning
the adapter down-projection ratio and using fewer trainable parameters, our
Polyhistor-Lite still obtains a higher performance gain against the baseline
methods with low trainable parameters (e.g., Compacter++, Bitfit, VPT, PHM
layer, LoRA, Adapter). These results show that our proposed Polyhistor-Lite
can derive a better trade-off against all existing parameter-efficient works
in different pretrained feature backbones.

###  B Ranks of hyper-network output.

Another important hyper-parameter in our model is ranks of hyper-network
outputs. We thus experiment Polyhistor and vary the rank of output matrices,
and, as shown in Table 5, we find the model can derive a better results with a
large rank.

###  C Experiment results of tuning baseline methods.

In Fig. 3b of the main paper, we presented the results of different baseline
methods with different hyper-parameters. For a clearer comparison between
baseline methods, We also provide the exact values of all results in Table 6.
It is worth noting that simply tuning the baseline methods only leads to
limited improvements of baseline methods, and our proposed Polyhistor and
Polyhistor-Lite still achieve a better trade-off between performance gain and
trainable parameters.

Table 3:  Experimental results on Multi-Task Adaptation. We use
SwinTransformer-Base pretrained on ImageNet-1k as the feature backbone.
$\Delta_{up}$ represents relative improvement or gap to the Single-task Full
Fine-tuning. Results with the symbol $\uparrow/\downarrow$ indicate
higher/lower is better. $\rho$ represents the down-projection ratio of
adapters (i.e.,. the ratio of adapter input $d$ to hidden vectors $n$,
$\rho=d/n$).

| Number of Trainable Parameters |  | Performance of Each Downstream Task |  | Averaged Results
---|---|---|---|---|---
| Encoder/All |  | Seg. $\uparrow$ | H.Part $\uparrow$ | Sal. $\uparrow$ | Normals $\downarrow$ |  | $\Delta_{up}$
Single-task Full Fine-tuning | 346.96 / 350.01 |  | 67.88 | 64.47 | 61.26 | 18.85 |  | 0.00$\%$
Fine-tuning Decoders | 0.00 / 3.04 |  | 68.98 | 55.57 | 58.37 | 21.36 |  | -7.55$\%$
Bitfit [7] | 0.20 / 3.24 |  | 71.93 | 59.12 | 60.67 | 20.08 |  | -2.46$\%$
Relative bias [13] | 0.06 / 3.11 |  | 68.44 | 55.70 | 57.27 | 21.63 |  | -8.51$\%$
VPT-deep [9] | 2.41 / 5.45 |  | 68.80 | 55.85 | 58.25 | 21.37 |  | -7.57$\%$
PHM layer [12] | 1.89 / 4.94 |  | 71.93 | 59.11 | 59.71 | 20.35 |  | -3.21$\%$
Compacter++ [12] | 0.25 / 3.29 |  | 72.00 | 59.11 | 59.73 | 20.41 |  | -3.25$\%$
LoRA [10] | 0.87 / 4.31 |  | 74.10 | 61.57 | 63.87 | 18.55 |  | 2.63$\%$
Adapter [21] | 3.64 / 6.68 |  | 73.29 | 60.30 | 62.42 | 18.66 |  | 1.10$\%$
Low-rank adapter | 0.34 / 2.89 |  | 72.13 | 59.10 | 59.81 | 20.28 |  | -3.01$\%$
Hyperformer [16] |  60.88 / 63.92 |  | 73.60 | 63.82 | 67.31 | 16.90 |  | 6.91$\%$
Polyhistor-Lite (Ours; $\rho=1$) | 1.29 / 4.34 |  | 73.70 | 63.32 | 66.50 | 16.93 |  | 6.38$\%$
Polyhistor-Lite (Ours; $\rho=2$) | 0.62 / 3.67 |  | 73.69 | 63.04 | 66.56 | 17.30 |  | 5.80$\%$
Polyhistor-Lite (Ours; $\rho=4$) | 0.39 / 3.43 |  | 73.57 | 62.04 | 65.84 | 17.70 |  | 4.55$\%$
Polyhistor-Lite (Ours; $\rho=8$) | 0.29 / 3.34 |  | 73.92 | 62.15 | 65.37 | 17.70 |  | 4.53$\%$
Polyhistor-Lite (Ours; $\rho=32$) | 0.24 / 3.28 |  | 73.80 | 61.32 | 64.64 | 17.92 |  | 3.57$\%$

Table 4:  Experimental results on Multi-Task Adaptation. We use
SwinTransformer-Base pretrained on ImageNet-22k as the feature backbone.
$\Delta_{up}$ represents relative improvement or gap to the Single-task Full
Fine-tuning. Results with the symbol $\uparrow/\downarrow$ indicate
higher/lower is better. $\rho$ represents the down-projection ratio of
adapters (i.e.,. the ratio of adapter input $d$ to hidden vectors $n$,
$\rho=d/n$).

| Number of Trainable Parameters |  | Performance of Each Downstream Task |  | Averaged Results
---|---|---|---|---|---
| Encoder/All |  | Seg. $\uparrow$ | H.Part $\uparrow$ | Sal. $\uparrow$ | Normals $\downarrow$ |  | $\Delta_{up}$
Single-task Full Fine-tuning | 346.96 / 350.01 |  | 70.72 | 67.47 | 61.00 | 18.73 |  | 0.00$\%$
Fine-tuning Decoders | 0.00 / 3.04 |  | 73.33 | 60.56 | 59.13 | 21.38 |  | -5.94$\%$
Bitfit [7] | 0.20 / 3.24 |  | 76.42 | 64.89 | 62.05 | 19.03 |  | 1.09$\%$
Relative bias [13] | 0.06 / 3.11 |  | 72.86 | 60.64 | 58.44 | 21.51 |  | -6.53$\%$
VPT-deep [9] | 2.41 / 5.45 |  | 74.21 | 61.41 | 58.80 | 21.61 |  | -5.90$\%$
PHM layer [12] | 1.89 / 4.94 |  | 76.33 | 64.59 | 60.43 | 20.23 |  | -1.32$\%$
Compacter++ [12] | 0.25 / 3.29 |  | 75.99 | 64.65 | 60.42 | 20.01 |  | -1.13$\%$
LoRA [10] | 0.87 / 4.31 |  | 78.24 | 66.95 | 64.70 | 18.07 |  | 4.86$\%$
Adapter [21] | 3.64 / 6.68 |  | 77.22 | 65.95 | 63.80 | 18.38 |  | 3.35$\%$
Low-rank adapter | 0.34 / 2.89 |  | 75.65 | 64.75 | 60.50 | 20.03 |  | -1.21$\%$
Hyperformer [16] |  60.88 / 63.92 |  | 78.41 | 68.94 | 67.50 | 16.80 |  | 6.91$\%$
Polyhistor-Lite (Ours; $\rho=1$) | 1.29 / 4.34 |  | 77.91 | 68.02 | 66.89 | 16.54 |  | 8.08$\%$
Polyhistor-Lite (Ours; $\rho=32$) | 0.24 / 3.28 |  | 77.74 | 66.33 | 65.03 | 17.65 |  | 5.15$\%$

Table 5:  Ablation study on the sizes of ranks in hypernetwork output
matrices. We vary the dimensions of ranks $r$ from $1$ to $\frac{n}{2}$ on our
Polyhistor. Note that $n$ is the dimension of hidden vectors in adapters. All
results in this table are based on SwinTransformer-Tiny pretrained on
ImageNet-1k.

| Dimension of | Num. of Trainable Parameters |  | Performance of Each Downstream Task |  | Averaged Results
---|---|---|---|---|---|---
| Ranks $r$ | Encoder/ All |  | Seg. $\uparrow$ | H.Seg. $\uparrow$ | Sal. $\uparrow$ | Normals $\downarrow$ |  | $\Delta_{up}$
Single-task Full Fine-tuning | - | 110.07 / 112.62 |  | 67.21 | 61.93 | 62.35 | 17.97 |  | 0.00$\%$
Polyhistor | 1 | 2.38 / 4.93 |  | 70.31 | 58.61 | 64.14 | 17.98 |  | 0.52$\%$
Polyhistor | n/8 | 4.08 / 6.63 |  | 71.18 | 59.52 | 65.04 | 17.81 |  | 1.70$\%$
Polyhistor | n/4 | 6.41 / 8.96 |  | 70.87 | 59.54 | 65.47 | 17.47 |  | 2.34$\%$
Polyhistor | n/2 | 11.08 / 13.63 |  | 71.31 | 60.15 | 65.46 | 17.40 |  | 2.84$\%$

Table 6:  Limited improvements from tuning hyper-parameters on baseline
method. $\Delta_{up}$ represents relative improvement or gap to the Single-
task Full Fine-tuning. Results with the symbol $\uparrow/\downarrow$ indicate
higher/lower is better. Results with the symbol $\uparrow/\downarrow$ indicate
higher/lower is better. $\rho$ represents the down-projection ratio of
adapters (i.e.,. the ratio of adapter input $d$ to hidden vectors $n$,
$\rho=d/n$).

| Num. of Trainable Parameters |  | Performance of Each Downstream Task |  | Averaged Results
---|---|---|---|---|---
| Encoder / All |  | Seg. $\uparrow$ | H.Seg. $\uparrow$ | Sal. $\uparrow$ | Normals $\downarrow$ |  | $\Delta_{up}$
Single-task Full Fine-tuning | 110.07 / 112.62 |  | 67.21 | 61.93 | 62.35 | 17.97 |  | 0.00$\%$
Fine-tuning Decoders | 0.00 / 2.55 |  | 63.14 | 52.37 | 58.39 | 20.89 |  | -11.02$\%$
Compacter++ ($\rho=1$) [12] | 0.14 / 2.69 |  | 67.33 | 55.68 | 59.50 | 19.66 |  | -5.98$\%$
Compacter++ ($\rho=2$) [12] | 0.11 / 2.66 |  | 67.26 | 55.69 | 59.47 | 19.54 |  | -5.84$\%$
Compacter++ ($\rho=8$) [12] | 0.09 / 2.64 |  | 67.19 | 55.85 | 59.48 | 19.56 |  | -5.96$\%$
Compacter ($\rho=1$) [12] | 0.28 / 2.83 |  | 67.94 | 56.23 | 60.18 | 19.25 |  | -4.69$\%$
Compacter ($\rho=2$) [12] | 0.23 / 2.78 |  | 68.08 | 56.41 | 60.08 | 19.22 |  | -4.55$\%$
Compacter ($\rho=8$) [12] | 0.19 / 2.74 |  | 68.15 | 56.16 | 60.12 | 19.37 |  | -4.83$\%$
Adapter ($\rho=1$) [21] | 17.32 / 19.87 |  | 69.13 | 57.35 | 61.17 | 18.79 |  | -2.75$\%$
Adapter ($\rho=2$) [21] | 8.69 / 11.24 |  | 69.21 | 57.38 | 61.28 | 18.83 |  | -2.71$\%$
Adapter ($\rho=4$) [21] | 4.37 / 6.92 |  | 68.93 | 57.33 | 61.24 | 18.95 |  | -3.03$\%$
Adapter ($\rho=8$) [21] | 2.21 / 4.76 |  | 69.04 | 57.34 | 61.25 | 18.86 |  | -2.86$\%$
Adapter ($\rho=16$) [21] | 1.13 / 3.68 |  | 69.03 | 57.22 | 61.17 | 18.91 |  | -3.01$\%$
Shared Adapter ($\rho=1$) [20] | 4.35 / 6.89 |  | 70.57 | 59.43 | 62.54 | 19.07 |  | -1.21$\%$
Shared Adapter ($\rho=2$) [20] | 2.20 / 4.74 |  | 70.21 | 59.15 | 62.29 | 19.26 |  | -1.83$\%$
Shared Adapter ($\rho=4$) [20] | 1.12 / 3.66 |  | 70.02 | 58.87 | 62.09 | 19.35 |  | -2.22$\%$
Shared Adapter ($\rho=8$) [20] | 0.58 / 3.12 |  | 69.63 | 58.54 | 61.74 | 19.61 |  | -2.99$\%$
Hyperformer ($\rho=8$) [16] | 72.77 / 75.32 |  | 71.43 | 60.73 | 65.54 | 17.77 |  | 2.64$\%$
Hyperformer ($\rho=16$) [16] | 37.69 / 40.24 |  | 71.28 | 60.19 | 65.82 | 17.89 |  | 2.31$\%$
Hyperformer ($\rho=32$) [16] | 20.15 / 22.70 |  | 71.12 | 59.71 | 64.41 | 19.06 |  | -0.14$\%$
Polyhistor(Ours) | 6.41 / 8.96 |  | 70.87 | 59.54 | 65.47 | 17.47 |  | 2.34$\%$
Polyhistor-Lite(Ours) | 0.41 / 2.96 |  | 70.24 | 59.12 | 64.75 | 17.40 |  | 1.74$\%$

###  D Experiment results of other hierarchical vision transformers.

As shown in Table 7, we further apply our method and other baseline methods to
the Pyramid Vision Transformer [26]. We find our Polyhistor can achieve
comparable results to Hyperformer while using much fewer trainable parameters.
Polyhistor-Lite can further reduce trainable parameters and achieves higher
accuracy than all other methods using a similar amount of trainable parameters
(e.g., BitFit, PHM layer, Compacter, LoRA, and Low-rank Adapter). This trend
is aligned with what we found in Swin Transformer. We show that our method
generalizes to different backbones.

Table 7:  Experimental results on Multi-Task Adaptation. We use Pryramid
Vision Transformer-Small as the feature backbone. $\Delta_{up}$ represents
relative improvement or gap to the Single-task Full Fine-tuning. Results with
the symbol $\uparrow/\downarrow$ indicate higher/lower is better. $\rho$
represents the down-projection ratio of adapters (i.e.,. the ratio of adapter
input $d$ to hidden vectors $n$, $\rho=d/n$).

| Number of Trainable Parameters |  | Performance of Each Downstream Task |  | Averaged Results
---|---|---|---|---|---
| Encoder/All |  |  Seg. $\uparrow$ |  H.Part $\uparrow$ |  Sal. $\uparrow$ |  Normals $\downarrow$ |  |  $\Delta_{up}$
Single-task Full Fine-tuning | 95.88 / 97.99 |  | 68.81 | 61.27 | 62.67 | 17.55 |  |  0.00$\%$
Fine-tuning Decoders | 0.00 / 2.11 |  | 64.86 | 51.18 | 61.54 | 19.55 |  |  -8.85$\%$
Bitfit [7] | 0.22 / 2.34 |  | 71.41 | 55.71 | 64.08 | 18.69 |  |  -2.38$\%$
Adapter [21] | 0.79 / 2.90 |  | 71.94 | 56.38 | 64.16 | 18.75 |  |  -1.97$\%$
LoRA [10] | 0.30 / 2.41 |  | 71.89 | 56.90 | 64.27 | 18.48 |  |  -1.35$\%$
Low-rank adapter | 0.25 / 2.36 |  | 70.72 | 55.34 | 63.39 | 18.70 |  |  -3.08$\%$
PHM layer [12] | 0.42 / 2.53 |  | 70.81 | 55.02 | 63.51 | 18.75 |  |  -3.20$\%$
Compacter++ [12] | 0.09 / 2.20 |  | 70.29 | 54.80 | 63.16 | 18.82 |  |  -3.71$\%$
Hyperformer [16] |  14.03 / 16.14 |  | 70.81 | 57.76 | 65.49 | 17.75 |  | 0.14$\%$
Polyhistor-Lite (Ours; $\rho=1$) | 5.21 / 7.32 |  | 71.00 | 57.52 | 65.83 | 17.83 |  | 0.13$\%$
Polyhistor-Lite (Ours; $\rho=32$) | 0.29 / 2.40 |  | 70.93 | 56.71 | 65.00 | 17.95 |  | -0.73$\%$

###  E Experiment results of self-supervised models.

We conduct an experiment using the self-supervised Swim Transformer-Tiny
(MoBY-Tiny [33]), and, for a fair comparison, we also run all baseline with
MoBY-Tiny and report the results in the Table 8. We find our proposed method
can achieve similar or even better results compared to the Hyperformer [2]
while using much less trainable parameters.

Table 8:  Experimental results on Multi-Task Adaptation. We use MoBY-Tiny [33]
as the feature backbone. $\Delta_{up}$ represents relative improvement or gap
to the Single-task Full Fine-tuning. Results with the symbol
$\uparrow/\downarrow$ indicate higher/lower is better. $\rho$ represents the
down-projection ratio of adapters (i.e.,. the ratio of adapter input $d$ to
hidden vectors $n$, $\rho=d/n$).

| Number of Trainable Parameters |  | Performance of Each Downstream Task |  | Averaged Results
---|---|---|---|---|---
| Encoder/All |  |  Seg. $\uparrow$ |  H.Part $\uparrow$ |  Sal. $\uparrow$ |  Normals $\downarrow$ |  |  $\Delta_{up}$
Single-task Full Fine-tuning | 110.07 / 112.62 |  | 65.52 | 61.78 | 62.05 | 18.14 |  |  0.00$\%$
Fine-tuning Decoders | 0.00 / 2.55 |  | 59.64 | 52.97 | 59.60 | 19.88 |  |  -9.21$\%$
Bitfit [7] | 0.30 / 2.85 |  | 63.43 | 54.90 | 59.50 | 19.80 |  |  -6.90$\%$
VPT-shallow [9] | 0.02 / 2.57 |  | 59.50 | 52.84 | 59.48 | 19.88 |  |  -9.36$\%$
VPT-deep [9] | 0.88 / 3.43 |  | 56.15 | 50.30 | 57.22 | 20.71 |  |  -13.72$\%$
Adapter [21] | 8.69 / 11.24 |  | 65.00 | 56.66 | 60.84 | 18.64 |  |  -3.45$\%$
LoRA [10] | 0.32 / 2.87 |  | 65.64 | 57.66 | 62.29 | 18.47 |  |  -1.99$\%$
Low-rank adapter | 0.34 / 2.89 |  | 63.30 | 55.24 | 59.72 | 19.14 |  |  -5.82$\%$
PHM layer [12] | 0.59 / 3.14 |  | 63.21 | 54.99 | 59.70 | 19.13 |  |  -5.95$\%$
Compacter++ [12] | 0.11 / 2.66 |  | 62.31 | 54.69 | 59.43 | 19.58 |  |  -7.14$\%$
Hyperformer [16] |  19.29 / 44.25 |  | 66.50 | 58.97 | 66.02 | 17.61 |  | 1.56$\%$
Polyhistor (Ours) | 6.41 / 8.96 |  | 67.69 | 59.32 | 65.15 | 17.43 |  | 2.05$\%$
Polyhistor-Lite (Ours) | 0.41 / 2.96 |  | 67.23 | 58.90 | 64.62 | 17.72 |  | 1.09$\%$

###  F Discussion on difference to Visual Prompt Tuning [9]

We summarize the difference between Visual Prompt Tuning and our method in the
following points.

Different Problem Settings: Visual Prompt Tuning focuses on single-task
parameter-efficient adaptation, while our proposed method focuses on multi-
task parameter-efficient adaptation. Our goal is to perform a parameter-
efficient adaptation for multiple tasks and share the beneficial information
across multiple vision tasks.

Different types of parameter-efficient methods: Visual Prompt Tuning adds
learnable parameters along with the visual embeddings, while our proposed
method utilizes a shared hyper-network to produce the adapter weights for
different tasks. Also, the insertion locations of learnable parameters are
different (VPT: input space, Ours: parallel to fully-connected layers).

###  G Implementation Details

For a fair comparison between different methods, we use batch size $12$ and
train for $60$ epochs for each task. We use Adam optimizer [34] with the
learning rate $1e-4$ and the weight decay $1e-4$, and the learning rate is
linearly decreased with respect to the training iteration.

We followed the prior multi-tasking learning work [31] to use task-wise
weighting on different losses, while we found that using the uniform weights
on the losses has similar results as the task-wise weighting. We also applied
the same data augmentations, RandomHorizontalFlip, RandomScale with the range
$[0.75,1.25]$, RandomRotate with the range $[-20,20]$, and Resize to
$(512,512)$, which are used in the prior work [31].

For the hyper-parameters of Polyhistor, we set the input dimension of adapter
$d$ as the dimension of hidden vectors in SwinTransformers, and the down-
projection ratio is set as $\rho=d/n=16$. For the low-rank output matrix of
hyper-networks, we set the rank as $n/4$, where $n$ is bottleneck size. We set
the size of task embeddings as $64$.

As for the hyper-parameters of Polyhistor-Lite, we also set the input
dimension of adapter $d$ as the dimension of hidden vectors in
SwinTransformers, and the down-projection ratio is set as $\rho=d/n=2$. For
the low-rank output matrix of hyper-networks, we set the rank as $n/4$, where
$n$ is bottleneck size. We set the size of task embeddings as $64$.


11institutetext: Carnegie Mellon University, Pittsburgh, USA
11email: {pcarragh, huixiann<EMAIL_ADDRESS>
# Simulation of Stance Perturbations

Peter Carragher  0000-0003-4053-1001 Lynnette Hui Xian Ng  0000-0002-2740-7818
Kathleen M. Carley  0000-0002-6356-0238

###### Abstract

In this work, we analyze the circumstances under which social influence
operations are likely to succeed. These circumstances include the selection of
Confederate agents to execute intentional perturbations and the selection of
Perturbation strategies. We use Agent-Based Modelling (ABM) as a simulation
technique to observe the effect of intentional stance perturbations on scale-
free networks. We develop a co-evolutionary social influence model to
interrogate the tradeoff between perturbing stance and maintaining influence
when these variables are linked through homophily. In our experiments, we
observe that stances in a network will converge in sufficient simulation
timesteps, influential agents are the best Confederates and the optimal
Perturbation strategy involves the cascade of local ego networks. Finally, our
experimental results support the theory of tipping points and are in line with
empirical findings suggesting that 20-25% of agents need to be Confederates
before a change in consensus can be achieved.

###### Keywords:

Agent Based Modelling Social Influence Simulation

##  1 Background

ABM has a rich history in social network analysis [5, 8, 9, 11, 12, 13]. Will
et. al. [13] categorized such studies into those that investigate endogenously
emerging networks, exogenously imposed networks and co-evolutionary networks.
Endogenous studies look at how social network structure evolves as a function
of the set of agent states. Conversely, exogenous studies keep the network
structure constant and model changes in agent states based on this structure.
The co-evolutionary approach is a hybrid of both and models the interplay
between agent states and network structure. Despite being a closer fit to
genuine SI processes, this approach is relatively understudied [13]. Differing
timescales between endogenous & exogenous effects complicate the matter.

Ng et. al. [10] demonstrated that endogenous and exogenous features are
equally important in predicting pro / anti-vaccine stance flips on Twitter.
The SI model used to make this prediction is an exogenous one based on
Friedkin’s foundational social influence (SI) model [5], where the influence
weight matrix $W$ is static and pre-calculated from self-reports. In contrast
Macy et. al. [9] develop the Hopfield model, a co-evolutionary approach for
the simulation of stance change. It has been shown that both exogenous [11]
and co-evolutionary models can converge to a hyper-polarized state [9].

ABM has also been used to highlight the vulnerability of the SI model to
manipulative actors Ṙoss et. al. [12] find that as little as 2-4% well-
positioned bots are capable of tipping the majority opinion. Of note is the
variance in estimations of tipping points, with Centola et. al. [4] showing
that 25% of users must commit to certain language use before it gains
traction. This motivates the investigation of a wide variety of environmental
conditions in the simulation of intentional stance perturbation.

##  2 Simulation Objectives

In our simulation, we wish to observe the effect of intentional stance
perturbations on the overall stances in the network. The process of a group of
Confederate agents perturbing a social network simulates the real-world
scenario of an influence operation, wherein disingenuous operatives
strategically promote subversive and provocative content with the intention of
catalyzing a shift in or preventing the formation of consensus.

###  2.1 Research Questions

In this study, our objective is to explore deliberate perturbations’ impact on
influence networks and their effect on individual stances. Using a co-
evolutionary SI model that simulates how stance changes with respect to
interpersonal influence, we seek to understand strategies that maintain
influence while perturbing the network.

R1 How can Confederates manipulate stances effectively without losing
influence?

R2 Considering factors such as susceptibility, network structure, and stance,
which agents make for effective Confederates?

Finally, we investigate the potential for intentional perturbations to disrupt
the prevailing consensus, shedding light on opinion dynamics within the
influence network.

R2 To what extent can intentional perturbations change an established
consensus?

By addressing these research questions, we will gain a deeper understanding of
intentional perturbations in influence networks, their impact on stances, and
strategies to optimize their effectiveness.

###  2.2 Contributions

By adopting a co-evolutionary SI model, we explore the tradeoff between
influence and stance within social networks. Our simulations capture the
intricate relationship between the emergence of the influence network and its
impact on individual stances.

Secondly, we propose an evaluation criterion based on consensus, which serves
as a measure to assess intentional stance perturbation. By analyzing changes
in consensus over time, we can gauge the success of perturbations by the
magnitude of stance change. Furthermore, our findings reveal the efficacy of
targeted nudging strategies in perturbing stances. Well-targeted nudges yield
the largest stance perturbation, underscoring the importance of utilizing the
structure of the influence network.

Lastly, we find support for empirical findings on the theory of tipping points
that suggest 20-25% of agents need to be Confederates to bring about a new
consensus [4]. These contributions significantly enhance our understanding of
influence dynamics, intentional stance perturbation, and consensus formation
within social networks.

Table 1: Definitions of Terminology used in this study Terminology | Definition
---|---
Stance | Opinion on a topic using a scale of two extremes, -1.0 and 1.0
Susceptibility | How vulnerable an agent is to the changes of stance / influence
Influence | How much impact / sway an agent has towards its neighbors
Confederates | The group of agents that are perturbing the network
Perturbation | The “nudging" of stances of other agents in the network
Tipping point | The timestep of the simulation where majority stance changes
Table 2: Definitions of Symbols used in this study Symbol | Definition
---|---
$i,j$ | identification of agents
$y(t)$ | stance at timestep $t$
$W$ | influence weight matrix, row-normalized
$A$ | diagonal matrix of actor susceptibilities to influence
$\alpha$ | stance update rate
$\lambda$ | influence update rate
$\mu_{y}$ | average stance at time $t$ over $N$ non-Confederate agents
$\theta$ | threshold of Confederate influence
$k$ | node at timestep $t$ with maximum global influence
$w_{i}^{g}$ | Confederate $i$’s total (‘global’) network influence
$w_{i}^{l}$ |  $i$’s influence over the top $M$ agents in $i$’s ego-network

##  3 Model Definition

This simulation study performs an examination of the effect of Confederate
agent selection and stance perturbation strategies on the overall stance in
the network. In Table 1 and Table 2, we define some of the terminology and
symbols used in this study, including the scale-free influence network. At
each timestep $t$, the stance vector is updated, followed by the influence
matrix, using two interdependent recurrence relations.

Following Friedkin’s model [5], the stance update equation represents the
exogenous effects of the influence network; an agent’s stance is incrementally
nudged towards the average stance of those who have influence over them. The
model relates influence matrix $W$ to stances $y$ at time $t$ as per Equation
1. Note, the diagonal susceptibility matrix A is scaled by a stance learning
rate $\alpha=0.001$.

$y(t)=AW(t)y(t-1)+(I-A)y(1)$ (1)

Equation 2 describes the endogenous update rule for influence matrix $W$.
Based on the Hopfield model [9], the influence update equation makes this a
co-evolutionary model. We introduce an additional reverse relation between $W$
& $y$ based on the homophily process, to enable influence to co-evolve with
stance. We also introduces the influence update rate, $\lambda=0.01$, also
known as the rate of structural learning [9].

$W(t+1)=\lambda y_{t}y^{\intercal}_{t}+(1-\lambda)W(t)$ (2)

The relation between the influence and stance update rates, $\lambda$ and
$\alpha$, is crucial. Since the model dynamics are extremely sensitive to
these update rates, it is also important to choose $\lambda$ and $\alpha$ such
that performance comparisons between various perturbation strategies can be
made. That is, when the simulation converges, it has reached a stable
polarised state where there is not an absolute majority. We find that the
proposed values achieve this, as demonstrated in figure 2. In a stable
polarised state we can compare strategies based on how many agents change
stances, as approximated by the average stance of non-Confederate agents.

Concretely, the case for $\alpha>\lambda$ is trivial, as the simulation
converges quickly as defined by the influence network topology. Here agents
adjust their stance to match their ego networks faster than they can update
their influence weights based on the homophily principle of the influence
update rule. Our experiments use $\lambda>>\alpha$ so that agents adjust their
influence based on homophily first and foremost. The result is a changing
network structure that introduces a tradeoff for Confederates between
maintaining influence and perturbing stance.

###  3.1 Influence-Stance Tradeoff

Adopting a radical position risks triggering resistance and skepticism,
potentially leading to a loss of trust and influence. Operatives may establish
trust by posing as like-minded individuals, sharing relatable experiences, and
providing seemingly credible information. Therein, they face a dilemma; they
may adopt a less extreme stance that aligns with users’ existing beliefs to
potentially increase their influence, or they attempt to change the narrative
with extremist content and risk triggering resistance in return.

The homophily principle in our model encapsulates this tradeoff ($\lambda
y_{t}y^{\intercal}_{t}$ in Equation 2); as Confederate stances diverge from
the average stance of the network, their influence decreases. Figure 1
illustrates this point with a single Confederate trying to perturb an 80-node
network using the conversion perturbation strategy.

Figure 1: In an 80-node network, a single Confederate struggles to perturb
consensus while maintaining influence using the conversion perturbation
strategy. The Confederate maintains a -1 stance until its influence begins to
drop at timestep 90. It then raises its stance until its influence rebuilds.
This repeats at timestep 130, the beginning of a distinctly cyclic pattern.

###  3.2 Confederate Selection

With this trade-off in mind, we propose three strategies for determining
Confederates.

Maximum influence: we select the most influential agents according to weighted
out-degree in the influence matrix at time $t=0$;
$W_{max}=argmax_{j}\Sigma^{N}_{i}W(0)_{ij}$.

Minimum susceptibility: we select the least susceptible agents, skewing the
remaining agents towards being more susceptible; $A_{min}=argmin_{j}A_{jj}$

Random selection (control): Confederates are selected uniformly at random.

###  3.3 Perturbation Strategy

Confederates perturb the social network, causing the average stance of the
agents to change. We define three strategies for the perturbation of a scale-
free network: the conservative strategy, the conversion strategy, and the
cascade strategy.

The conservative approach (Equation 3) is to perturb stance if the
Confederate’s influence is above some threshold $\theta$. Whenever influence
drops below $\theta$, we set the stance to $\mu_{y}$; the average over $N$
non-Confederate stances at time t: $\mu_{y}=\Sigma^{N}_{i}y(i,t)/N$.

$\\!\begin{aligned}
y(i,t)=\begin{cases}\mu_{y}&\Sigma^{N}_{j}w(j,i)\leq\theta\\\
-1&\Sigma^{N}_{j}w(j,i)>\theta\\\ \end{cases}\end{aligned}$ (3)

The conversion strategy (Equation 4) is based on “nudging" the network towards
the desired stance (-1). It is therefore a continuous function where we scale
the magnitude of the perturbation by the current level of influence as in
figure 1. When influence is low, the stance perturbation is less aggressive
but when influence is high, the perturbation results in a more extreme stance
(closer to -1). Here, $w^{g}_{i}$ is the global influence factor of
Confederate $i$ on all $N$ non-Confederate agents (Equation 5). We normalize
using the influence of $k$, the agent at timestep t with the maximum global
influence.

$y(i,t)=\mu^{g}_{y}+w^{g}_{i}*(-1-\mu^{g}_{y})$ (4)

$w^{g}_{i}=\Sigma^{N}_{j}w(j,i)/\max_{k^{\prime}}\Sigma^{N}_{j}w(j,k^{\prime})$
(5)

The cascade strategy (Equation 6) is derived from the conversion strategy
(Equation 4). The key difference for cascade is that we take a local approach
when calculating Confederate influence, looking only at each Confederate’s ego
network. We rank all agents by the influence the Confederate has over them and
sum the influence weights from the top $M$ most influenced agents (Equation 7)
where $M=N/10$. This more targeted approach is named after the well known
cascade effect which highlights the role of network structure in information
diffusion.

$y(i,t)=\mu^{l}_{y}+w^{l}_{i}*(-1-\mu^{l}_{y})$ (6)

$w^{l}_{i}=\Sigma^{M}_{j}w(j,i)/\max_{k^{\prime}}\Sigma^{M}_{j}w(j,k^{\prime})$
(7)

##  4 Methodology

Our simulation experiment involves perturbations by a group of Confederates
across scale-free networks. The ABM is implemented in the Construct framework
[8, 2, 3]. This framework reads in a scale-free network and simulates the SI
model as defined in Equations 1 & 2 until convergence. Results are averaged
over five replicates.

To begin our experiments, we construct a series of generalized scale-free
networks. A scale-free network is a network where the node degrees follow a
power law distribution [1]. This leads to a characteristic hub structure that
mimics a social network setting, where there are some nodes that have many
connections, while others are more isolated. These networks are constructed
using the preferential attachment model, with five replicates per network
size. Experiments on alternative network constructions such as Small World,
Erdos-Renyi, and Core Periphery, are beyond the scope of this study.

We construct networks with $N$ agents, ranging from 10 to 150. We provide two
attributes to each agent: stance and susceptibility. Stance, $y$, is initially
1, reflecting a state of consensus. Susceptibility, $s$ is a random variable
drawn from a Normal distribution; $s\sim N(0.1,0.1)$. Next we choose a
Confederate selection strategy, the percentage of agents to choose as
Confederates, and a Perturbation strategy which defines the evolution of each
Confederates stance. Confederate agent susceptibility is set to zero.

Finally, we run the simulation until convergence; that is, the point where the
mean stance change of all $N$ non-Confederate agents over the previous 30
timesteps is less than 0.001. At that point, the simulation is terminated.

To measure the optimality of Confederate strategies, we calculate
$\hat{\mu}_{y}$, the mean stance of non-Confederate agents at convergence. As
such lower is better, indicating the success of Confederate perturbations in
driving network stance towards -1.

###  4.1 Virtual Experiments Setup

Our virtual experiment setup, detailing the independent, dependent and control
variables are summarized in Table 3.

Table 3: Definitions of Terminology and Symbols used in this study Variable | type | Range | Value | Number
---|---|---|---|---
Number of agents, $N$ | Independent | [10, 150] | 10, 20, 30… 150 | 14
Percent Confederates | Independent | [0, 100] | 5, 10, 15,…, 40 | 8
Perturbation strategy | Independent |  | Equations 3, 4, and 6 | 3
Agent selection strategy | Independent |  |  $W_{max},A_{min},$ Random | 3
Mean stance at convergence, $\hat{\mu}_{y}$ | Dependent | [-1.0, 1.0] | [-1.0, 1.0] | $\mathbb{R}$
Convergence timestep | Dependent | $\mathbb{N}$ | [50, 200] | $\mathbb{N}$
Initial stance, $y$ | Control | [-1.0, 1.0] | -1, 1 | 2
Susceptibility, $s$ | Control | [0, 1.0] | $s\sim N(0.1,0.1)$ | $\mathbb{N}$
Influence update rate, $\lambda$ | Control | [0, 1.0] | 0.01 | 1
Stance update rate, $\alpha$ | Control | [0, 1.0] | 0.001 | 1

##  5 Results and Discussion

We detail four key results; convergence to a polarized state, optimal
Perturbation strategies, optimal Confederate selection strategies, and the
observation of tipping points.

#### Convergence to a state of stable polarization

We first observe that stances in a network do eventually come to convergence.
Figure 2 illustrates the change in agent stances across time for an 80-node
network, where each line represents an agent. The stances in the simulation
eventually converge into one of the two extremes, 1.0 and -1.0.

Figure 2: Simulation of the change in agent stances for an 80-node network.
Each line represents an agent’s stance over time. Stances eventually converge
into one of two extremes, 1 and -1.

#### Cascade is optimal

In examining the perturbation strategies against network size and means
stances, we observe that the optimal Perturbation strategy is the cascade
strategy. That is, an optimal strategy for the changing of stances involves
the nudging of agents that are in a Confederate agent’s direct neighborhood.
This is illustrated in Figure 4, where the line of mean stance for the cascade
strategy is the lowest (best). We also note that the conversion strategy fares
the worst out of the three strategies, indicating that global nudging
strategies are not optimal in general and require precise targeting.

#### Influential Confederates are best

Out of the three Confederate selection strategies, influential agents are the
best Confederates. This is observed in Figure 4, where Confederate agent
selection strategies are plotted against each other. We also note a clear
trend of diminishing returns where Confederate strategies are less successful
as the network size increases. This hints at the underlying resilience of
scale free networks.

Figure 3: Comparison of the perturbation strategies, lower is better. The
cascade strategy is optimal.

Figure 4: Comparison of the agent selection strategies, lower is better.
Influential Confederates are optimal.

#### Minority stance tipping points exist

We observe that Confederate agents can create a tipping point wherein the
network converges to the stance of the minority Confederate agents. Figure 5
shows tipping point ranges between 20-25% for the three different Perturbation
strategies applied, mirroring empirical evidence from Centola et. al. [4]. The
cascade strategy requires the least Confederates to cause the tip, while the
conversion strategy requires the most Confederates to tip the overall stance.
This mirrors the findings for the stance perturbation experiment (Figure 4).

Figure 5: With $>$20-25% of network agent as Confederates, mean stance shifts
rapidly. Results are averaged over five 80-node networks, using the maximum
influence selection strategy.

##  6 Validation

Our results match stylized facts that can be used to determine known
phenomena. States of consensus are reached when users achieve social
conformity with each other [11, 2, 3], which is observed through stance
convergence (Figure 2) and tipping points (Figure 5) within our work. This is
also observed within the principles of homophily, giving rise to the adage
“birds of a feather flock together".

We find that influential people are best positioned to affect other people’s
stances (Figure 4). Studies on targeted attacks on scale-free networks have
shown that higher degree nodes have larger effects on the overall network and
our experimental results provide the same observation [12, 6]. Similarly, the
optimal perturbation strategy in our model is to begin by effecting a change
towards people that are close to you, i.e. your ego-network (Figure 4). This
is in-line with the theory of information cascades [6].

Finally, the tipping points observed in our experimental runs are similar to
previous research in the literature [4]; 20-25% of the network are required as
Confederates to change the stance of the network. These results lend our model
theoretic validity and face validity.

##  7 Limitations and Future Work

Validation is challenging; it is clearly unethical to precipitate consensus
change in a real social network. A lack of real-world data further complicates
the matter, as observations of stance changes in real-world networks are
extremely rare. Ng et. al. [10] find that only 1% of Twitter users changed
their stances towards the Coronavirus vaccine over a one-year period. Most
Twitter users maintained the same stance. An area of future research is to
identify and process real event data with stance and opinion annotation change
across time and adjust our formulation to the observed data.

Despite these limitations, it is important to understand the circumstances
under which agents in a social situation will change their opinion, which has
implications for persuasion and disinformation research. Differences in social
networking platforms may play a role; where account histories are easily
accessible, for instance, modeling suspicion as a function of stance
volatility would counteract certain Confederate strategies. To account for
this, Equation 2 could be modified to incorporate a regularization parameter
that penalizes highly volatile agents. Additionally, experimenting with other
models for the influence network such as Small World, Erdos-Renyi, and Core
Periphery, may explain how network structure affects the success of the
Cascade strategy.

Given the emergence of stable polarized states within this model, perhaps the
most natural and pertinent avenue for future work is the recovery from such a
state. A body of theory, and in particular Krackhardt’s notion of simmelian
ties, supports such an effort [7]. The core building block of our model is
that of the dyadic tie. Extending this model to capture triadic relationships
would enable the interrogation of the role non-prejudiced third parties play
in recovering from the polarized state.

##  8 Conclusion

In this work, we formulate and test a co-evolutionary social influence model
to simulate intentional stance perturbation in scale-free networks. We define
three strategies for selecting a set of Confederates that will perturb the
network: maximum influence, minimum susceptibility, and a random selection for
a control experiment. We design three Perturbation strategies that
Confederates use to probe the network: conservative, conversion, and cascade
strategies. Our results show that influential agents are the best choice of
Confederate, that the optimal Perturbation strategy involves the targeted
nudging of local ego networks and that there exists a range of tipping points
for group consensus. We hope that this simulation sheds light on the
effectiveness of intentional stance change within a social network and the
manner in which successful social influence operations are run.

##  9 Acknowledgements

This work was supported in part by the Office of Naval Research grant
(N000141812106) and the Knight Foundation. Additional support was provided by
the Center for Computational Analysis of Social and Organizational Systems
(CASOS) at Carnegie Mellon University. The views and conclusions in this
document are those of the authors and should not be interpreted as
representing the official policies, either expressed or implied, of the Knight
Foundation, Office of Naval Research, or the U.S. Government.

## References

  * [1] Broido, A.D., Clauset, A.: Scale-free networks are rare. Nature communications 10(1), 1–10 (2019)
  * [2] Carley, K.M.: Group stability: A socio-cognitive approach. Advances in group processes 7(1), 44 (1990)
  * [3] Carley, K.M., Martin, M.K., Hirshman, B.R.: The etiology of social change. Topics in Cognitive Science 1(4), 621–650 (2009)
  * [4] Centola, D., Becker, J., Brackbill, D., Baronchelli, A.: Experimental evidence for tipping points in social convention. Science 360(6393), 1116–1119 (Jun 2018). https://doi.org/10.1126/science.aas8827
  * [5] Friedkin, N.E., Johnsen, E.C.: Social influence and opinions. Journal of Mathematical Sociology 15(3-4), 193–206 (1990), publisher: Taylor & Francis
  * [6] Jalili, M., Perc, M.: Information cascades in complex networks. Journal of Complex Networks 5(5), 665–693 (2017)
  * [7] Krackhardt, D.: The ties that torture: Simmelian tie analysis in organizations. Research in the Sociology of Organizations 16(1), 183–210 (1999)
  * [8] Lanham, M.J., Morgan, G.P., Carley, K.M.: Social network modeling and agent-based simulation in support of crisis de-escalation. IEEE Transactions on Systems, Man, and Cybernetics: Systems 44(1), 103–110 (2013)
  * [9] Macy, M.W., Kitts, J.A., Flache, A., Benard, S.: Polarization in dynamic networks: A Hopfield model of emergent structure. Dynamic Social Network Modeling and Analysis pp. 162–173 (2003), washington DC: National Academies Press
  * [10] Ng, L.H.X., Carley, K.M.: Pro or Anti? a Social Influence Model of Online Stance Flipping. IEEE Transactions on Network Science and Engineering pp. 1–18 (2022). https://doi.org/10.1109/TNSE.2022.3185785
  * [11] Rajabi, A., Gunaratne, C., Mantzaris, A.V., Garibay, I.: On countering disinformation with caution: Effective inoculation strategies and others that backfire into community hyper-polarization. In: Social, Cultural, and Behavioral Modeling: 13th International Conference, SBP-BRiMS 2020, Washington, DC, USA, October 18–21, 2020, Proceedings 13. pp. 130–139. Springer (2020)
  * [12] Ross, B., Pilz, L., Cabrera, B., Brachten, F., Neubaum, G., Stieglitz, S.: Are social bots a real threat? An agent-based model of the spiral of silence to analyse the impact of manipulative actors in social networks. European Journal of Information Systems 28(4), 394–412 (Jul 2019). https://doi.org/10.1080/0960085X.2018.1560920
  * [13] Will, M., Groeneveld, J., Frank, K., Müller, B.: Combining social network analysis and agent-based modelling to explore dynamics of human interaction: A review. Socio-Environmental Systems Modelling 2, 16325 (Feb 2020). https://doi.org/10.18174/sesmo.2020a16325, https://sesmo.org/article/view/16325