# Training-Free Pretrained Model Merging

Zhengqi Xu1, Ke Yuan1, Huiqiong Wang2, Yong Wang3, Mingli Song1, Jie Song1∗
1Zhejiang University
2Ningbo Innovation Center, Zhejiang University
3State Grid Shandong Electric Power Company
{xuzhengqi,keyuan,huiqiong_wang,brooksong,sjie}<EMAIL_ADDRESS>
###### Abstract

Recently, model merging techniques have surfaced as a solution to combine
multiple single-talent models into a single multi-talent model. However,
previous endeavors in this field have either necessitated additional training
or fine-tuning processes, or require that the models possess the same pre-
trained initialization. In this work, we identify a common drawback in prior
works w.r.t. the inconsistency of unit similarity in the weight space and the
activation space. To address this inconsistency, we propose an innovative
model merging framework, coined as merging under dual-space constraints
(MuDSC). Specifically, instead of solely maximizing the objective of a single
space, we advocate for the exploration of permutation matrices situated in a
region with a unified high similarity in the dual space, achieved through the
linear combination of activation and weight similarity matrices. In order to
enhance usability, we have also incorporated adaptations for group structure,
including Multi-Head Attention and Group Normalization. Comprehensive
experimental comparisons demonstrate that MuDSC can significantly boost the
performance of merged models with various task combinations and architectures.
Furthermore, the visualization of the merged model within the multi-task loss
landscape reveals that MuDSC enables the merged model to reside in the
overlapping segment, featuring a unified lower loss for each task. Our code is
publicly available at https://github.com/zju-vipa/training_free_model_merging.

11footnotetext: Corresponding author.

##  1 Introduction

Great success has been achieved on various challenging tasks in computer
vision by using deep neural networks, and a plethora of deep neural networks
are developed and released publicly, with either their architectures and
trained parameters (_e.g_., Pytorch Hub111https://pytorch.org/hub/, Hugging
Hub222https://huggingface.co/HUB). These off-the-shelf models are finely-tuned
for a wide range of tasks, providing users with substantial convenience.
However, these models are restricted to the tasks they were trained on, and
this limitation poses significant challenges in terms of model storage [11,
63] and computation as the parameters of models continue to grow rapidly.

Figure 1: We use an intermediate layer of ResNet50 [16] to construct two
groups of units as two parents, and then merge them pairwise. In the figure,
each point represents a merged unit. The x-axis and the y-axis denote the
weight and activation similarity between their two parents respectively. The
color indicates the smaller activation similarity between them and their two
parents. The star depicts the best merged unit which is most activation-
similar to its parents. More details are provided in the supplementary
material.

Given the abundance of well-trained models across various tasks, numerous
research studies propose the amalgamation of multiple models into a single
model that possesses multiple functionalities simultaneously. The existing
body of literature can be broadly classified into two schools: training-based
knowledge amalgamation [37, 56, 30, 54, 31, 45, 20, 7, 14, 46] and training-
free model merging [18, 52, 53]. The former is inspired by knowledge
distillation and adopts the outputs from multiple pretrained teachers to train
a student model, which amalgamates the knowledge from multiple models into a
single one. However, these methods usually require an additional expensive
training process to facilitate the transfer of knowledge from the teacher
models to the student model. In contrast, The latter fully utilize the
parameters of the pretrained models to construct the multi-talent model,
thereby avoiding additional training. The mainstream involves constructing
multi-task models by adding task vectors, which are computed by subtracting
the pre-trained model’s parameter values from those of the fine-tuned model,
together. However, these methods can only merge fine-tuned models derived from
the same pre-trained initialization, that is an extremely limited
prerequisites. Emerging work w.r.t. unit matching [1, 22, 40, 44] shows
promise to breaking the extra-training and pre-training curse, which has
already reached zero-loss barriers in single-task model merging [1]. However,
the permutation invariance [10], a cornerstone for most unit matching methods,
seems not applicable to merge models from multiple tasks and leads to a poor
artifact.

In this work, we have cast doubt upon the matched units identified by existing
unit matching algorithms. Reviewing prior work, current unit matching methods
either represent units using weights or activation values [27]. Nonetheless,
using either weights or activation values as the matching criterion is
incomplete. In terms of weights, the weight of a unit reflects its connection
strength with the units in the previous layer, but it cannot capture the
information of specific features (i.e. scale, offset) due to the absence of
input data. As shown in Fig. 1, there are many pairs of units with highly
similar weights but entirely different activation values, leading to a
significant change in the function of their merged units. In terms of
activations, it is compelling to measure units by their activations since two
models must learn similar features to accomplish the same task [28], yet the
same task can be achieved by distinct parameters [12]. Moreover, a phenomenon
is shown in Fig. 1 that the best-merged unit is one whose parents
simultaneously exhibits high similarity in both weight and activation.

Motivated by this, we propose a concise and effective model merging framework,
termed merging under dual-space constraints(MuDSC) to achieve a more precise
matching of relevant units. Specifically, the proposed MuDSC constrains the
matching algorithm to find more suitable matches by linearly combining the
similarity matrix of representations of units in both activation space and
weight space. For the sake of rigor, we provide a formula derivation to prove
the equivalence of this approach with a linear combination of optimization
objectives in separate spaces. Moreover, we devise a novel iterative algorithm
that simultaneously maximize the global similarity of matched units in both
weight space and activation space. In addition, we devise two variant of
matching algorithms proposed in [1] and [43] to support the use of unit
matching in networks with a group structure i.e. ViT [9] and Group
Normalization [50].

In summary, the key contributions of this paper are:

  1. 1.

We highlight the inconsistency of unit similarity in weight space and
activation space, demonstrating through a straightforward experiment the
impact of this inconsistency on model merging.

  2. 2.

To take into account this inconsistency, we propose a novel model merging
framework MuDSC (Merging under Dual-Space Constraints) that encourages both
the activation spaces and weight spaces between two models be of high
similarity.

  3. 3.

We present a vigorous experimental study and show that MuDSC can significantly
boost the merged performance in manifold multi-task scenarios for various
architectures. Further loss landscape visualization verifies that our solution
locates in the overlapping part with unified lower loss of each task.

##  2 Related Work

Training-based model merging. In order to reuse trained models and reduce the
cost of training the new models from scratch, Shen et al. [37] propose
knowledge amalgamation (KA), the goal of which is to learn a versatile student
model from multiple task-specific teachers, but it is only verified on the
comprehensive classification tasks. Following the knowledge amalgamation, Ye
et al. [56] validate the effectiveness of KA in complex scenarios such as deep
estimation and scene analysis, and in another work [57], they customized
student models by amalgamating filtered knowledge from different teachers,
even surpassing the those teachers in certain cases. There are more learning
strategies have been proposed to enhance the performance [30, 38, 31] and
usability [58, 20, 61, 14, 21] of KA. Although the success achieved by KA, a
completely re-parameterized student model evidently leads to inefficient model
merging. In contrast, model stitching [26] more efficiently leverages pre-
trained models. model stitching aims to “plug-in” the bottom layers of one
network into the top layers of another network, thus forming a stitched
network [2, 6]. For instance, Yang et al. [55] introduce a novel two-stage
strategy for reassembling customized networks from a zoo of pre-trained models
under user-specified constraints, and Pan et al. [34] propose Stitchable
Neural Networks for elastic deep learning by directly utilising the pretrained
model families in model zoo via model stitching. Regrettably, the fine-tuning
process is still necessary for bridging the gaps between pre-trained blocks in
model stitching. Unlike KA and model stitching, we do not involve any
additional training or fine-tuning processes in our merging methods.

Training-free model merging. Considering the data privacy and the costly
retraining for expanding model capabilities, recent efforts have sparked
significant interest in exploring how to merge models from multiple tasks into
the single multi-task model without additional training [19, 18]. Ilharco et
al. [18] present a paradigm for editing neural networks, namely Task
Arithmetic, that constructs multi-task models with performing simple addition
operations on task vectors. Expanding on this groundwork, a series of work
expand the scope of the task arithmetic framework [62] or solve the potential
issues of task vector-based method [52, 53]. However, task vector-based
methods can only be applied to models fine-tuned from the same pre-trained
initialization, and this assumption severely restricts the applicability of
model merging. Fortunately, unit matching technologies show promise in
relaxing this assumption [1]. Guided by the existence of the permutation
symmetries of hidden units in neural networks [17, 13, 10, 15, 39, 4, 44],
unit matching aims at mapping all models to the same basin by permutating
neuron units, Thought we find it widely applied for merging models in the
single-task scenarios [59, 48, 40, 1, 22], single-task zero-loss barrier
merging [1] highlights the immense potential of unit matching in combining
models from multiple task-specific tasks. Current unit matching methods
primarily consider two types of matching spaces [27]: one is the activation
space[43, 44, 40, 42, 33, 3, 1, 28], and the other is the weight space [48,
59, 1, 51, 35, 24]. Nevertheless, we demonstrate through a straightforward
experiment that the similarity of units in the activation space and weight
space is not always consistent, and these discrepancies manifest various pros
and cons during model merging. Furthermore, simultaneously considering the
similarity of models in both weight space and activation space can offer
potential advantages in reducing merging barriers. Therefore, our method
simultaneously constrains unit matching in both the weight space and
activation space.

##  3 Methodology

In Sec. 3.1, we introduce a general procedure of merging model via unit
matching. Afterward, we provide the theoretical basis for merging models under
dual-space constraints and then elaborate on how to merge models under dual-
space constraints in Sec 3.2. Two solutions for adaption to group structures
are presented in Sec. 3.3.

###  3.1 Preliminaries

Assume there are $N$ pre-trained models in the same architecture. The $n$-th
model is parameterized by weights $\\{\boldsymbol{W}_{l}^{(n)}\\}_{l=1}^{L}$,
where $L$ represents the number of layers of the models. Without loss of
generality, we omit the bias term in parameters and simply regard the model as
MLP model, so $\boldsymbol{W}_{l}^{(n)}\in\mathbb{R}^{D_{l}D_{l-1}}$, where
$D_{l}$ and $D_{l-1}$ represent the number of units in $l$-th layer and
$(l-1)$-th layer, respectively.

The main idea behind model merging revolves around the fusion of units that
exhibit significant similarity, aiming to maximize the cumulative global
similarity of the merged units within each layer. The unit parameters after
merging are obtained by averaging the unit parameters before merging. The
solution can be summarized as computing the maximum-weighted matching of the
graph, where the nodes refer to the units and the edges refer to their
similarities. The goal is to find a subset of edges with the maximum weight,
and in which no node occurs more than $N$ times. Formally, in the $l$-th
layer, let $\mathbb{C}_{l}\in\mathbb{R}^{(N\cdot D_{l})(N\cdot D_{l})}$ be the
similarity matrix of merging a pair of units in the N models, where each
element $(\mathbb{C}_{l})_{i,j}$ denotes the similarity of the $i$-th unit and
the $j$-th unit, _i.e_., the weights of edges as mentioned above. Then the
goal of model merging can be formulated as follows:

$\displaystyle\arg\max_{\mathbb{P}_{l}}$
$\displaystyle\sum_{l=1}^{L}\langle\mathbb{C}_{l},\mathbb{P}_{l}\rangle,~{}\mathrm{s.t.}\sum_{i=1}^{N\cdot
D_{l}}(\mathbb{P}_{l})_{i,j}\leq N~{}\forall~{}j,$ (1)
$\displaystyle(\mathbb{P}_{l})_{i,j}=1\implies(\mathbb{P}_{l})_{i,k}=(\mathbb{P}_{l})_{j,k},\forall~{}i,j,k$
$\displaystyle(\mathbb{P}_{l})_{i,i}=1~{}\forall~{}i.$

$\mathbb{P}_{l}$ denotes the merging matrix which is symmetrical and binary,
and where $(\mathbb{P}_{l})_{i,j}=1$ represents that the $i$-th unit and
$j$-th unit are merged together.
$\langle\boldsymbol{X},\boldsymbol{Y}\rangle=\sum_{i,j}\boldsymbol{X}_{i,j}\boldsymbol{Y}_{i,j}$
denotes the Frobenius inner product between real-valued matrices
$\boldsymbol{X}$ and $\boldsymbol{Y}$.

To merge the similar units from different models, we can construct a set of
permutation matrices $\\{\boldsymbol{P}_{l}\\}_{l=1}^{L}$. Then the merge
operation can be formulated as follows:

${\boldsymbol{W}_{l}}^{\prime}=\frac{1}{N}\sum_{n=1}^{N}(\boldsymbol{P}_{l}^{(n)})^{\top}{\boldsymbol{W}_{l}^{(n)}}\boldsymbol{P}_{l-1}^{(n)},~{}l\in\\{1,2,...,L\\}$
(2)

To clarify the relation between $\boldsymbol{P}_{l}^{(n)}$ and
$\mathbb{P}_{l}$, we define
$\boldsymbol{P}_{l}=\boldsymbol{P}_{l}^{(1)}\circ\boldsymbol{P}_{l}^{(2)}\circ...\circ\boldsymbol{P}_{l}^{(N)}$
where $\boldsymbol{P}_{l}^{(n)}\in\mathbb{R}^{D_{l}D_{l}}$ and operation
$\circ$ denotes concatenating the matrices in the first dimension. Then
$\boldsymbol{P}_{l}$ can be straightforwardly derived from $\mathbb{P}_{l}$ by
removing duplicate columns of $\mathbb{P}_{l}$ because of the constraints of
Eq. (1).

###  3.2 Merging under Dual-Space Constraints

As introduced in Section 1, prior works usually measure the unit similarity
within a single space (i.e. the activation space or the weight space),
resulting in inconsistencies between the two spaces. In this work, we assess
unit similarity by considering similarities in both activation space and
weight space. In this study, we propose an innovative merging framework that
addresses disparities between models in both the activation space and weight
space. The fundamental challenge lies in reconciling the distinctions between
weight-based matching and activation-based matching during the matching
procedure.

Given the $\\{\boldsymbol{A}_{l}\\}_{l=1}^{L}$ and
$\\{\boldsymbol{Z}_{l}\\}_{l=1}^{L}$ as the representation vectors of the
$l$-th layer in the activation space and the weight space, respectively, and
with a balanced factor $\alpha\in[0,1]$, the expected objective can be written
as follows:

$\arg\max_{\mathbb{P}_{l}}\alpha\sum_{l=1}^{L}\langle\mathbb{C}(\boldsymbol{Z}_{l}),\mathbb{P}_{l}\rangle\\\
+(1-\alpha)\sum_{l=1}^{L}\langle\mathbb{C}(\boldsymbol{A}_{l}),\mathbb{P}_{l}\rangle$
(3)

Here, $\mathbb{C}:\mathbb{R}^{(N\cdot
D_{l})R_{l}}\rightarrow\mathbb{R}^{(N\cdot D_{l})(N\cdot D_{l})}$ denote the
function that calculate the similarity matrix by $\boldsymbol{A}_{l}$ or
$\boldsymbol{Z}_{l}$. Specially, $R_{l}$ refers to the quantity of related
paramaters when it comes to $\boldsymbol{Z}_{l}$, or $R_{l}$ training data
points when it comes to $\boldsymbol{A}_{l}$. We omit the constraint
conditions here. Then we can re-express Eq. (3) as follows:

$\arg\max_{\mathbb{P}_{l}}\sum_{l=1}^{L}\langle\alpha\mathbb{C}(\boldsymbol{Z}_{l})+(1-\alpha)\mathbb{C}(\boldsymbol{A}_{l}),\mathbb{P}_{l}\rangle$
(4)

Manifestly, the aforementioned derivation demonstrates that the dual-space
constraints can be realized through a linear combination of the similarity
matrices in both spaces. This formulation presents an efficacious approach to
address unit matching under the dual-space constraints. The remaining question
is how to implement Eq. (4) in the matching procedure. To this end, we delve
into the solutions within two distinct spaces.

In term of the activations, activation-based matching can be solved in a
single-pass. This procedure is described as:

$\boldsymbol{P}_{l}=\Psi(\mathbb{C}(\boldsymbol{A}_{l})),l\in\\{1,2,\dots,L\\}$
(5)

Here, $\Psi(\cdot)$ is the function that derives the permutation matrix by Eq.
(1) from the similarity matrix.

In the realm of weight space, there are three distinct perspectives on how to
obtain representation vectors. One viewpoint advocates the use of output
vectors in the next layer to represent the units [59]. Another approach leans
towards utilizing the weights of the current layer [48, 41]. An advanced
solution involves considering all weights associated with the layer, including
those from the preceding and subsequent layers. Such a method has achieved
zero-loss barriers in the merging of single-task models [1]. In our work, we
adopted the third method to obtain representations of units in the weight
space. Consequently, for $n$-th model, the representation vectors in the
$l$-th layer is conducted by:

$\boldsymbol{Z}_{l}^{(n)}=\boldsymbol{W}_{l}^{(n)}\boldsymbol{P}_{l-1}^{(n)}\|\boldsymbol{P}_{l+1}^{(n)}(\boldsymbol{W}_{l+1}^{(n)})^{\top}$
(6)

where $\|$ denote the operation which concatenates the given matrices in the
second dimension. Note that aforementioned $\boldsymbol{W}_{l}$ and
$\boldsymbol{Z}_{l}$ contain the units of all models in the $l$-th layer like
$\boldsymbol{P}_{l}$ and can be expressed as
$\boldsymbol{W}_{l}^{(1)}\circ...\circ\boldsymbol{W}_{l}^{(N)}$ and
$\boldsymbol{Z}_{l}^{(1)}\circ...\circ\boldsymbol{Z}_{l}^{(N)}$, respectively.

There is a contradiction that a valid $\boldsymbol{Z}_{l}$ for matching in
$l$-th layer requires the units in the $(l+1)$-th and $(l-1)$-th layers to
reach the optimal match. This contradiction can be resolved through an
iterative algorithm, and its recursive expression is as follows:

$\begin{cases}\boldsymbol{P}_{l}^{1}=\mathbb{I}\\\
{\boldsymbol{Z}_{l}}^{t+1}=\upsilon(\boldsymbol{P}_{l-1}^{t},\boldsymbol{P}_{l+1}^{t},\boldsymbol{W}_{l}^{t},\boldsymbol{W}_{l+1}^{t})\\\
\boldsymbol{P}_{l}^{t+1}=\Psi(\mathbb{C}({\boldsymbol{Z}_{l}}^{t+1}))\end{cases},$
(7)

where $l\in\\{1,2,\dots,L\\}$ and $t\in\mathbb{N}^{+}$. When the number of
iterations approaches infinity, the model reaches its optimal match in weight
space. In Eq. (7), $\mathbb{I}$ is a permutation matrix where the submatrix of
each model is identity matrix. And $\upsilon(\cdot,\cdot,\cdot,\cdot)$ is the
function that applies permutation matrices to the weights of all models by Eq.
(6) and then concatenate all the $\boldsymbol{Z}_{l}^{(n)}$ into
$\boldsymbol{Z}_{l}$.

Afterwards, it is evident to implement the merging method under the dual-space
constraints based on Eq. (4). We substitute Eq. (4) and Eq. (5) into Eq. (7),
yielding:

$\begin{cases}\boldsymbol{P}_{l}^{1}=\Psi(\mathbb{C}(\boldsymbol{A}_{l}))\\\
{\boldsymbol{Z}_{l}}^{t+1}=\upsilon(\boldsymbol{P}_{l-1}^{t},\boldsymbol{P}_{l+1}^{t},\boldsymbol{W}_{l}^{t},\boldsymbol{W}_{l+1}^{t})\\\
\boldsymbol{P}_{l}^{t+1}=\Psi(\alpha\mathbb{C}({\boldsymbol{Z}_{l}}^{t+1})+(1-\alpha)\mathbb{C}(\boldsymbol{A}_{l}))\end{cases},$
(8)

Lastly, we present the overall procedure of our MuDSC merging in Alg. 1. We
further illustrate the convergence and the complexity of Alg. 1 in the
supplementary material.

Algorithm 1 MuDSC Merging

0: The weights $\\{\boldsymbol{W}_{l}\\}_{l=1}^{L}$ and the activations
$\\{\boldsymbol{A}_{l}\\}_{l=1}^{L}$ of the models, the function for computing
the similarity matrix $\mathbb{C}$, selected matching algorithm $\Psi$, the
balanced factor $\alpha$. $RP$ returns a random permutation of the sequence.

for $l=1,2,\dots,L$ do

${\boldsymbol{C}_{l}}^{\prime}\leftarrow\mathbb{C}(\boldsymbol{A}_{l})$

$\boldsymbol{P}_{l}\leftarrow\Psi({\boldsymbol{C}_{l}}^{\prime})$

end for

repeat

for $l=RP(1,2,\dots,L)$ do

${\boldsymbol{Z}_{l}}\leftarrow\upsilon(\boldsymbol{P}_{l-1},\boldsymbol{P}_{l+1},\boldsymbol{W}_{l},\boldsymbol{W}_{l+1})$

${\boldsymbol{C}_{l}}^{\prime}\leftarrow\alpha\mathbb{C}(\boldsymbol{Z}_{l})+(1-\alpha){\boldsymbol{C}_{l}}^{\prime}$

$\boldsymbol{P}_{l}\leftarrow\Psi({\boldsymbol{C}_{l}}^{\prime})$

end for

until convergence

Get the merged weights $\\{\boldsymbol{W}_{l}^{\prime}\\}_{l=1}^{L}$ by Eq.
(2)

return $\\{\boldsymbol{W}_{l}^{\prime}\\}_{l=1}^{L}$

###  3.3 Adaptation to Group Structure

As mentioned previously, the abstractions of current matching algorithm do not
generalize to networks with the following structure:

  1. 1.

External Permutation Invariance: There are $g$ separate groups which can be
permuted with each other. For instance, Group Normalization [50] divides
channels into $g$ groups and normalizes the features within each group.

  2. 2.

Internal Permutation Invariance: Each of those $g$ groups contains $k$ units
which can be permuted. For instance, the features within each group of Group
Normalization can be permuted.

This structure is commonly encountered within contemporary neural network
modules, in particularly Group Normalization [50] and Multi-Head Attention
[47]. Consequently the application of matching algorithms will face
significant hindrance if they cannot adapt to this structure. Here, we propose
viable adaptation strategies for a alignment algorithm based on linear sum
assignment [1] (namely group alignment) and the greedy algorithm used in zip
operation [43] (namely group zip).

Among them, group alignment first performs internal alignment of pairwise
groups and then conducts external alignment based on the average similarity
obtained from the internal alignment. We provide an illustrative diagram in
Figure 2. While this algorithm has quadratic time complexity when it comes to
external matching, we have found this cost to be acceptable for the
application in our experiments. In addition, it is worth noting that this
algorithm is capable of achieving a global optimal match.

Figure 2: Left. Examples of vanilla structure and group structure.Right. An
example of group alignment. First, we calculate the similarity between units.
Next, we compute permutation and then calculate the average of matched
similarity within each pairs of groups. Finally, we compute permutation for
each pairs of groups and then set the permutation of unmatched pairs to zeros.

For group zip, unlike alignment-based approaches, zip operation employs a
greedy matching algorithm to merge similar units. However, this method is
challenging to extend to group merging. In response to this, we propose a
simplified matching scheme. We initially determine which groups to merge
together through a group matching strategy, and then we "zip" these groups
individually. As finding a suitable grouping strategy is a challenging task,
we did not introduce a efficient grouping strategy in this work, leaving the
task for future work. In subsequent experiments, we directly use the group
matching results of the group alignment as the outcome of our grouping
strategy, enabling a fair performance comparison between the alignment methods
and the zip methods.

##  4 Experiments

We first validate the superiority of MuDSC to existing methods in merging
models of homogenous tasks (_i.e_., classification tasks with different sets
of categories) on both the small- and the large-scale datasets (Section 4.1).
Then, we verify the effectiveness of MuDSC in merging models of heterogenous
tasks (_e.g_., segmentation and classification) with the pre-trained models
released in Taskonomy[60] (Section 4.2). Finally, we visualize multi-task loss
landscape to further demonstrate the superiority of MuDSC over prior single-
space methods (Section 4.3).

Model | Resnet20 | Resnet20GN
---|---|---
Dataset | CIFAR100(50+50) | CIFAR10(5+5) | CIFAR100(50+50)
Method | Joint | Avg | T. A | T. B | Joint | Avg | T. A | T. B | Joint | Avg | T. A | T. B
Average | 16.52 | 24.22 | 23.22 | 25.21 | 54.42 | 75.24 | 79.58 | 70.90 | 5.63 | 11.06 | 9.67 | 12.44
Rebasin | 41.33 | 56.94 | 57.31 | 56.58 | 60.61 | 88.57 | 88.46 | 88.68 | 13.85 | 22.18 | 22.99 | 21.37
A. Align | 44.33 | 61.13 | 61.61 | 60.66 | 61.71 | 89.21 | 88.63 | 89.78 | 29.37 | 42.05 | 41.05 | 43.05
MuDSCAlign | 45.50 | 62.81 | 63.06 | 62.56 | 60.84 | 89.34 | 89.04 | 89.63 | 31.84 | 45.31 | 45.34 | 45.29
Zipit | 54.69 | 66.78 | 67.11 | 66.44 | 82.44 | 94.61 | 94.22 | 95.00 | 29.93 | 41.20 | 39.99 | 42.41
W.Zip | 55.16 | 67.65 | 68.58 | 66.71 | 82.85 | 94.71 | 94.42 | 94.99 | 14.28 | 20.95 | 19.17 | 22.72
MuDSCZip | 56.01 | 68.13 | 68.80 | 67.47 | 83.09 | 94.88 | 94.56 | 95.21 | 30.05 | 41.52 | 40.39 | 42.65
Table 1: The joint accuracy and the per-task accuracy of the merged multitask
model. Two original models are trained from scratch on two subtasks of the
same classification task. We emphasize the data achieving the best accuracy.
Std is provided in the supplementary material.

###  4.1 Merging Models of Homogeneous Tasks

####  4.1.1 Experimental Settings

Model | Resnet26 | Resnet50GN | ViT
---|---|---|---
Method | Joint | Avg | T. A | T. B | Joint | Avg | T. A | T. B | Joint | Avg | T. A | T. B
Average | 61.44 | 74.75 | 74.46 | 75.05 | 74.52 | 84.78 | 85.06 | 84.50 | 70.16 | 84.32 | 84.32 | 84.32
Rebasin | 61.39 | 74.79 | 74.48 | 75.10 | 74.52 | 84.78 | 85.06 | 84.50 | 70.16 | 84.32 | 84.32 | 84.32
A. Align | 61.91 | 75.41 | 75.03 | 75.79 | 74.44 | 84.77 | 84.99 | 84.56 | 69.99 | 84.22 | 84.20 | 84.24
MuDSCAlign | 62.84 | 76.14 | 75.87 | 76.40 | 74.66 | 84.91 | 85.25 | 84.58 | 70.09 | 84.39 | 84.38 | 84.40
Zipit | 60.23 | 73.68 | 73.20 | 74.17 | 72.05 | 82.99 | 83.06 | 82.92 | 68.57 | 83.05 | 82.79 | 83.30
W.Zip | 61.28 | 74.69 | 74.42 | 74.96 | 74.52 | 84.78 | 85.06 | 84.50 | 70.16 | 84.32 | 84.32 | 84.32
MuDSCZip | 61.58 | 75.01 | 74.61 | 75.41 | 74.71 | 84.88 | 85.14 | 84.62 | 70.10 | 84.38 | 84.41 | 84.36
Table 2: The joint accuracy and the per-task accuracy of the merged multitask model. Two original models that well-pretrained on ImageNet are fintuned on two subtasks of CIFAR100. We emphasize the data achieving the best accuracy. Std is in the supplementary material. Method | Joint Acc | Avg Acc | T. A | T. B
---|---|---|---|---
Average | 44.85 | 62.31 | 61.93 | 62.68
Rebasin | 44.85 | 62.31 | 61.93 | 62.68
A. Align | 44.86 | 62.62 | 62.56 | 62.67
MuDSCAlign | 44.87 | 62.66 | 62.57 | 62.74
Zipit | 44.22 | 62.25 | 62.23 | 62.26
W.Zip | 44.85 | 62.31 | 61.93 | 62.68
MuDSCZip | 44.86 | 62.59 | 62.45 | 62.72
Table 3: Results on ImageNet400(200+200). Merging ResNet-50 models trained
with DINO pretrained backbone on disjoint 200 category subsets (Task A and B)
of ImageNet-1k.

Datasets and models The experiments are conducted on the small-scale CIFAR-10
[23], CIFAR-100 [23] and the large-scale ImageNet [8]. In the selection of
models for merging, we emphasize the initial state of the model and whether it
has a group structure i.e. Multi-Head Attention [9] and Group Normalization
[50]. These two aspects of configuration significantly reflect the
effectiveness and applicability of the merging algorithm. For randomly
initialized models, ResNet-20 [16] and ResNet-20 with Group Normalization [50]
are selected. Additionally, we choose several pre-trained models from PyTorch
Image Models (timm) [49], including ResNet-26 [16], ResNet-50 with Group
Normalization[50], ViT-Small [9], DINO-Small [5] and Swin-Tiny [29].

Baselines. Our method is applied to two different types of matching algorithms
referred to as alignment-based matching and zip-based matching. We compare our
method with the current state-of-the-art methods of these two types. For
alignment-based matching, the algorithm aligns the units of all models with
those of the first model among them [1]. We implement a alignment-based
version of our method MuDSCAlign and compare it with Git Rebasin [1].
Additionally, we incorporated an activation-based alignment
algorithm(abbreviated as A. Align). For zip-based matching, the units from
different models are “zipped” together in the same layer which is referred to
as zip operation [43]. Similarly, we implement MuDSCZip based on zip
operation, comparing it to Zipit [43]and a weight-based variant(abbreviated as
W.Zip) implemented by us.

Experimental details. We randomly partition a classification dataset into two
non-overlapping sub-classification tasks, trained respective models for each,
and subsequently merged the models into one. Then we evaluate performance of
merged model with joint accuracy and per-task accuracy. Joint accuracy is the
overall accuracy of a model when it is evaluated on all classes within a
combined dataset. It is similar to a continual learning setting where our
objective is to enhance the knowledge of the model. For per-task accuracy, we
provided the accuracy of the merged multi-task model on two individual tasks,
along with their average performance. Each model is trained with a CLIP-style
loss [36] using CLIP text encodings of the class names as targets. For fair
comparisons, we train 5 pairs of models and report the average accuracy. Three
zip-based methods share the best hyperparameters (mentioned in [43]) on Zipit.
In addition, for each pair of models, we search the balance factor $\alpha$ of
MuDSC for the best per-task accuracy on the training data and then apply it to
test.

####  4.1.2 Results and Analysis

| Method
---|---
Visual Task | Weight Average | Rebasin | Act. Align | MuDSCAlign | Zipit | Weight Zip | MuDSCZip
Class Object | 76.15 | 80.76 | 89.75 |  89.75 +0.00 | 84.84 | 80.93 |  86.04 +1.20
Segment Semantic | 23.01 | 30.59 | 52.54 |  55.24 +2.69 | 32.51 | 33.77 |  36.22 +2.45
Rgb2depth | 95.42 | 95.90 | 98.69 |  98.70 +0.01 | 99.30 | 99.07 |  99.35 +0.05
Rgb2mist | 94.56 | 95.00 | 98.28 |  98.28 +0.00 | 99.08 | 98.59 |  99.15 +0.00
Edge3D | 79.79 | 80.44 | 91.04 | 90.77 -0.27 | 93.22 | 86.42 | 92.81 -0.40
Edge2D | 68.54 | 75.48 | 81.37 | 81.24 -0.12 | 90.39 | 88.85 |  93.09 +2.69
Keypoints2D | 71.33 | 75.89 | 81.99 |  82.30 +0.31 | 93.38 | 93.62 |  94.73 +1.11
Keypoints3D | 91.54 | 91.99 | 96.16 |  96.17 +0.02 | 96.99 | 96.00 | 96.95 -0.04
Reshading | -4.72 | 8.52 | 60.23 |  61.85 +1.61 | 69.18 | 41.66 | 68.71 -0.47
Rgb2sfnorm | 26.62 | 29.43 | 65.46 |  65.56 +0.09 | 76.62 | 64.43 |  77.50 +0.88
Autoencoding | 21.96 | 35.65 | 51.84 |  54.21 +2.37 | 86.55 | 77.58 |  87.99 +1.45
Denoising | 33.92 | 33.57 | 52.37 |  54.40 +2.03 | 84.28 | 76.14 |  85.49 +1.22
Total Average | 56.51 | 61.10 | 76.64 |  77.37 +0.73 | 83.86 | 78.09 |  84.84 +0.98
Table 4: The average scaled performance of multiple heterogenous-task models
after merged with each other. We emphasize the data achieving the best
accuracy and the improvement of our methods.

Results on CIFAR. We first test the performance of the methods on merging
randomly initialized models in Table 1. For experiments with Resnet20 on
CIFAR100 and CIFAR10, the zip-based methods achieve relatively higher merged
accuracy than alignment-based methods and our MuDSCZip achieve further
improvements. In experiments with Resnet20GN, MuDSCAlign outperform all
methods and significantly surpass the second-best method by 8.41%, 7.75%,
10.45%, 5.20% on joint accuracy and three per-task accuracy. Secondly, we
conduct the experiments on CIFAR100 with Resnet26, Resnet50GN and ViT all of
which are well-pretrained on ImageNet [8]. Unlike randomly initialized models,
finetuned models exhibit a high similarity in weight space as they are
initialized with the same pretrained model, making it challenging to discover
better matching results. As observed in Table 2, the results for W. Zip and
Rebasin are completely consistent with those of Average. Another regrettable
fact is that when merging finetuned models, activation-based methods perform
even worse than direct averaging. Fortunately, from Table 2, we can see that
merging models under the dual-space constraints has overcome the obstacles of
the aforementioned single-space methods, achieving further improvements in
merged accuracy. Results of DINO-S and Swin-T (Table LABEL:supplement-
tab:more_vits) also validate the superiority of our method.

Results on ImageNet. To test our method on the large scale data, we train 5
ResNet-50 models initialized with DINO pretrained backbone [5] on disjoint 200
class subsets of ImageNet-1k [8]. Then we conduct experiments on exhaustively
merging pairs from the 5 models. As shown in Tab. 3, the proposed MuDSC still
surpasses existing methods, demonstrating its capability for model merging on
the large-scale data.

###  4.2 Merging Models of Heterogenous Tasks

####  4.2.1 Experimental Settings

Tasks and models. We adopt 12 pre-trained models from Taskonomy [60] trained
on various tasks (including Autoencoder, Denoise, Edge 2D, Edge 3D, Keypoint
2D, Keypoint 3D, Reshade, Rgb2depth, Rgb2mist, Rgb2sfnorm, Segmentation, and
Classification.). The architectures of these models follows an encoder-decoder
scheme, in which the encoder is implemented by fully convolutional layers and
the decoder varies according to the tasks. Please refer to [60] for more
detailed information. In our experiments, only the encoders of these models
are adopted for merging.

Baselines and Metric.  The settings of baselines are the same as that of Sec.
4.1. In order to reflect the gap between the merged model and the original
model more intuitively, we define a scaled performance of model $\theta$,
which is expressed as follows:

$\mathcal{L}_{SP}=\frac{\mathcal{L}_{\theta}-\mathcal{L}_{0}}{\mathcal{L}_{1}-\mathcal{L}_{0}},$
(9)

where $\mathcal{L}_{\theta}$, $\mathcal{L}_{1}$ and $\mathcal{L}_{0}$
represent the original performance metrics for the model $\theta$, the
pretrained model and the average estimator. Notably, when $\mathcal{L}_{SP}$
approaches 1, the performance of the model $\theta$ is similar to the pre-
trained model, and when $\mathcal{L}_{SP}$ approaches 0, the performance of
the model $\theta$ is similar to the average value of the labels. In other
words, we aim for the performance of the merged model to be closer to, or even
better than 1. For the original performance, we maintain consistency with [60]
except for adopting error rate in Segmentation, and we are unable to access
any GAN loss due to the lack of discriminators.

Experimental Details. We merge the adopted pretrained models with each other
and ultimately report the average scaled performance per and all tasks for all
merging methods. Due to the remarkably large size of the complete Taskonomy
dataset, we conduct a reasonable sampling. For evaluation, we used the test
split of official partitioned tiny subset, which contains 54513 samples, to
evaluate the performance of the models. In order to reset the batch
normalization of models and obtain the activations of the model, we sampled
data from training dataset, which included 8 buildings and a total of 61520
samples. Additionally, the entire encoder is used to match but only the layers
before the last layer are merged. In this experiment, we simply adopted 0.5 as
the balanced factor for our MuDSC.

Figure 3: The loss landscape visualization of three zip-based
methods(MuDSCZip, Zipit and W. Zip). Stars, squares, and diamond mark the
positions of the merged models, and black spots mark the positions of their
parent models.

####  4.2.2 Results and Analysis

Tab. 4 provide a quantitative comparison between the proposed MuDSC with
existing works in scaled performance. It can be easily seen that the MuDSC
yields significantly superior performance to existing two types of matching
methods. In alignment-based algorithms, MuDSCAlign maintains or surpasses the
current SOTA in 83% of tasks, Furthermore, in certain tasks i.e. Segment
Semantic, Reshading, Autoencoding, and Denoise, MuDSCAlign brings improvements
of 5.12%, 2.68%, 4.58%, and 3.88%, respectively. Similarly, in zip-based
methods, MuDSC outperforms the current SOTA methods in 75% of tasks, with
improvements exceeding 1% in 8 tasks, notably achieving a remarkable 11.42%
improvement in the Segment Semantic task. Overall, alignment-based methods and
zip-based methods have their respective strengths in merging tasks. Alignment-
based methods excel at maintaining performance in semantic tasks(i.e. Class
Object and Segment Semantic), while zip-based methods have an advantage in
pixel-to-pixel tasks. The reasons for this disparity are worth investigating
further (but is out of role of our method). However, regardless of the merging
methods, MuDSC brings significant improvements to both.

###  4.3 Loss Landscape Visualization

In this subsection, we demonstrate an example of how MuDSC improves the zip-
based method. We conduct visualizations on a multi-task experiment constructed
with MNIST [25]. Two binary classification tasks involve determining whether a
sample is a prime number (i.e. Task A) and whether a sample is odd (i.e. Task
B). The architecture of the model is a four-layer MLP and each model is
trained as CLIP image encoder. The results are reported at Tab.
LABEL:supplement-tab:mt_mnist. Then, we visualize merged models and their
original models obtained by various methods onto a single diagram to
intuitively compare their differences in the flatness of the loss landscape.
To be more specific, we construct a set of high-dimensional vectors obtained
by flattening models’ parameters and use the PCA dimension reduction algorithm
[32] to generate the two-dimensional coordinates. In Fig. 3, there are the
average loss landscape, the loss landscape for Task A, and the loss landscape
for Task B from left to right. Then we mark the positions of activation-based
methods (i.e. Zipit), weight-based methods, and MuDSCZip.

As shown in Fig. 3, in the average loss landscape, our approach positions the
merged model directly at the lowest point of the basin, in contrast to the
activation-based method which places the merged model near the lowest point,
and the weight-based method, which positions the merged model further away
from the center of the basin. Then observe the loss landscape of different
tasks. Upon further examination of the loss landscapes for different tasks, we
observe that MuDSC facilitates the matching algorithm to discover a set of
overlapping loss basins at lower points. This implies that our approach can
better balance the performance of different tasks when merging for multi-task
scenarios. Overall, with the support of a more precise constraint, MuDSC
achieves an enhancement of the matching algorithm.

##  5 Conclusion

We introduce Merging under Dual-Space Constraints (MuDSC) to balance the
inconsistency of unit similarity in weight space and activation space when
merging models. MuDSC linearly combines the similarity matrices both of the
weights and the activations of the units to seek a better permutation matrix.
We find experimentally that MuDSC enhances the performance of the merged
multi-task model across various tasks and architecture. We conduct the
visualization of the merged model in the multi-task loss landscape which shows
that MuDSC makes the merged model locate in the overlapping part with unified
lower loss of each task.

##  6 Acknowledgements

This work is supported by the Science and Technology Project of SGCC: Research
and Digital Application of High-precision Electric Power Super-scale Pre-
trained Visual Model (5108-202218280A-2-395-XG).

## References

  * Ainsworth et al. [2023] Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa.  Git re-basin: Merging models modulo permutation symmetries.  In _The Eleventh International Conference on Learning Representations_ , 2023.
  * Bansal et al. [2021] Yamini Bansal, Preetum Nakkiran, and Boaz Barak.  Revisiting model stitching to compare neural representations.  _Advances in neural information processing systems_ , 34:225–236, 2021.
  * Benzing et al. [2022] Frederik Benzing, Simon Schug, Robert Meier, Johannes Von Oswald, Yassir Akram, Nicolas Zucchet, Laurence Aitchison, and Angelika Steger.  Random initialisations performing above chance and how to find them.  _arXiv preprint arXiv:2209.07509_ , 2022.
  * Brea et al. [2019] Johanni Brea, Berfin Simsek, Bernd Illing, and Wulfram Gerstner.  Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape.  _arXiv preprint arXiv:1907.02911_ , 2019.
  * Caron et al. [2021] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 9650–9660, 2021.
  * Csiszárik et al. [2021] Adrián Csiszárik, Péter Kőrösi-Szabó, Akos Matszangosz, Gergely Papp, and Dániel Varga.  Similarity and matching of neural network representations.  _Advances in Neural Information Processing Systems_ , 34:5656–5668, 2021.
  * de Carvalho et al. [2023] Marcus de Carvalho, Mahardhika Pratama, Jie Zhang, and Yajuan Sun.  Class-incremental learning via knowledge amalgamation.  In _Machine Learning and Knowledge Discovery in Databases_ , pages 36–50, Cham, 2023. Springer Nature Switzerland.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , pages 248–255, 2009.
  * Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _International Conference on Learning Representations_ , 2021.
  * Entezari et al. [2021] Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur.  The role of permutation invariance in linear mode connectivity of neural networks.  _arXiv preprint arXiv:2110.06296_ , 2021.
  * Fifty et al. [2021] Chris Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn.  Efficiently identifying task groupings for multi-task learning.  _Advances in Neural Information Processing Systems_ , 34:27503–27516, 2021.
  * Frankle and Carbin [2018] Jonathan Frankle and Michael Carbin.  The lottery ticket hypothesis: Finding sparse, trainable neural networks.  _arXiv preprint arXiv:1803.03635_ , 2018.
  * Fukumizu and Amari [2000] K. Fukumizu and S. Amari.  Local minima and plateaus in hierarchical structures of multilayer perceptrons.  _Neural Networks_ , 13(3):317–327, 2000.
  * Gao et al. [2023] Shangde Gao, Yichao Fu, Ke Liu, and Yuqiang Han.  Contrastive knowledge amalgamation for unsupervised image classification.  In _Artificial Neural Networks and Machine Learning – ICANN 2023_ , pages 192–204, Cham, 2023. Springer Nature Switzerland.
  * Grigsby et al. [2023] Elisenda Grigsby, Kathryn Lindsey, and David Rolnick.  Hidden symmetries of ReLU networks.  In _Proceedings of the 40th International Conference on Machine Learning_ , pages 11734–11760. PMLR, 2023.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Hecht-Nielsen [1990] Robert Hecht-Nielsen.  On the algebraic structure of feedforward network weight spaces.  In _Advanced Neural Computers_ , pages 129–135. North-Holland, Amsterdam, 1990.
  * Ilharco et al. [2023] Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi.  Editing models with task arithmetic.  In _The Eleventh International Conference on Learning Representations_ , 2023.
  * Jin et al. [2023] Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng.  Dataless knowledge fusion by merging weights of language models.  In _The Eleventh International Conference on Learning Representations_ , 2023.
  * Jing et al. [2021] Yongcheng Jing, Yiding Yang, Xinchao Wang, Mingli Song, and Dacheng Tao.  Amalgamating knowledge from heterogeneous graph neural networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021.
  * Jing et al. [2023] Yongcheng Jing, Chongbin Yuan, Li Ju, Yiding Yang, Xinchao Wang, and Dacheng Tao.  Deep graph reprogramming.  In _CVPR_ , 2023.
  * Jordan et al. [2023] Keller Jordan, Hanie Sedghi, Olga Saukh, Rahim Entezari, and Behnam Neyshabur.  REPAIR: REnormalizing permuted activations for interpolation repair.  In _The Eleventh International Conference on Learning Representations_ , 2023.
  * Krizhevsky et al. [2009] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * Lam et al. [2021] Thanh Chi Lam, Nghia Hoang, Bryan Kian Hsiang Low, and Patrick Jaillet.  Model fusion for personalized learning.  In _Proceedings of the 38th International Conference on Machine Learning_ , pages 5948–5958. PMLR, 2021.
  * LeCun et al. [2010] Yann LeCun, Corinna Cortes, and CJ Burges.  Mnist handwritten digit database.  _ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist_ , 2, 2010.
  * Lenc and Vedaldi [2015] Karel Lenc and Andrea Vedaldi.  Understanding image representations by measuring their equivariance and equivalence.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 991–999, 2015.
  * Li et al. [2023] Weishi Li, Yong Peng, Miao Zhang, Liang Ding, Han Hu, and Li Shen.  Deep model fusion: A survey.  _arXiv preprint arXiv:2309.15698_ , 2023.
  * Li et al. [2015] Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft.  Convergent learning: Do different neural networks learn the same representations?  _arXiv preprint arXiv:1511.07543_ , 2015.
  * Liu et al. [2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 10012–10022, 2021.
  * Luo et al. [2019] Sihui Luo, Xinchao Wang, Gongfan Fang, Yao Hu, Dapeng Tao, and Mingli Song.  Knowledge amalgamation from heterogeneous networks by common feature learning.  In _Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI)_ , 2019.
  * Luo et al. [2020] Sihui Luo, Wenwen Pan, Xinchao Wang, Dazhou Wang, Haihong Tang, and Mingli Song.  Collaboration by competition: Self-coordinated knowledge amalgamation for multi-talent student learning.  In _European Conference on Computer Vision_ , 2020.
  * Maćkiewicz and Ratajczak [1993] Andrzej Maćkiewicz and Waldemar Ratajczak.  Principal components analysis (pca).  _Computers & Geosciences_, 19(3):303–342, 1993.
  * Nguyen et al. [2023] Dang Nguyen, Trang Nguyen, Khai Nguyen, Dinh Phung, Hung Bui, and Nhat Ho.  On cross-layer alignment for model fusion of heterogeneous neural networks.  In _ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 1–5, 2023.
  * Pan et al. [2023] Zizheng Pan, Jianfei Cai, and Bohan Zhuang.  Stitchable neural networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 16102–16112, 2023.
  * Peña et al. [2023] Fidel A Guerrero Peña, Heitor Rapela Medeiros, Thomas Dubail, Masih Aminbeidokhti, Eric Granger, and Marco Pedersoli.  Re-basin via implicit sinkhorn differentiation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 20237–20246, 2023.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Shen et al. [2019a] Chengchao Shen, Xinchao Wang, Jie Song, Li Sun, and Mingli Song.  Amalgamating knowledge towards comprehensive classification.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 3068–3075, 2019a.
  * Shen et al. [2019b] Chengchao Shen, Mengqi Xue, Xinchao Wang, Jie Song, Li Sun, and Mingli Song.  Customizing student networks from heterogeneous teachers via adaptive knowledge amalgamation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3504–3513, 2019b.
  * Simsek et al. [2021] Berfin Simsek, François Ged, Arthur Jacot, Francesco Spadaro, Clément Hongler, Wulfram Gerstner, and Johanni Brea.  Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances.  In _International Conference on Machine Learning_ , pages 9722–9732. PMLR, 2021.
  * Singh and Jaggi [2020a] Sidak Pal Singh and Martin Jaggi.  Model fusion via optimal transport.  _Advances in Neural Information Processing Systems_ , 33, 2020a.
  * Singh and Jaggi [2020b] Sidak Pal Singh and Martin Jaggi.  Model fusion via optimal transport.  _Advances in Neural Information Processing Systems_ , 33, 2020b.
  * Song et al. [2023] Jie Song, Zhengqi Xu, Sai Wu, Gang Chen, and Mingli Song.  Modelgif: Gradient fields for model functional distance.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2023.
  * Stoica et al. [2023] George Stoica, Daniel Bolya, Jakob Bjorner, Taylor Hearn, and Judy Hoffman.  Zipit! merging models from different tasks without training, 2023.
  * Tatro et al. [2020] Norman Tatro, Pin-Yu Chen, Payel Das, Igor Melnyk, Prasanna Sattigeri, and Rongjie Lai.  Optimizing mode connectivity via neuron alignment.  _Advances in Neural Information Processing Systems_ , 33:15300–15311, 2020.
  * Thadajarassiri et al. [2021] Jidapa Thadajarassiri, Thomas Hartvigsen, Xiangnan Kong, and Elke A Rundensteiner.  Semi-supervised knowledge amalgamation for sequence classification.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 35(11):9859–9867, 2021.
  * Thadajarassiri et al. [2023] Jidapa Thadajarassiri, Thomas Hartvigsen, Walter Gerych, Xiangnan Kong, and Elke Rundensteiner.  Knowledge amalgamation for multi-label classification via label dependency transfer.  37:9980–9988, 2023.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wang et al. [2020] Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni.  Federated learning with matched averaging.  In _International Conference on Learning Representations_ , 2020.
  * Wightman [2019] Ross Wightman.  Pytorch image models.  https://github.com/rwightman/pytorch-image-models, 2019.
  * Wu and He [2018] Yuxin Wu and Kaiming He.  Group normalization.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018.
  * Xiao et al. [2023] Peng Xiao, Biao Zhang, Samuel Cheng, Ke Wei, and Shuqin Zhang.  Probabilistic fusion of neural networks that incorporates global information.  In _Proceedings of The 14th Asian Conference on Machine Learning_ , pages 1149–1164. PMLR, 2023.
  * Yadav et al. [2023] Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal.  Resolving interference when merging models.  In _NeurIPS_ , New Orleans, USA, 2023. Proceedings of Machine Learning Research.
  * Yang et al. [2023] Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao.  Adamerging: Adaptive model merging for multi-task learning.  _arXiv preprint arXiv:2310.02575_ , 2023.
  * Yang et al. [2022a] Xingyi Yang, Jingwen Ye, and Xinchao Wang.  Factorizing knowledge in neural networks.  In _European Conference on Computer Vision_ , pages 73–91. Springer, 2022a.
  * Yang et al. [2022b] Xingyi Yang, Daquan Zhou, Songhua Liu, Jingwen Ye, and Xinchao Wang.  Deep model reassembly.  _Advances in neural information processing systems_ , 35:25739–25753, 2022b.
  * Ye et al. [2019a] Jingwen Ye, Yixin Ji, Xinchao Wang, Kairi Ou, Dapeng Tao, and Mingli Song.  Student becoming the master: Knowledge amalgamation for joint scene parsing, depth estimation, and more.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2829–2838, 2019a.
  * Ye et al. [2019b] Jingwen Ye, Xinchao Wang, Yixin Ji, Kairi Ou, and Mingli Song.  Amalgamating filtered knowledge: Learning task-customized student from multi-task teachers.  In _International Joint Conference on Artificial Intelligence_ , 2019b.
  * Ye et al. [2020] Jingwen Ye, Yixin Ji, Xinchao Wang, Xin Gao, and Mingli Song.  Data-free knowledge amalgamation via group-stack dual-gan.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12516–12525, 2020.
  * Yurochkin et al. [2019] Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni.  Bayesian nonparametric federated learning of neural networks.  In _Proceedings of the 36th International Conference on Machine Learning_ , pages 7252–7261. PMLR, 2019.
  * Zamir et al. [2018] Amir R. Zamir, Alexander Sax, William Shen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese.  Taskonomy: Disentangling task transfer learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * Zhang et al. [2023a] Haofei Zhang, Feng Mao, Mengqi Xue, Gongfan Fang, Zunlei Feng, Jie Song, and Mingli Song.  Knowledge amalgamation for object detection with transformers.  _IEEE Transactions on Image Processing_ , 32:2093–2106, 2023a.
  * Zhang et al. [2023b] Jinghan Zhang, Shiqi Chen, Junteng Liu, and Junxian He.  Composing parameter-efficient modules with arithmetic operations.  _arXiv preprint arXiv:2306.14870_ , 2023b.
  * Zhang and Yang [2021] Yu Zhang and Qiang Yang.  A survey on multi-task learning.  _IEEE Transactions on Knowledge and Data Engineering_ , 34(12):5586–5609, 2021.