# ZipIt! Merging Models from Different Tasks without Training

George Stoica Daniel Bolya*

Jakob Bjorner Pratik Ramesh Taylor Hearn Judy Hoffman

Georgia Tech

<EMAIL_ADDRESS>
Equal Contribution. Code: https://github.com/gstoica27/ZipIt.

###### Abstract

Typical deep visual recognition models are capable of performing the one task
they were trained on. In this paper, we tackle the extremely difficult problem
of combining distinct models with different initializations, each solving a
separate task, into one multi-task model without any additional training.
Prior work in model merging permutes one model to the space of the other then
averages them together. While this works for models trained on the same task,
we find that this fails to account for the differences in models trained on
disjoint tasks. Thus, we introduce “ZipIt!”, a general method for merging two
arbitrary models of the same architecture that incorporates two simple
strategies. First, in order to account for features that aren’t shared between
models, we expand the model merging problem to allow for merging features
within each model by defining a general “zip” operation. Second, we add
support for partially zipping the models up until a specified layer, naturally
creating a multi-head model. We find that these two changes combined account
for 20-60% improvement over prior work, making it more feasible to merge
models trained on disjoint tasks without retraining.

##  1 Introduction

Ever since AlexNet (Krizhevsky et al., 2017) popularized deep learning in
computer vision, the field has thrived under the reign of massive models with
an ever increasing number of parameters. Many vision problems once considered
difficult or impossible are now benchmark tasks: classification with tens of
thousands of classes (Deng et al., 2009; Zhou et al., 2017; Gemmeke et al.,
2017), fast instance segmentation (He et al., 2017; Bolya et al., 2019),
realistic image generation (Karras et al., 2018; Ho et al., 2020; Rombach et
al., 2022), and more.

There are an abundance of independent, carefully tuned models out there for
many tasks. However, if we want to expand an existing model’s capabilities, we
run into many potential issues. If we try training the model on an additional
task, we face catastrophic forgetting (Kirkpatrick et al., 2017; Li & Hoiem,
2017; De Lange et al., 2021). If we evaluate the same model on different data
without adaptation, we often find it doesn’t generalize to out of domain
samples (Blanchard et al., 2011; Muandet et al., 2013; Wang et al., 2022). We
can try so called “intervention” strategies (Wang et al., 2022; De Lange et
al., 2021) to mitigate these effects, but these often require further training
which can be expensive (Dosovitskiy et al., 2020; Zhai et al., 2022; Dehghani
et al., 2023). Instead, it would be nice if we could expand a model’s capacity
to solve new tasks by simply “zipping” it with other models trained on those
tasks without additional training.

Figure 1:  Setting and ZipIt! (a) Prior work merges differently initialized
models from the same dataset with the same label sets: e.g., merging two
models both trained to classify dog breeds. (b) Our setting expands this to
merging models from different datasets with different label sets: e.g.,
merging a model that classifies dog breeds with one that classifies bird
species. (c) ZipIt! merges these models without retraining by identifying
shared features.

Combining multiple models into one has recently started to gain traction in
the vision community. Model Soups (Wortsman et al., 2022a) can add multiple
models finetuned from the same pretrained initialization to improve accuracy
and robustness. Git Re-Basin (Ainsworth et al., 2022) generalizes further to
models trained on the same data but with different initializations, though
with a significant accuracy drop. REPAIR (Jordan et al., 2022) improves on Git
Re-Basin by adding new parameters and adjusting model batch norms where
applicable. However, all of these methods only combine models trained on the
same task. In this paper, we take this line of work to a logical extreme:
merging differently initialized models trained on completely separate tasks
(see Fig. 1ab). We show that this is an incredibly difficult problem for prior
work and employ two simple strategies to make it feasible.

First, we note that prior work focuses on permuting one model to the other
when merging them. This creates a 1-1 mapping between the two models,
inherently assuming that most features across them are correlated. Since this
isn’t necessarily the case for models trained on different tasks, we cannot
rely on permutation alone. Instead, we generalize model merging to support
“zipping” any combination of correlated features within and across each model.
We find that on some tasks, this alone improves accuracy by up to 20% vs.
permutation-based approaches. Moreover, we prove that merging within models
can yield a better result in the theoretical setting of Entezari et al.
(2021).

Second, existing methods merge the entire network. While this might work for
extremely similar models trained in the same setting, the features of models
trained on disjoint tasks become less correlated over the course of the
network (Kornblith et al., 2019). To solve this, we introduce partial zipping,
where we only “zip” up to a specified layer. Afterward, we feed the merged
model’s outputs to the remaining unmerged layers of the original networks,
creating a multi-head model. Depending on task difficulty, this can improve
accuracy by over 15% while still keeping most layers merged.

Incorporating both of these strategies, we introduce ZipIt! (Fig. 1c), a
general method for “zipping” any number of models trained on different tasks
into a single multitask model without retraining. By deriving a general graph-
based algorithm for merging and unmerging (Sec. 4), we can zip models of the
same architecture together, merge features within each model, and partially
zip them to create a multi-task model. We validate our approach by merging
models trained on entirely disjoint sets of CIFAR (Krizhevsky et al., 2009)
and ImageNet (Deng et al., 2009) categories, as well as merging several models
trained on completely independent datasets into one, significantly
outperforming prior work (Sec. 5). Finally, we ablate and analyze our method’s
capabilities on these scenarios (Sec. 6).

##  2 Related Work

Model merging combines the weights of two or more models into a one. Our work
differs from prior work in that we adapt mode connectivity techniques to
target models trained on disjoint tasks (Fig. 1).

##### Merging Finetuned Models.

If two models are finetuned from the same pretrained checkpoint, they often
lie in the same error basin (Neyshabur et al., 2020). Several works (Huang et
al., 2017; Izmailov et al., 2018; Von Oswald et al., 2020; Wortsman et al.,
2022b; Ilharco et al., 2022b; Don-Yehiya et al., 2023) have exploited this
property to average together the weights of a model at different stages of
training. Tarvainen & Valpola (2017); Cai et al. (2021); Grill et al. (2020);
Caron et al. (2021); Baevski et al. (2022) use an “exponential moving average”
of training checkpoints as a teacher for self-supervised learning. Other works
merge models initialized from the same pretrained base, but that were
finetuned independently, either by simply averaging their weights (McMahan et
al., 2017; Wortsman et al., 2022a; Choshen et al., 2022; Ramé et al., 2022),
permuting one model to the other (Ashmore & Gashler, 2015; Yurochkin et al.,
2019; Wang et al., 2020), combining meaningful weight regions (Ilharco et al.,
2022a; Gueta et al., 2023; Yadav et al., 2023; Sung et al., 2023), or
maximizing an objective (Matena & Raffel, 2021). Our setting differs, as we do
not assume the same initialization.

##### Merging Differently Initialized Models.

Merging models with different initializations is a much more challenging
problem. Works in this space often rely on mode connectivity (Freeman & Bruna,
2016; Garipov et al., 2018; Draxler et al., 2018; Frankle et al., 2020),
attempting to interpolate between models along a low loss path (e.g., Tatro et
al. (2020); Singh & Jaggi (2020); Liu et al. (2022)). Most recent work follow
the intuition, later formalized by Entezari et al. (2021), that models
permuted to the same loss basin can be merged by averaging their weights. Most
notably, Git Re-Basin (Ainsworth et al., 2022) permutes models by comparing
the similarity between their weights. REPAIR (Jordan et al., 2022) improves
the accuracy of Git Re-Basin by instead computing the correlation between
their intermediate layer feature activations, and adding several batch norms
to the network. Peña et al. (2022) find permutations using global rather than
local optimization, though they don’t support skip connections. Some of these
works (e.g., Singh & Jaggi (2020); Ainsworth et al. (2022)) evaluate on on a
setting where each model sees varying numbers of instances per class. And Peña
et al. (2022) evaluates on a continual learning setting with disjoint
categories, but their method requires training optimization. Similarly, He et
al. (2018) merges models of different tasks, but requires jointly finetuning
after each layer merge. As far as we are aware, we present the first general
method to successfully merge models trained on disjoint tasks without
additional training.

##  3 Background and Motivation

Model merging stems from mode connectivity (Garipov et al., 2018), where it is
conjectured that models trained with SGD on the same dataset lying in the same
loss basin (i.e., region or mode of low loss) can be combined into a single
model that’s just as performant as the original models. If these models can be
combined well by linearly interpolating their weights, they are said to be
linearly mode connected (LMC) (Garipov et al., 2018; Entezari et al., 2021).
Our work is similar to finding LMC, but across different datasets with
disjoint label sets (i.e., separate tasks as in De Lange et al. (2021)).

Consider a model $\mathcal{L}$ as a collection of layers
$L_{i}\in\mathcal{L}$, each of which may have some parameters (e.g.,
$W_{i},b_{i}$ for a linear layer). If $\mathcal{L}^{A}$ and $\mathcal{L}^{B}$
are finetuned from the same checkpoint, several works (e.g., Izmailov et al.
(2018); Wortsman et al. (2022a)) find merging them is as easy as linearly
interpolating their weights (i.e., they are LMC). E.g., if $L_{i}$ is a linear
layer, the new weight matrix $W_{i}^{*}$ is simply

${\color[rgb]{0.20703125,0.109375,0.45703125}\definecolor[named]{pgfstrokecolor}{rgb}{0.20703125,0.109375,0.45703125}W_{i}^{*}}=\gamma{\color[rgb]{0.5234375,0.125,0.046875}\definecolor[named]{pgfstrokecolor}{rgb}{0.5234375,0.125,0.046875}W_{i}^{A}}+(1-\gamma){\color[rgb]{0.04296875,0.32421875,0.58203125}\definecolor[named]{pgfstrokecolor}{rgb}{0.04296875,0.32421875,0.58203125}W_{i}^{B}}$
(1)

with an interpolation constant $\gamma\in[0,1]$, usually set to
$\nicefrac{{1}}{{2}}$. However, if $\mathcal{L}^{A}$ and $\mathcal{L}^{B}$
were not finetuned from the same checkpoint, they often do not lie in the same
mode (Entezari et al., 2021; Ainsworth et al., 2022) and cannot be
interpolated. Indeed, Eq. 1 typically results in random accuracy.

To fix this, Entezari et al. (2021) conjecture that large enough models are
likely LMC modulo permutation. This is because (1) many neural networks can be
permuted internally without affecting their outputs and (2) permutation can
move two models into the same basin, allowing much of the lost accuracy to be
recovered. More concretely, let $P_{i}$ be a permutation matrix that permutes
outputs of layer $L_{i}^{B}$ to the space of $L_{i}^{A}$. Then for each layer,
permutation works apply

${\color[rgb]{0.20703125,0.109375,0.45703125}\definecolor[named]{pgfstrokecolor}{rgb}{0.20703125,0.109375,0.45703125}W_{i}^{*}}=\gamma{\color[rgb]{0.5234375,0.125,0.046875}\definecolor[named]{pgfstrokecolor}{rgb}{0.5234375,0.125,0.046875}W_{i}^{A}}+(1-\gamma){\color[rgb]{0.04296875,0.32421875,0.58203125}\definecolor[named]{pgfstrokecolor}{rgb}{0.04296875,0.32421875,0.58203125}P_{i}W_{i}^{B}P_{i-1}^{T}}$
(2)

Note that here we permute the output space of $W_{i}^{B}$, but we also need to
permute its input space to undo the permutation from the previous layer (hence
the use of $P_{i-1}^{T}$).

Figure 2: Task Loss Landscapes for models in Tab. LABEL:tab:cifar50+50. Model
A and Model B lie in low loss basins for their own tasks, but not for the
other task. Thus, any interpolation between Model A and a permuted Model B
(e.g., Git Re-basin) lies outside the minima for both tasks and thus performs
poorly. In contrast, ZipIt! improves the merge by finding a model that lies in
a low loss basin for both.

##### Problems with Permutation.

Eq. 2 relies on the likelihood of model B lying in the same basin as model A
after permutation being high. However, this is far less likely when the models
are trained on different tasks, as each model optimizes for basins containing
distinct task-specific information. In this case the optimal permutation of
model B to model A still lies in a strong basin on task B but doesn’t lie in a
basin on task A, as shown in Figure 2. This causes the interpolated model to
perform worse than either the two original models. Thus, we explore
alternative merging methods.

##  4 ZipIt!

In this work, we treat model merging as combining the checkpoints (i.e.,
collection of weights) of multiple models into a single checkpoint that can
perform all the tasks of its constituents. We do this by merging the layers of
the models together. For instance, suppose $L_{i}\in\mathcal{L}$ is a linear
layer with parameters $W_{i}\in\mathbb{R}^{n_{i}\times
m_{i}},b_{i}\in\mathbb{R}^{n_{i}}$ with input features
$x\in\mathbb{R}^{m_{i}}$ and outputs features $f_{i}\in\mathbb{R}^{n_{i}}$:

$f_{i}=L_{i}(x)=W_{i}x+b_{i}$ (3)

Our goal is to take
${\color[rgb]{0.5234375,0.125,0.046875}\definecolor[named]{pgfstrokecolor}{rgb}{0.5234375,0.125,0.046875}L_{i}^{A}}\in{\color[rgb]{0.5234375,0.125,0.046875}\definecolor[named]{pgfstrokecolor}{rgb}{0.5234375,0.125,0.046875}\mathcal{L}^{A}}$
from model A and
${\color[rgb]{0.04296875,0.32421875,0.58203125}\definecolor[named]{pgfstrokecolor}{rgb}{0.04296875,0.32421875,0.58203125}L_{i}^{B}}\in{\color[rgb]{0.04296875,0.32421875,0.58203125}\definecolor[named]{pgfstrokecolor}{rgb}{0.04296875,0.32421875,0.58203125}\mathcal{L}^{B}}$
from model B and merge them into a layer $L_{i}^{*}$ that combines their
feature spaces such that information from both $f_{i}^{A}$ and $f_{i}^{B}$ is
retained in $f_{i}^{*}$. We accomplish this by merging each layer of one model
with the corresponding layer in the other, both merging features in one across
both layers or within the same layer. This is in contrast to permutation-based
merging method, which only combine features across layers.

##### Why should we merge within?

Features of models trained on different tasks may be dissimilar, as the models
solve different problems. Forcibly combining these dissimilar features can
yield merges that don’t perform well on either original task (Fig 2). Instead,
those features may be more compatible with others within the same model, which
would better retain performance when combined.

In fact, we can prove that methods which allow merges within each model (as
well as across both) perform equal to or better than those which only merge
across models (e.g., permutation-reliant approaches) in a limited but
prevalent setting. Specifically, we obtain a tighter bound over Theorem 3.1
from Entezari et al. (2021) when redundancy exists within a model and is
leveraged. Both Theorem 3.1 and our Theorem 1 (see Appendix G for
formalization and proof) bound the degree to which the loss of a merged two-
layer model with $d$-input dimensions and $h$-intermediate dimensions
increases compared to the losses of the original models. Theorem 3.1 bounds
this increase to $\tilde{O}(h^{-\nicefrac{{1}}{{(2d+4)}}})$. However, if
features within a model are redundant, then we reduce the bound to

$\text{Loss Increase of Merged
Model}\leq\begin{cases}\tilde{O}\left(\left(\frac{h}{1-2\Gamma}\right)^{-\frac{1}{2d+4}}\right)&\Gamma<0.5\\\
0&\text{otherwise}\end{cases}$ (4)

with $\Gamma\in[0,1]$ measuring what portion of features are redundant. This
bound is $\sqrt[2d+4]{1-2\Gamma}\leq 1$ times that of Theorem 3.1 when
$\Gamma<0.5$ (equal only when $\Gamma=0$) and explicitly zero when $\Gamma\geq
0.5$.

##### How do we merge features?

In prior work, each of the merged features $f_{i}^{*}$ is the result of
combining one feature from $f_{i}^{A}$ and one from $f_{i}^{B}$. However in
our case, we can also merge features by combining two from just $f_{i}^{A}$ or
two from just $f_{i}^{B}$. To account for this, we concatenate $f_{i}^{A}$ and
$f_{i}^{B}$ into a single feature vector:
${\color[rgb]{0.5234375,0.125,0.046875}\definecolor[named]{pgfstrokecolor}{rgb}{0.5234375,0.125,0.046875}f_{i}^{A}}\|{\color[rgb]{0.04296875,0.32421875,0.58203125}\definecolor[named]{pgfstrokecolor}{rgb}{0.04296875,0.32421875,0.58203125}f_{i}^{B}}\in\mathbb{R}^{2n_{i}}$.
Then, like prior work (e.g. Li et al. (2015); Jordan et al. (2022)), we define
feature similarity as the pairwise correlation between between neuron
activations over a small set of images (without labels). However, unlike those
works, we compute correlations between every activation in the full
concatenated space
${\color[rgb]{0.5234375,0.125,0.046875}\definecolor[named]{pgfstrokecolor}{rgb}{0.5234375,0.125,0.046875}f_{i}^{A}}\|{\color[rgb]{0.04296875,0.32421875,0.58203125}\definecolor[named]{pgfstrokecolor}{rgb}{0.04296875,0.32421875,0.58203125}f_{i}^{B}}$.
Our approach thus measures the similarity of every feature
${\color[rgb]{0.5234375,0.125,0.046875}\definecolor[named]{pgfstrokecolor}{rgb}{0.5234375,0.125,0.046875}f_{i}^{A}}$
and
${\color[rgb]{0.04296875,0.32421875,0.58203125}\definecolor[named]{pgfstrokecolor}{rgb}{0.04296875,0.32421875,0.58203125}f_{i}^{B}}$
to all features in both models, rather than solely between
${\color[rgb]{0.5234375,0.125,0.046875}\definecolor[named]{pgfstrokecolor}{rgb}{0.5234375,0.125,0.046875}f_{i}^{A}}$
and
${\color[rgb]{0.04296875,0.32421875,0.58203125}\definecolor[named]{pgfstrokecolor}{rgb}{0.04296875,0.32421875,0.58203125}f_{i}^{B}}$.

Next, if two features are well correlated, we can average them without losing
much information. Thus, we can construct $f_{i}^{*}$ by finding $n_{i}$ pairs
of similar features in
${\color[rgb]{0.5234375,0.125,0.046875}\definecolor[named]{pgfstrokecolor}{rgb}{0.5234375,0.125,0.046875}f_{i}^{A}}\|{\color[rgb]{0.04296875,0.32421875,0.58203125}\definecolor[named]{pgfstrokecolor}{rgb}{0.04296875,0.32421875,0.58203125}f_{i}^{B}}$
and averaging them together. By default, we do this greedily: i.e.,
iteratively match the features with the highest correlation without
replacement; though we explore extensions to this in Sec. 4.3 and test other
methods in Tab. 4. Then we can use these matches to construct $f_{i}^{*}$.
Formally, we define a “merge matrix”
${\color[rgb]{0.20703125,0.109375,0.45703125}\definecolor[named]{pgfstrokecolor}{rgb}{0.20703125,0.109375,0.45703125}M_{i}}\in\mathbb{R}^{n_{i}\times
2n_{i}}$ s.t.

${\color[rgb]{0.20703125,0.109375,0.45703125}\definecolor[named]{pgfstrokecolor}{rgb}{0.20703125,0.109375,0.45703125}f_{i}^{*}}={\color[rgb]{0.20703125,0.109375,0.45703125}\definecolor[named]{pgfstrokecolor}{rgb}{0.20703125,0.109375,0.45703125}M_{i}}\left({\color[rgb]{0.5234375,0.125,0.046875}\definecolor[named]{pgfstrokecolor}{rgb}{0.5234375,0.125,0.046875}f_{i}^{A}}\|{\color[rgb]{0.04296875,0.32421875,0.58203125}\definecolor[named]{pgfstrokecolor}{rgb}{0.04296875,0.32421875,0.58203125}f_{i}^{B}}\right)$
(5)

${\color[rgb]{0.20703125,0.109375,0.45703125}\definecolor[named]{pgfstrokecolor}{rgb}{0.20703125,0.109375,0.45703125}M_{i}}$
averages the matched features, with each match corresponding to one output
feature in
${\color[rgb]{0.20703125,0.109375,0.45703125}\definecolor[named]{pgfstrokecolor}{rgb}{0.20703125,0.109375,0.45703125}f_{i}^{*}}$.
For instance, if $u$th match is between indices $s,t\in\\{1,\ldots,2n_{i}\\}$
of
${\color[rgb]{0.5234375,0.125,0.046875}\definecolor[named]{pgfstrokecolor}{rgb}{0.5234375,0.125,0.046875}f_{i}^{A}}\|{\color[rgb]{0.04296875,0.32421875,0.58203125}\definecolor[named]{pgfstrokecolor}{rgb}{0.04296875,0.32421875,0.58203125}f_{i}^{B}}$,
then the $u$th row of
${\color[rgb]{0.20703125,0.109375,0.45703125}\definecolor[named]{pgfstrokecolor}{rgb}{0.20703125,0.109375,0.45703125}M_{i}}$
would be $\nicefrac{{1}}{{2}}$ at columns $s$ and $t$ and 0 elsewhere. This
results in
${\color[rgb]{0.20703125,0.109375,0.45703125}\definecolor[named]{pgfstrokecolor}{rgb}{0.20703125,0.109375,0.45703125}f_{i}^{*}}[u]=\frac{1}{2}({\color[rgb]{0.5234375,0.125,0.046875}\definecolor[named]{pgfstrokecolor}{rgb}{0.5234375,0.125,0.046875}f_{i}^{A}}\|{\color[rgb]{0.04296875,0.32421875,0.58203125}\definecolor[named]{pgfstrokecolor}{rgb}{0.04296875,0.32421875,0.58203125}f_{i}^{B}})[s]+\frac{1}{2}({\color[rgb]{0.5234375,0.125,0.046875}\definecolor[named]{pgfstrokecolor}{rgb}{0.5234375,0.125,0.046875}f_{i}^{A}}\|{\color[rgb]{0.04296875,0.32421875,0.58203125}\definecolor[named]{pgfstrokecolor}{rgb}{0.04296875,0.32421875,0.58203125}f_{i}^{B}})[t]$.
Thus, applying
${\color[rgb]{0.20703125,0.109375,0.45703125}\definecolor[named]{pgfstrokecolor}{rgb}{0.20703125,0.109375,0.45703125}M_{i}}$
has the effect of interpolating with $\gamma=\nicefrac{{1}}{{2}}$ but is more
general (e.g., allows for merging more than 2 models at once, see Sec. 4.3).

##### What about the next layer?

After merging features in one layer, we now have the problem that the next
layers,
${\color[rgb]{0.5234375,0.125,0.046875}\definecolor[named]{pgfstrokecolor}{rgb}{0.5234375,0.125,0.046875}L_{i+1}^{A}},{\color[rgb]{0.04296875,0.32421875,0.58203125}\definecolor[named]{pgfstrokecolor}{rgb}{0.04296875,0.32421875,0.58203125}L_{i+1}^{B}}$,
are incompatible with
${\color[rgb]{0.20703125,0.109375,0.45703125}\definecolor[named]{pgfstrokecolor}{rgb}{0.20703125,0.109375,0.45703125}f_{i}^{*}}$.
Instead, we need to undo the merge operation before passing the features to
the next layer. Thus, we define an “unmerge” matrix
${\color[rgb]{0.20703125,0.109375,0.45703125}\definecolor[named]{pgfstrokecolor}{rgb}{0.20703125,0.109375,0.45703125}U_{i}}\in\mathbb{R}^{2n_{i}\times
n_{i}}$ s.t.

${\color[rgb]{0.20703125,0.109375,0.45703125}\definecolor[named]{pgfstrokecolor}{rgb}{0.20703125,0.109375,0.45703125}U_{i}f_{i}^{*}}\approx{\color[rgb]{0.5234375,0.125,0.046875}\definecolor[named]{pgfstrokecolor}{rgb}{0.5234375,0.125,0.046875}f_{i}^{A}}\|{\color[rgb]{0.04296875,0.32421875,0.58203125}\definecolor[named]{pgfstrokecolor}{rgb}{0.04296875,0.32421875,0.58203125}f_{i}^{B}}$
(6)

${\color[rgb]{0.20703125,0.109375,0.45703125}\definecolor[named]{pgfstrokecolor}{rgb}{0.20703125,0.109375,0.45703125}U_{i}}$
is the pseudoinverse of $M_{i}$ and in the case of the matching from earlier
is simply
$2{\color[rgb]{0.20703125,0.109375,0.45703125}\definecolor[named]{pgfstrokecolor}{rgb}{0.20703125,0.109375,0.45703125}M_{i}}^{T}$.
Note that strict equality is unlikely here. Like in prior work, merging models
is a lossy operation.

We split this unmerge matrix in half along its rows into
${\color[rgb]{0.5234375,0.125,0.046875}\definecolor[named]{pgfstrokecolor}{rgb}{0.5234375,0.125,0.046875}U_{i}^{A}},{\color[rgb]{0.04296875,0.32421875,0.58203125}\definecolor[named]{pgfstrokecolor}{rgb}{0.04296875,0.32421875,0.58203125}U_{i}^{B}}\in\mathbb{R}^{n_{i}\times
n_{i}}$ that act individually to produce $f_{i}^{A}$ and $f_{i}^{B}$. With
this, we can evaluate the next layers using the merged features:

${\color[rgb]{0.5234375,0.125,0.046875}\definecolor[named]{pgfstrokecolor}{rgb}{0.5234375,0.125,0.046875}f_{i+1}^{A}}\approx{\color[rgb]{0.5234375,0.125,0.046875}\definecolor[named]{pgfstrokecolor}{rgb}{0.5234375,0.125,0.046875}L_{i+1}^{A}}({\color[rgb]{0.5234375,0.125,0.046875}\definecolor[named]{pgfstrokecolor}{rgb}{0.5234375,0.125,0.046875}U_{i}^{A}}{\color[rgb]{0.20703125,0.109375,0.45703125}\definecolor[named]{pgfstrokecolor}{rgb}{0.20703125,0.109375,0.45703125}f_{i}^{*}})\qquad{\color[rgb]{0.04296875,0.32421875,0.58203125}\definecolor[named]{pgfstrokecolor}{rgb}{0.04296875,0.32421875,0.58203125}f_{i+1}^{B}}\approx{\color[rgb]{0.04296875,0.32421875,0.58203125}\definecolor[named]{pgfstrokecolor}{rgb}{0.04296875,0.32421875,0.58203125}L_{i+1}^{B}}({\color[rgb]{0.04296875,0.32421875,0.58203125}\definecolor[named]{pgfstrokecolor}{rgb}{0.04296875,0.32421875,0.58203125}U_{i}^{B}}{\color[rgb]{0.20703125,0.109375,0.45703125}\definecolor[named]{pgfstrokecolor}{rgb}{0.20703125,0.109375,0.45703125}f_{i}^{*}})$
(7)

###  4.1 The “Zip” Operation

We now have all the necessary pieces, and can derive a general operation to
merge $L_{i}^{A}$ and $L_{i}^{B}$ at an arbitrary point in the network (Fig.
3). First, we compute $M_{i}$ and $U_{i}$ by matching features between
$f_{i}^{A}$ and $f_{i}^{B}$. We then pass $U_{i}$ to the next layer and
receive $U_{i-1}$ from the previous layer. Using $M_{i}$ and $U_{i-1}$, we
“fuse” the merge and unmerge operations into the layer’s parameters. For a
linear layer:

${\color[rgb]{0.20703125,0.109375,0.45703125}\definecolor[named]{pgfstrokecolor}{rgb}{0.20703125,0.109375,0.45703125}W^{*}_{i}}={\color[rgb]{0.5234375,0.125,0.046875}\definecolor[named]{pgfstrokecolor}{rgb}{0.5234375,0.125,0.046875}M_{i}^{A}W_{i}^{A}U^{A}_{i-1}}+{\color[rgb]{0.04296875,0.32421875,0.58203125}\definecolor[named]{pgfstrokecolor}{rgb}{0.04296875,0.32421875,0.58203125}M_{i}^{B}W_{i}^{B}U^{B}_{i-1}}$
(8)

where $M_{i}^{A}$ and $M_{i}^{B}$ are $M_{i}$ split along its columns.
$b_{i}^{*}$ has the same equation but without unmerging.

Note the similarity between Eq. 8 and Eq. 2. This isn’t a coincidence: if we
only allowed merging across models and not within models, our “zip” operation
would be identical to Git Re-Basin’s permute-then-interpolate approach. Thus,
Eq. 8 can be thought of as a generalization of prior work.

Figure 3: ZipIt! merges models layer-wise by exploiting redundancy in their
features. (a) Output features $f^{A}$ and $f^{B}$ from two disjoint layers are
(b) paired with other features based on the similarity of their activations.
(c) We produce a merge matrix M to combine the pairs into a single shared
output feature space, and a corresponding unmerge matrix U that undoes this
operation. (d) We then propagate U up the network to align the next layer’s
input space, and simultaneously receive the previous layer’s U to align our
input space. (e) We apply Eq. 8 to “zip” the layers together using the M for
the output and U for the input, producing a single layer (f). We then repeat
(a) on the next layer.

Figure 4: Zip Propagation. We propagate $M_{i}$ backward until we hit a layer
with weights, merging merging element-wise layers (e.g., BatchNorm) along the
way.

Figure 5:  Partial Zip. (a) If we stop zipping early and (b) apply the latest
U from the zip propagation to the inputs of the first unmerged layer in each
model, (c) we get a multi-head model with a head for each task.

###  4.2 Zip Propagation

However, most modern neural networks are not simply collections of linear
layers stacked on top of each other. In practice, we cannot combine merge and
unmerge matrices into every layer of the network, as a local zip (Eq. 8)
expects the layer to have a weight matrix—i.e., the layer has to have separate
input and output spaces so that we can unmerge the input space and merge the
output space. Other layers (e.g., BatchNorm, ReLU) don’t have such a weight
matrix.

Thus, we “propogate” $M_{i}$ and $U_{i}$ through these layers. For instance,
in Fig. 5, we show a common stack of layers found in a typical ConvNet.
Following Jordan et al. (2022), we compute $M_{i}$ and $U_{i}$ using the
activations of the network (i.e., after each ReLU). We can’t fuse $M_{i}$ with
the ReLU layer, as it doesn’t have any parameters. Similarly, we can merge the
parameters of the preceding BatchNorm layer (i.e., in the same way as bias).
But it doesn’t have a weight matrix, so we also can’t fuse $M_{i}$ into it.
Only once we’ve reached the Conv layer can we fuse $M_{i}$ and $U_{i}$ into it
using Eq. 8 (in this case, treating each kernel element as independent).

Similar care needs to be taken with skip connections, as every layer that
takes input from or outputs to a skip connection shares the same feature
space. However, this too can be dealt with during propagation—we just need to
propagate $M_{i}$ backward and $U_{i}$ forward to each layer connected by the
same skip connection. In general, we can define propagation rules to handle
many different types of network modules (see Appendix C).

###  4.3 Extensions

##### Partial Zip.

We don’t always want to zip every layer of the two networks, especially if
their output spaces are incompatible, or if doing so would lose too much
accuracy. Instead, we can perform a partial zip. That is, we zip most of the
layers together, but leave the later ones unzipped (Fig. 5).

Implementing this operation is simple in our framework: zip as normal until
the specified layer $i$, then the remaining unzipped layers will receive
$U_{i}$ through zip propagation. If we apply $U_{i}^{A}$ to $L_{i+1}^{A}$ and
$U_{i}^{B}$ to $L_{i+1}^{B}$, the remaining unzipped layers will form “heads”
that expect merged features as input. We can then ensemble the heads or choose
one to evaluate at runtime.

##### Repeated Matching ($\alpha$).

In some cases, we’d like to merge more than two models together. To do this,
we allow “repeated matches”. That is, when two features are matched in our
greedy algorithm, they are removed and replaced with the resulting merged
feature. To ensure that one feature doesn’t get merged endlessly, we set the
correlations of the new feature to be the minimum of the old features’
similarities weighted by $\alpha\in(0,1]$. We find a small value of $\alpha$
typically works best.

##### Same-model Budget ($\beta$).

To demonstrate the effectiveness of same-model merges, we introduce a “budget”
parameter $\beta\in[0,1]$ that denotes what percent of total merged features
can come from models merging within themselves, with each model receiving an
equal portion of this budget. Note that a budget of 0 results in Eq. 2, as in
that case no features can be merged within models.

##  5 Results

There is no standard benchmark to evaluate merging approaches on models from
distinct tasks, so we construct our own. We evaluate our approach in two
different settings. (1) A versatile test-bed: disjoint category splits of the
same dataset (i.e., same dataset and different label sets). (2) A very
challenging setting: completely different datasets and tasks (i.e., different
datasets and label sets).

##### Experimental Details.

For each experiment where we sample multiple disjoint splits of categories, we
hold one split out for hyperparameter search and report mean and standard
deviation on the rest. For experiments with models trained on different
datasets, we subsample the validation set into a validation and test set to
use for the same purpose. To compute correlations, we use a portion of the
training set for each dataset as in Li et al. (2015) (see Appendix B). For a
fair comparison, we reset the batch norms for all methods (including the
original models) using the training data (following the recommendation in
Jordan et al. (2022)). For our method, ZipIt!${}_{\text{n/m}}$ indicates that
$n$ out of the $m$ layers in the network have been zipped (Sec. 4.3). Note,
all our models have different initializations.

##### Evaluation.

For the setting with disjoint class splits of the same dataset, we evaluate
performance in two ways: joint accuracy and per task accuracy. For joint
accuracy, we evaluate each model over all classes in the combined dataset. For
per task accuracy, we compute the accuracy of each task individually (i.e.,
supposing we had task labels at runtime) and then report the average. The
former is similar to a continual learning setting where we want to augment the
knowledge of the model, while the latter is akin to a multi-task setting where
we know which task we’re using at test time. For the scenario where we merge
models trained on different datasets, we use the per task accuracy metric, as
the label spaces are not comparable.

##### Baselines.

In addition to the default Weight Matching version of Git Re-Basin (Ainsworth
et al., 2022), we compare to two baselines: Weight Averaging (Eq. 1) and
Permute (Eq. 2) with $\gamma=\nicefrac{{1}}{{2}}$ using our framework (i.e.,
we set $M_{i}$ and $U_{i}$ such that Eq. 8 is equivalent). For Permute, we use
linear sum assignment to find optimal permutations (following Li et al.
(2015)). Note that our Permute is a strong baseline we create using our
framework and is more accurate than Git Re-Basin in our settings. It’s also
similar to REPAIR (Jordan et al., 2022), but without adding extra parameters
to the model. Finally, with perfect merging, the merged model’s outputs would
be identical to the originals. Thus we include Ensemble as an upper bound
(executing and concatenating the results of both models).

|  | Accuracies (%)
---|---|---
Method |  FLOPs (G) |  Joint |  Task A |  Task B |  Avg
Model A |  0.68 |  48.2$\pm$1.0 |  97.0$\pm$0.6 |  45.1$\pm$8.6 |  71.0$\pm$4.4
Model B |  0.68 |  48.4$\pm$3.8 |  49.1$\pm$9.3 |  96.1$\pm$1.1 |  72.6$\pm$4.9
W. Avg (Eq. 1) |  0.68 |  43.0$\pm$1.6 |  54.1$\pm$1.4 |  67.5$\pm$1.2 |  60.8$\pm$4.5
Git Re-Basin‡ |  0.68 |  46.2$\pm$0.8 |  76.8$\pm$8.9 |  82.7$\pm$5.1 |  79.8$\pm$6.5
Permute (Eq. 2) |  0.68 |  58.4$\pm$6.8 |  86.6$\pm$2.1 |  87.4$\pm$1.1 |  87.4$\pm$1.4
ZipIt!${}_{\text{20/20}}$ |  0.68 |  79.1$\pm$1.1 |  92.9$\pm$1.1 |  91.2$\pm$1.4 |  92.1$\pm$1.0
Ensemble |  1.37 |  87.4$\pm$2.6 |  97.0$\pm$0.6 |  96.1$\pm$1.1 |  96.6$\pm$0.4
ZipIt!${}_{\text{13/20}}$ |  0.91 |  83.8$\pm$3.1 |  95.1$\pm$0.7 |  94.1$\pm$1.5 |  94.6$\pm$0.6

(a)

|  | Accuracies (%)
---|---|---
Method |  FLOPs (G) |  Joint |  Task A |  Task B |  Avg
Model A |  2.72 |  41.6$\pm$0.3 |  82.9$\pm$0.7 |  24.8$\pm$0.4 |  53.9$\pm$0.5
Model B |  2.72 |  41.6$\pm$0.2 |  25.1$\pm$1.2 |  82.8$\pm$0.2 |  54.0$\pm$0.6
W. Avg (Eq. 1) |  2.72 |  17.0$\pm$1.7 |  23.8$\pm$6.9 |  24.8$\pm$5.9 |  24.3$\pm$1.9
Git Re-Basin‡ |  2.72 |  40.9$\pm$0.2 |  57.3$\pm$1.5 |  56.7$\pm$0.7 |  57.0$\pm$0.8
Permute (Eq. 2) |  2.72 |  42.8$\pm$0.7 |  61.6$\pm$1.4 |  60.5$\pm$0.5 |  61.0$\pm$0.8
ZipIt!${}_{\text{20/20}}$ |  2.72 |  54.9$\pm$0.8 |  68.2$\pm$0.8 |  67.9$\pm$0.6 |  68.0$\pm$0.4
Ensemble |  5.45 |  73.5$\pm$0.4 |  82.9$\pm$0.7 |  82.8$\pm$0.2 |  82.8$\pm$0.4
ZipIt!${}_{\text{13/20}}$ |  3.63 |  70.2$\pm$0.4 |  80.3$\pm$0.8 |  80.1$\pm$0.7 |  80.2$\pm$0.6

(b)

Table 1: CIFAR Results. ZipIt! vs. baselines on combining a model trained on
half the classes (Task A) with one trained on the other half (Task B) without
extra training. We report both joint (10/100-way) and per-task (5/50-way)
accuracy. ZipIt! significantly outperforms its baseline and closes in on the
upper bound (ensemble accuracy). ${\ddagger}$ refers to Ainsworth et al.
(2022).

###  5.1 CIFAR-10 and CIFAR-100

We train 5 pairs of ResNet-20 (He et al., 2016) from scratch with different
initializations on disjoint halves of the CIFAR-10 and CIFAR-100 classes
(Krizhevsky et al., 2009). While ZipIt! supports “partial zipping” to merge
models with different outputs (in this case, disjoint label sets), prior
methods without retraining do not. To make a fair comparison, we train these
CIFAR models with a CLIP-style loss (Radford et al., 2021) using CLIP text
encodings of the class names as targets. This way, both models output into the
same CLIP-space regardless of the category. Note, this means the models are
capable of some amount of zero-shot classification on the tasks they were not
trained on.

##### CIFAR-10 (5+5).

In Tab. LABEL:tab:cifar5+5, we merge models trained on disjoint 5 class
subsets of CIFAR-10 using ResNet-20 with a $4\times$ width multiplier (denoted
as ResNet-20$\times$4). In joint classification (i.e., 10-way), Git Re-Basin
is unable to perform better than using either of the original models alone,
while our Permute baseline performs slightly better. In stark contrast, our
ZipIt! performs a staggering 32.9% better than Git Re-Basin and 20.7% better
than our baseline. If allow the last stage of the network to remain unzipped
(i.e., zip up to 13 layers), our method obtains 83.8%, which is only 3.6%
behind an ensemble of model A and model B (which is practically the upper
bound for this setting). We also achieve similar results when merging VGG11
models in this setting (Appendix D).

##### CIFAR-100 (50+50).

We find similar results on disjoint 50 class splits of CIFAR-100 in Tab.
LABEL:tab:cifar50+50, this time using an $8\times$ width multiplier instead.
Like with CIFAR-10, Git Re-Basin fails to outperform even the unmerged models
themselves in joint classification (i.e., 100-way), and this time Permute is
only 1.2% ahead. ZipIt! again significantly outperforms prior work with +14%
accuracy over Git Re-Basin for all layers zipped, and a substantial +29.2% if
zipping 13/20 layers. At this accuracy, ZipIt!13/20 is again only 3.3% behind
the ensemble for joint accuracy and 2.6% behind for average per task accuracy,
landing itself in an entirely different performance tier compared to prior
work.

|  | Accuracies (%)
---|---|---
Method |  FLOPs (G) |  Joint |  Task A |  Task B |  Avg
Model A |  4.11 |  37.2$\pm$2.0 |  74.3$\pm$4.0 |  0.5$\pm$0.1 |  37.4$\pm$2.0
Model B |  4.11 |  35.3$\pm$1.6 |  0.5$\pm$0.1 |  70.5$\pm$3.2 |  35.5$\pm$1.6
W. Avg (Eq. 1) |  4.11 |  0.3$\pm$0.1 |  0.6$\pm$0.1 |  0.7$\pm$0.1 |  0.6$\pm$0.1
Git Re-Basin‡ |  4.11 |  3.1$\pm$1.2 |  5.3$\pm$2.6 |  5.7$\pm$2.4 |  5.5$\pm$1.7
Permute (Eq. 2) |  4.11 |  8.6$\pm$5.8 |  10.1$\pm$4.4 |  15.3$\pm$11.1 |  12.7$\pm$7.7
ZipIt!${}_{\text{50/50}}$ |  4.11 |  8.6$\pm$4.7 |  12.4$\pm$5.9 |  14.7$\pm$7.8 |  13.5$\pm$6.6
Ensemble |  8.22 |  63.3$\pm$4.9 |  74.3$\pm$4.0 |  70.5$\pm$3.2 |  72.4$\pm$2.5
ZipIt!${}_{\text{22/50}}$ |  6.39 |  55.8$\pm$4.1 |  65.9$\pm$2.5 |  64.1$\pm$3.0 |  65.0$\pm$2.3
ZipIt!${}_{\text{10/50}}$ |  7.43 |  60.9$\pm$4.1 |  70.7$\pm$3.0 |  69.0$\pm$2.9 |  69.9$\pm$1.9

Table 2: ImageNet-1k (200+200) Results. Merging ResNet-50 models trained from
scratch on disjoint 200 category subsets (Task A and B) of ImageNet-1k. Prior
work performs poorly, but ZipIt! makes this task feasible. ‡Ainsworth et al.
(2022).

###  5.2 ImageNet-1k (200+200)

To test our method on the much harder setting of large-scale data, we train 5
differently initialized ResNet-50 models with cross entropy loss on disjoint
200 class subsets of ImageNet-1k (Deng et al., 2009). To compare to prior work
that doesn’t support partial zipping, we initialize the models with capacity
for all 1k classes, but only train each on their subset.

In Tab. 2 we show results on exhaustively merging pairs from the 5 models. To
compute joint (i.e., 400-way) accuracy, we softmax over each task’s classes
individually (like in Ahn et al. (2021)), and take the argmax over the
combined 400 class vector. On this extremely difficult task, Git Re-Basin only
obtains 3.1% for joint accuracy (with random accuracy being 0.25%). Both the
Permute baseline and ZipIt! with all layers zipped perform better, but with
each at 8.6%, are still clearly lacking. Note that we find the same-model
merging budget $\beta$ to not matter for this set of models (see Fig. 7),
which suggests that there’s not a lot of redundant information within each
model in this setting. Thus, ZipIt! chooses to merge mostly across models
instead, performing similarly to the permute baseline. We find this same trend
in CIFAR with smaller models (see Fig. 7), and may be an artifact of model
capacity. The story changes when we increase the capacity of the merged model
by partial zipping: ZipIt!10/50 reaches close to upper bound ensemble accuracy
on this extremely difficult task, while saving on FLOPs.

###  5.3 Multi-Dataset Merging

We now take our model merging framework one step further by merging
differently initialized models trained on completely separate datasets and
tasks. We present two settings: merging multiple classification datasets and
merging semantic segmentation with image classification.

|  | Per-Task Accuracies (%)
---|---|---
Method |  FLOPs (G) |  SD |  OP |  CUB |  NAB |  Avg
Merging Pairs
W. Avg (Eq. 1) |  4.11 |  12.9 |  18.2 |  13.9 |  0.2 |  11.3
Permute (Eq. 2) |  4.11 |  46.2 |  47.6 |  35.6 |  13.5 |  35.7
ZipIt!${}_{\text{49/50}}$ |  4.11 |  46.9 |  50.7 |  38.0 |  12.7 |  37.1
Ensemble |  8.22 |  72.7 |  81.1 |  71.0 |  77.2 |  75.5
ZipIt!${}_{\text{22/50}}$ |  6.39 |  62.6 |  71.2 |  62.8 |  53.0 |  62.4
ZipIt!${}_{\text{10/50}}$ |  7.42 |  66.5 |  75.8 |  65.6 |  66.8 |  68.7
Merging All 4
W. Avg (Eq. 1) |  4.12 |  0.8 |  3.0 |  0.6 |  0.3 |  1.2
Permute (Eq. 2) |  4.12 |  15.7 |  26.1 |  14.0 |  5.3 |  15.3
ZipIt!${}_{\text{49/50}}$ |  4.12 |  21.1 |  33.3 |  8.6 |  3.9 |  16.8
Ensemble |  16.4 |  72.7 |  81.2 |  71.0 |  77.2 |  75.5
ZipIt!${}_{\text{22/50}}$ |  11.0 |  50.2 |  55.9 |  44.0 |  32.0 |  45.5
ZipIt!${}_{\text{10/50}}$ |  14.1 |  63.5 |  70.8 |  63.7 |  63.1 |  65.3

Table 3: Multi-Dataset Results. Merging ResNet-50 models trained on completely
different datasets: Stanford Dogs (SD), Oxford Pets (OP), CUB200 (CUB), and
NABirds (NAB). We report average per-task accuracy over merging model pairs,
and all four.

##### Image Classification Datasets.

Merging ResNet-50 models trained on: Stanford Dogs (Khosla et al., 2011),
Oxford Pets (Parkhi et al., 2012), CUB200 (Welinder et al., 2010), and NABirds
(Van Horn et al., 2015). In Tab. 3, we show the average per task accuracy from
exhaustively merging each pair and the much more difficult setting of merging
all four at once. We report the accuracy of our baselines by applying them up
until the last layer, but we can’t compare to prior work as they don’t support
this setting. As in all our previous experiment we merge without retraining.

For pairs of models, ZipIt! slightly outperforms our permute baseline across
all tasks and performs similarly when merging all 4 models at once. However,
if we add capacity to the merged model through partial zipping, we perform up
to 33% better on merging pairs and 50% better on merging all four models than
the permute baseline. Partial zipping is a significant factor to obtain strong
performance, especially with more than 2 models.

##### Multiple Output Modalities.

In Appendix F, we combine across modalities by merging the ResNet-50 backbone
of a DeeplabV3 (Chen et al., 2017) segmentation model with an ImageNet-1k
classification model. The resulting combined model can perform both semantic
segmentation and image classification. Even with half of layers merged, ZipIt!
retains good performance on both tasks.

Figure 7: Varying $\beta$ and Model Scale. Left: We find when the model has
enough capacity for the task, a high budget (Sec. 4.3) improves performance.
Right: ZipIt! makes effective use of extra model capacity to quickly reach the
ensemble on CIFAR-100 (50+50) when we increase the width of ResNet-20 models.
In contrast, our baselines only slightly benefit from the extra scale.

##  6 Analysis

##### Merging within Models.

A critical piece of ZipIt! compared to prior work is the ability to merge
within models, not just across models. In Sec. 4.3, we introduce a budget
parameter $\beta$ to limit the number of same-model merges, and here use
CIFAR-100 (50+50) and ImageNet-1k (200+200) to illustrate its effectiveness
(Fig. 7a). On CIFAR, same-model merges are very important, with the optimal
budget being above 0.8, meaning 80% of merges are allowed to be within the
same model. This is not the case, however, on ImageNet, where the difficulty
of the task means there likely are much fewer redundant features within each
model.

##### Model Scale.

In Fig. 7b, we test the effect of model scale directly by evaluating joint
accuracy on our CIFAR-100 (50+50) setting with ResNet-20 models of increasing
width. Here, we explicitly see that when the width of the models are too small
for the task (e.g., $<4\times$), ZipIt! and the Permute baseline perform
identically (though both much better than Git Re-Basin). However, when the
scale increases, ZipIt! trends toward the ensemble upper bound of 75%, while
both the Permute baseline and Git Re-Basin plateau at around 45%. This
corroborates Eq. 4 and indicates our method uses the extra model capacity
effectively, much better than prior work.

Algorithm |  A$\leftrightarrow$A/B$\leftrightarrow$B? |  Acc |  Time
---|---|---|---
Identity (Eq. 1) |  ✗ |  43.0$\pm$3.1 |  1.8 ms
Permute (Eq. 2) |  ✗ |  58.4$\pm$1.3 |  28 ms
K-Means |  ✓ |  29.1$\pm$5.5 |  19 sec
Zip (Eq. 8)
Optimal Match |  ✓ |  79.6$\pm$1.7 |  11 min
Greedy Match |  ✓ |  79.0$\pm$1.8 |  1.1 sec
Greedy, $\alpha$=0.1 |  ✓ |  79.1$\pm$2.1 |  1.2 sec

Table 4: Comparing Matching Algorithms to use for $M_{i}$ on CIFAR-10 (5+5)
joint 10-way accuracy. Permuting B$\rightarrow$A as in prior work (Eq. 2)
performs poorly. We significantly improve by merging features within each
model (Eq. 8). Our greedy approach is nearly as accurate as the optimal
algorithm while being two orders of magnitude faster.

##### Matching Algorithm.

In Tab. 4, we compare matching algorithms used to compute $M_{i}$ in Eq. 8.
Using either the identity (weight averaging) or a permutation (as in prior
work) underperforms on CIFAR-10 (5+5) joint 10-way classification. In
contrast, we obtain up to 21.2% higher accuracy if we allow both permutations
and merging within models. However, doing this optimally is difficult, as the
standard linear sum assignment algorithm assumes bipartite matches. We could
use a optimal graph-based solver (e.g., Hagberg et al. (2008)) instead, but
doing so is prohibitively slow (11 minutes to transform a ResNet-20$\times$4
model). Thus, we find matches greedily by repeatedly taking the most
correlated pair of features without replacement. This performs almost as well,
and is multiple orders of magnitude faster. If we allow repeated matches (Sec.
4.3), we obtain a slightly better result. Like Bolya et al. (2023), we find
that matching is better for merging features than clustering (K-Means).

##  7 Conclusion

In this paper, we tackle the extremely difficult task of merging models
trained on completely disjoint tasks without additional training. We find that
prior work underperforms in this setting and posit that they neither fully (1)
exploit model similarities nor (2) account for model dissimilarities. We
introduce ZipIt!, a general framework for merging models that addresses these
issues, and show it to significantly outperform prior work across several
difficult settings, comprehensively analyzing each.

##### Reproducibility Statement.

To ensure reproducibility, we will release code for our algorithm,
experiments, and baselines. We also include algorithm details in Section 4 and
further in Appendix C, experimental details in Section 5 and Appendix B, and a
proof of our Theorem 1 in Appendix G.

##### Acknowledgements.

This work was supported in part by funding from NSF CAREER #2144194, ARL,
Google, and NSF GRFP. All views and conclusions expressed in this work are
those of the authors and not a reflection of these sources.

## References

  * Ahn et al. (2021) Hongjoon Ahn, Jihwan Kwak, Subin Lim, Hyeonsu Bang, Hyojun Kim, and Taesup Moon.  Ss-il: Separated softmax for incremental learning.  In _ICCV_ , 2021.
  * Ainsworth et al. (2022) Samuel K Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa.  Git re-basin: Merging models modulo permutation symmetries.  _arXiv:2209.04836_ , 2022.
  * Ashmore & Gashler (2015) Stephen Ashmore and Michael Gashler.  A method for finding similarity between multi-layer perceptrons by forward bipartite alignment.  In _IJCNN_ , 2015.
  * Baevski et al. (2022) Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli.  Data2vec: A general framework for self-supervised learning in speech, vision and language.  In _ICML_ , 2022.
  * Blanchard et al. (2011) Gilles Blanchard, Gyemin Lee, and Clayton Scott.  Generalizing from several related classification tasks to a new unlabeled sample.  In _NeurIPS_ , 2011.
  * Bolya et al. (2019) Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong Jae Lee.  Yolact: Real-time instance segmentation.  In _ICCV_ , 2019.
  * Bolya et al. (2023) Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman.  Token merging: Your vit but faster.  _ICLR_ , 2023.
  * Cai et al. (2021) Zhaowei Cai, Avinash Ravichandran, Subhransu Maji, Charless Fowlkes, Zhuowen Tu, and Stefano Soatto.  Exponential moving average normalization for self-supervised and semi-supervised learning.  In _CVPR_ , 2021.
  * Caron et al. (2021) Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In _ICCV_ , 2021.
  * Chen et al. (2017) Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam.  Rethinking atrous convolution for semantic image segmentation.  _arXiv preprint arXiv:1706.05587_ , 2017.
  * Choshen et al. (2022) Leshem Choshen, Elad Venezian, Noam Slonim, and Yoav Katz.  Fusing finetuned models for better pretraining.  _arXiv:2204.03044_ , 2022.
  * De Lange et al. (2021) Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars.  A continual learning survey: Defying forgetting in classification tasks.  _TPAMI_ , 2021.
  * Dehghani et al. (2023) Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al.  Scaling vision transformers to 22 billion parameters.  _arXiv:2302.05442_ , 2023.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _CVPR_ , 2009.
  * Don-Yehiya et al. (2023) Shachar Don-Yehiya, Elad Venezian, Colin Raffel, Noam Slonim, and Leshem Choshen.  ColD fusion: Collaborative descent for distributed multitask finetuning.  Association for Computational Linguistics, 2023.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv:2010.11929_ , 2020.
  * Draxler et al. (2018) Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht.  Essentially no barriers in neural network energy landscape.  In _ICML_ , 2018.
  * Entezari et al. (2021) Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur.  The role of permutation invariance in linear mode connectivity of neural networks.  _arXiv:2110.06296_ , 2021.
  * Everingham et al. (2010) M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman.  The pascal visual object classes (voc) challenge.  _International Journal of Computer Vision_ , 88(2):303–338, June 2010.
  * Frankle et al. (2020) Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin.  Linear mode connectivity and the lottery ticket hypothesis.  In _ICML_ , 2020.
  * Freeman & Bruna (2016) C Daniel Freeman and Joan Bruna.  Topology and geometry of half-rectified network optimization.  _arXiv:1611.01540_ , 2016.
  * Garipov et al. (2018) Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson.  Loss surfaces, mode connectivity, and fast ensembling of dnns.  _NeurIPS_ , 2018.
  * Gemmeke et al. (2017) Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter.  Audio set: An ontology and human-labeled dataset for audio events.  In _ICASSP_ , 2017.
  * Grill et al. (2020) Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent-a new approach to self-supervised learning.  _NeurIPS_ , 2020.
  * Gueta et al. (2023) Almog Gueta, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, and Leshem Choshen.  Knowledge is a region in weight space for fine-tuned language models.  _arXiv:2302.04863_ , 2023.
  * Hagberg et al. (2008) Aric A. Hagberg, Daniel A. Schult, and Pieter J. Swart.  Exploring network structure, dynamics, and function using networkx.  In _Proceedings of the 7th Python in Science Conference_ , 2008.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  2016\.
  * He et al. (2017) Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In _ICCV_ , 2017.
  * He et al. (2018) Xiaoxi He, Zimu Zhou, and Lothar Thiele.  Multi-task zipping via layer-wise neuron sharing.  _NeurIPS_ , 2018.
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _NeurIPS_ , 2020.
  * Huang et al. (2017) Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E Hopcroft, and Kilian Q Weinberger.  Snapshot ensembles: Train 1, get m for free.  _arXiv:1704.00109_ , 2017.
  * Ilharco et al. (2022a) Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi.  Editing models with task arithmetic.  _arXiv:2212.04089_ , 2022a.
  * Ilharco et al. (2022b) Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Kornblith, Ali Farhadi, and Ludwig Schmidt.  Patching open-vocabulary models by interpolating weights.  _NeurIPS_ , 2022b.
  * Izmailov et al. (2018) Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson.  Averaging weights leads to wider optima and better generalization.  _UAI_ , 2018.
  * Jordan et al. (2022) Keller Jordan, Hanie Sedghi, Olga Saukh, Rahim Entezari, and Behnam Neyshabur.  Repair: Renormalizing permuted activations for interpolation repair.  _arXiv:2211.08403_ , 2022.
  * Karras et al. (2018) Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In _CVPR_ , 2018.
  * Khosla et al. (2011) Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li.  Novel dataset for fine-grained image categorization: Stanford dogs.  In _CVPR Workshop on Fine-Grained Visual Categorization (FGVC)_ , 2011\.
  * Kirkpatrick et al. (2017) James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  _PNAS_ , 2017.
  * Kornblith et al. (2019) Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton.  Similarity of neural network representations revisited.  In _ICML_ , 2019.
  * Krizhevsky et al. (2009) Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * Krizhevsky et al. (2017) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  _Communications of the ACM_ , 2017.
  * Li et al. (2015) Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft.  Convergent Learning: Do different neural networks learn the same representations?  _arXiv:1511.07543_ , 2015.
  * Li & Hoiem (2017) Zhizhong Li and Derek Hoiem.  Learning without forgetting.  _TPAMI_ , 2017.
  * Liu et al. (2022) Chang Liu, Chenfei Lou, Runzhong Wang, Alan Yuhan Xi, Li Shen, and Junchi Yan.  Deep neural network fusion via graph matching with applications to model ensemble and federated learning.  In _ICML_ , 2022.
  * Matena & Raffel (2021) Michael Matena and Colin Raffel.  Merging models with fisher-weighted averaging.  _arXiv:2111.09832_ , 2021.
  * McMahan et al. (2017) Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas.  Communication-efficient learning of deep networks from decentralized data.  In _Artificial intelligence and statistics_. PMLR, 2017.
  * Muandet et al. (2013) Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf.  Domain generalization via invariant feature representation.  In _ICML_ , 2013.
  * Neyshabur et al. (2020) Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang.  What is being transferred in transfer learning?  _NeurIPS_ , 2020.
  * Parkhi et al. (2012) Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar.  Cats and dogs.  In _CVPR_ , 2012.
  * Peña et al. (2022) Fidel A Guerrero Peña, Heitor Rapela Medeiros, Thomas Dubail, Masih Aminbeidokhti, Eric Granger, and Marco Pedersoli.  Re-basin via implicit sinkhorn differentiation.  _arXiv:2212.12042_ , 2022.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _ICML_ , 2021.
  * Ramé et al. (2022) Alexandre Ramé, Kartik Ahuja, Jianyu Zhang, Matthieu Cord, Léon Bottou, and David Lopez-Paz.  Recycling diverse models for out-of-distribution generalization.  _arXiv:2212.10445_ , 2022.
  * Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _CVPR_ , 2022.
  * Simsek et al. (2021) Berfin Simsek, François Ged, Arthur Jacot, Francesco Spadaro, Clement Hongler, Wulfram Gerstner, and Johanni Brea.  Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances.  In _Proceedings of the 38th International Conference on Machine Learning_. PMLR, 2021.
  * Singh & Jaggi (2020) Sidak Pal Singh and Martin Jaggi.  Model fusion via optimal transport.  _NeurIPS_ , 2020.
  * Sung et al. (2023) Yi-Lin Sung, Linjie Li, Kevin Lin, Zhe Gan, Mohit Bansal, and Lijuan Wang.  An empirical study of multimodal model merging, 2023.
  * Tarvainen & Valpola (2017) Antti Tarvainen and Harri Valpola.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  _NeurIPS_ , 2017.
  * Tatro et al. (2020) Norman Tatro, Pin-Yu Chen, Payel Das, Igor Melnyk, Prasanna Sattigeri, and Rongjie Lai.  Optimizing mode connectivity via neuron alignment.  _NeurIPS_ , 2020.
  * Van Horn et al. (2015) Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge Belongie.  Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection.  In _CVPR_ , 2015.
  * Von Oswald et al. (2020) Johannes Von Oswald, Seijin Kobayashi, Joao Sacramento, Alexander Meulemans, Christian Henning, and Benjamin F Grewe.  Neural networks with late-phase weights.  _arXiv:2007.12927_ , 2020.
  * Wang et al. (2020) Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni.  Federated learning with matched averaging.  _arXiv:2002.06440_ , 2020.
  * Wang et al. (2022) Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip Yu.  Generalizing to unseen domains: A survey on domain generalization.  _TKDE_ , 2022.
  * Welinder et al. (2010) P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona.  Caltech-UCSD Birds 200.  Technical Report CNS-TR-2010-001, California Institute of Technology, 2010\.
  * Wortsman et al. (2022a) Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al.  Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.  In _ICML_ , 2022a.
  * Wortsman et al. (2022b) Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al.  Robust fine-tuning of zero-shot models.  In _CVPR_ , 2022b.
  * Yadav et al. (2023) Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal.  Resolving interference when merging models.  _arXiv:2306.01708_ , 2023.
  * Yamada et al. (2023) Masanori Yamada, Tomoya Yamashita, Shin’ya Yamaguchi, and Daiki Chijiwa.  Revisiting permutation symmetry for merging models between different datasets, 2023.
  * Yurochkin et al. (2019) Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni.  Bayesian nonparametric federated learning of neural networks.  In _ICML_ , 2019.
  * Zhai et al. (2022) Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer.  Scaling vision transformers.  In _CVPR_ , 2022.
  * Zhou et al. (2017) Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba.  Scene parsing through ade20k dataset.  In _CVPR_ , 2017.

##  Appendix A Partial Zipping

In Fig. 9 we plot the average per task accuracy by the number of layers zipped
in ResNet-20$\times$8 for CIFAR-100 (50+50) and ResNet-50 for ImageNet-1k
(200+200). Note that to avoid adding extra unmerge modules into the network,
our stopping point while unzipping has to be the end of a stage.

In Table 5, we show the average neuron correlations at each partial-zipping
stage between the layers of ResNet-20 ($8\times$) models trained on the
CIFAR-100 (50+50) task. We collect results using the same models used in Table
LABEL:tab:cifar50+50, and compute correlations as described in Section 4.
Overall, we find the correlations between models consistently decreases
through successive partial zipping locations. This corroborates the finding of
Kornblith et al. (2019) that model layers become increasingly dissimilar with
depth, as they encode more task-specific features. Coupling Table 5 with
Figure LABEL:fig:partial_zip_cifar100, we observe a direct correlation between
layer-(dis)similarities and performance decrease. This illustrates the
importance of layer similarity between two networks and strong performance.

(a)

(b)

Figure 8: Varying Partial Zip. By leaving some layers unzipped (Sec. 4.3), we
can recover a significant amount of performance while still merging most of
the model.

Average Stage Correlations
---
Layer $\nicefrac{{7}}{{20}}$ |  Layer $\nicefrac{{13}}{{20}}$ |  Layer $\nicefrac{{19}}{{20}}$
0.50$\pm$0.01 |  0.37$\pm$0.00 |  0.27$\pm$0.00

Table 5: CIFAR-100 (50+50) Zipping Correlations. We show the average
correlations between two ResNet-20 ($8\times$ width) models at each partial
zipping stage. Correlations consistently decrease at each successive stage,
indicating that the layers of the two models increasingly diverge.

(a)

(b)

Figure 9:  Data Usage. How much data do we need to use to compute activations?
We find that only a few hundred images are needed to obtain the best
performance. Data augmentation is not always useful.

##  Appendix B Data Usage

In our approach, we use a sample of the training set in order to compute
activations and match features together. For the main paper, we used the full
training set for CIFAR, 1% of the training set for ImageNet, and the number of
images in the smallest training set for the Multi-Dataset classification
experiment (so that we could use the same number of images from each dataset).
In each case, we used the same data augmentations from training.

That begs the question: how much data do we actually need, and how necessary
are data augmentations? Here we ablate the amount of data used for our
CIFAR-100 (50+50) ResNet-20 ($8\times$ width) and ImageNet (200+200) Resnet-50
($\nicefrac{{22}}{{50}}$ layers) experiments. In Fig. 9, we test how much data
is actually necessary to obtain a good accuracy on CIFAR and ImageNet with or
without data augmentation.

We ultimately find that the amount of data doesn’t actually matter that much.
In the main paper, we use the entire training set for CIFAR-100 with a batch
size of 500 (100 batches, or 50,000 images), but it seems like as little as 2
batches (100 images) produces the same result. Similarly on ImageNet, using
0.05% of the data (640 images) produces the same result as 5% (64,048 images).

In fact, the main consideration is whether or not to use data augmentation.
For less diverse datasets like CIFAR-100, data augmentation seems essential
(giving an almost 4% boost in average task accuracy), and well above the
variance of results without augmentation. However, for ImageNet, which has
much more diverse images, data augmentation actually hurts slightly on
average—though the two are within variance. Note that despite this result, for
consistency we use data augmentation in all experiments.

##  Appendix C Zip Propagation Details

In the main paper we described general rules for zip propagation—namely,
propagate through layers until you reach a module with a weight matrix. Here,
we describe rules more concretely for each layer type needed to define most
convnets.

##### Linear.

Apply $M_{i}$ and $U_{i}$. Stop propagation.

##### Conv.

Apply $M_{i}$ and $U_{i}$ to each kernel location (i.e., move the $k\times k$
kernel dimensions to the batch dimension). Stop propagation.

##### BatchNorm.

Apply $M_{i}$ to all parameters (weight, bias, mean, variance), squaring it
for the variance term. Continue propagation. As Jordan et al. (2022) points
out, we cannot compute the correct variance without knowing the covariance
between the two models (which we don’t have access to). Thus, we reset batch
norms after merging to evaluate the variance correctly.

##### LayerNorm.

Apply $M_{i}$ to all parameters (weight, bias). Continue propagation. Since
LayerNorm computes mean and standard deviation on the fly, we don’t need to do
anything special.

##### ReLU.

Nothing to merge. Continue propagation. Note that passing the merge matrix
unchanged through the ReLU layers is an approximation, since we’re using a
linear merge operation on nonlinear features. Addressing this issue could be
an interesting topic for future work, as even the permute and add approach of
prior work has this issue (ReLU is invariant to permutation, but certainly not
adding).

##### Avg / Max Pool.

Nothing to Merge. Continue propagation.

##### Skip Connection.

Continue propagation through every input to the skip connection (using the
same $M_{i}$ and $U_{i}$ for each).

##  Appendix D Cross Entropy on CIFAR

|  | Accuracies (%)
---|---|---
Method |  FLOPs (G) |  Joint |  Task A |  Task B |  Avg
Model A |  10.88 |  37.9 |  74.15 |  1.7 |  36.4
Model B |  10.88 |  36.7 |  2.2 |  75.2 |  38.7
W. Avg |  10.88 |  2.7 |  5.0 |  4.9 |  4.9
Git Re-Basin${\ddagger}$ |  10.88 |  3.1 |  5.8 |  5.3 |  5.6
Permute |  10.88 |  20.0 |  30.8 |  32.8 |  31.8
ZipIt!${}_{\text{20/20}}$ |  10.88 |  27.9 |  40.1 |  39.7 |  39.9
Ensemble |  21.76 |  60.5 |  74.2 |  75.2 |  74.7
ZipIt!${}_{\text{13/20}}$ |  14.52 |  38.6 |  51.8 |  52.0 |  51.9
ZipIt!${}_{\text{7/20}}$ |  18.14 |  47.0 |  60.6 |  60.5 |  60.6

Table 6: CIFAR-100 (50+50) Cross Entropy. ZipIt! vs. baselines using ResNet-20
($16\times$ width). Merging the entire model as in prior work produces bad
results when using cross-entropy, hence we use CLIP in the main draft. If we
use partial zipping, we can recover a lot of the lost performance.
${\ddagger}$ refers to (Ainsworth et al., 2022)

In the main paper, we train our CIFAR models with a CLIP (Radford et al.,
2021) loss (using CLIP embeddings of class names as targets). This ensures
that the output spaces of the two models are aligned, which is necessary to
get good accuracy for prior work that merge the entire model together.

##### ResNet.

In Tab. 6, we show results for CIFAR-100 (50+50) where we train with the
normal one-hot targets (i.e., like we did for ImageNet), instead. Immediately,
we see that accuracies of the merged models are much lower across the board,
with no method able to outperform just using one of the original models when
merging the entire network. In fact, Git Re-Basin (Ainsworth et al., 2022)
does almost no better than weight averaging, which gets close to random
accuracy. While ZipIt! without partial zipping also performs worse than the
original models, it still greatly outperforms all prior work. And with partial
zipping, ZipIt! is able to exceed the accuracy of the original models.

Thus, in the case of using cross-entropy loss, partial zipping is extremely
important. Merging the entire model as in prior work fails, since the later
layers of the model are incompatible with each other due to each model having
a different output space. Partial zipping, on the other hand, can mitigate
that issue.

|  | Accuracies (%)
---|---|---
Method |  FLOPs (G) |  Joint |  Task A |  Task B |  Avg
Model A |  0.15 |  44.6 |  89.2 |  21.0 |  55.1
Model B |  0.15 |  44.0 |  23.1 |  88.1 |  55.6
W. Avg |  0.15 |  10.2 |  20.8 |  20.9 |  20.9
Permute |  0.15 |  25.4 |  47.2 |  48.5 |  47.8
ZipIt!${}_{\text{22/22}}$ |  0.15 |  33.2 |  53.8 |  59.9 |  56.5
Ensemble |  0.30 |  66.6 |  89.2 |  88.1 |  88.6
ZipIt!${}_{\text{14/22}}$ |  0.17 |  35.2 |  56.7 |  60.2 |  58.4
ZipIt!${}_{\text{7/22}}$ |  0.27 |  44.5 |  66.0 |  65.1 |  65.5

Table 7: CIFAR-10 (5+5) CE with VGG. ZipIt! vs. baselines using VGG11
($1\times$ width) using Cross Entropy instead of CLIP loss. ZipIt! displays
the same behavior here as it does for ResNet-20 with low width.

##### VGG.

In the main paper, we use ResNets for each experiment, since they are easy to
train and produce strong results. However, in principle ZipIt! can work on any
architecture. For completeness, in Tab. 7, we show results on the CIFAR-10
(5+5) setting with VGG11 ($1\times$ width). Note that this is a much smaller
and weaker model than the ResNet-20s we use in the main paper, so its results
on CIFAR-10 aren’t as strong. Furthermore, we conducted this experiment with a
cross entropy loss, so merging the entire model performs worse than the
original models.

Despite this, we observe a very similar trend to the ResNet-20 models in that
ZipIt! outperforms all baselines and that partial zipping is important for
reaching the accuracy of the original models (in this case, matching not
exceeding). In fact, these results continue a more general trend in that
ZipIt! greatly benefits from larger model scales, making effective use of the
extra capacity. In this case, the scale of the model is quite small, so there
is not as much room in the weights to store the potentially disjoint features
of both models.

|  | Accuracies (%)
---|---|---
Method |  FLOPs (G) |  Joint |  Task A |  Task B |  Avg
$1\times$ Width
Ensemble |  8.22 |  63.3 |  74.3 |  70.5 |  72.4
ZipIt!${}_{\text{50/50}}$ |  4.11 |  8.6 |  12.4 |  14.7 |  13.5
ZipIt!${}_{\text{37/50}}$ |  4.92 |  33.1 |  41.8 |  42.3 |  42.0
ZipIt!${}_{\text{22/50}}$ |  6.39 |  55.8 |  65.9 |  64.1 |  65.0
ZipIt!${}_{\text{10/50}}$ |  7.43 |  60.9 |  70.7 |  69.0 |  69.9
$1.5\times$ Width
Ensemble |  32.6 |  67.8 |  76.7 |  72.6 |  74.7
ZipIt!${}_{\text{50/50}}$ |  16.3 |  9.7 |  13.2 |  16.0 |  14.6
ZipIt!${}_{\text{37/50}}$ |  19.5 |  49.0 |  56.2 |  56.7 |  56.4
ZipIt!${}_{\text{22/50}}$ |  25.5 |  64.1 |  71.6 |  70.4 |  71.0
ZipIt!${}_{\text{10/50}}$ |  29.7 |  66.8 |  74.9 |  72.1 |  73.5

Table 8: ImageNet-1k (200+200) Width Comparison. We show how ZipIt! is able to
make use of the extra model width when merging models together. For instance,
merging 37 layers goes from 33% joint accuracy with $1\times$ width to 49%
with $1.5\times$, while the ensemble only improves by 4%. These models use
cross-entropy, so merging the entire network results in poor performance.

##  Appendix E ImageNet with 1.5x Width

In the main paper, we show that ZipIt! scales very well with increased width
of the model for the CIFAR-100 (50+50) setting. While CIFAR-100 is a
challenging dataset on its own, the natural question is if that same trend
occurs the much harder ImageNet-1k (200+200) setting.

In Tab. 8, we test this by comparing ZipIt! on the original $1\times$ width
ResNet-50 in the main paper with a $1.5\times$ width one. In all cases, except
for the fully zipped model (likely because of the Cross-Entropy loss), ZipIt!
enjoys a large jump in performance from the extra width. For 37 layers, 33.1%
joint accuracy becomes 49.0%. For 22 layers, 55.8% becomes 64.1%. And for 10
layers, 60.9% becomes 66.8%, now only 1% away from the ensemble. Thus, even in
this much more challenging setting, ZipIt! is able to make full use of the
extra model capacity.

##  Appendix F Merging models with different output modalities

|  Accuracy (%) |  mIoU (%)
---|---|---
Method |  ImageNet-1k |  Pascal VOC
W. Avg |  0.8 |  3.3
ZipIt!${}_{\text{49/50}}$ |  23.1 |  6.0
Ensemble |  77.8 |  76.8
ZipIt!${}_{\text{37/50}}$ |  47.7 |  35.0
ZipIt!${}_{\text{22/50}}$ |  60.9 |  64.4
ZipIt!${}_{\text{10/50}}$ |  64.9 |  71.7

Table 9: PASCAL VOC and ImageNet-1k merging models with different output
modalities using a DeepLabV3 ResNet-50 backbone and ImageNet-1k Resnet-50
model.

In this experiment we use ZipIt! to merge two models with different
initializations trained on different tasks with different output modalities:
semantic segmentation and image classification. Specifically, we merge the
ResNet-50 backbone of a DeepLabV3 (Chen et al., 2017) model finetuned on the
Pascal VOC (Everingham et al., 2010) dataset, with a ResNet-50 model trained
on ImageNet-1k. While the DeepLabV3 backbone was itself pre-trained on
ImageNet-1k, it was further finetuned on Pascal VOC and does not share the
initialization of our classification model. Table 9 shows the results of
combining the two ResNet-50 models with ZipIt! at various partial merging
locations. We evaluate the performance of each merged by reporting its
ImageNet-1k accuracy, and its Pascal VOC mIoU as is standard. Overall, we
observe that ZipIt! is capable of merging nearly half the number of ResNet-50
layers between both models while still maintaining good performance on both
tasks, all without any training.

##  Appendix G A Tighter Bound for Linear Mode Connectivity

In this section we demonstrate that merging models by supporting feature
merges both across and within each, yields a tighter bound than Theorem 3.1 in
(Entezari et al., 2021) in its limited setting. We first introduce necessary
background from prior work, including Theorem 3.1 and a particular
formalization for within-model merging borrowed from (Simsek et al., 2021).
Second, we introduce Theorem 1, which produces a tighter bound on Theorem 3.1
when merging within models is allowed, and prove its validity (Section G.2 &
G.3). Third, we provably extend Theorem 1 to a less restrictive setting,
retaining its bounds (Section G.4).

###  G.1 Background

We first introduce Theorem 3.1 from (Entezari et al., 2021). Second, we
formalize a restricted version of within-model merging necessary for our proof
using the definitions from (Simsek et al., 2021).

####  G.1.1 Thoerem 3.1

##### The Theorem.

Let
$f_{\\{{\mathbf{v}},{\mathbf{W}}\\}}(x)={\mathbf{v}}^{T}\sigma({\mathbf{W}}x)$,
$f_{\\{{\mathbf{v^{\prime}}},{\mathbf{W^{\prime}}}\\}}(x)={\mathbf{v}}^{\prime
T}\sigma({\mathbf{W}}^{\prime}x)$ be two fully-connected networks with $h$
hidden units where $\sigma(\cdot)$ is ReLU activation,
${\mathbf{v}}\in\mathbb{R}^{h}$ and ${\mathbf{W}}\in\mathbb{R}^{h\times d}$
are the parameters and $x\in\mathbb{R}^{d}$ is the input. If each element of
${\mathbf{W}}$ and ${\mathbf{W^{\prime}}}$ is sampled uniformly from
$[-1/\sqrt{d},1/\sqrt{d}]$ and each element of ${\mathbf{v}}$ and
${\mathbf{v^{\prime}}}$ is sampled uniformly from $[-1/\sqrt{h},1/\sqrt{h}]$,
then for any $x\in\mathbb{R}^{d}$ such that $\lVert x\rVert_{2}=\sqrt{d}$,
with probability $1-\delta$ over
${\mathbf{W}},{\mathbf{W^{\prime}}},{\mathbf{v}},{\mathbf{v^{\prime}}}$, there
exist a permutation such that

$|f_{\\{\alpha{\mathbf{v}}+(1-\alpha){\mathbf{v^{\prime\prime}}},\alpha{\mathbf{W}}+(1-\alpha){\mathbf{W^{\prime\prime}}}\\}}(x)-\alpha
f_{\\{{\mathbf{v}},{\mathbf{W}}\\}}(x)-(1-\alpha)f_{\\{{\mathbf{v^{\prime}}},{\mathbf{W^{\prime}}}\\}}(x)|=\tilde{O}(h^{-\frac{1}{2d+4}})$
(9)

where ${\mathbf{v^{\prime\prime}}},{\mathbf{W^{\prime\prime}}}$ are permuted
version of ${\mathbf{v^{\prime}}},{\mathbf{W^{\prime}}}$, $\alpha\in[0,1]$ is
an arbitrary interpolation constant, and the left-hand-side of the equality is
the amount an interpolated model differs in output compared to the
interpolation of the original models. (Entezari et al., 2021) show that
minimizing this quantity is analogous to minimizing the barrier (as defined by
Entezari et al. (2021)) in this setting. This is important because it states
that achieving a zero output difference is equivalent to achieving zero-
barrier, which implies that two models are linearly mode connected (LMC).

##### Implications

Theorem 3.1 states that given any two two-layer models with different random
initializations, there exists a permutation for one model such that applying
the permutation makes it linearly mode connected to the second with high
probability, given that the networks are wide enough (i.e. $h$ is large
enough). In other words, it states that any two randomly initialized two-layer
networks are LMC modulo permutation with high likelihood. Entezari et al.
(2021) use this result to conjecture that most well-trained neural networks
with the same architecture and trained on the same task are also LMC modulo
permutation with high likelihood.

Notably however, permutations only allow for merging across models. We will
show how adding the ability to merge within models leads to a tighter bound
than Theorem 3.1 with the same likelihood.

####  G.1.2 A Restricted Formalization of Merging Within Models

##### The Formalization.

Let $\bm{\theta}_{h}=\\{{\mathbf{v}},{\mathbf{W}}\\}$ represent a parameter-
set such that $f_{\\{{\mathbf{v}},{\mathbf{W}}\\}}=f_{\bm{\theta}_{h}}$, and
likewise let
$\bm{\theta^{\prime}}_{h}=\\{{\mathbf{v^{\prime}}},{\mathbf{W^{\prime}}}\\},\text{
s.t.
},f_{\\{{\mathbf{v^{\prime}}},{\mathbf{W^{\prime}}}\\}}=f_{\bm{\theta^{\prime}}_{h}}$.
Given $\bm{\theta}_{h}$, let $\Theta_{h}$ denote the set of all parameter-sets
with functional equivalence to $\bm{\theta}_{h}$. This means that
$\forall\theta\in\Theta_{h}\text{, and }\forall x\in\\{x\in\mathbb{R}^{d}|\
\lVert
x\rVert_{2}=\sqrt{d}\\},f_{\bm{\theta}}(x)=f_{\bm{\bm{\theta}_{h}}}(x)$.
Similarly, let $\Theta_{h}^{\prime}$ be the set of all parameter-sets with
functional equivalence to $\bm{\theta}_{h}^{\prime}$. Following
$\bm{\theta_{h}}$, let $\bm{\theta}_{r}$ be an arbitrary parameter-set for $f$
which has $r$ hidden units instead. Assume $\bm{\theta}_{h}$ can be reduced to
some $\bm{\theta}_{r},r\leq h$ in a function-invariant manner using the
definition of zero-type neurons from (Simsek et al., 2021). This means that
there are $h-r$ total combinations of (1) rows in ${\mathbf{W}}$ that are
copies of one another whose corresponding ${\mathbf{v}}$ elements sum to $0$,
and (2) some zero-elements in ${\mathbf{v}}$. Thus, following Simsek et al.
(2021) there exists a function-and-loss-preserving affine transformation that
reduces $\bm{\theta}_{h}$ to $\bm{\theta}_{r}$. We denote this function as
$M_{h\rightarrow r}\in\mathbb{R}^{r\times h}$, with $M_{h\rightarrow
r}(\bm{\theta}_{h})=\bm{\theta}_{r}$. Note that when $r=h$, $M_{h\rightarrow
r}$ can simply be the identity transformation.

By definition, $\bm{\theta}_{h}$ lies in the expansion manifold of
$\bm{\theta}_{r}$ (Simsek et al., 2021). This means there is a similarly
defined affine transformation $U_{r\rightarrow h}\in\mathbb{R}^{h\times r}$
that can expand $\bm{\theta}_{r}$ back to arbitrary
$\bm{\tilde{\theta}_{h}}\in\Theta_{h}$ lying on the expansion manifold. One
simple way is to extend $\bm{\theta}_{r}$ to $\bm{\tilde{\theta}_{h}}$ by
filling the remaining $h-r$ ${\mathbf{v}}$ elements with $0$ and the $h-r$
${\mathbf{W}}$ rows with arbitrary values. Because the associated
${\mathbf{v}}$ elements for each ${\mathbf{W}}$ row are zero, the values of
each row don’t impact the function output. Note that because $h\geq r$,
$U_{r\rightarrow h}$ can assign $\bm{\theta}_{r}$ into arbitrary new indices
in $\bm{\theta}_{h}$. Thus, $U_{r\rightarrow h}$ act as both a permutation and
expansion transformation. Let $T=U\circ M=U_{r\rightarrow h}(M_{h\rightarrow
r}(\bm{\theta}_{h}))$ be the coupling of the reduction and expansion affine-
transformations that produce new networks of width $h$ from $\bm{\theta}_{r}$.
By definition, any $T$ is a permutation when $M$ is the identity and $U$ is
the permutation matrix. For the remainder of this section, we assume that $T$
further contains a permutation (i.e. $T=P\circ U\circ M$ for some permutation
matrix $P\in\mathbb{R}^{h\times h}$).

We will leverage the concept of zero-type neurons presented in (Simsek et al.,
2021) to obtain a tighter bound on Theorem 3.1.

##### A Note on Novelty.

While we borrow ideas from Simsek et al. (2021), our Theorem is a differs in
theoretical application. First, Simsek et al. (2021) restrict their attention
to overall connectivity across points within expansion manifolds. This is
important because our Theorem and proof do not require models to lie on the
same expansion manifold to be linearly mode connected. Second, our models need
not be reducible to the same $r$. That is, we allow for arbitrary reducibility
between any two network parameter-sets. Our theorem also differs from Theorem
3.1 in (Entezari et al., 2021) in that we extend function-invariant
transformations beyond the permutation matrix, and show that tighter bounds
are achievable in the process. Furthermore, we show that uniformity
assumptions may be relaxed while retaining the same bounds (Section G.4).

###  G.2 A theoretical Result

We now introduce Theorem 1, an extension of Theoerem 3.1 that yields a
strictly tighter bound when the transformations $T$ from Section G.1.2 are
included and $r<h$, and exactly as tight when $r=h$. We leave the proof to the
next section.

##### Theorem 1.

Let
$f_{\\{{\mathbf{v}},{\mathbf{W}}\\}}(x)={\mathbf{v}}^{T}\sigma({\mathbf{W}}x)$,
$f_{\\{{\mathbf{v^{\prime}}},{\mathbf{W^{\prime}}}\\}}(x)={\mathbf{v}}^{\prime
T}\sigma({\mathbf{W}}^{\prime}x)$ be two fully-connected networks with $h$
hidden units where $\sigma(\cdot)$ is ReLU activation,
${\mathbf{v}}\in\mathbb{R}^{h}$ and ${\mathbf{W}}\in\mathbb{R}^{h\times d}$
are the parameters and $x\in\mathbb{R}^{d}$ is the input. If each element of
${\mathbf{W}}$ and ${\mathbf{W^{\prime}}}$ is sampled uniformly from
$[-1/\sqrt{d},1/\sqrt{d}]$ and each element of ${\mathbf{v}}$ and
${\mathbf{v^{\prime}}}$ is sampled uniformly from $[-1/\sqrt{h},1/\sqrt{h}]$,
then for any $x\in\mathbb{R}^{d}$ such that $\lVert x\rVert_{2}=\sqrt{d}$,
with probability $1-\delta$ over
${\mathbf{W}},{\mathbf{W^{\prime}}},{\mathbf{v}},{\mathbf{v^{\prime}}}$, there
exist transformations $T,T^{\prime}$ such that

$\displaystyle|f_{\\{\alpha\tilde{{\mathbf{v}}}+(1-\alpha)\tilde{{\mathbf{v}}}^{\prime},\alpha\tilde{{\mathbf{W}}}+(1-\alpha)\tilde{{\mathbf{W}}}^{\prime}\\}}(x)-\alpha
f_{\\{{\mathbf{v}},{\mathbf{W}}\\}}(x)-(1-\alpha)f_{\\{{\mathbf{v}}^{\prime},{\mathbf{W}}^{\prime}\\}}(x)|$
$\displaystyle\leq\begin{cases}\tilde{O}\left(\left(\frac{h^{2}}{(r+r^{\prime})-h}\right)^{-\frac{1}{2d+4}}\right)&\text{,
}(r+r^{\prime})-h>0\\\ 0&\text{, otherwise}\end{cases}$ (10)

where $\tilde{{\mathbf{v}}},\tilde{{\mathbf{W}}}$ are transformed versions of
${\mathbf{v}},{\mathbf{W}}$ from $T$ and
$\tilde{{\mathbf{v}}}^{\prime},\tilde{{\mathbf{W}}}^{\prime}$ are transformed
versions of ${\mathbf{v^{\prime}}},{\mathbf{W^{\prime}}}$ from $T^{\prime}$
respectively. $0<r,r^{\prime}\leq h$ are the hidden unit amounts each network
can be reduced to via its respective $M,M^{\prime}$ transformation before
being expanded back to width $h$ via $P\circ U,P^{\prime}\circ U^{\prime}$,
where $P,P^{\prime}$ are permutation matrices.

##### Implications.

Theorem 1 states that when redundancy exists and can be leveraged in a
network, one can find a transformation that yields strictly lower barrier than
with permutation with any $h$. Moreover, it approaches zero-barrier faster
with increase in $h$ compared to permutations. Although it only explicitly
holds for random initializations—like Theorem 3.1, this theoretical intuition
is supported by our experimental observations. For instance it explains why
algorithms like ZipIt! appear to converge to the ensemble exponentially faster
than permutation methods in Figure 7b). The ensemble achieves zero-barrier,
and ZipIt! is faster to approach it than permutation counterparts because it
can reduce models with minimal deduction in performance.

###  G.3 Theorem 1 Proof

We now derive our proposed Theorem 1. Theorem 1 is very similar to Theorem
3.1— we just add the reducibility property from Section G.1.2. Thus, our
derivation is nearly identical to their Appendix D proof. We fully derive the
novel components of Theorem 1 for clarity, while referring to Appendix D in
(Entezari et al., 2021) for the remaining identical derivations to avoid
redundancy.

Let $\bm{\theta}_{h}=\\{{\mathbf{v}},{\mathbf{W}}\\}$ and
$\bm{\theta}_{h}^{\prime}=\\{{\mathbf{v^{\prime}}},{\mathbf{W^{\prime}}}\\}$
respectively as defined in Section G.1. Suppose each can be reduced to some
$\bm{\theta}_{r},\bm{\theta}_{r}^{\prime}$ respectively with $r\leq h$, via an
appropriate $M_{h\rightarrow r},M_{h\rightarrow r^{\prime}}$ transformation.
Further, let $U_{r\rightarrow h},U_{r^{\prime}\rightarrow h}$ be as defined in
Section G.1, expanding $M_{h\rightarrow r}(\bm{\theta}_{h}),M_{h\rightarrow
r^{\prime}}(\bm{\theta}_{h})$ back to width $h$ by filling in all
$h-r,h-r^{\prime}$ dimensions in ${\mathbf{v}},{\mathbf{v^{\prime}}}$ with $0$
respectively and all $h-r,h-r^{\prime}$ dimensions in
${\mathbf{W}},{\mathbf{W^{\prime}}}$ with some specific values. Finally, let
$T,T^{\prime}$ be transformations defined in Section G.1 with $T=P\circ U\circ
M,T^{\prime}=P^{\prime}\circ U^{\prime}\circ M^{\prime}$ respectively.

Let $\tilde{\bm{\theta}_{h}}=T(\bm{\theta}_{h})$,
$\tilde{\bm{\theta}_{h}^{\prime}}=T^{\prime}(\bm{\theta}_{h}^{\prime})$ be the
new parameter-sets obtained from $T,T^{\prime}$ respectively, and let
$\tilde{\bm{\theta}_{h}}=\\{\tilde{{\mathbf{v}}},\tilde{{\mathbf{W}}}\\}$ and
$\tilde{\bm{\theta}_{h}^{\prime}}=\\{\tilde{{\mathbf{v^{\prime}}}},\tilde{{\mathbf{W^{\prime}}}}\\}$.
By definition, $\tilde{\bm{\theta}_{h}}\in\Theta_{h}$, and
$\tilde{\bm{\theta}_{h}^{\prime}}\in\Theta_{h}^{\prime}$. From the definitions
of $T,T^{\prime}$, ${\mathbf{W}}$ has $h-r$ zero $\tilde{{\mathbf{v}}}$
elements and ${\mathbf{W^{\prime}}}$ has $h-r^{\prime}$
zero-$\tilde{{\mathbf{v}}}^{\prime}$ elements. Now, let us suppose that the
corresponding $h-r$ rows in $\tilde{{\mathbf{W}}}$ are set to copy rows in
$\tilde{{\mathbf{W^{\prime}}}}$, and similarly $h-r^{\prime}$ rows in
$\tilde{{\mathbf{W^{\prime}}}}$ rows are set to copy rows in
$\tilde{{\mathbf{W}}}$. Now, interpolating between any non-zero element and a
zero-element is equivalent to simply scaling the non-zero element: $z:\alpha
0+(1-\alpha)z=(1-\alpha)z$. Thus, so long as $h\leq(h-r)+(h-r^{\prime})$, we
can achieve perfect interpolation by placing $h-r$ elements from
$\tilde{\bm{\theta}}_{h}^{\prime}$ into the zero-elements of
$\tilde{\bm{\theta}}_{h}$ and $h-(h-r)\leq h-r^{\prime}$ elements from
$\tilde{\bm{\theta}}_{h}$ into the zero-elements of
$\tilde{\bm{\theta}}_{h}^{\prime}$. This yields the zero-part of our piece-
wise bound. However, the proof is more involved for the second case when
$h>(h-r)+(h-r^{\prime})\rightarrow(r+r^{\prime})-h>0$. We continue the proof
for this case below.

First, note that we only need to worry about the $(r+r^{\prime})-h$ rows in
${\mathbf{W}},{\mathbf{W^{\prime}}}$ that cannot necessarily be matched with
perfect interpolation as shown above. Let $\mathbb{K}\text{ , and
}\mathbb{K}^{\prime}$ be the set of these rows for each network respectively,
where $|\mathbb{K}|=|\mathbb{K}^{\prime}|=(r+r^{\prime})-h$. These are the
only rows within the two models that must still be considered. For any given
$\xi>0$, we consider the set
$S_{\xi}=\\{-1/\sqrt{d}+\xi,-1/\sqrt{d}+3\xi,\ldots,1/\sqrt{d}-\xi\\}^{d}$, a
discretization of the $\mathbb{R}^{d}$ which has size
$(\frac{1}{\xi\sqrt{d}})^{d}$111Like (Entezari et al., 2021), we choose $\xi$
such that it is a factor of $1/\sqrt{d}$.. For any $s\in S_{\xi}$, let
$C_{s}(\tilde{{\mathbf{W}}})$ be the set of indices of rows in $\mathbb{K}$ of
$\tilde{{\mathbf{W}}}$ that are closest in Euclidean distance to $s$ than any
other element in $S_{\xi}$:

$\displaystyle C_{s}(\tilde{{\mathbf{W}}})$
$\displaystyle=\\{i|\bm{w}_{i}\in\mathbb{K},s=\text{arg min}_{s^{\prime}\in
S_{\xi}}\lVert\mathbf{w}_{i}-s^{\prime}\rVert_{\infty}\\}$ $\displaystyle
C_{s}(\tilde{{\mathbf{W}}}^{\prime})$
$\displaystyle=\\{i|\bm{w}_{i}\in\mathbb{K}^{\prime},s=\text{arg
min}_{s^{\prime}\in S_{\xi}}\lVert\mathbf{w}_{i}-s^{\prime}\rVert_{\infty}\\}$

where for simplicity we assume that arg min returns a single element. These
are the same definitions and assumptions as in (Entezari et al., 2021).

Now for every $s\in\mathcal{S}$ consider a random 1-1 matching (permutation)
of elements in $C_{s}(\tilde{{\mathbf{W}}})\text{ and
}C_{s}(\tilde{{\mathbf{W}}}^{\prime})$. Whenever
$|C_{s}(\tilde{{\mathbf{W}}})|\neq|C_{s}(\tilde{{\mathbf{W}}}^{\prime})|$, we
will inevitably have unmatched indices because permutations only allow 1-1
mappings. Let $I\text{, and }I^{\prime}$ denote the set of total unmatched
indices from $\mathbb{K}\text{, and }\mathbb{K}^{\prime}$ respectively. If
$|C_{s}(\tilde{{\mathbf{W}}})|-|C_{s}(\tilde{{\mathbf{W}}}^{\prime})|\geq 0$,
we add these extra indices that are not matched to $I$ and otherwise we add
them to $I^{\prime}$. Because $\mathbb{K}\text{ , and }\mathbb{K}^{\prime}$
are the same size, $|I|=|I^{\prime}|$ after adding all unmatched indices
across all $C_{s}$. Thus, by definition
$|I|=|I^{\prime}|\leq(r+r^{\prime})-h$.

Pick an arbitrary $s\in\mathcal{S}$. Since each element of $\mathbb{K}$ and
$\mathbb{K}^{\prime}$ is sampled uniformly from $[-1/\sqrt{d},1/\sqrt{d}]$,
for each row in the respective sets, the probability of being assigned to each
$s\in\mathcal{S}_{\xi}$ is a multinomial distribution with equal probability
for each $s$: $\nicefrac{{1}}{{|\mathcal{S}_{\xi}|}}$. Pick an arbitrary $s$.
$|C_{s}(\tilde{{\mathbf{W}}})|$ is the sum over all indicator variables
$W_{i}^{s}=\mathbbm{1}\\{w_{i}\in C_{s}(\tilde{{\mathbf{W}}})\\}$, and the
expected value of this sum,
$E|[C_{s}(\tilde{{\mathbf{W}}})|]=\nicefrac{{[(r+r^{\prime})-h]}}{{|\mathcal{S}_{\xi}|}}$
as there are $(r+r^{\prime})-h$ total rows. Let $(r+r^{\prime})-h=n$. Since
each $W_{i}^{s}$ is between $[0,1]$, we can use Hoeffding’s Inequality to
bound the size of $C_{s}(\tilde{{\mathbf{W}}})$ with high probability

$\displaystyle P(|S_{n}-E[S_{n}]|\geq t)\leq
2\exp\left(\frac{-2t^{2}}{n}\right)$ $\displaystyle(\text{Hoeffding's
Inequality})$ (11) $\displaystyle
P(||C_{s}(\tilde{{\mathbf{W}}})|-E[|C_{s}(\tilde{{\mathbf{W}}})|]|\geq t)\leq
2\exp\left(\frac{-2t^{2}}{n}\right)$ $\displaystyle\because
S_{n}=|C_{s}(\tilde{{\mathbf{W}}})|$ (12) $\displaystyle
P\left(||C_{s}(\tilde{{\mathbf{W}}})|-\frac{(r+r^{\prime})-h}{|\mathcal{S}_{\xi}|}|\geq
t\right)\leq 2\exp\left(\frac{-2t^{2}}{n}\right)$ (13) $\displaystyle
P\left(||C_{s}(\tilde{{\mathbf{W}}})|-\frac{(r+r^{\prime})-h}{|\mathcal{S}_{\xi}|}|\geq
t\right)\leq 2\exp\left(\frac{-2t^{2}}{(r+r^{\prime})-h}\right)$
$\displaystyle\because n=(r+r^{\prime})-h$ (14)

Let $n=(r+r^{\prime})-h$. By taking a union bound over all elements of
$\mathbb{K}$ and $\mathbb{K}^{\prime}$, with probability $1-\delta/3$, the
following holds for all choices of $s$:

$\frac{n}{|S_{\xi}|}-\sqrt{\frac{n}{2}\log{(12|S_{\xi}|/\delta)}}\leq|C_{s}(\tilde{{\mathbf{W}}})|,|C_{s}(\tilde{{\mathbf{W}}}^{\prime})|\leq\frac{n}{|S_{\xi}|}+\sqrt{\frac{n}{2}\log{(12|S_{\xi}|/\delta)}}$
(15)

Note this derivation almost exactly follows (Entezari et al., 2021), except
that we have $n\leq h$, yielding a tighter size-bound.

Using Eq. (15), we can obtain a bound on the cardinality differences between
$C_{s}(\tilde{{\mathbf{W}}})$ and $C_{s}(\tilde{{\mathbf{W}}}^{\prime})$ by
subtracting the minimum value of one from the maximum value of the other:

$\displaystyle||C_{s}(\tilde{{\mathbf{W}}})|-|C_{s}(\tilde{{\mathbf{W}}}^{\prime})||$
$\displaystyle\leq\sup(|C_{s}(\tilde{{\mathbf{W}}})|)-\inf(|C_{s}(\tilde{{\mathbf{W}}})|)$
(16)
$\displaystyle||C_{s}(\tilde{{\mathbf{W}}})|-|C_{s}(\tilde{{\mathbf{W}}}^{\prime})||$
$\displaystyle\leq\left(\frac{n}{|S_{\xi}|}+\sqrt{\frac{n}{2}\log(12|S_{\xi}|/\delta)}\right)$
$\displaystyle-\left(\frac{n}{|S_{\xi}|}-\sqrt{\frac{n}{2}\log(12|S_{\xi}|/\delta)}\right)$
(17)
$\displaystyle||C_{s}(\tilde{{\mathbf{W}}})|-|C_{s}(\tilde{{\mathbf{W}}}^{\prime})||$
$\displaystyle\leq 2\sqrt{\frac{n}{2}\log(12|S_{\xi}|/\delta)}$ (18)

Using Eq. (18) we can bound the size of $I,I^{\prime}$ with probability
$1-\nicefrac{{\delta}}{{3}}$ as follows:

$\displaystyle\sum_{s\in
S_{\xi}}||C_{s}(\tilde{{\mathbf{W}}})|-|C_{s}(\tilde{{\mathbf{W}}}^{\prime})|$
$\displaystyle\leq\sum_{s\in
S_{\xi}}2\sqrt{\frac{n}{2}\log(12|S_{\xi}|/\delta)}$ (19)
$\displaystyle\sum_{s\in
S_{\xi}}||C_{s}(\tilde{{\mathbf{W}}})|-|C_{s}(\tilde{{\mathbf{W}}}^{\prime})|$
$\displaystyle\leq 2|S_{\xi}|\sqrt{\frac{n}{2}\log(12|S_{\xi}|/\delta)}$ (20)
$\displaystyle|I|+|I^{\prime}|=\sum_{s\in
S_{\xi}}||C_{s}(\tilde{{\mathbf{W}}})|-|C_{s}(\tilde{{\mathbf{W}}}^{\prime})|$
$\displaystyle\leq 2|S_{\xi}|\sqrt{\frac{n}{2}\log(12|S_{\xi}|/\delta)}$ (21)
$\displaystyle|I|=|I^{\prime}|=\frac{1}{2}\sum_{s\in
S_{\xi}}||C_{s}(\tilde{{\mathbf{W}}})|-|C_{s}(\tilde{{\mathbf{W}}}^{\prime})|$
$\displaystyle\leq|S_{\xi}|\sqrt{\frac{n}{2}\log(12|S_{\xi}|/\delta)}$
$\displaystyle\because|I|=|I^{\prime}|$ (22)

Note, our Eq. (22) equivalent to Eq. (6) in (Entezari et al., 2021), but in
terms of $n$ instead of $h$.

The remainder of our derivation exactly follows (and achieves identical bounds
to) (Entezari et al., 2021) until directly after their substitution of $\xi$.
To avoid writing an identical derivation, we refer readers to their derivation
following Eq. (6) in their Appendix D until the substitution of $\xi$, and
instead pick up immediately before the substitution of $\xi$:

Let $\epsilon\geq 0$ denote the value of
$|f_{\alpha\tilde{{\mathbf{v}}}+(1-\alpha)\tilde{{\mathbf{v}}}^{\prime},\alpha{\mathbf{W}}+(1-\alpha)\tilde{{\mathbf{W}}}^{\prime}\\}}(x)-\alpha
f_{\\{{\mathbf{v}},{\mathbf{W}}\\}}(x)-(1-\alpha)f_{\\{{\mathbf{v^{\prime}}},{\mathbf{W^{\prime}}}\\}}(x)|$.
Following the derivation of (Entezari et al., 2021), we bound $\epsilon$ as
follows:

$\displaystyle\epsilon=|f_{\\{\alpha\tilde{{\mathbf{v}}}+(1-\alpha)\tilde{{\mathbf{v}}}^{\prime}\\},\\{\alpha{\mathbf{W}}+(1-\alpha)\tilde{{\mathbf{W}}}^{\prime}\\}}(x)-\alpha
f_{\\{{\mathbf{v}},{\mathbf{W}}\\}}(x)-(1-\alpha)f_{\\{{\mathbf{v^{\prime}}},{\mathbf{W^{\prime}}}\\}}(x)|$
$\displaystyle\leq\sqrt{2\log(12/\delta)\log(12h/\delta)\left(\frac{|I|}{h}+\xi^{2}d\right)}$
(23)

Setting $\xi=\epsilon/\sqrt{4d\log(12/\delta)\log(12h/\delta)}$ gives the
following bound on $h$:

$\displaystyle h$
$\displaystyle\leq\frac{4\log(12/\delta)\log(12h/\delta)|I|}{\epsilon^{2}}$
$\displaystyle\leq\frac{4\log(12/\delta)\log(12h/\delta)|S_{\xi}|\sqrt{\frac{n}{2}\log(12|S_{\xi}|/\delta)}}{\epsilon^{2}}$

Therefore, we have:

$\displaystyle h^{2}$
$\displaystyle\leq\left(\frac{4\log(12/\delta)\log(12h/\delta)|S_{\xi}|\sqrt{\frac{n}{2}\log(12|S_{\xi}|/\delta)}}{\epsilon^{2}}\right)^{2}$
(24)
$\displaystyle\leq\left(\frac{4\log(12/\delta)\log(12h/\delta)|S_{\xi}|\sqrt{\log(12|S_{\xi}|/\delta)}}{\epsilon^{2}}\right)^{2}\left(\frac{n}{2}\right)$
$\displaystyle\because|S_{\xi})|=\left(\frac{1}{\xi\sqrt{d}}\right)^{d}$ (25)
$\displaystyle\leq\left(\frac{4\log(12/\delta)\log(12h/\delta)}{\epsilon^{2}}\right)^{d+2}(\log(12/\delta)+d\log(1/\epsilon))(n)$
(26) $\displaystyle\frac{h^{2}}{n}$
$\displaystyle\leq\left(\frac{4\log(12/\delta)\log(12h/\delta)}{\epsilon^{2}}\right)^{d+2}(\log(12/\delta)+d\log(1/\epsilon))$
(27)

Using the inequality in equation (27), we have
$\epsilon=\tilde{O}((\frac{h^{2}}{n})^{-\frac{1}{2d+4}})=\tilde{O}((\frac{h^{2}}{r+r^{\prime}-h})^{-\frac{1}{2d+4}})$.

Thus, we obtain the following piece-wise bound over the barrier:

$\displaystyle|f_{\\{\alpha\tilde{{\mathbf{v}}}+(1-\alpha)\tilde{{\mathbf{v}}}^{\prime},\alpha\tilde{{\mathbf{W}}}+(1-\alpha)\tilde{{\mathbf{W}}}^{\prime}\\}}(x)-\alpha
f_{\\{\tilde{{\mathbf{v}}},\tilde{{\mathbf{W}}}\\}}(x)-(1-\alpha)f_{\\{\tilde{{\mathbf{v}}}^{\prime},\tilde{{\mathbf{W}}}^{\prime}\\}}(x)|$
$\displaystyle\leq\begin{cases}\tilde{O}\left(\left(\frac{h^{2}}{(r+r^{\prime})-h}\right)^{-\frac{1}{2d+4}}\right)&\text{,
}(r+r^{\prime})-h>0\\\ 0&\text{, otherwise}\end{cases}$

$\hfill\square$

###  G.4 Uniformity is not needed: An Extension of Theorem 1

Although Theorem 1 demonstrates a tighter bound compared to Theorem 3.1 is
possible when merging within a model is allowed, its reliance on
${\mathbf{W}},{\mathbf{v}},{\mathbf{W^{\prime}}},{\mathbf{v^{\prime}}}$ being
uniform random variables is unnecessary. Instead, we can assume that
${\mathbf{W}},{\mathbf{W^{\prime}}}$ are sampled from an arbitrary probability
distribution that is bounded on $[-1/\sqrt{d},1/\sqrt{d}]$. Similarly, assume
that ${\mathbf{v}},{\mathbf{v^{\prime}}}$ is sampled from an arbitrary
probability distribution that is both centered and bounded on
$[-1/\sqrt{h},1/\sqrt{h}]$. Note how for both ${\mathbf{v}}\text{, and
}{\mathbf{W}}$ any continuous probability distribution is valid, so long as it
satisfies the stated conditions. We formalize this as follows:

##### Theorem 1.1.

Let $f_{{\mathbf{v}},{\mathbf{W}}}(x)={\mathbf{v}}^{T}\sigma({\mathbf{W}}x)$,
$f_{{\mathbf{v^{\prime}}},{\mathbf{W^{\prime}}}}(x)={\mathbf{v}}^{\prime
T}\sigma({\mathbf{W}}^{\prime}x)$ be two fully-connected networks with $h$
hidden units where $\sigma(\cdot)$ is ReLU activation,
${\mathbf{v}}\in\mathbb{R}^{h}$ and ${\mathbf{W}}\in\mathbb{R}^{h\times d}$
are the parameters and $x\in\mathbb{R}^{d}$ is the input. If each element of
${\mathbf{W}}$ and ${\mathbf{W^{\prime}}}$ is sampled from an continuous
probability distribution that is bounded on $[-1/\sqrt{d},1/\sqrt{d}]$, and
each element of ${\mathbf{v}}$ and ${\mathbf{v^{\prime}}}$ is sampled from an
continuous probability distribution that is centered and bounded on
$[-1/\sqrt{h},1/\sqrt{h}]$, then for any $x\in\mathbb{R}^{d}$ such that
$\lVert x\rVert_{2}=\sqrt{d}$, with probability $1-\delta$ over
${\mathbf{W}},{\mathbf{W^{\prime}}},{\mathbf{v}},{\mathbf{v^{\prime}}}$, there
exist transformations $T,T^{\prime}$ such that

$\displaystyle|f_{\\{\alpha\tilde{{\mathbf{v}}}+(1-\alpha)\tilde{{\mathbf{v}}}^{\prime},\alpha\tilde{{\mathbf{W}}}+(1-\alpha)\tilde{{\mathbf{W}}}^{\prime}\\}}(x)-\alpha
f_{\\{{\mathbf{v}},{\mathbf{W}}\\}}(x)-(1-\alpha)f_{\\{{\mathbf{v}}^{\prime},{\mathbf{W}}^{\prime}\\}}(x)|$
$\displaystyle\leq\begin{cases}\tilde{O}\left(\left(\frac{h^{2}}{(r+r^{\prime})-h}\right)^{-\frac{1}{2d+4}}\right)&\text{,
}(r+r^{\prime})-h>0\\\ 0&\text{, otherwise}\end{cases}$ (28)

where $\tilde{{\mathbf{v}}},\tilde{{\mathbf{W}}}$ are transformed versions of
${\mathbf{v}},{\mathbf{W}}$ from $T$ and
$\tilde{{\mathbf{v}}}^{\prime},\tilde{{\mathbf{W}}}^{\prime}$ are transformed
versions of ${\mathbf{v^{\prime}}},{\mathbf{W^{\prime}}}$ from $T^{\prime}$
respectively. $0<r,r^{\prime}\leq h$ are the hidden unit amounts each network
can be reduced to via its respective $M,M^{\prime}$ transformation before
being expanded back to width $h$ via $U,U^{\prime}$.

##### Proof.

The proof for Theorem 1.1 is takes a very similar form to Theorem 1, with two
differences. For what follows, we assume the same notation and definitions as
in Section G.3 up to Eq. (11), with one change: each element of
$\mathbb{K},\mathbb{K}^{\prime}$ need not be assigned to each $s\in S_{\xi}$
with equal probability.

Despite this change, for a given $s\in S_{\xi}$, we can use the Hoeffding’s
Inequality to bound the size of $C_{s}(\tilde{{\mathbf{W}}})$ with high
probability:

$\displaystyle P(|S_{n}-E[S_{n}]|\geq t)\leq
2\exp\left(\frac{-2t^{2}}{n}\right)$ (29) $\displaystyle
P(||C_{s}(\tilde{{\mathbf{W}}})|-E[|C_{s}(\tilde{{\mathbf{W}}})|]|\geq t)\leq
2\exp\left(\frac{-2t^{2}}{n}\right)$ (30)

Despite $E[|C_{s}(\tilde{{\mathbf{W}}})|]$ no longer being equal for each $s$,
we can take the union bound (1) over the rows of
$\tilde{{\mathbf{W}}},\tilde{{\mathbf{W}}}^{\prime}$ and (2) for each $s\in
S_{\xi}$ to obtain,

$||C_{s}(\tilde{{\mathbf{W}}})|-|C_{s}(\tilde{{\mathbf{W}}}^{\prime})||\leq
2\sqrt{\frac{n}{2}\log(12|S_{\xi}|/\delta)}$ (32)

with probability $1-\nicefrac{{\delta}}{{3}}$. Thus, we achieve the same bound
on the size of $I,I^{\prime}$ as in Section G.3.

The second difference is that $\tilde{{\mathbf{v}}}$ is no longer sampled
uniformly from $[-1/\sqrt{h},1/\sqrt{h}]$. However, this does not affect
anything because we assume $E[\tilde{{\mathbf{v}}}]=0$. Noting these changes,
we can follow the derivation in Section G.3 and achieve the same bounds.

###  G.5 Simplifying variables in Theorem 1

Theorem 1 can be simplified to match its introduction in Section 4. First, let
$\tau,\tau^{\prime}$ denote the proportions of features from $h$ that are
reduced under the $M,M^{\prime}$ transformation in each model. By definition,
$\tau,\tau^{\prime}\in[0,1]$. Then, we can define $r\text{, and }r^{\prime}$
in terms of $\tau,\tau^{\prime}\text{, and }h$:

$r=h(1-\tau)\text{, and }r^{\prime}=h(1-\tau^{\prime})$

This means $(r+r^{\prime})-h=h(1-\tau-\tau^{\prime})\leq
h(1-2\min(\tau,\tau^{\prime}))$. Let $\Gamma=\min(\tau,\tau^{\prime})$, then
we achieve $h(1-2\Gamma)$ in the denominator of our bound, which simplifies to
what is written in Section 4.

##  Appendix H Experiments in Settings of Concurrent Works

Table 10 shows the results of merging $16\times$ width ResNet20 models trained
with cross-entropy loss on the CIAR5+5 task. This setting is equivalent to
that of Table 2 from the concurrent work of (Yamada et al., 2023), except for
two important differences. First, Yamada et al. (2023) add the REPAIR Jordan
et al. (2022) algorithm to each merging method, which significantly improves
the performance of each merging algorithm by adding new parameters to the
merged model.

|  | Accuracies (%)
---|---|---
Method |  FLOPs (G) |  Joint |  Task A |  Task B |  Avg
Model A |  10.88 |  46.7$\pm$0.6 |  94.4$\pm$0.8 |  18.4$\pm$3.1 |  56.4$\pm$1.7
Model B |  10.88 |  40.2$\pm$9.6 |  22.3$\pm$6.9 |  93.8$\pm$1.7 |  58.0$\pm$2.9
W. Avg |  10.88 |  30.4$\pm$2.4 |  46.2$\pm$13.4 |  47.0$\pm$9.3 |  46.6$\pm$3.9
Git Re-Basin$~{}{\ddagger}$ |  10.88 |  27.3$\pm$0.5 |  46.3$\pm$1.6 |  46.3$\pm$9.3 |  38.9$\pm$7.7
Permute |  10.88 |  45.4$\pm$5.3 |  81.3$\pm$7.9 |  79.5$\pm$10.0 |  80.4$\pm$3.8
ZipIt!${}_{\text{20/20}}$ |  10.88 |  66.0$\pm$3.9 |  88.0$\pm$0.9 |  86.7$\pm$4.0 |  87.4$\pm$2.0
Ensemble |  21.76 |  80.0$\pm$2.4 |  94.4$\pm$0.8 |  93.8$\pm$1.7 |  94.1$\pm$0.7
ZipIt!${}_{\text{19/20}}$ |  10.89 |  72.0$\pm$2.5 |  90.0$\pm$1.3 |  88.2$\pm$3.2 |  89.1$\pm$1.7
ZipIt!${}_{\text{13/20}}$ |  14.52 |  73.4$\pm$2.7 |  91.5$\pm$0.9 |  89.6$\pm$3.4 |  90.5$\pm$1.6
ZipIt!${}_{\text{7/20}}$ |  18.14 |  75.5$\pm$2.9 |  92.7$\pm$1.0 |  90.7$\pm$3.3 |  91.7$\pm$1.5

Table 10: CIFAR-10 (5+5) Cross Entropy. ZipIt! vs. baselines using ResNet-20
($16\times$ width). Merging with ZipIt! up to the last layer (ZipIt!19/20)
nearly achieves the ensemble “Task Avg.” performance with half the FLOPs and
vastly outperforms the nearest baseline. Partially merging, brings ZipIt! even
closer to the ensemble. ${\ddagger}$ refers to (Ainsworth et al., 2022)

Second, Yamada et al. (2023) include merging methods that require training in
their Table 2. In contrast, we report the performance of each merging method
using its original capabilities (i.e., without REPAIR), and without any
training (as it is outside our setting). Thus, all results shown in Table 10
are a lower-bound to what is achievable either with REPAIR, or with training.
To make Table 10 as close to Table 2 in (Yamada et al., 2023) as possible, we
report “Joint Acc” as the average of each method’s logits for the ensemble. To
the best of our knowledge, “Joint Acc” is thus the same metric used by (Yamada
et al., 2023). Overall, we observe that ZipIt! fully-merged outperforms the
nearest baseline by over 20% in “Joint Acc”, and zipping up to the
classification layers (ZipIt!19/20) nearly matches the ensemble “Task Avg.”
accuracy without requiring any training. Interestingly, Git Re-Basin performs
especially poorly in this setting, likely requiring REPAIR to achieve the
performance reported in Table 2 by (Yamada et al., 2023).