# Spurious Feature Diversification Improves Out-of-distribution Generalization

Yong Lin1 indicates equal contributions. Corresponding to: Yong
<EMAIL_ADDRESS>Lu Tan2∗ Yifan Hao1∗ Honam Wong1 Hanze Dong1
Weizhong Zhang3 Yujiu Yang2 Tong Zhang1

###### Abstract

Generalization to out-of-distribution (OOD) data is a critical challenge in
machine learning. Ensemble-based methods, like weight space ensembles that
interpolate model parameters, have been shown to achieve superior OOD
performance. However, the underlying mechanism for their effectiveness remains
unclear.

In this study, we closely examine WiSE-FT, a popular weight space ensemble
method that interpolates between a pre-trained and a fine-tuned model. We
observe an unexpected “FalseFalseTrue” phenomenon, in which WiSE-FT
successfully corrects many cases where each individual model makes incorrect
predictions, which contributes significantly to its OOD effectiveness. To gain
further insights, we conduct theoretical analysis in a multi-class setting
with a large number of spurious features. Our analysis predicts the above
phenomenon and it further shows that ensemble-based models reduce prediction
errors in the OOD settings by utilizing a more diverse set of spurious
features. Contrary to the conventional wisdom that focuses on learning
invariant features for better OOD performance, our findings suggest that
incorporating a large number of diverse spurious features weakens their
individual contributions, leading to improved overall OOD generalization
performance. Empirically we demonstrate the effectiveness of utilizing diverse
spurious features on a MultiColorMNIST dataset, and our experimental results
are consistent with the theoretical analysis.

Building upon the new theoretical insights into the efficacy of ensemble
methods, we further identify an issue of WiSE-FT caused by the overconfidence
of fine-tuned models in OOD situations. This overconfidence magnifies the
fine-tuned model’s incorrect prediction, leading to deteriorated OOD ensemble
performance. To remedy this problem, we propose a novel method called BAlaNced
averaGing (BANG) to mitigate the overconfidence problem, which significantly
enhances the OOD performance of WiSE-FT.

##  1 Introduction

Machine learning has seen significant advancements recently. However, the
assumption that testing samples follow the same distribution as training
samples, known as the Identically Independent Distributed (IID) assumption,
can be violated in real-world applications. When a machine learning model
encounters novel testing samples that it hasn’t seen during training, it faces
the out-of-distribution (OOD) generalization problem.

Ensemble-based models (ESM) have achieved significant success in addressing
OOD problems in recent years. Specifically, denote the input as $\bm{x}$ and
the model as $f_{\theta}$ with parameter $\theta$. Given two models
$f_{\bar{\theta}}$ and $f_{\tilde{\theta}}$, existing ESM works typically
consider the output space ensemble (OSE) which outputs
$f_{\bar{\theta}}(\bm{x})+f_{\tilde{\theta}}(\bm{x})$ and the weight space
ensemble (WSE) which outputs $f_{(\bar{\theta}+\tilde{\theta})/2}(\bm{x})$.
WSE is also called weight averaging in literature. [60, 59, 49] show that ESM
can significantly improve the OOD performance and WSE outperforms OSE. Many
works, e.g., [12, 49, 6, 46, 59, 56, 33], adopt WSE to repeatedly improve the
SOTA performance on many OOD benchmarks such as DomainBed [27] and ImageNet
variants [60]. See Appendix B for a detailed discussion on related works.

Consider two types of features for OOD: (1) invariant features that
consistently predict the label across distributions, and (2) spurious features
that have unstable correlations with the label. Existing OOD theories [5, 51,
57, 2, 67] show that an ERM-trained model relying on spurious features can
fail in worst-case. ESM, which combines multiple ERM-trained models, may still
heavily depend on such features and potentially fail in worst-case scenarios
as well. There have been some previous attempts to explain the effectiveness
of model ensemble, but they do not offer satisfactory explanations on the
overall OOD improvement of ESM. Furthermore, the difference between weight and
output space ensemble remains under-explored (a thorough discussion on related
works in Appendix B).

An intriguing phenomenon. To understand the benefits of ESM, we examine the
WiSE-FT [60], which interpolates between a pre-trained and fine-tuned model.
When evaluating OOD datasets, we divided them into four groups based on the
correctness of predictions made by the individual models. Surprisingly, we
found a “FalseFalseTrue” phenomenon: WiSE-FT can correct predictions on
samples where both individual models make incorrect predictions. Further, we
show that two individual models learn different feature sets, and WiSE-FT
utilizes more diverse features. Based on these observations, we then motivate
our theory by a toy example (shown in Figure 1). Suppose we have two models,
$\bar{f}$ and $\tilde{f}$, for a 3-class classification task. For a sample
from the first class, $\bar{f}$ produces logits of (0.4, 0.6, 0), and
$\tilde{f}$ produces logits of (0.4, 0, 0.6). The ensemble model’s prediction
would be (0.4, 0.3, 0.3). This phenomenon can happen when $\bar{f}$ and
$\tilde{f}$ learn different subsets of spurious features, represented as
$\bar{\mathcal{S}}$ and $\tilde{\mathcal{S}}$, respectively. Recall that the
spurious correlations change in OOD. In the example, $\bar{f}$ generates a
high logit (0.6) for the second class influenced by $\bar{\mathcal{S}}$, while
$\tilde{f}$ produces a high logit (0.6) for the third class influenced by
$\tilde{\mathcal{S}}$ (details in Section 2).

Figure 1: Illustration of FalseFalseTrue phenomenon. Consider to classify
camels, cows, and dogs. The invariant feature $\bm{x}_{v}$ is the shape of the
animal. There are 2 spurious features, i.e., 1) the background $\bm{x}_{s,1}$,
e.g., camels are always on the sand, cows are on grass and dogs are on the
floor. 2) the fur of the animals $\bm{x}_{s,2}$, e.g., camels have brown fur,
cows have dotted fur and dogs are all in black in the training dataset.
Suppose we fit two models, $\bar{f}$ and $\tilde{f}$, on the training dataset
independently. Assume that $\bar{f}$ uses the invariant feature $\bm{x}_{v}$
and $\bm{x}_{s,1}$, and $\tilde{f}$ uses $\bm{x}_{v}$ and $\bm{x}_{s,2}$.
$\bar{f}$ and $\tilde{f}$ both correctly predict the label of a sample from
the training distribution. Consider an OOD testing sample of a dog with brown
fur on the grass. $\bar{f}$ puts a large logit for the cow class since the
background(grass) is spuriously correlated with cows, i.e.,
$\bar{f}(\bm{x}_{v},\bm{x}_{s,1})=[0.4,0.6,0]$. $\tilde{f}$ puts a large logit
for the camel class since the texture(brown fur) is spuriously correlated with
camels, i.e., $\tilde{f}(\bm{x}_{v},\bm{x}_{s,2})=[0.4,0,0.6]$. Both $\bar{f}$
and $\tilde{f}$ make mistakes on this sample. However, the average of them can
make correct prediction, i.e.,
$1/2\bar{f}(\bm{x}_{v},\bm{x}_{s,1})+1/2\tilde{f}(\bm{x}_{v},\bm{x}_{s,2})=[0.4,0.3,0.3]$.

A new perspective on OOD generalization. In Section 3, we extend a popular
theoretical setting [51, 57] to a 3-class classification with multiple
spurious features. Our theoretical results predicts the aforementioned
phenomenon. We show that ESM incorporates more diverse spurious features,
which weakens the contributions of individual spurious feature and further
leads to improved overall OOD performance. We also shed light on the
difference between the weight and output space ensemble. Recall that there has
been a significant effort in OOD community to learn invariant features and
discard spurious features [5]. However, these approaches have not shown
satisfactory performance when applied to real-world datasets [27], which may
be due to the fact that invariant learning requires numerous domains [51],
strong regularization [67], and faces additional difficulties induced by non-
linearity [51], overparameterization [35], and optimization challenges [15].
In contrast, our findings offer a new perspective that spurious features
diversification actually improves OOD performance, which can be easily
implemented as shown in ensemble-based models and has achieved remarkable
empirical success. To further verify our findings, we introduce
MultiColorMNIST in Section 3.3, a novel variant of CMNIST [5], with multiple
spurious features. Through empirical analysis, we show that individual models
trained on MultiColorMNIST utilize different spurious features, and their
ensemble achieves superior OOD performance by leveraging this diversity.
Notably, while several methods promote feature diversity to enhance empirical
performance, none of them have explored the spurious features diversification
from a perspective similar to ours (details in Appendix B.2).

An improved method. Our theoretical results indicate that the scaling of
$\bar{f}$ and $\tilde{f}$ should be similar to maintain the improvement of the
model ensemble. If $\tilde{f}$ is much more confident than $\bar{f}$,
resulting in a larger scaling for $\tilde{f}$, the ensemble model can become
biased towards $\tilde{f}$. Unfortunately, the scaling issue arises in WiSE-
FT, which combines a pre-trained model and a fine-tuned model in the weight
space. Empirical evidence shows that the pre-trained model is well calibrated,
whereas the fine-tuned model is highly over-confident on OOD datasets,
indicating a larger scaling compared to the pre-trained model. Based on these
findings, we propose BAlaNced averaGing (BANG), which combines the pre-trained
model with a model fine-tuned by over-confidence preventing methods like Label
Smoothing and MixUp. We demonstrate that BANG improves vanilla WiSE-FT by
approximately 1.9pp in average OOD performance across five ImageNet variants.

To summarize, the following are the main contributions of the paper:

  * •

By examining WiSE-FT, a popular method of ensemble-based models (EBM) that
combines the pre-trained and fine-tuned model in the weight space, we discover
an unexpected ‘FalseFalseTrue’ phenomenon that WiSE-FT can correct a large
fraction of OOD samples on which both individual models make wrong
predictions. We further show that two individual models use different sets of
features and WiSE-FT utilizes more diverse features.

  * •

Through theoretical analysis on a multi-class classification problem with
multiple spurious features, we provide a natural explanation for the observed
phenomenon and show EBM can improve OOD performance by leveraging more diverse
spurious features. Furthermore, we provide novel insights into the distinction
between weight and output space ensemble.

  * •

Contrary to the traditional belief that emphasizes the exclusive learning of
invariant features for OOD, our findings suggest that incorporating diverse
spurious features weakens their individual contributions, leading to improved
overall OOD generalization performance. Through experiments on our
MultiColorMNIST dataset, which contains multiple spurious features, we provide
concrete evidence for the effectiveness of diverse spurious features.

  * •

Based on our theoretical and empirical findings, we show that WiSE-FT can
suffer from the over-confidence problem of the fine-tuned model, which skews
the ensemble and deteriorates the OOD performance. We further propose a novel
method BANG to remedy this problem, and it significantly improves the OOD
performance.

##  2 Understanding Ensemble-based Models via Examining WiSE-FT

The FalseFalseTrue phenomenon. In this section, we closely examine WiSE-FT
[60] to obtain intuition on why EBM can improve OOD performance. Specifically,
[60] ensemble pre-trained CLIP and the model fine-tuned on ImageNet in the
weight space. In Appendix C.1, we divide each dataset (ImageNet as ID dataset
and five ImageNet variants 111They are ImageNet-V2 [50], ImageNet-R [28],
ImageNet-A [29], ImageNet Sketch [58] and ObjectNet [7]. We refer to them as
IN-V2, IN-R, IN-A, IN-S, and ObjNet for short. More details in Appendix E as
OOD datasets) into 8 groups by whether the pre-trained, fine-tuned and
averaged models make correct predictions. We surprisingly find that WiSE-FT
can correct a substantial part of samples on which both the pre-trained and
fine-tuned models make mistakes. Specifically, we calculate the number of
“FalseFalseTrue” samples, i.e., samples on which WiSE-FT is correct while both
the pre-trained and fine-tuned models are incorrect. We then calculate the
FalseFalseTrue ratio by dividing FalseFalseTrue number over the dataset size.
Figure 2(Left) shows FalseFalseTrue ratio on each OOD dataset and compares it
with “overall improvement”, which is the accuracy improvement of WiSE-FT over
the best of pre-trained and fine-tuned model. We can see that there are
substantial parts of FalseFalseTrue samples in each dataset. Refer to Appendix
C.1 for more details. It is interesting that the FalseFalseTrue ratio is even
higher than the overall improvement in IN-R and IN-A, we provide in-depth
analysis and explanation in Appendix C.1 and E.6.

Figure 2: (Left) FalseFalseTrue ratio; (Right) GradCAM feature visualization.

Illustration on when FalseFalseTrue occurs. In this part, we try to understand
the FalseFalseTrue phenomenon. We first consider the output space ensemble to
be similar to the weight space ensemble in this part and will present an
analysis of their difference in Section 3. Suppose we want to distinguish from
camels, cows, and dogs. There is one invariant feature $\bm{x}_{v}$ (the shape
of the animal) and two spurious features (the background $\bm{x}_{s,1}$ and
the fur of the animal $\bm{x}_{s,2}$). Camels are typically found on sand,
cows on grass, and dogs on the floor. Camels have brown fur, cows have dotted
fur, and dogs are all black in the training dataset. See Fig. 1 for
illustration. Suppose we fit two different models, $\bar{f}$ and $\tilde{f}$
on the training dataset. Further assume $\bar{f}$ uses the feature
$\bm{x}_{v}$ and $\bm{x}_{s,1}$, and $\tilde{f}$ uses $\bm{x}_{v}$ and
$\bm{x}_{s,2}$ 222For simplicity of illustration, we assume that $\bar{f}$ and
$\tilde{f}$ learn the same invariant feature. However, this is not necessary
for EBM to outperform both individual models, as demonstrated in Section 3 .
Both $\bar{f}$ and $\tilde{f}$ correctly predict samples from the training
distribution. Whereas, for a sample from the testing distribution, e.g., a dog
with brown fur ($\bm{x}_{s,2}$) on the grass ($\bm{x}_{s,1}$): $\bar{f}$ puts
a large logit for the cow class since the background, grass, is spuriously
correlated with cow, i.e., $\bar{f}(\bm{x}_{v},\bm{x}_{s,1})=[0.4,0.6,0]$;
$\tilde{f}$ puts a large logit for the camel class since the texture, brown
fur, is spuriously correlated with camel, i.e.,
$\tilde{f}(\bm{x}_{v},\bm{x}_{s,2})=[0.4,0,0.6]$. Both $\bar{f}$ and
$\tilde{f}$ make different mistakes under distributional shifts due to using
different spurious features. However, the ensemble of them can make a correct
prediction, i.e.,
$1/2f_{1}(\bm{x}_{v},\bm{x}_{s,1})+1/2f_{1}(\bm{x}_{v},\bm{x}_{s,2})=[0.4,0.3,0.3]$.

Feature visualization. The reasoning above assumes that individual models
utilize different features. GradCam [53] visualization of the features used by
the pre-trained (zero-shot), fine-tuned, and WiSE-FT in Figure 2(Right)
confirms this assumption. The visualization shows that the pre-trained and
fine-tuned models rely on different features, while WiSE-FT utilizes more
diverse features. Additionally, [3] provides empirical evidence supporting the
use of diverse features by different DNNs with the same architecture trained
on the same datasets (with different initialization). They also provide formal
theoretical proof for 2-layer DNNs. We include some of [3]’s empirical results
in Appendix C.2. Additionally, there is more evidence suggesting that DNNs
favor sparse feature representations and discard redundant features [40, 4].

##  3 Analysis on Spurious Feature Diversification

###  3.1 Theoretical Settings

Notation. For simplicity of presentation, we consider a 3-class classification
problem, i.e., $\bm{y}\in\\{\bm{e}_{1},\bm{e}_{2},\bm{e}_{3}\\}$, where
$\bm{e}_{i}$ denotes the 3-dimensional unit vector with $i$th element equaling
1, e.g., $\bm{e}_{2}=[0,1,0]^{\top}$. In Appendix F.2, we extend the setting
to $K$-class classification. $\bm{a}(k)$ means the $k$th element of vector
$\bm{a}$, $\bm{A}(k)$ means the $k$th column of matrix $\bm{A}$. We use
$\bm{I}_{K}$ to represent a $K\times K$ identity matrix, e.g.,
$\bm{I}_{3}=[\bm{e}_{1},\bm{e}_{2},\bm{e}_{3}]$. We omit the subscript of
$\bm{I}$ when no confusion arises.

Suppose we have $d_{v}$ invariant features $\\{\bm{x}_{v,i}\\}_{i=1}^{d_{v}}$
and $d_{s}$ spurious features $\\{\bm{x}_{s,j}\\}_{j=1}^{d_{s}}$ where
$\bm{x}_{v,i},\bm{x}_{s,j}\in\mathbb{R}^{d}$ and the whole feature
$\bm{x}\in\mathbb{R}^{d\times(d_{s}+d_{v})}$ is the concatenation of them,
i.e.,

$\bm{x}=\mbox{Concat}\Big{(}\\{\bm{x}_{v,i}\\}_{i=1}^{d_{v}}\cup\\{\bm{x}_{s,j}\\}_{j=1}^{d_{s}}\Big{)}=[\bm{x}_{v,1},\dots,\bm{x}_{v,d_{v}},\bm{x}_{s,1},\dots,\bm{x}_{s,d_{s}}].$

Consider that each model $f$ is composed of a featurizer
$\Phi\in\\{0,1\\}^{d_{v}+d_{s}}$ and a classifier
$\bm{w}\in\mathbb{R}^{d\times 3}$. $\Phi$ first selects feature by
$\bm{x}\Phi$. For example, suppose $\bm{x}=[\bm{x}_{1},\bm{x}_{2},\bm{x}_{3}]$
and $\Phi=[1,1,0]^{\top}$, then $\bm{x}\Phi=\bm{x}_{1}+\bm{x}_{2}$. Then the
classifier $\bm{w}\in\mathbb{R}^{d\times 3}$ is fit based on the features
selected by $\Phi$ as

$\bm{w}=\operatorname*{arg\,min}_{\bm{v}\in\mathbb{R}^{d\times
3}}\mathcal{R}_{{id}}(\bm{v},\Phi)=\operatorname*{arg\,min}_{\bm{v}\in\mathbb{R}^{d\times
3}}\mathbb{E}_{(\bm{x},\bm{y})\sim\mathcal{D}_{{id}}}[\ell(\bm{v}^{\top}(\bm{x}\Phi),\bm{y})],$

where $\ell$ is the cross-entropy loss function and $\mathcal{D}_{{id}}$ is
the ID distribution.

Remark. Our theoretical models utilize a simplified two-layer structure [5,
51, 67] on concatenated features to mimic the feature learning of modern deep
learning models like Vision Transformers [22], which process input images as
patches. Refer to Appendix D.1 for further discussions.

Following [51, 57], we consider that each $\bm{x}_{v,i}$ and $\bm{x}_{s,j}$
are generated from the label $\bm{y}$ with the latent invariant features
$\bm{\mu}_{v,i}$ and spurious features $\bm{\mu}_{s,i}$, where
$\bm{\mu}_{v,i},\bm{\mu}_{s,j}\in\mathbb{R}^{d\times 3}$. The full data
generation process is:

######  Definition 1 (Data Generation Process).

The whole data generation process is as follows:

$\displaystyle\bm{y}\sim\text{Unif}\left\\{\bm{e}_{1},\bm{e}_{2},\bm{e}_{3}\right\\},\bm{x}=\mbox{Concat}\Big{(}\\{\bm{x}_{v,i}\\}_{i=1}^{d_{v}}\cup\\{\bm{x}_{s,j}\\}_{j=1}^{d_{s}}\Big{)},$
$\displaystyle\mathbb{P}_{\theta}(\bm{x}_{v,i}\mid\bm{y})=\mathcal{N}\left({\bm{\mu}}_{v,i}\bm{Q}_{v,i}\bm{y},\sigma^{2}\bm{I}_{d}\right),\mathbb{P}_{\theta}(\bm{x}_{s,j}\mid\bm{y})=\mathcal{N}\left({\bm{\mu}}_{s,j}\bm{Q}_{s,j}\bm{y},\sigma^{2}\bm{I}_{d}\right),\forall
i,j.$ (1)

where $\bm{Q}_{v,i},\bm{Q}_{s,j}\in\\{0,1\\}^{3\times 3}$. Further,
$\bm{Q}_{v,i}=\bm{I}_{3}=[\bm{e}_{1},\bm{e}_{2},\bm{e}_{3}]$ always hold. In
the ID distribution $\mathcal{D}_{\mbox{id}}$, $\bm{Q}_{s,j}=\bm{I}_{3}$; and
in OOD $\mathcal{D}_{\mbox{ood}}$, the $k$th column of $\bm{Q}$, i.e.,
$\bm{Q}_{s,j}(k)$, is as follows for $k=1,2,3$:

$\bm{Q}_{s,j}(k)=\begin{cases}\bm{e}_{k},\mbox{ with probability }1-p\\\
\mbox{Unif}\\{\bm{e}_{1},\bm{e}_{2},\bm{e}_{3}\\},\mbox{ with probability
}p.\end{cases}$

Figure 3: (a) $\bm{\mu}_{s,j}\in\mathbb{R}^{d\times 3}$ represents a spurious
feature, e.g., the background. Each column of $\bm{\mu}_{s,j}$ is an attribute
of the spurious feature, e.g., $\bm{\mu}_{s,j}(1)$, $\bm{\mu}_{s,j}(2)$ and
$\bm{\mu}_{s,j}(3)$ are the floor, grass, and sand, respectively. (b)
$\bm{Q}_{s,j}\in\\{0,1\\}^{3\times 3}$ represents the relationship between
labels and spurious features. In the ID distribution, $\bm{Q}_{s,j}$ equals
$\bm{I}$, indicating that each spurious feature is perfectly correlated with
the corresponding class. (c) In the OOD distribution, spurious correlation can
fail, e.g., $\bm{Q}_{s,j}(1)$ equals $\bm{e}_{2}$ with probability $p/3$,
indicating the background of the dog is the grass.

The intuition of the data generation process. We consider the example in
Figure 1. Figure 3 shows the intuition of $\bm{\mu}_{s,j}$ and $\bm{Q}_{s,j}$.
Suppose the spurious feature $\bm{\mu}_{s,j}$ is the background in Figure 1.
Here
$\bm{\mu}_{s,j}=[\bm{\mu}_{s,j}(1),\bm{\mu}_{s,j}(2),\bm{\mu}_{s,j}(3)]\in\mathbb{R}^{d\times
3}$ and each column $\bm{\mu}_{s,j}(k)$ for $k=1,2,3$ represents a specific
attribute that is associated with class $k$ in the training set. In other
words, $\bm{\mu}_{s,j}(1),\bm{\mu}_{s,j}(2)$, and $\bm{\mu}_{s,j}(3)$
represent 3 attributes of background, namely, floor, grass, and sand, which
are correlated with dog, cow, and camel, respectively. Consider a dog image
(i.e., $\bm{y}=\bm{e}_{1}=[1,0,0]$ ). We have
$\bm{\mu}_{s,j}\bm{Q}\bm{y}|_{\bm{y}=\bm{e}_{1}}=\bm{\mu}_{s,j}\bm{Q}_{s,j}(1)$
and 333Specifically,
$\bm{Q}\bm{y}|_{\bm{y}=\bm{e}_{1}}=\bm{Q}[1,0,0]^{\top}=\bm{Q}_{s,j}(1)$,
where $\bm{Q}_{s,j}(1)$ is the first column of $\bm{Q}_{s,j}$. further

  * (a)

In the ID distribution $\mathcal{D}_{\mbox{id}}$, $\bm{Q}_{s,j}(1)=\bm{e}_{1}$
and
$\bm{\mu}_{s,j}\bm{Q}_{s,j}\bm{y}|_{\bm{y}=\bm{e}_{1}}=\bm{\mu}_{s,j}\bm{e}_{1}=\bm{\mu}_{s,j}(1).$
Then $\bm{x}_{s,j}=\mathcal{N}(\bm{\mu}_{s,j}(1),\sigma\bm{I})$, indicating
that in $\mathcal{D}_{\mbox{id}}$ the background of the dog (i.e.,
$\bm{y}=\bm{e}_{1}$) is the floor (i.e., $\bm{\mu}_{s,j}(1)$).

  * (b)

In the OOD distribution $\mathcal{D}_{\mbox{ood}}$,
$\bm{Q}_{s,j}(1)=\bm{e}_{1}$ with probability $1-p$ and
$\bm{Q}_{s,j}(1)\sim\mbox{Unif}\\{\bm{e}_{1},\bm{e}_{2},\bm{e}_{3}\\}$ with
probability $p$. Then we have the following:

$\displaystyle\bm{\mu}_{s,j}\bm{Q}_{s,j}\bm{y}|_{\bm{y}=\bm{e}_{1}}=\begin{cases}\bm{\mu}_{s,j}(1),\mbox{
with probability }1-p\\\
\mbox{Unif}\\{\bm{\mu}_{s,j}(1),\bm{\mu}_{s,j}(2),\bm{\mu}_{s,j}(3)\\},\mbox{
with probability }p,\end{cases}$

indicating that in the OOD distribution the background of the dog (i.e.,
$\bm{y}=\bm{e}_{1}$) is the floor (i.e., $\bm{\mu}_{s,j}(1)$) with probability
$1-p$ and is randomly drawn from floor, grass, and sand (i.e.,
$\bm{\mu}_{s,j}(1)$, $\bm{\mu}_{s,j}(2)$, and $\bm{\mu}_{s,j}(3)$) with $p$.
In other words, $p$ is the probability that spurious correlation no-longer
holds and a larger $p$ indicates larger distributional shift.

Remark. Our data generation process extends the setting of [57, 51] to a
3-class classification problem with multiple features. This extension aligns
with the intuition behind popular multi-class datasets used in empirical
studies on OOD generalization, such as FullColorMNIST, ColoredObject, and
CifarMNIST [61, 35, 67, 66, 1]. Take ColoredObject for example, correlations
between classes and background colors exist in the training dataset but fail
with a certain probability in OOD.

######  Definition 2 (Individual models).

Denote the whole invariant feature set as
$\mathcal{V}:=\\{\bm{x}_{v,i}\\}_{i=1}^{d_{v}}$ and spurious feature set
$\mathcal{S}:=\\{\bm{x}_{s,j}\\}_{j=1}^{d_{s}}$. Consider
$\bar{f}=(\bar{\Phi},\bar{\bm{w}})$ and
$\tilde{f}=(\tilde{\Phi},\tilde{\bm{w}})$. Suppose $\bar{\Phi}$ learns
$\bar{\mathcal{V}}\subset\mathcal{V}$ and
$\bar{\mathcal{S}}\subset\mathcal{S}$, and $\tilde{\Phi}$ learns
$\tilde{\mathcal{V}}\subset\mathcal{V}$ and
$\tilde{\mathcal{S}}\subset\mathcal{S}$. Denote
$|\tilde{\mathcal{V}}|=\tilde{n}_{v}$, $|\tilde{\mathcal{S}}|=\tilde{n}_{s}$,
$|\bar{\mathcal{V}}|=\bar{n}_{v}$, $|\bar{\mathcal{S}}|=\bar{n}_{s}$,
$|\tilde{\mathcal{V}}\cap\bar{\mathcal{V}}|=n_{vo}$, and
$|\tilde{\mathcal{S}}\cap\bar{\mathcal{S}}|=n_{so}$. Specifically, we have

$\displaystyle\bm{x}\bar{\Phi}=\sum_{\bm{x}_{v}\in\bar{\mathcal{V}}}\bm{x}_{v}+\sum_{\bm{x}_{s}\in\bar{\mathcal{S}}}\bm{x}_{s},\bar{\bm{w}}=\operatorname*{arg\,min}_{\bm{v}\in\mathbb{R}^{d\times
3}}\mathcal{R}_{\mbox{id}}(\bm{v},\bar{\Phi}),$
$\displaystyle\bm{x}\tilde{\Phi}=\sum_{\bm{x}_{v}\in\tilde{\mathcal{V}}}\bm{x}_{v}+\sum_{\bm{x}_{s}\in\tilde{\mathcal{S}}}\bm{x}_{s},\tilde{\bm{w}}=\operatorname*{arg\,min}_{\bm{v}\in\mathbb{R}^{d\times
3}}\mathcal{R}_{\mbox{id}}(\bm{v},\tilde{\Phi}).$

######  Definition 3 (Output space ensemble (OSE)).

Given the two individual models defined in Definition 2, the prediction of the
the output space ensemble (OSE) is

$f_{\mbox{ose}}(\bm{x})=\frac{1}{2}(\bar{\bm{w}}^{\top}(\bm{x}\bar{\Phi})+\tilde{\bm{w}}^{\top}(\bm{x}\tilde{\Phi})).$

The predicted class of the sample $(\bm{x},\bm{y})$ is the class with the
maximum logit. Specifically, denote the logit as $\hat{l}=f(\bm{x})$. The
predicted class is
$\hat{k}=\operatorname*{arg\,max}_{h\in\\{1,2,3\\}}\hat{l}(h)$ where
$\hat{l}(h)$ of the $h$th dimension of the logit $\hat{l}$. The model makes
correct prediction if $\mathbb{I}(e_{\hat{k}}=\bm{y})$ holds where
$\mathbb{I}$ is the indicator function. The accuracy is
$\mathcal{A}(f)=\mathbb{E}_{\bm{x},\bm{y}}[\mathbb{I}(\bm{e}_{\hat{k}}=\bm{y})]$.
We denote the OOD accuracy as
$\small\mathcal{A}_{\mbox{ood}}(f)=\mathbb{E}_{\bm{Q}_{s}}\left[\mathbb{E}_{\bm{x},\bm{y}}[\mathbb{I}(\bm{e}_{\hat{k}}=\bm{y})|\bm{Q}_{s}]\right],$
where we use $\bm{Q}_{s}$ as a short hand for
$\bm{Q}_{s,1},\dots,\bm{Q}_{s,d_{s}}$. We discuss the metric in Appendix D.4.
We defer the analysis of ID accuracy to Appendix D.5 since we consider
infinite samples and the ID accuracy of all considered models are all close to
1.

######  Assumption 1 (Small Noise).

Denote $n_{v}^{\prime}$ and $n_{s}^{\prime}$ as the the maximum number of
invariant features and spurious features that a model can learn, respectively.
We need the overall noise to be small to satisfy
$\bm{F}^{K}(\frac{1}{\sigma(n_{v}^{\prime}+n_{s}^{\prime})})\geq 1-\epsilon,$
in which $\bm{F}$ is the cumulative distribution function of standard Gaussian
random variable, and $K$ refers to the class number (here we analyze the case
$K=3$).

Remark. Since we impose random noise on each feature, e.g.,
$\bm{x}_{v,i}=\bm{\mu}_{v,i}+\bm{z}$ where
$\bm{z}\sim\mathcal{N}(0,\sigma^{2}\bm{I}_{d})$ where $\bm{I}_{d}$ is a
d-dimensional identity matrix and $d\gg d_{v}+d_{s}$, it is natural to assume
the overall noise is controlled, e.g., we have $\epsilon\leq 10^{-6}$ when
$K=10$, $\sigma=1/100$, $n_{v}^{\prime}+n_{s}^{\prime}=20$.

######  Assumption 2 (Orthogonal features [57, 3]).

(1) $\|\bm{\mu}_{v,i}(k)\|_{2}=1$ and $\|\bm{\mu}_{s,j}(k)\|_{2}=1$ for
$i=1,\cdots,d_{v}$, $j=1,\cdots,d_{s}$, $k=1,2,3$. (2)
${\bm{v}}_{i}(k)\perp{\bm{v}}_{i^{\prime}}(k^{\prime})$ for any
$(i,k)\neq(i^{\prime},k^{\prime})$, $k,k^{\prime}=1,2,3,$
${\bm{v}}_{i},{\bm{v}}_{i^{\prime}}\in\\{\bm{\mu}_{v,1},\cdots,\bm{\mu}_{v,d_{v}},\bm{\mu}_{s,1},\dots,\bm{\mu}_{s,d_{s}}\\}$.

###  3.2 Theoretical Results

We first show the intuition on the simple Example 1 and then extend to the
general setting in Def. 3:

######  Example 1 (Illustrative examples).

Consider that there are totally 4 invariant features
$\\{\bm{x}_{v,i}\\}_{i=1}^{4}$ and 6 spurious features
$\\{\bm{x}_{s,j}\\}_{j=1}^{6}$, and two individual models
$(\bar{\bm{w}},\bar{\Phi})$ and $(\tilde{\bm{w}},\tilde{\Phi})$ learn non-
overlapped features as

$\displaystyle\bm{x}\bar{\Phi}=\sum_{i=1,2}\bm{x}_{v,i}+\sum_{j=1,2,3}\bm{x}_{s,j},\text{
and }\bm{x}\tilde{\Phi}=\sum_{i=3,4}\bm{x}_{v,i}+\sum_{j=4,5,6}\bm{x}_{s,j}.$

######  Proposition 1 (Illustrative examples).

Consider Example 1, suppose Assumption 1 and 2 hold, and there are infinite ID
and OOD samples. Omitting small terms containing $\epsilon$, we have

$\mathcal{A}_{ood}(\bar{f})=\mathcal{A}_{ood}(\tilde{f})=1-\frac{1}{9}p^{3},\text{
and
}\mathcal{A}_{ood}(f_{\mbox{ose}})=1-\frac{2p^{5}}{81}-\frac{17p^{6}}{729}.$

We can see that OSE improves OOD by
$\mathcal{A}_{ood}(f_{\mbox{ose}})-\max\\{\mathcal{A}_{ood}(\bar{f}),\mathcal{A}_{ood}(\tilde{f})\\}>{1}/{81}p^{3}$.

Intuition of the proof (Full proof in Appendix F.1). Let’s consider the
samples of first class $\bm{y}=\bm{e}_{1}=[1,0,0]$. Model
$(\bar{\bm{w}},\bar{\Phi})$ has
$\bm{x}\bar{\Phi}|_{\bm{y}=\bm{e}_{1}}=\sum_{i=1}^{2}\bm{\mu}_{v,i}\bm{Q}_{v,i}(1)+\sum_{j=1}^{3}{\bm{\mu}}_{s,j}\bm{Q}_{s,j}(1)+z$
where $z\sim\mathcal{N}(0,5\sigma^{2}\bm{I}_{d})$. By Lemma 5, we have
$\bar{\bm{w}}(k)=\sum_{i=1}^{2}\bm{\mu}_{v,i}(k)+\sum_{j=1}^{3}{\bm{\mu}}_{s,j}(k)$
for each class $k=1,2,3$. Omitting the small noise term, the predicted logit
for class $k$ is
$\bar{\bm{w}}(k)^{\top}(\bm{x}\bar{\Phi})|_{\bm{y}=\bm{e}_{1}}=\sum_{i=1}^{2}\bm{\mu}_{v,i}(k)^{\top}(\bm{\mu}_{v,i}\bm{Q}_{v,i}(1))+\sum_{j=1}^{3}{\bm{\mu}}_{s,j}(k)^{\top}({\bm{\mu}}_{s,j}\bm{Q}_{s,j}(1))$
. The model will mistakenly predict $\bm{e}_{2}$ on the samples with true
label $\bm{e}_{1}$ when
$\bar{\bm{w}}(1)^{\top}\bm{x}\bar{\Phi}|_{y=\bm{e}_{1}}<\bar{\bm{w}}(2)^{\top}\bm{x}\bar{\Phi}|_{y=\bm{e}_{1}}$.
This will happen when the three events
$\\{\bm{Q}_{s,j}(1)=\bm{e}_{2}\\}_{j=1}^{3}$ simultaneously happen in OOD (see
Appendix D.7 for detailed discussion). Each event occurs with a probability of
$p/3$, resulting in a combination probability of $p^{3}/27$. This means that
with a probability of $p^{3}/27$, we encounter an OOD scenario where the model
$\bar{f}=(\bar{\bm{w}},\bar{\Phi})$ incorrectly predicts almost all samples
from the first class $\bm{e}_{1}$ as the second class $\bm{e}_{2}$. This
failure occurs because all three spurious features happen to have values that
are spuriously correlated with $\bm{e}_{2}$ in the training dataset. In other
words, the three spurious features dominate the prediction of $\bm{e}_{2}$,
overshadowing the two invariant features that predict the true label
$\bm{e}_{1}$. For the OSE model, we have
$\bar{\bm{w}}(k)^{\top}(\bm{x}\bar{\Phi})+\tilde{\bm{w}}(k)^{\top}(\bm{x}\tilde{\Phi})|_{\bm{y}=\bm{e}_{1}}=\sum_{i=1}^{4}\bm{\mu}_{v,i}(k)^{\top}(\bm{\mu}_{v,i}\bm{Q}_{v,i}(1))+\sum_{j=1}^{6}{\bm{\mu}}_{s,j}(k)^{\top}({\bm{\mu}}_{s,j}\bm{Q}_{s,j}(1))$.
The model will mistakenly predict $\bm{e}_{2}$ on the samples with true label
$\bm{e}_{1}$ when at least five of the six events
$\\{\bm{Q}_{s,j}(1)=\bm{e}_{2}\\}_{j=1}^{6}$ simultaneously happen in OOD (see
Appendix D.6 for details), whose probability is much less than that of
$\bar{f}$. Intuitively, the failure probability of the averaged model is
smaller as it utilizes more spurious features, which are less likely to make
the same mistakes.

######  Proposition 2 (General Results for OSE).

Consider Definition 1-3, Assumption 1-2 hold, and infinite ID and OOD samples.
Omitting small constants involving $\epsilon$, we have

$\mathcal{A}_{\mbox{ood}}(\bar{f})~{}=~{}F_{p}\left(\frac{(1-p)\bar{n}_{s}+\bar{n}_{v}}{\sqrt{\bar{n}_{s}}}\right),\quad\mathcal{A}_{\mbox{ood}}(\tilde{f})=F_{p}\left(\frac{(1-p)\tilde{n}_{s}+\tilde{n}_{v}}{\sqrt{\tilde{n}_{s}}}\right),$
$\text{ and
}\mathcal{A}_{\mbox{ood}}(f_{\mbox{ose}})~{}=~{}F_{p}\left(\frac{(1-p)(\tilde{n}_{s}+\bar{n}_{s})+(\tilde{n}_{v}+\bar{n}_{v})}{\sqrt{\tilde{n}_{s}+\bar{n}_{s}+2n_{so}}}\right).$

Figure 4:  (a) Illustration of of $F(x)$; (b)
$\mathcal{A}_{\mbox{ood}}(f_{\mbox{ose}})-\mathcal{A}_{\mbox{ood}}(\bar{f})$
in Example 2;

Here $F_{p}(x)$ is a cumulative density function (CDF) parameterized by $p$ as
defined in Appendix F.2, which is monotonically increasing with $x$ as shown
in Figure 4(a). Suppose two individuals learns the same number of features
with no-overlap, i.e., $\tilde{n}_{v}=\bar{n}_{v}=n_{v}$,
$\bar{n}_{s}=\tilde{n}_{s}=n_{s}$, and $n_{vo}=n_{so}=0$, we have
$\mathcal{A}_{\mbox{ood}}(f_{\mbox{ose}})=F_{p}\left(\sqrt{2}t\right)$ and
$\mathcal{A}_{\mbox{ood}}(\bar{f})=\mathcal{A}_{\mbox{ood}}(\bar{f})=F_{p}(t)$
where $t=(1-p)\sqrt{n_{s}}+\frac{n_{v}}{\sqrt{n_{s}}}$, indicating that
$f_{\mbox{ose}}$ is better than $\bar{f}$ since $F(\cdot)$ is monotonically
increasing.

######  Example 2.

Consider $p=0.9$ and two individual models learn none overlapped, i.e.,
$n_{vo}=n_{so}=0$, fixing $\bar{n}_{v}=5,\bar{n}_{s}=20$, and vary
$\tilde{n}_{v}=0,1,..,5$ and $\tilde{n}_{s}=0,1,...,20$.

Figure 4(b) illustrates
$\mathcal{A}_{\mbox{ood}}(f_{\mbox{ose}})-\mathcal{A}_{\mbox{ood}}(\bar{f})$
on Example 2. $f_{\mbox{ose}}$ achieves better OOD performance than $\bar{f}$
in most cases. One exception is that if $\tilde{f}$ is much weaker than
$\bar{f}$, e.g., $\bar{f}$ learns 5 invariant features but $\tilde{f}$ learns
0 invariant features, the ensemble model $f_{\mbox{ose}}$ is inferior than
$\bar{f}$.

The difference between the output and weight space ensemble. It is an open
problem on the difference betwee output space ensemble (OSE) and WSE (referred
as OSE-WSE difference) and why WSE is better than OSE in OOD [60, 59, 49]. We
shed light on this by our bilinear theoretical model
$\bm{w}^{\top}\bm{x}\Phi$:

######  Definition 4 (Weight space ensemble (WSE)).

Given the two individual models defined in Definition 2, the prediction of the
weight space ensemble( WSE) is

$f_{\mbox{wse}}(\bm{x})=\frac{1}{4}(\bar{\bm{w}}+\tilde{\bm{w}})^{\top}\left(\bm{x}(\bar{\Phi}+\tilde{\Phi})\right).$

In Appendix D.2, we show that the OSE-WSE difference in a 2-layer DNN is
closely connected with the OSE-WSE difference captured by our models in
Definition 3\- 4.

######  Proposition 3 (General Results for WSE).

Consider Definition 1-3, Assumption 1-2, and infinite ID and OOD samples.
Omittimg small constants involving $\epsilon$, we have

$\mathcal{A}_{\mbox{ood}}(f_{\mbox{wse}})=F_{p}(\frac{(1-p)(\tilde{n}_{s}+\bar{n}_{s}+2n_{so})+(\tilde{n}_{v}+\bar{n}_{v}+2n_{vo})}{\sqrt{\tilde{n}_{s}+\bar{n}_{s}+14n_{so}}}).$

Comparing Proposition 2 and 3, we can see that the only difference between
$\mathcal{A}_{\mbox{ood}}(f_{\mbox{wse}})$ and
$\mathcal{A}_{\mbox{ood}}(f_{\mbox{ose}})$ is the number of overlapped
invariant and spurious features learned by individual models, i.e., $n_{vo}$
and $n_{so}$. Specifically, when $\bar{\Phi}$ and $\tilde{\Phi}$ selects no
overlapped features, $f_{\mbox{wse}}$ and $f_{\mbox{ose}}$ makes the same
prediction since $\bm{x}\bar{\Phi}\perp\tilde{\bm{w}}$ and
$\bm{x}\tilde{\Phi}\perp\bar{\bm{w}}$ by Assumption 2 and further
$(\bar{\bm{w}}+\tilde{\bm{w}})^{\top}\left(\bm{x}(\bar{\Phi}+\tilde{\Phi})\right)\propto\bar{\bm{w}}^{\top}\bm{x}\bar{\Phi}+\tilde{\bm{w}}^{\top}\bm{x}\tilde{\Phi}$.
When there is overlapped features: (a) for WSE, the coefficient of overlapped
features is amplified by 2 in $\bar{\Phi}+\tilde{\Phi}$, and further amplified
twice in $\bar{\bm{w}}+\tilde{\bm{w}}$. This results in coefficient of the
overlapped feature becoming 4 in
$(\bar{\bm{w}}+\tilde{\bm{w}})^{\top}\bm{x}(\bar{\Phi}+\tilde{\Phi})$. (b) for
OSE, i.e.,
$\bar{\bm{w}}^{\top}\bm{x}\bar{\Phi}+\tilde{\bm{w}}^{\top}\tilde{\bm{x}}\Phi$,
the coefficient of the overlapped feature is 2. See Appendix D.7.1 for a
detailed discussion. In Appendix D.7.2, we provide conditions when
$f_{\mbox{wse}}$ outperforms $f_{\mbox{ose}}$, in addition with simulation
results and supportive experiments.

###  3.3 Experimental Verification on MultiColorMNIST

Previous efforts in OOD community have focused on learning invariant features
and discarding spurious features [5]. However, these approaches have not
performed well on real-world datasets [51]. This could be due to the
requirements of invariant learning, such as the need for numerous domains
[51], strong regularization [67], and the challenges posed by non-linearity,
overparameterization, and optimization [51, 35, 15]. In contrast, our findings
show that learning diverse spurious features also help with OOD
generalization. This approach, as shown in ensemble-based models, is easily
implementable and has shown remarkable empirical success.

To further verify our findings, we contruct MultiColorMNIST, a 10-class
variant of CMNIST [5] with 32 spurious features, following Definition 1. As
shown in Figure 5, each sample in MultiColorMNIST consists of 32 color
patches, each serving as a spurious feature. We train two neural networks,
denoted as $f_{\theta_{1}}$ and $f_{\theta_{2}}$, with the same architecture
but different initializations on MultiColorMNIST. The results in Table 1 show
that the OSE model ($f_{\theta_{1}}(\bm{x})$ \+ $f_{\theta_{2}}(\bm{x})$)
improve OOD performance over individual models ($f_{\theta_{1}}(\bm{x})$ and
$f_{\theta_{2}}(\bm{x})$). In Appendix C.3, (1) we show that each individual
model learn a subset of spurious features in MultiColorMNIST and OSE utilizes
more diverse spurious features (2) we construct SingleColorMNIST with only one
spurious feature and show OSE yields little performance gain since both
individual models learn the same spurious feature (similar to the results in
[49]).

Figure 5: Two samples from MultiColorMNIST

$p$ | 0.70 | 0.75 | 0.80 | 0.85 | 0.90
---|---|---|---|---|---
model 1 |  71.05$\pm$1.04 |  60.07$\pm$1.04 |  48.57$\pm$0.92 |  36.93$\pm$0.70 |  26.01$\pm$0.45
model 2 |  71.77$\pm$0.94 |  60.75$\pm$0.91 |  49.26$\pm$0.83 |  37.74$\pm$0.66 |  26.63$\pm$0.42
model ensemble | 78.64$\pm$0.73 | 67.61$\pm$0.80 | 55.25$\pm$0.75 | 42.34$\pm$0.64 | 29.28$\pm$0.40

Table 1: OOD performance of (output space) model ensemble on MultiColorMNIST.
The spurious correlation is $1$ and $1-p$ in the training and testing set,
respectively. A larger $p$ indicates larger distributional shift

##  4 BAlaNced averaGing (BANG)

Our previous results show that EBM can boost the OOD performance. An implicit
requirement is that the scaling of the two models should be roughly the same.
If the two models have different scalings, e.g., one model is much more
confident than the other, the EBM improvement is weakened.

######  Proposition 4 (Imbalanced scaling weakens WSE).

Consider the Example 1, Definition 1-4, Assumption 1-2. Consider an WSE of two
imbalanced models, $\bar{f}=(\bar{\bm{w}},\bar{\Phi})$ and
$\tilde{f}_{\lambda}=(\lambda\tilde{\bm{w}},\lambda\tilde{\Phi})$, where
$\lambda\geq 1$. Specifically,
$f_{\mbox{wse}}(\bm{x})=0.25(\bar{\bm{w}}+\lambda\tilde{\bm{w}})\bm{x}(\bar{\Phi}+\lambda\tilde{\Phi})$.
We have

$\mathcal{A}_{ood}(f_{\mbox{wse}})|_{\lambda>\sqrt{5}}~{}-~{}\mathcal{A}_{ood}(f_{\mbox{wse}})|_{\lambda=1}~{}<~{}-34/729p^{3}.$

See Appendix F.3 for proofs and Appendix D.8 for an illustration of the over-
confidence characterized by $\lambda$. When $\lambda=1$, indicating similar
confidence levels between $\bar{f}$ and $\tilde{f}_{\lambda}$, the WSE is
balanced. However, when $\lambda>\sqrt{5}$ and $\tilde{f}_{\lambda}$ is
significantly more confident than $\bar{f}$, $f_{\mbox{wse}}$ becomes biased
towards $\tilde{f}_{\lambda}$, resulting in a performance drop of over
$34/729p^{3}$. Here we set $\lambda=\sqrt{5}$ for illustration purposes and
similar results can be similarly obtained for other $\lambda>1$.
Unfortunately, we find WiSE-FT, which is the WSE of the pre-trained model (PM)
and fine-tuned model (FM), suffers from the imbalanced confidence issue.
Specifically, we compare the PM and FM on their confidence and accuracy. The
confidence is defined as the largest probability that a model assigns to a
class (details in Appendix E.3). Figure 6 shows that the fine-tuned model is
highly over-confident, especially on OOD datasets, e.g., ImageNetA have only
0.37 accuracy while the average confidence is over 0.7. Such overconfidence
magnifies the FM’s incorrect prediction, leading to deteriorate OOD ensemble
performance (details in Appendix E.6).

A direct fix to the issue of over-confidence is to tune the temperature of the
softmax of the fine-tuned model [33]. However, this method can not be directly
applied to WiSE-FT since WiSE-FT ensemble model weights instead of the outputs
(see Appendix E.6 for some adaptations of [33], which also do not yield
satisfactory results). Moreover, the temperature scaling tuned on the ID
dataset [33] fails to calibrate the fine-tuned model on OOD datasets, where
over-confidence is more severe (Appendix E.5). We propose BAlaNced averaGing
(BANG), which adopt label smoothing and Mixup during fine-tuning to prevent
overconfidence. 1) Label smoothing: in label smoothing, instead of setting 1
as the target label for the true class and 0 for all other classes, a positive
value(smoothing paramter, e.g., 0.8) is assigned to the true class and the
smoothing parameter(e.g., 0.2) is distributed evenly among the other classes
[37]. 2) Mixup [62] generates new samples during fine-tuning by linearly
mixing pairs of training data and their labels.

Figure 6: Comparison of confidence and accuracy between zero-shot and
finetuned model. In the figure, $\triangledown$ refers to IN, $\circ$ for
IN-A, $\square$ for IN-R, $+$ for IN-S, $\diamondsuit$ for IN-V2 and $\times$
for ObjNet.

We conduct experiments with CLIP ViT-B/16[45]. We impose Mixup or Label
Smoothing during fine-tuning the pre-trained CLIP on ImageNet (IN), and test
OOD performance on ImageNetV2 (IN-V2), ImageNetR (IN-R), ImageNet-A(IN-A),
ImageNetSketch (IN-S) and ObjectNet. Following WiSE-FT [60], BANG averages the
pre-trained CLIP model and the model finetuned with LS and MixUp (more details
in Appendix E.4). The results in Table 2 show that BANG effectively improve
the performance over WiSE-FT. Specifically, BANG(LS+Mixup), where both LS and
MixUp are adopted, achieves 1.9% higher average OOD accuracy than WiSE-FT.
Further experimental results in the appendix show that Mixup and Label
Smoothing can effectively alleviate the over-confidence of the fine-tuned
model on both ID and OOD datasets. In Appendix E.5, we include comparison with
[33], and also show that the performance gain of BANG can not be explained by
other data augmentation methods.

Since Mixup and LS also improve the performance of the fine-tuned model, so a
curious reader would wonder whether the improvement of BANG comes from better
calibration or just due to the improvement in the fine-tuned model. We conduct
further investigation in Appendix E.6 to confirm the contribution of better
calibration: (1) Dividing the weight of the vanilla fine-tuned model by
multiple scalars significantly enhances the performance of weight averaging,
which nearly matches the performance of BANG. (2) BANG can correct
substantially more samples that is mis-classified by the fine-tuned model.

Methods | Model Averaging | IN | IN-V2 | IN-R | IN-A | IN-S | ObjectNet | Avg OOD
---|---|---|---|---|---|---|---|---
Zero-shot [60] | No | 68.3 | 61.9 | 77.6 | 49.8 | 48.2 | 53.0 | 58.1
Fine-tuning [60] | No | 81.3 | 70.9 | 65.6 | 36.7 | 46.3 | 49.6 | 53.8
Fine-tuning (LS) | No | 82.0 | 72.3 | 63.3 | 38.3 | 46.5 | 51.1 | 54.3
Fine-tuning (Mixup) | No | 83.0 | 72.7 | 66.4 | 43.7 | 48.8 | 52.4 | 56.8
Fine-tuning (Mixup + LS) | No | 82.9 | 72.7 | 65.8 | 43.6 | 48.5 | 52.2 | 56.6
WiSE-FT [60] | Yes | 81.7 | 72.8 | 78.7 | 52.2 | 53.9 | 57.3 | 63.0
BANG (LS) | Yes | 82.1 | 73.3 | 78.2 | 55.2 | 53.7 | 58.9 | 63.9
BANG (Mixup) | Yes | 81.5 | 73.0 | 79.5 | 57.9 | 54.5 | 58.7 | 64.7
BANG (Mixup + LS) | Yes | 81.6 | 73.1 | 79.7 | 58.2 | 54.8 | 58.9 | 64.9

Table 2: Results of fine-tuning CLIP VIT-B/16 on ImageNet. LS is short for
Label Smoothing. The performance of the baseline methods are from the Table 8
of [60].

##  5 Conclusion and Discussion

In this paper, we investigate why ESM has superior OOD performance. Our
empirical analysis of WiSE-FT, coupled with theoretical insights, reveals that
ESM learns more diverse features, thereby reducing the probability of
encountering OOD distributions that fail the model. Further, we improve WiSE-
FT by alleviating the fine-tuned model’s overconfidence problem, which is
identified via theoretical analysis and empirical observations. A limitation
of the present work is the lack of theoretic guarantee with general nonlinear
feature extractors, which is also an open problem in OOD.

## References

  * [1] Faruk Ahmed, Yoshua Bengio, Harm Van Seijen, and Aaron Courville.  Systematic generalisation with group invariant predictions.  In International Conference on Learning Representations, 2021.
  * [2] Kartik Ahuja, Jun Wang, Amit Dhurandhar, Karthikeyan Shanmugam, and Kush R Varshney.  Empirical or invariant risk minimization? a sample complexity perspective.  arXiv preprint arXiv:2010.16412, 2020.
  * [3] Zeyuan Allen-Zhu and Yuanzhi Li.  Towards understanding ensemble, knowledge distillation and self-distillation in deep learning.  arXiv preprint arXiv:2012.09816, 2020.
  * [4] Maksym Andriushchenko, Aditya Vardhan Varre, Loucas Pillaud-Vivien, and Nicolas Flammarion.  Sgd with large step sizes learns sparse features.  In International Conference on Machine Learning, pages 903–925. PMLR, 2023.
  * [5] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz.  Invariant risk minimization.  arXiv preprint arXiv:1907.02893, 2019.
  * [6] Devansh Arpit, Huan Wang, Yingbo Zhou, and Caiming Xiong.  Ensemble of averages: Improving model selection and boosting performance in domain generalization.  Advances in Neural Information Processing Systems, 35:8265–8277, 2022.
  * [7] Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz.  Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models.  Advances in neural information processing systems, 32, 2019.
  * [8] Eric Bauer and Ron Kohavi.  An empirical comparison of voting classification algorithms: Bagging, boosting, and variants.  Machine learning, 36:105–139, 1999.
  * [9] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool.  Food-101–mining discriminative components with random forests.  In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pages 446–461. Springer, 2014.
  * [10] Leo Breiman.  Bagging predictors.  Machine learning, 24:123–140, 1996.
  * [11] Rich Caruana, Alexandru Niculescu-Mizil, Geoff Crew, and Alex Ksikes.  Ensemble selection from libraries of models.  In Proceedings of the twenty-first international conference on Machine learning, page 18, 2004.
  * [12] Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park.  Swad: Domain generalization by seeking flat minima.  Advances in Neural Information Processing Systems, 34:22405–22418, 2021.
  * [13] Yimeng Chen, Tianyang Hu, Fengwei Zhou, Zhenguo Li, and Zhi-Ming Ma.  Explore and exploit the diverse knowledge in model zoo for domain generalization.  In International Conference on Machine Learning, pages 4623–4640. PMLR, 2023.
  * [14] Yongqiang Chen, Wei Huang, Kaiwen Zhou, Yatao Bian, Bo Han, and James Cheng.  Towards understanding feature learning in out-of-distribution generalization.  arXiv preprint arXiv:2304.11327, 2023.
  * [15] Yongqiang Chen, Kaiwen Zhou, Yatao Bian, Binghui Xie, Bingzhe Wu, Yonggang Zhang, MA KAILI, Han Yang, Peilin Zhao, Bo Han, et al.  Pareto invariant risk minimization: Towards mitigating the optimization dilemma in out-of-distribution generalization.  In The Eleventh International Conference on Learning Representations, 2023.
  * [16] Xu Chu, Yujie Jin, Wenwu Zhu, Yasha Wang, Xin Wang, Shanghang Zhang, and Hong Mei.  Dna: Domain generalization with diversified neural averaging.  In International Conference on Machine Learning, pages 4010–4034. PMLR, 2022.
  * [17] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi.  Describing textures in the wild.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606–3613, 2014.
  * [18] MMPreTrain Contributors.  Openmmlab’s pre-training toolbox and benchmark.  https://github.com/open-mmlab/mmpretrain, 2023.
  * [19] Thomas G Dietterich.  Ensemble methods in machine learning.  In Multiple Classifier Systems: First International Workshop, MCS 2000 Cagliari, Italy, June 21–23, 2000 Proceedings 1, pages 1–15. Springer, 2000.
  * [20] Thomas G Dietterich et al.  Ensemble learning.  The handbook of brain theory and neural networks, 2(1):110–125, 2002.
  * [21] Qishi Dong, Awais Muhammad, Fengwei Zhou, Chuanlong Xie, Tianyang Hu, Yongxin Yang, Sung-Ho Bae, and Zhenguo Li.  Zood: Exploiting model zoo for out-of-distribution generalization.  Advances in Neural Information Processing Systems, 35:31583–31598, 2022.
  * [22] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [23] Zhili Feng, Anna Bair, and J Zico Kolter.  Leveraging multiple descriptive features for robust few-shot image learning.  arXiv preprint arXiv:2307.04317, 2023.
  * [24] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin.  Linear mode connectivity and the lottery ticket hypothesis.  In International Conference on Machine Learning, pages 3259–3269. PMLR, 2020.
  * [25] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky.  Domain-adversarial training of neural networks.  The journal of machine learning research, 17(1):2096–2030, 2016.
  * [26] Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann.  Shortcut learning in deep neural networks.  Nature Machine Intelligence, 2(11):665–673, 2020.
  * [27] Ishaan Gulrajani and David Lopez-Paz.  In search of lost domain generalization.  arXiv preprint arXiv:2007.01434, 2020.
  * [28] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al.  The many faces of robustness: A critical analysis of out-of-distribution generalization.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8340–8349, 2021.
  * [29] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song.  Natural adversarial examples.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15262–15271, 2021.
  * [30] Saachi Jain, Dimitris Tsipras, and Aleksander Madry.  Combining diverse feature priors.  In International Conference on Machine Learning, pages 9802–9832. PMLR, 2022.
  * [31] Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson.  Last layer re-training is sufficient for robustness to spurious correlations.  arXiv preprint arXiv:2204.02937, 2022.
  * [32] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In Proceedings of the IEEE international conference on computer vision workshops, pages 554–561, 2013.
  * [33] Ananya Kumar, Tengyu Ma, Percy Liang, and Aditi Raghunathan.  Calibrated ensembles can mitigate accuracy tradeoffs under distribution shift.  In Uncertainty in Artificial Intelligence, pages 1041–1051. PMLR, 2022.
  * [34] Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao.  Deep domain generalization via conditional invariant adversarial networks.  In Proceedings of the European conference on computer vision (ECCV), pages 624–639, 2018.
  * [35] Yong Lin, Hanze Dong, Hao Wang, and Tong Zhang.  Bayesian invariant risk minimization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16021–16030, 2022.
  * [36] Yong Lin, Shengyu Zhu, Lu Tan, and Peng Cui.  Zin: When and how to learn invariance without environment partition?  Advances in Neural Information Processing Systems, 35:24529–24542, 2022.
  * [37] Rafael Müller, Simon Kornblith, and Geoffrey E Hinton.  When does label smoothing help?  Advances in neural information processing systems, 32, 2019.
  * [38] Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang.  What is being transferred in transfer learning?  Advances in neural information processing systems, 33:512–523, 2020.
  * [39] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek.  Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift.  Advances in neural information processing systems, 32, 2019.
  * [40] Vardan Papyan, XY Han, and David L Donoho.  Prevalence of neural collapse during the terminal phase of deep learning training.  Proceedings of the National Academy of Sciences, 117(40):24652–24663, 2020.
  * [41] Seo Yeon Park and Cornelia Caragea.  On the calibration of pre-trained language models using mixup guided by area under the margin and saliency.  arXiv preprint arXiv:2203.07559, 2022.
  * [42] Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen.  Causal inference by using invariant prediction: identification and confidence intervals.  Journal of the Royal Statistical Society Series B: Statistical Methodology, 78(5):947–1012, 2016.
  * [43] Aahlad Puli, Lily H Zhang, Eric K Oermann, and Rajesh Ranganath.  Out-of-distribution generalization in the presence of nuisance-induced spurious correlations.  arXiv preprint arXiv:2107.00520, 2021.
  * [44] Shikai Qiu, Andres Potapczynski, Pavel Izmailov, and Andrew Gordon Wilson.  Simple and fast group robustness by automatic feature reweighting.  arXiv preprint arXiv:2306.11074, 2023.
  * [45] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [46] Alexandre Rame, Kartik Ahuja, Jianyu Zhang, Matthieu Cord, Leon Bottou, and David Lopez-Paz.  Model ratatouille: Recycling diverse models for out-of-distribution generalization.
  * [47] Alexandre Ramé, Kartik Ahuja, Jianyu Zhang, Matthieu Cord, Léon Bottou, and David Lopez-Paz.  Recycling diverse models for out-of-distribution generalization.  arXiv preprint arXiv:2212.10445, 2022.
  * [48] Alexandre Rame, Kartik Ahuja, Jianyu Zhang, Matthieu Cord, Léon Bottou, and David Lopez-Paz.  Model ratatouille: Recycling diverse models for out-of-distribution generalization.  2023\.
  * [49] Alexandre Rame, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy, Patrick Gallinari, and Matthieu Cord.  Diverse weight averaging for out-of-distribution generalization.  arXiv preprint arXiv:2205.09739, 2022.
  * [50] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar.  Do imagenet classifiers generalize to imagenet?  In International conference on machine learning, pages 5389–5400. PMLR, 2019.
  * [51] Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski.  The risks of invariant risk minimization.  arXiv preprint arXiv:2010.05761, 2020.
  * [52] Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski.  Domain-adjusted regression or: Erm may already learn features sufficient for out-of-distribution generalization.  arXiv preprint arXiv:2202.06856, 2022.
  * [53] Ramprasaath R Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra.  Grad-cam: Why did you say that?  arXiv preprint arXiv:1611.07450, 2016.
  * [54] Baochen Sun and Kate Saenko.  Deep coral: Correlation alignment for deep domain adaptation.  In Computer Vision–ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III 14, pages 443–450. Springer, 2016.
  * [55] Damien Teney, Ehsan Abbasnejad, Simon Lucey, and Anton Van den Hengel.  Evading the simplicity bias: Training a diverse set of models discovers solutions with superior ood generalization.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16761–16772, 2022.
  * [56] Junjiao Tian, Xiaoliang Dai, Chih-Yao Ma, Zecheng He, Yen-Cheng Liu, and Zsolt Kira.  Trainable projected gradient method for robust fine-tuning.  arXiv preprint arXiv:2303.10720, 2023.
  * [57] Yoav Wald, Gal Yona, Uri Shalit, and Yair Carmon.  Malign overfitting: Interpolation and invariance are fundamentally at odds.  In NeurIPS 2022 Workshop on Distribution Shifts: Connecting Methods and Applications, 2022.
  * [58] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing.  Learning robust global representations by penalizing local predictive power.  Advances in Neural Information Processing Systems, 32, 2019.
  * [59] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al.  Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.  In International Conference on Machine Learning, pages 23965–23998. PMLR, 2022.
  * [60] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al.  Robust fine-tuning of zero-shot models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7959–7971, 2022.
  * [61] Dinghuai Zhang, Kartik Ahuja, Yilun Xu, Yisen Wang, and Aaron Courville.  Can subnetwork structure be the key to out-of-distribution generalization?  In International Conference on Machine Learning, pages 12356–12367. PMLR, 2021.
  * [62] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  arXiv preprint arXiv:1710.09412, 2017.
  * [63] Jianyu Zhang and Léon Bottou.  Learning useful representations for shifting tasks and distributions.  In International Conference on Machine Learning, pages 40830–40850. PMLR, 2023.
  * [64] Jianyu Zhang, David Lopez-Paz, and Léon Bottou.  Rich feature construction for the optimization-generalization dilemma.  In International Conference on Machine Learning, pages 26397–26411. PMLR, 2022.
  * [65] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba.  Places: A 10 million image database for scene recognition.  IEEE transactions on pattern analysis and machine intelligence, 40(6):1452–1464, 2017.
  * [66] Xiao Zhou, Yong Lin, Renjie Pi, Weizhong Zhang, Renzhe Xu, Peng Cui, and Tong Zhang.  Model agnostic sample reweighting for out-of-distribution learning.  In International Conference on Machine Learning, pages 27203–27221. PMLR, 2022.
  * [67] Xiao Zhou, Yong Lin, Weizhong Zhang, and Tong Zhang.  Sparse invariant risk minimization.  In International Conference on Machine Learning, pages 27222–27244. PMLR, 2022.

###### Contents

  1. 1 Introduction
  2. 2 Understanding Ensemble-based Models via Examining WiSE-FT
  3. 3 Analysis on Spurious Feature Diversification
    1. 3.1 Theoretical Settings
    2. 3.2 Theoretical Results
    3. 3.3 Experimental Verification on MultiColorMNIST
  4. 4 BAlaNced averaGing (BANG)
  5. 5 Conclusion and Discussion
  6. A Social Impact
  7. B Related Works
    1. B.1 A review on the existing methods
    2. B.2 On our difference with existing works
  8. C Supportive Empirical Results for the Theory
    1. C.1 FalseFalseTrue Phenomenon
    2. C.2 Deep neural networks learn different features
    3. C.3 Experiments on MultiColorMNIST
    4. C.4 Simulation
  9. D Discussions, illustrations, and supportive results for the theoretical parts.
    1. D.1 Discussion on the theoretical models
    2. D.2 Comparison on our model with a 2-layer DNN
    3. D.3 Illustration of the transformation matrix Q
    4. D.4 On the pessimism of worst-case theoretical analysis for OOD
    5. D.5 ID performance
    6. D.6 Intuition of OOD Performance Improvement of OSE
    7. D.7 The Difference Between WSE and OSE in OOD
      1. D.7.1 Explaining the difference between WSE and OSE
      2. D.7.2 The theoretical condition of WSE outperforming OSE
      3. D.7.3 Empirical Verification
    8. D.8 Illustrating the over-confidence.
  10. E More experimental details and results on BANG
    1. E.1 Details on ImageNet Variants
    2. E.2 Details of Places365, StanfordCars, DTD and Food101 (PSDF)
    3. E.3 Details on calculating the confidence
    4. E.4 Experimental Details
    5. E.5 More Results on BANG and Discussions
    6. E.6 WiSE-FT benefits significantly from better calibration
  11. F Proofs
    1. F.1 Proof of Proposition 1
    2. F.2 Proof of Proposition 2
      1. F.2.1 Proof for Single Model
      2. F.2.2 Proof for Weight Space Ensemble
      3. F.2.3 Proof for Output Space Ensemble
      4. F.2.4 Case Study for $K=3$
      5. F.2.5 Close Form of $F_{p}(\cdot)$
      6. F.2.6 Close Form of $G(n_{v},n_{s},n_{vo},n_{so},C)$
    3. F.3 Proof of Proposition 4
    4. F.4 Proof of Proposition 6
    5. F.5 Auxiliary Lemmas

##  Appendix A Social Impact

We investigate how to enable machine learning models to generalize in OOD
scenarios, which makes machine learning models more reliable in real-world
applications.

##  Appendix B Related Works

###  B.1 A review on the existing methods

##### Out-of-distribution generalization

Machine learning models are based on the I.I.D. (independently and identically
distribution) assumption. Whereas, the I.I.D. assumption can be easily
violated since the model can easily encounter novel testing samples that are
from distributions different with the training distribution. This is also
known as the out-of-distribution generalization (OOD) problem. Existing works
find that the model performance deteriorates dramatically under distributional
shift. This is especially the case when the model rely on spurious features
that are unstable in a new domain [26, 5]. OOD problem has attracted great
attention in recent years and there are a rich line of works in this
direction, such as Invariant Risk Minimization (IRM) [5, 36], model averaging
[59, 60, 47, 12], feature alignment methods [25, 54, 34] and so on.

Among them, Invariant Risk Minimization (IRM) has gained significant attention
from the researchers [5] and inspires a great line of works. Recall that there
are two kinds of features for OOD generalization: invariant features that can
stably predict the labels, and spurious features whose correlation with the
labels is unstable. IRM tries to build robust models by extracting only
invariant features. IRM has strong theoretical guarantees in linear system and
clear connection with causal theory. Nevertheless, IRM methods face challenges
in dealing with large scale real-world datasets, as it has been repeatedly
observed that IRM cannot outperform ERM on various datasets. Some works have
provided explanation for this, e.g., [51] shows that IRM needs a very great
number of domains, [51] shows IRM lacks theretical guarantees on non-linear
models, [36] shows it is difficult to learn invariance without domain
partition and [35, 15] show the difficulty of optimizing IRM objects on deep
neural networks. In contrast, model averaging is exceptionally powerful and
achieve SOTA performance in a lot of benchmark with various models and
architecture [12, 60, 59, 49, 16, 6].

##### Output and weight space ensemble

The ensemble of multiple models is a powerful idea that often leads to
stronger predictive performance [11, 19, 8, 10]. Typically, conventional
ensemble methods aggregate the outputs of models, as known as the output space
ensemnle. The recent application usually average the parameters of models
which is generated from the same pre-training model by finetuning [59, 60,
12], also known as the weight space ensemble. While averaging two models
trained from scratch by different initialization often yields poor results
(close to random guessing). [38] finds that fine-tuning two models from the
same pre-trained initialization results in two different models that were
connected via a linear path in weight-space, along which the performance
remains high. This is also known as linear mode connectivity [24]. A notable
difference between ensemble in ID and OOD study is that the improvement of
ensemble in OOD is much more significant than that in IID. [60] shows that an
ensemble the finetuned model with the pre-trained model improve near 1pp in
ImageNet (the ID domain) and over 6-8pp on the variants of ImageNet (the OOD
Domain). Actually, model averaging is still among strongest methods for OOD
generalization. It still remains mysterious on why averaging methods are so
effective for OOD.

##### Theory of Out-of-distribution generalization.

Existing theory mostly focus on the worst case metric on analyzing the OOD
performance [57, 5, 51, 43, 67]. The worst case metric requires a model to be
robust at any OOD testing distribution. Typically, a model that only uses
invariant features can minimize the worst case metric. However, as we discuss
above, invariance learning is hard in practice and performs ineffectively on
real world datasets [27, 51, 36]. The worst case metric based theory can not
explain the success of model averaging methods. To be specific, the averaging
of two models can use spurious features that learnt by each individual model
due to its pessimism as described in Appendix D.4. In contrast, our
theoretical results characterize the probability of the model failure due to
distributional shift, which can successfully explain the experimental results.

###  B.2 On our difference with existing works

##### Difference with existing works on learning diverse features.

There have been some works that promote feature diversity to enhance empirical
performance OOD generalization by weight average [16, 49, 48], feature
concatenation [63], boosted rich feature learning [64, 14, 30, 55, 23], and
utilizing model zoo [21, 13]. However, existing explanations either do not
distinguish the invariant or spurious features [16, 49, 48, 63, 21, 13], or
focus only on learning the potentially missing invariant features [14, 23]. In
fact, according to existing invariance learning perspective [5] arguing that
models relying on spurious features are prone to failure in OOD scenarios,
these methods that learn diverse features while also incorporating spurious
features may not be able to generalize effectively under distributional shift.
In contrast, our spurious feature diversification viewpoint provides a
explanation by characterizing why and when incorporating more diverse spurious
feature diversification can improve OOD performance.

##### Difference with the existing theoretical results on ensemble in IID
settings.

There are existing explanations for the effectiveness of model ensemble in the
IID setting, which is mainly from the perspective of variance due to over-
fitting the label noise in finite samples cases [20]. Specifically, model
ensemble can have smaller variance in prediction compared with each single
model. Whereas, we consider infinite sample case, where the variance of the
model due to fitting label noise is zero. So model ensemble can not bring
significant IID improvement in this case. However, the model trained on
infinite samples can still fail due to distributional shift [5]. This is
because the model utilizes the spurious features, which are also considered as
a kind of bias [57]. Our results show that model ensemble can reduce the risk
of model failure and lead to better expected performance under distributional
shift by spurious feature diversification. In other words, model ensemble
reduces the probability of the model failure due to the bias. This is a new
result in the OOD problem as shown in Proposition 1 and 2. Notably, [3] also
considers ensemble in the IID setting, however, their theory can not explain
the OOD performance improvement of ESM models on the data in Definition 1,
explain the FalseFalseTrue phenomenon, or explain the difference of weight and
output space ensemble.

##### Difference with existing explanations on the OOD performance of
ensemble-based methods (EBM).

There are some previous attempts that try to explain the effectiveness of EBM
for OOD. [12] shows that the loss landscape changes under distributional shift
and model averaging can lead to flatter minima. However, as discussed in [49],
the upper bound of [12] is uncontrolled and their analysis based on flat
minima fails to explain many experimental results. [49] decomposes the OOD
loss into the bias, variance and covariance terms. They show that the variance
term can benefit from EBM. Different from the results of [49] that only
tackles with the variance term, our results provide a concise characterization
on the overall OOD performance. Further, [49]’s results can not differentiate
between the weight and output space ensemble.

##  Appendix C Supportive Empirical Results for the Theory

###  C.1 FalseFalseTrue Phenomenon

In this subsection, we take a deeper look at WiSE-FT [60], a popular model
averaging method that averages the weights of the pre-trained and fine-tuned
model. [60] obtains the fine-tuned model by fine-tuning the pre-trained CLIP
model on ImageNet. They have shown that the averaging of the pre-trained and
the fine-tuned model can outweigh both of them on ImageNet (ID dataset) as
well as five OOD datasets (ImageNetV2, ImageNetA, ImageNetR, ImageNetSketch
and ObjectNet). We denote the pre-trained model as PM, fine-tuned model as FM,
and averaged model as AM.

To understand why model averaging is effective, we divide each dataset into
eight groups of samples according to whether the PM, FM and AM make correct
predictions, respectively. We further use T/F to denote whether a model makes
correct predictions, i.e., T for True and F for False. For example, we use
PM(T)-FM(F)-AM(T) to denote the group of samples on which the predictions of
PM , FM and AM are correct, wrong, and correct, respectively. A simple
explanation for the improvement of the averaging of two models is that when
one model makes a mistake and the other one is correct, the correct model can
rectify the mistakes made by the other model. So we evaluate the performance
on the group of data where one model makes a wrong prediction while the other
model makes a correct prediction, i.e., the group containing PM(T)-FM(F) and
PM(F)-FM(T). We refer to this group of data as TF+FT for short in the
following discussion. We also look into another subset TT+FF which contains
PM(T)-FM(T) and PM(F)-FM(F).

Given a subset $\mathcal{G}$ of a dataset $\mathcal{D}$, we use
$\mbox{CorrectNum}(\mathcal{G};f)$ to denote the number samples in
$\mathcal{G}$ that are correctly predicted by a model $f$, e.g.,
$\mbox{CorrectNum}(\mbox{TF+FT};\mbox{PM})$ stands for the number of samples
that are correctly classified by the pre-trained model PM. We propose the
metric $\mbox{ImproveContri}(\mathcal{G})$ which estimates how much AM
performs better than PM and FM on the group $\mathcal{G}$ and how much the
improvement on $\mathcal{G}$ contributes to the overall accuracy improvement
on the whole dataset $\mathcal{D}$:

$\displaystyle\small\mbox{ImproveContri}(\mathcal{G})=\frac{\mbox{CorrectNum}(\mathcal{G};\mbox{AM})-\max\\{\mbox{CorrectNum}(\mathcal{G};\mbox{PM}),\mbox{CorrectNum}(\mathcal{G};\mbox{FM})\\}}{|\mathcal{D}|}$
(2)

For Example, suppose $\mathcal{D}$ contains 1,000 samples and its subset
$\mathcal{G}$ contains 200 samples. PM, FM and AM correctly predict 120, 118,
130 samples in $\mathcal{G}$, i.e.,
$\mbox{CorrectNum}(\mathcal{G};\mbox{PM})=120$,
$\mbox{CorrectNum}(\mathcal{G};\mbox{FM})=118$,
$\mbox{CorrectNum}(\mathcal{G};\mbox{PM})=130$. AM outperform PM and FM by
making 10 more correct predictions on $\mathcal{G}$, further these 10 samples
contribute to $\frac{10}{1,000}\times 100\%=1.0\%$ accuracy improvement on the
whole dataset. Note that $\mbox{ImproveContri}(\mathcal{D})$ denotes the
accuracy improvement of model averaging on the dataset $\mathcal{D}$, which is
also denoted as ImproveContri(ALL) in the following discussion. The results of
ImproveContri(TT+FF), ImproveContri(TF+FT) and ImproveContri(ALL) are
illustrated in Figure 7(a).

We surprisingly find that $\mbox{ImproveContri}(\mathcal{G})$ is significant
on TT+FF in all the datasets, which means the averaged model AM can exceed PM
and FM on the groups where PM and FM are both right or wrong. Recall that the
subset TT+FF contains four groups, PM(T)-FM(T)-AM(T), PM(T)-FM(T)-AM(F),
PM(F)-FM(F)-FM(F), and PM(F)-FM(F)-FM(T). We further plot the ratio of the
sample size in PM(T)-FM(T)-AM(F) and PM(F)-FM(F)-FM(T) over $|\mathcal{G}|$ in
Figure 7(b), respectively. We find that PM(T)-FM(T)-AM(F) is nearly the same
(about 0.5% ) in all datasets. The group PM(F)-FM(F)-AM(T) is much larger than
PM(T)-FM(T)-AM(F), especially in OOD datasets. It indicates that AM can make
correct predictions on many samples where the both PM and FM make wrong
predictions when distributional shift occurs! Interestingly, we find
$\mbox{ImproveContri}(TF+FT)$ is negative on some datasets, e.g, IN-R and
IN-S. In Section 4 and Appendix E.6, we find that this is because the fine-
tuned model is highly over-confident and the fine-tuned model dominate WiSE-FT
even when it make mistakes.

Figure 7: A closer look at WiSE-FT, which averages the pre-trained CLIP and
the model obtained by fine-tuning CLIP on ImageNet. Here ImageNet is regarded
as ID domain and the other 5 ImageNet variants are OOD domains, i.e., IN-V2
(ImageNetV2), IN-R(ImageNetR), IN-A(ImageNetA), IN-S (ImageNetSketch), and
ObjNet (ObjectNet). (Left) $\mbox{ImproveContri}(\mathcal{G})$ is defined in
Eqn. equation 2, which estimates how the AM (averaged model) performs better
than the PM (pre-trained) and FM (fine-tuned model) on the group $\mathcal{G}$
and how much the improvement on $\mathcal{G}$ contributes to the overall
accuracy improvement on the whole dataset $\mathcal{D}$. (Right) The ratio of
sample size in PM(T)-FM(T)-AM(F) and PM(F)-FM(F)-AM(T) over the same size of
the whole dataset. Here PM(T)-FM(T)-AM(F) denotes the group where the AM make
wrong predictions and the PM and FM models make correct predictions;
PM(F)-FM(F)-AM(T) denotes the group where AM make correct predictions while
both PM and FM make wrong predictions. Putting these two figures together, we
can see the AM can correct many samples on which PM and FM make wrong
predictions in OOD.

Remark: In Figure 2(Left) of Section 2, we present the results of
ImproveContri(TT+FF) to represent the samples where both individual models
make incorrect predictions, but the averaged model makes correct predictions.
ImproveContri(TT+FF) is calculated as the group ratio of PM(F)-FM(F)-AM(T)
subtracted by PM(T)-FM(T)-AM(F). We use ImproveContri(TT+FF) instead of
PM(F)-FM(F)-AM(T) because we believe that there is a certain proportion of
samples in PM(F)-FM(F)-AM(T) where the averaged model corrects mistakes due to
the randomness introduced by the non-linearity of deep neural networks (DNNs)
during weight averaging. To approximate such randomness, we use the size of
PM(T)-FM(T)-AM(F). This adjustment helps account for a more accurate
approximation of the sample ratios where the averaged model corrects the
samples due to its utilization of more diverse spurious features.

###  C.2 Deep neural networks learn different features

In Section 2, we have shown that the pre-trained and fine-tuned uses different
features and the averaged model can ultilize more diverse features. Actually,
[3] provides empirical evidence (e.g., Figure 3 and 4 in [3]) supporting the
use of diverse features by different deep neural networks with same
archetecture, even when trained on the same datasets (with different
initialization). We add their empirical observations in Figure 8 for easy of
reference.

Figure 8: Figures taken from [3] which show that different DNN (with the same
architecture) learns different features even trained on the same dataset.

###  C.3 Experiments on MultiColorMNIST

MultiColorMNIST. We extend the CMNIST [5] to MultiColorMNIST, which is
constructed following Definition 1. MultiColorMNIST contains 10 classes with
32 spurious features. Each image has $42\times 42\times 3$ pixels. There are
32 patches in each image and each patch can take one of 10 colors. Figure 9
illustrates two samples from MultiColorMNIST. Specifically, the label of the
sample is generated from the shape of the digit and each color patch is
perfected correlated with the label. Let $C_{i}$ denote $i$th color patch for
$i=1,2,...,32$. Each $C_{i}$ takes one of the color which is perfectly
correlated with $\bm{y}$. For example, the $1$st patch, i.e., $C_{1}$, always
takes ‘white’ on samples with label $5$; the $2$nd patch, i.e., $C_{2}$,
always takes ‘yellow’ on samples with label $5$. Each $C_{i}$ is independently
generated from the label $\bm{y}$ and we have $C_{i}\perp C_{j}|\bm{y}$ for
$i\neq j$. See Figure 11 for detailed illustration of the data generation
process which follows the theoretical Definition 1. In the OOD testing
distribution, the spurious correlation can fail with probability $p$. For
example, samples with label $5$ can randomly pick any color with probability
$p$ in OOD . The data generation process is analogous to the theoretical
setting in Definition 1, where each patch is a spurious feature and each color
is an attribute that the spurious feature can take.

SingleColorMNIST. We also introduce SingleColorMNIST for better comparision.
SingleColorMNIST has 10 classes and each image has $42\times 42\times 3$
pixels, which is the same with MultiColorMNIST. However, SingleColorMNIST only
contains 1 spurious features. In other words, the 32 patches in each image are
the same. The spurious correlation is defined similarly with MultiColorMNIST.
Figure 10 illustrates two samples from SingleColorMNIST.

Experimental Details. We use the following configuration for both
SingleColorMNIST and MultiColorMNIST. We use an 2 layer MLP with 64 hidden
units to perform classification. We adopt Adam with learning rate $10^{-3}$
and batch size 100. We train for 5000 steps and report the performance at the
last step. We train two individual models $\bar{f}$ and $\tilde{f}$ with
different random initialization on MultiColorMNIST. We also evaluate the
ensemble of the two models, i.e.,
$f_{\mbox{ose}}(\bm{x})=\bar{f}(\bm{x})+\tilde{f}(\bm{x})$. Each experiment is
repeated for $n=20$ random seeds.

Results. We vary $p$ in MultiColorMNIST and compare the performance of the
ensemble model with each individual model. $p$ is the probability that
spurious correlation no-longer holds in testing environment. A larger $p$
indicates larger distributional shift. The results of MultiColorMNIST are
summarized in Table 3. We can see that model ensemble consistently improve the
OOD performance. Figure 12 visualizes how much each model relies on each
patch. Specifically, Figure 12 shows how much the model changes its prediction
when we replace a patch with black color. We can see each individual models
uses different feature sets and model ensemble uses more diverse features.
Table 4 shows the results of SingleColorMNIST. We can see that model ensemble
can not improve the OOD performance in SingleColorMNIST (since there is only
one spurious feature in SingleColorMNIST and model ensemble can not utilize
more diverse spurious features).

Comparing Table 3 and Table 4, we can see that the performance of individual
model in MultiColorMNIST is higher than that in SingleColorMNIST when the $p$
is the same. This is because the individual model already learns multiple
spurious features (even though it is only a small subset of the whole feature
set as shown in Figure 12). This is also consistent with our theoretical
results that diverse spurious features leads to better OOD performance.

Figure 9: Two samples from MultiColorMNIST. MultiColorMNIST has 10 classes and
each sample contains 32 spurious features.

Figure 10: Two samples from SingleColorMNIST. SingleColorMNIST has 10 classes and each sample contains 1 spurious feature. Figure 11: The data generation process of MultiColorMNIST (follows Definition 1) Figure 12: Visualization of the features uses by each model and model ensemble. | model1 | model2 | model_ensemble
---|---|---|---
$p$ |  |  |
0.10 | 100.00$\pm$0.00 | 100.00$\pm$0.00 | 100.00$\pm$0.00
0.20 | 99.99$\pm$0.00 | 99.99$\pm$0.00 | 100.00$\pm$0.00
0.30 | 99.91$\pm$0.01 | 99.89$\pm$0.04 | 99.99$\pm$0.01
0.40 | 99.16$\pm$0.11 | 99.19$\pm$0.13 | 99.75$\pm$0.03
0.50 | 95.84$\pm$0.35 | 96.06$\pm$0.35 | 98.13$\pm$0.14
0.60 | 87.15$\pm$0.74 | 87.56$\pm$0.69 | 92.31$\pm$0.41
0.70 | 71.05$\pm$1.04 | 71.77$\pm$0.94 | 78.64$\pm$0.73
0.75 | 60.07$\pm$1.04 | 60.75$\pm$0.91 | 67.61$\pm$0.80
0.80 | 48.57$\pm$0.92 | 49.26$\pm$0.83 | 55.25$\pm$0.75
0.85 | 36.93$\pm$0.70 | 37.74$\pm$0.66 | 42.34$\pm$0.64
0.90 | 26.01$\pm$0.45 | 26.63$\pm$0.42 | 29.28$\pm$0.40
Table 3: Results on MultiColorMNIST | model1 | model2 | model_ensemble
---|---|---|---
$p$ |  |  |
0.1 | 91.04$\pm$0.03 | 91.07$\pm$0.05 | 91.04$\pm$0.04
0.2 | 82.34$\pm$0.02 | 82.39$\pm$0.07 | 82.34$\pm$0.04
0.3 | 72.93$\pm$0.08 | 72.99$\pm$0.10 | 72.93$\pm$0.09
0.4 | 64.08$\pm$0.09 | 64.21$\pm$0.19 | 64.10$\pm$0.11
0.5 | 54.89$\pm$0.12 | 54.99$\pm$0.18 | 54.88$\pm$0.14
0.6 | 45.91$\pm$0.16 | 46.09$\pm$0.31 | 45.92$\pm$0.19
0.7 | 37.39$\pm$0.15 | 37.55$\pm$0.27 | 37.39$\pm$0.17
0.8 | 27.86$\pm$0.19 | 28.06$\pm$0.32 | 27.87$\pm$0.22
0.9 | 19.28$\pm$0.18 | 19.50$\pm$0.34 | 19.29$\pm$0.20
Table 4: Results on SingleColorMNIST

Remark. Recall weight space ensemble (WSE) needs to be conducted between the
pre-trained and fine-tuned models or different fine-tuned models starting from
the same pre-trained model [60, 24]. Since we have suitable pre-trained model
for the synthetic dataset, we leave the investigation of WSE on
MultiColorMNIST to future work.

###  C.4 Simulation

In this section, we take some simulations to investigate the performance of
theoretical forecasting results of OOD accuracy. Following the data generation
process in Definition 1, here we consider four examples:

  1. 1.

example 1-1: $\bar{n}_{v}=2,\bar{n}_{s}=3$ in model 1;
$\tilde{n}_{v}=2,\tilde{n}_{s}=3$ in model 2; overlapped feature number
$n_{vo}=n_{so}=0$; noise variance $\sigma=0.01$; distribution shift
probability $p=0.9$.

  2. 2.

example 1-2: $\bar{n}_{v}=2,\bar{n}_{s}=3$ in model 1;
$\tilde{n}_{v}=2,\tilde{n}_{s}=3$ in model 2; overlapped feature number
$n_{vo}=n_{so}=1$; noise variance $\sigma=0.01$; distribution shift
probability $p=0.9$.

  3. 3.

example 2-1: $\bar{n}_{v}=5,\bar{n}_{s}=20$ in model 1;
$\tilde{n}_{v}=4,\tilde{n}_{s}=20$ in model 2; overlapped feature number
$n_{vo}=n_{so}=0$; noise variance $\sigma=0.01$; distribution shift
probability $p=0.9$.

  4. 4.

example 2-2: $\bar{n}_{v}=5,\bar{n}_{s}=20$ in model 1;
$\tilde{n}_{v}=5,\tilde{n}_{s}=20$ in model 2; overlapped feature number
$n_{vo}=4,n_{so}=1$; noise variance $\sigma=0.01$; distribution shift
probability $p=0.9$.

In each example, we take $1000$ simulations to report the mean OOD accuracy in
Table 5. To be precise, the training data size is $20000$ and the test data
size is $10000$ in each simulation.

|  | Model $1$ | Model $2$ | Model Average | Model Ensemble
---|---|---|---|---|---
Example 1-1 | Simulation Results | 0.866 | 0.866 | 0.974 | 0.974
Theoretical Results | 0.865 | 0.865 | 0.973 | 0.973
Example 1-2 | Simulation Results | 0.866 | 0.861 | 0.943 | 0.940
Theoretical Results | 0.865 | 0.865 | 0.948 | 0.946
Example 2-1 | Simulation Results | 0.940 | 0.894 | 0.978 | 0.978
Theoretical Results | 0.941 | 0.910 | 0.980 | 0.980
Example 2-2 | Simulation Results | 0.943 | 0.939 | 0.999 | 0.989
Theoretical Results | 0.943 | 0.943 | 0.992 | 0.983
Table 5: Simulation for the OOD accuracy in different models

Then comparing the results of theoretical results and simulation results, it
is safely to say that our theoretical analysis, as well as proper
approximations, could take an effective estimation for OOD accuracy.

##  Appendix D Discussions, illustrations, and supportive results for the
theoretical parts.

###  D.1 Discussion on the theoretical models

Our theoretical models in Section 3 is designed to mimic the modern deep
learning architectures such as Vision Transforms (ViT) [22]. Figure 13
provides a comparison between our theoretical models and Vision Transformers.

Similar to ViT, we process images as patches, where each patch corresponds to
a specific feature denoted as $\mbox{Patch}_{i}$. Each $\mbox{Patch}_{i}$ is
represented by high-dimensional vectors $x_{i}\in\mathbb{R}^{d}$ in the
embedding space. Consequently, the whole feature is obtained by concatenating
the embeddings of each patch, resulting in ${\bm{x}}=[x_{1},x_{2},...]$.
Assuming a total of $d_{t}$ features ($d_{t}=d_{v}+d_{s}$ in Section 3), we
have ${\bm{x}}\in\mathbb{R}^{d\times d_{t}}$. Notably, [3] also uses a similar
theoretical data model that concatenates the patches to analyze the
convolutional neural networks, e.g., Figure 5 in [3].

To simplify the model, we utilize a two-layer structure consisting of a binary
feature mask $\Phi$ as the feature encoder and a linear classifier ${\bm{w}}$,
analogous to ViT which uses the transformer feature encoder with an MLP
classifier. This two-layer simplification approach has been widely employed in
OOD literature [5, 51, 67, 42, 36]. The difference between our theoretical
model and ViT is that ViT process the features sequentially while we select
the feature at once.

The binary feature mask $\Phi$ is represented as $\\{0,1\\}^{d_{t}}$. For
instance, if we have three features, i.e., ${\bm{x}}=[x_{1},x_{2},x_{3}]$, and
$\Phi=[1,1,0]$, the learned feature would be
${\bm{x}}^{\top}\Phi=x_{1}+x_{2}$. Considering a 3-class classification task,
the linear classifier ${\bm{w}}\in\mathbb{R}^{d\times 3}$ takes the learned
feature ${\bm{x}}^{\top}\Phi$ as input and produces a 3-dimensional vector
whose elements represent the logits of the three classes. The classifier
${\bm{w}}$ is optimized to minimize the in-distribution (ID) loss based on the
learned feature. Therefore, we have:

$\displaystyle\bm{w}=\arg\min_{\bm{v}\in\mathbb{R}^{d\times
3}}\mathcal{R}_{id}({\bm{v}},\Phi),$

where $\mathcal{R}_{id}({\bm{v}},\Phi)$ represents the loss of
$({\bm{v}},\Phi)$ in the ID distribution.

Figure 13: Comparison of our theoretical models in Section 3 with Vision
Transformers [22]. Some parts of the figures are adopted from [22].

###  D.2 Comparison on our model with a 2-layer DNN

In this paper, we consider the model $\bm{w}^{\top}\bm{x}\Phi$, where
$\bm{w}\in\mathbb{R}^{d\times K}$ and $\Phi\in\\{0,1\\}^{d_{v}+d_{s}}$ are
paramters. Here the input $\bm{x}\in\mathbb{R}^{d\times(d_{v}+d_{s})}$ and see
App D.1 for detailed discussion. We then compare our model with a general
2-layer DNN to see why it can capture the difference between weight space
ensemble (WSE) and output space ensemble (OSE) in DNN.

Consider a general 2-layer DNN parameterized by
$(W_{a}\in\mathbb{R}^{d_{1}\times d_{2}},W_{b}\in\mathbb{R}^{d_{2}\times K})$
with ReLU activation $\delta(\cdot)$ and output
$f_{dnn}(X)=W_{b}^{\top}\delta(W_{a}^{\top}X)$ for $X\in\mathbb{R}^{d_{1}}$.
Here we use uppercase $X$ to avoid confusion with our previous $\bm{x}$ since
they have slightly different dimensions (App D.1). Since WSE is conducted on
the models that is close to a pre-trained model [59], e.g., $(W_{a0},W_{b0})$,
so we consider $f_{dnn}(X)=(W_{b0}+\Delta W_{b})^{\top}\delta((W_{a0}+\Delta
W_{a})^{\top}X)$ where $\Delta W_{a}$ and $\Delta W_{b}$ is small and
trainable. By Taylor expansion, we have

$\displaystyle\small
f_{dnn}(X)=\underbrace{W_{b0}^{\top}\delta(W_{a0}^{\top}X)}_{\mbox{(a)Fixed
Term}}+\underbrace{\Delta
W_{b0}^{\top}\delta(W_{a0}^{\top}X)+W_{b0}\delta^{\prime}(W_{a0}^{\top}X)(\Delta
W_{a}^{\top}X)}_{\mbox{(b)Linear Term}}+\underbrace{\Delta
W_{b}\delta^{\prime}(W_{a0}^{\top}X)(\Delta W_{a}^{\top}X)}_{\mbox{(c)Bilinear
Term}}+\xi$

Where $\delta^{\prime}(Y)$ is $\frac{\partial\delta(Y)}{\partial Y}$. Further,
we incorporate the fact that the second order derivative
$\frac{\partial^{2}\delta(Y)}{\partial^{2}Y}$ is zero almost everywhere for
ReLU activation function (except at $Y=0$). Then $\xi$ is the error term
induced by the non-linearity of ReLU activation function (while $W_{a0}^{T}X$
has some zero elements). To be precise, as here we just focus on fine-tuning
regime and $W_{a0}^{T}X$ is not sparse in general situations, it is safely to
say that $\xi$ is small. WSE and OSE are exactly the same for the (a) fixed
term and (b) linear term. We will show that WSE and OSE differs on the
(c)bilinear term, which is captured by our model in Definition 3-4.

Consider two models, $\bar{f}_{dnn}$ and $\tilde{f}_{dnn}$, both close to the
pre-trained model. Specifically,

$\displaystyle\bar{f}_{dnn}(X)$
$\displaystyle=(W_{b0}+\Delta\bar{W}_{b})^{\top}\delta((W_{a0}+\Delta\bar{W}_{a})^{\top}X);$
$\displaystyle\tilde{f}_{dnn}(X)$
$\displaystyle=(W_{b0}+\Delta\tilde{W}_{b})^{\top}\delta((W_{a0}+\Delta\tilde{W}_{a})^{\top}X);$

Then the output space ensemble of $\bar{f}_{dnn}(X)$ and $\tilde{f}_{dnn}(X)$
is

$\displaystyle f_{dnn,ose}$
$\displaystyle=0.5\left((W_{b0}+\Delta\bar{W}_{b})^{\top}\delta((W_{a0}+\Delta\bar{W}_{a})^{\top}X)+(W_{b0}+\Delta\tilde{W}_{b})^{\top}\delta((W_{a0}+\Delta\tilde{W}_{a})^{\top}X)\right)$
$\displaystyle=\underbrace{W_{b0}^{\top}\delta(W_{a0}^{\top}X)}_{\mbox{(a)Fixed
Term}}+\underbrace{0.5(\Delta\bar{W}_{b0}+\Delta\tilde{W}_{b0})^{\top}\delta(W_{a0}^{\top}X)+W_{b0}\delta^{\prime}(W_{a0}^{\top}X)(0.5(\Delta\bar{W}_{a}+\Delta\tilde{W}_{a})^{\top}X)}_{\mbox{(b)Linear
Term}}$
$\displaystyle+\underbrace{0.5\left(\Delta\bar{W}_{b}\delta^{\prime}(W_{a0}^{\top}X)(\Delta\bar{W}_{a}^{\top}X)+\Delta\tilde{W}_{b}\delta^{\prime}(W_{a0}^{\top}X)(\Delta\tilde{W}_{a}^{\top}X)\right)}_{\mbox{(c)Bilinear
Term}}$

$\displaystyle f_{dnn,wse}=$
$\displaystyle(W_{b0}+0.5(\Delta\bar{W}_{b}+\Delta\tilde{W}_{b}))^{\top}\delta((W_{a0}+0.5(\Delta\bar{W}_{a}+\Delta\tilde{W}_{a}))^{\top}X)$
$\displaystyle=$
$\displaystyle\underbrace{W_{b0}^{\top}\delta(W_{a0}^{\top}X)}_{\mbox{(a)Fixed
Term}}+\underbrace{0.5(\Delta\bar{W}_{b0}+\Delta\tilde{W}_{b0})^{\top}\delta(W_{a0}^{\top}X)+W_{b0}\delta^{\prime}(W_{a0}^{\top}X)(0.5(\Delta\bar{W}_{a}+\Delta\tilde{W}_{a})^{\top}X)}_{\mbox{(b)Linear
Term}}$
$\displaystyle+\underbrace{0.25(\Delta\bar{W}_{b}+\Delta\tilde{W}_{b})\delta^{\prime}(W_{a0}^{\top}X)((\Delta\bar{W}_{a}+\Delta\tilde{W}_{a})^{\top}X)}_{\mbox{(c)Bilinear
Term}}$

Comparing $f_{dnn,ose}$ with $f_{dnn,wse}$ , we can see that the difference of
them lies in the bilinear term:

$\displaystyle f_{dnn,wse}-f_{dnn,ose}=$
$\displaystyle\left(0.25(\Delta\bar{W}_{b}+\Delta\tilde{W}_{b})\delta^{\prime}(W_{a0}^{\top}X)((\Delta\bar{W}_{a}+\Delta\tilde{W}_{a})^{\top}X)\right)$
$\displaystyle-0.5\left(\Delta\bar{W}_{b}\delta^{\prime}(W_{a0}^{\top}X)(\Delta\bar{W}_{a}^{\top}X)+\Delta\tilde{W}_{b}\delta^{\prime}(W_{a0}^{\top}X)(\Delta\tilde{W}_{a}^{\top}X)\right)$
(3)

We can see the bilinear term difference has a clear analogy with our models in
Definition 3-4. Specifically, according to our Definition of OSE and WSE in
Definition 3-4, we have

$\displaystyle
f_{\mbox{wse}}-f_{\mbox{ose}}=0.25(\bar{\bm{w}}+\tilde{\bm{w}})^{\top}\bm{x}(\bar{\Phi}+\tilde{\Phi})-0.5(\bar{\bm{w}}^{\top}\bm{x}\bar{\Phi}+\tilde{\bm{w}}^{\top}\bm{x}\tilde{\Phi}).$
(4)

Comparing equation D.2 and equation 4, we can see that $\bm{w}$ is analogous
to $\Delta W_{b}$ and $\Phi$ is analogous to $\Delta W_{a}$. equation D.2 and
equation 4 differ by a scaling $\delta^{\prime}(W_{a0}^{\top}X)$, which is a
fixed matrix independent of the trainable parameter $(\Delta W_{a},\Delta
W_{b})$ .

###  D.3 Illustration of the transformation matrix Q

Consider the 3-class classification problem. In the ID distribution, we have,

$\displaystyle\bm{Q}_{s,i}=[\bm{e}_{1},\bm{e}_{2},\bm{e}_{3}]=\bm{I}_{3}=\begin{bmatrix}1,0,0\\\
0,1,0\\\ 0,0,0\end{bmatrix},$

This indicates that each spurious feature is perfectly correlated with the
invariant feature, as illustrated in Figure 14 (left). For instance,
$\bm{Q}_{s,j}=\bm{I}_{3}$ implies that the background of the dog, crow, and
camel are floor, grass, and sand, respectively.

In the OOD distribution, $\bm{Q}_{s,j}$ is no longer equal to $\bm{I}$,
indicating that the correlation between animals and the background may fail
with a certain probability. Figure 14 (right) illustrates $\bm{Q}_{s,j}(1)$,
which represents the first column of $\bm{Q}_{s,j}$. $\bm{Q}_{s,j}(1)$ can
take the value $\bm{e}_{2}$ with a probability of $p/3$, indicating that the
background of the dog is grass in this case. Similarly, $\bm{Q}_{s,j}(1)$ can
take the value $\bm{e}_{3}$ with a probability of $p/3$, indicating that the
background of the dog is sand with a probability of $p/3$.

Figure 14: Illustrations of the matrix $\bm{Q}_{s,j}$.

###  D.4 On the pessimism of worst-case theoretical analysis for OOD

Recall that our definition of Robust Accuracy for OOD , i.e.,

$\mathcal{A}_{\mbox{ood}}(f)=\mathbb{E}_{\bm{Q}_{s}}\left[\mathbb{E}_{\bm{x},\bm{y}}[\mathbb{I}(\bm{e}_{\hat{k}}=\bm{y})|\bm{Q}_{s}]\right],$

is different from the traditional definition of Robust Accuracy defined in
terms of worst case performance [5, 51, 57, 67, 2], i.e.,
$\min_{\bm{Q}_{s}}\mathbb{E}_{\bm{x},\bm{y}}[\mathbb{I}(\bm{e}_{\hat{k}}=\bm{y})|\bm{Q}_{s}]$.
As shown in the previous subsection, $\mathbb{E}_{\bm{Q}_{s}}$ characterizes
the probability that we will encounter an OOD distribution which fails our
model in our setting. Instead, $\min_{\bm{Q}_{s}}$ cares about whether there
exists a $\bm{Q}_{s}$ that induces an OOD distribution in which our model will
fail. We find that $\min_{\bm{Q}_{s}}$ is too pessimistic too draw meaningful
conclusions for EBMs. We then define the matric

$\mathcal{A}_{{ood},worst}(f)=\min_{\bm{Q}_{s}}\left[\mathbb{E}_{\bm{x},\bm{y}}[\mathbb{I}(\bm{e}_{\hat{k}}=\bm{y})|\bm{Q}_{s}]\right],$

######  Proposition 5 (Pessimism of worst case metric).

Suppose Assumption 1 and 2 hold, and there are infinite ID and OOD samples.
Consider Example 1, we have
$\mathcal{A}_{{ood},worst}(f_{\mbox{ose}})-\max\\{\mathcal{A}_{{ood},worst}(\bar{f}),\mathcal{A}_{{ood},worst}(\tilde{f})\\}\leq\epsilon$.

In our setting, the worst case metric can not capture the interesting
improvement of ensemble-based models in OOD. Intuitively, since the worst case
metric only cares about whether there exists a $\bm{Q}_{s}$ that a model will
fail. As long as a model learns more spurious features than invariant
features, the worst case accuracy is all close to 0. However, as we have shown
previously, model averaging can use more diverse spurious features, resulting
in a much lower probability of encountering a $\bm{Q}_{s}$ that fails the
model.

###  D.5 ID performance

Recall that in Section 3 the OOD accuracy is defined by

$\mathcal{A}_{\mbox{ood}}(f)=\mathbb{E}_{\bm{Q}_{s}}\left[\mathbb{E}_{\bm{x},\bm{y}}[\mathbb{I}(\bm{e}_{\hat{k}}=\bm{y})|\bm{Q}_{s}]\right].$

The ID accuracy $\mathcal{A}_{\mbox{id}}(f)$ is defined similarly by fixing
$[\bm{Q}_{s,1},\dots,\bm{Q}_{s,d_{s}}]=[\bm{I},\dots,\bm{I}]$. According to
Lemma 3, we know that the ID accuracy of all models involved in Definition 2,
Example 1-2 are larger than $1-\epsilon$.

###  D.6 Intuition of OOD Performance Improvement of OSE

We use Example 1 to show the main intuition of the output space ensemble
(OSE). In Example 1, two individual models learn non-overlapped feature, so
model ensemble and averaging are the same. According to the proof in Appendix
F.1, consider the samples from the first class, i.e., $\bm{y}=\bm{e}_{1}$, the
predicted logit of the each class is

$\displaystyle\bm{w}(1)^{\top}\bm{x}\bar{\Phi}|_{\bm{y}=\bm{e}_{1}}=$
$\displaystyle\sum_{i=1}^{2}\bm{\mu}_{v,i}(1)^{\top}\left(\bm{\mu}_{v,i}\bm{Q}_{v,i}(1)\right)+\sum_{j=1}^{3}\bm{\mu}_{s,j}(1)^{\top}\left(\bm{\mu}_{s,j}\bm{Q}_{s,j}(1)\right),$
$\displaystyle\bm{w}(2)^{\top}\bm{x}\bar{\Phi}|_{\bm{y}=\bm{e}_{1}}=$
$\displaystyle\sum_{i=1}^{2}\bm{\mu}_{v,i}(2)^{\top}\left(\bm{\mu}_{v,i}\bm{Q}_{v,i}(1)\right)+\sum_{j=1}^{3}\bm{\mu}_{s,j}(2)^{\top}\left(\bm{\mu}_{s,j}\bm{Q}_{s,j}(1)\right),$
$\displaystyle\bm{w}(3)^{\top}\bm{x}\bar{\Phi}|_{\bm{y}=\bm{e}_{1}}=$
$\displaystyle\sum_{i=1}^{2}\bm{\mu}_{v,i}(3)^{\top}\left(\bm{\mu}_{v,i}\bm{Q}_{v,i}(1)\right)+\sum_{j=1}^{3}\bm{\mu}_{s,j}(3)^{\top}\left(\bm{\mu}_{s,j}\bm{Q}_{s,j}(1)\right),$

where we omit the noise term whose impact on the accuracy is less than
$\epsilon$ according to Lemma 3. Further, let $\bm{\mu}$ denote any
$\bm{\mu}_{v,i}$ and $\bm{\mu}_{s,j}$ and $\bm{Q}$ denote its corresponding
transformation matrix. For example, $\bm{\mu}=\bm{\mu}_{s,j}$ and
$\bm{Q}=\bm{Q}_{s,j}$. Suppose $\bm{Q}(1)=\bm{e}_{k_{2}}$, we have

$\bm{\mu}(k_{1})^{\top}\left(\bm{\mu}\bm{Q}(1)\right)=\bm{\mu}(k_{1})^{\top}\left(\bm{\mu}\bm{e}_{k_{2}}\right)=\bm{\mu}(k_{1})^{\top}\bm{\mu}(k_{2})=\begin{cases}1,\mbox{
if }k_{1}=k_{2},\\\ 0,\mbox{ otherwise.}\end{cases}$

For the invariant features, $\bm{Q}_{v,i}(1)=\bm{e}_{1}$ always hold and for
spurious features, $\bm{Q}_{s,j}(1)$ takes $\bm{e}_{1}$ with probability
$1-\frac{2p}{3}$, and takes $\bm{e}_{2}$ or $\bm{e}_{3}$ with $\frac{p}{3}$,
respectively. So the predicted logit of the each class is simply

$\displaystyle\bar{\bm{w}}(1)^{\top}\bm{x}\bar{\Phi}|_{\bm{y}=\bm{e}_{1}}=$
$\displaystyle 2+\sum_{j=1}^{3}\mathbb{I}(\bm{Q}_{s,j}(1)=\bm{e}_{1}),$
$\displaystyle\bar{\bm{w}}(2)^{\top}\bm{x}\bar{\Phi}|_{\bm{y}=\bm{e}_{1}}=$
$\displaystyle 0+\sum_{j=1}^{3}\mathbb{I}(\bm{Q}_{s,j}(1)=\bm{e}_{2}),$
$\displaystyle\bar{\bm{w}}(3)^{\top}\bm{x}\bar{\Phi}|_{\bm{y}=\bm{e}_{1}}=$
$\displaystyle 0+\sum_{j=1}^{3}\mathbb{I}(\bm{Q}_{s,j}(1)=\bm{e}_{3}),$

Let us consider the probability of
$\bar{\bm{w}}(2)^{\top}\bm{x}\bar{\Phi}|_{\bm{y}=\bm{e}_{1}}>\bar{\bm{w}}(1)^{\top}\bm{x}\bar{\Phi}|_{\bm{y}=\bm{e}_{1}}$,
i.e., the model $(\bar{\Phi},\bar{w})$ mistakenly predicts the second the
class $\bm{e}_{2}$ even if the true class is $\bm{e}_{1}$. This will happen
when $\\{\mathbb{I}(\bm{Q}_{s,j}(1)=\bm{e}_{2})\\}_{j=1,2,3}$ holds
simultaneously, whose probability would be $\frac{p^{3}}{27}$. Intuitively, 3
spurious features takes the value in OOD that is correlated with the second
class $\bm{e}_{2}$, overwhelming the two invariant features correlated with
$\bm{e}_{1}$.

As for the averaged model, we have

$\displaystyle\bar{\bm{w}}(1)^{\top}\bm{x}\bar{\Phi}|_{\bm{y}=\bm{e}_{1}}=$
$\displaystyle 4+\sum_{j=1}^{6}\mathbb{I}(\bm{Q}_{s,j}(1)=\bm{e}_{1}),$
$\displaystyle\bar{\bm{w}}(2)^{\top}\bm{x}\bar{\Phi}|_{\bm{y}=\bm{e}_{1}}=$
$\displaystyle 0+\sum_{j=1}^{6}\mathbb{I}(\bm{Q}_{s,j}(1)=\bm{e}_{2}),$
$\displaystyle\bar{\bm{w}}(3)^{\top}\bm{x}\bar{\Phi}|_{\bm{y}=\bm{e}_{1}}=$
$\displaystyle 0+\sum_{j=1}^{6}\mathbb{I}(\bm{Q}_{s,j}(1)=\bm{e}_{3}).$

We will have
$\bar{\bm{w}}(2)^{\top}\bm{x}\bar{\Phi}|_{\bm{y}=\bm{e}_{1}}>\bar{\bm{w}}(1)^{\top}\bm{x}\bar{\Phi}|_{\bm{y}=\bm{e}_{1}}$
if either of the following occurs

  * •

$\\{\mathbb{I}(\bm{Q}_{s,j}(1)=\bm{e}_{2})\\}_{j=1}^{6}$ holds simultaneously,
whose probability would be $\frac{p^{6}}{729}$

  * •

Five of $\\{\mathbb{I}(\bm{Q}_{s,j}(1)\\}_{j=1}^{6}$ takes $\bm{e}_{2}$ and
the remaining one takes $\bm{e}_{3}$, i.e.,
$\sum_{j=1}^{6}\mathbb{I}(\bm{Q}_{s,j}(1)=\bm{e}_{2})=5$ and
$\sum_{j=1}^{6}\mathbb{I}(\bm{Q}_{s,j}(1)=\bm{e}_{3})=1$. Such probability is
$\frac{6p^{6}}{729}$.

The total probability is then
$\frac{7p^{6}}{729}\approx\frac{p^{6}}{104}\leq\frac{p^{3}}{104}<\frac{p^{3}}{27}$.
See Figure 15 for a visualization of the main intuition.

Figure 15: Comparison of the failure probability of individual model and
averaged model. With probability $p^{3}/27$, the individual model
$(\bar{\Phi},\bar{\bm{w}})$ will encounter an OOD distribution where it
mistakenly predicting the second class $\bm{e}_{2}$ on the samples from the
first class $\bm{e}_{1}$. For the averaged model, such probability would be
roughly about $p^{6}/729$. Refer to Appendix D.6 for detailed explanation.

###  D.7 The Difference Between WSE and OSE in OOD

####  D.7.1 Explaining the difference between WSE and OSE

We use the following Example 3 to show the main intuition of the difference
between model averaging and ensemble.

######  Example 3.

Two individual models learn overlapped features
${\color[rgb]{0,0,1}\bm{x}_{v,2}}$ and ${\color[rgb]{0,0,1}\bm{x}_{s,3}}$ as

$\bm{x}\bar{\Phi}^{\top}=\bm{x}_{v,1}+{\color[rgb]{0,0,1}\bm{x}_{v,2}}+\bm{x}_{s,1}+\bm{x}_{s,2}+{\color[rgb]{0,0,1}\bm{x}_{s,3}},\quad\bm{x}\tilde{\Phi}^{\top}={\color[rgb]{0,0,1}\bm{x}_{v,2}}+\bm{x}_{v,3}+{\color[rgb]{0,0,1}\bm{x}_{s,3}}+\bm{x}_{s,4}+\bm{x}_{s,5},$

######  Proposition 6.

Consider the Example 3, suppose Assumption 1 and 2 hold, and there are
infinite ID and OOD samples, the averaged and ensemble models are defined as
Definition 3. Omitting small terms containing $\epsilon$, we have
$\mathcal{A}_{ood}(\bar{f})=\mathcal{A}_{ood}(\tilde{f})=1-\frac{1}{9}p^{3}$
and $\mathcal{A}_{ood}(f_{\mbox{ose}})=1-\frac{4p^{4}}{81}-\frac{8p^{5}}{243}$
and $\mathcal{A}_{ood}(f_{\mbox{wse}})=1-\frac{4p^{4}}{81}-\frac{p^{5}}{27}$.

Full Proof in Appendix F.4. In Example 3, two individual models learn
overlapped feature, $\bm{x}_{v,i}(k)$ and $\bm{x}_{s,3}(k)$. By Lemma 5 , for
$k=1,2,3$, we have

$\displaystyle\bar{\bm{w}}(k)$
$\displaystyle=\sum_{i=1}^{2}\bm{\mu}_{v,i}(k)+\sum_{j=1}^{3}\bm{\mu}_{s,j}(k),$
$\displaystyle\tilde{\bm{w}}(k)$
$\displaystyle=\sum_{i=2}^{3}\bm{\mu}_{v,i}(k)+\sum_{j=3}^{5}\bm{\mu}_{s,j}(k),$

So we have

$\displaystyle\bar{\bm{w}}(k)+\tilde{\bm{w}}(k)=\sum_{i=1,3}\bm{\mu}_{v,i}(k)+{\color[rgb]{0,0,1}2\bm{\mu}_{v,2}(k)}+\sum_{j=1,2,4,5}\bm{\mu}_{s,j}(k)+{\color[rgb]{0,0,1}2\bm{\mu}_{s,3}(k)}$

For samples from the first class, we also have

$\bm{x}(\bar{\Phi}+\tilde{\Phi})|_{\bm{y}=\bm{e}_{1}}=\sum_{i=1,3}\bm{\mu}_{v,i}\bm{Q}_{v,i}(1)+{\color[rgb]{0,0,1}2\bm{\mu}_{v,2}\bm{Q}_{v,2}(1)}+\sum_{j=1,2,4,5}\bm{\mu}_{s,j}\bm{Q}_{s,j}(1)+{\color[rgb]{0,0,1}2\bm{\mu}_{s,3}\bm{Q}_{s,3}(1)}+\sum_{i=1}^{10}\bm{z}_{i}$

where $\bm{z}_{i}\sim\mathcal{N}(0,\sigma^{2}\bm{I}_{d}),\forall i$. We then
have

$\displaystyle(\bar{\bm{w}}(k)+\tilde{\bm{w}}(k))^{\top}\bm{x}(\bar{\Phi}+\tilde{\Phi})|_{\bm{y}=\bm{e}_{1}}$
$\displaystyle=$
$\displaystyle\sum_{i=1,3}\bm{\mu}_{v,i}(k)^{\top}\left(\bm{\mu}_{v,i}\bm{Q}_{v,i}(1)\right)+{\color[rgb]{0,0,1}4\bm{\mu}_{v,2}(k)^{\top}\left(\bm{\mu}_{v,2}\bm{Q}_{v,2}(1)\right)}$
$\displaystyle+\sum_{j=1,2,4,5}\bm{\mu}_{s,j}(k)^{\top}\left(\bm{\mu}_{s,j}\bm{Q}_{s,j}(1)\right)+{\color[rgb]{0,0,1}{4\bm{\mu}_{s,3}(k)^{\top}\left(\bm{\mu}_{s,3}\bm{Q}_{s,3}(1)\right)}}+\xi$
(5)

As for model ensemble, we have

$\displaystyle\bar{\bm{w}}(k)^{\top}\bm{x}\bar{\Phi}+\tilde{\bm{w}}(k)^{\top}\bm{x}\tilde{\Phi}$
$\displaystyle=$
$\displaystyle\sum_{i=1,2}\bm{\mu}_{v,i}(k)^{\top}\left(\bm{\mu}_{v,i}\bm{Q}_{v,i}(1)\right)+\sum_{j=1,2,3}\bm{\mu}_{s,j}(k)^{\top}\left(\bm{\mu}_{s,j}\bm{Q}_{s,j}(1)\right)$
$\displaystyle+\sum_{i=2,3}\bm{\mu}_{v,i}(k)^{\top}\left(\bm{\mu}_{v,i}\bm{Q}_{v,i}(1)\right)+\sum_{j=3,4,5}\bm{\mu}_{s,j}(k)^{\top}\left(\bm{\mu}_{s,j}\bm{Q}_{s,j}(1)\right)$
(6) $\displaystyle=$
$\displaystyle\sum_{i=1,3}\bm{\mu}_{v,i}(k)^{\top}\left(\bm{\mu}_{v,i}\bm{Q}_{v,i}(1)\right)+{\color[rgb]{0,0,1}2\bm{\mu}_{v,2}(k)^{\top}\left(\bm{\mu}_{v,2}\bm{Q}_{v,2}(1)\right)}$
$\displaystyle+\sum_{j=1,2,4,5}\bm{\mu}_{s,j}(k)^{\top}\left(\bm{\mu}_{s,j}\bm{Q}_{s,j}(1)\right)+{\color[rgb]{0,0,1}{2\bm{\mu}_{s,3}(k)^{\top}\left(\bm{\mu}_{s,3}\bm{Q}_{s,3}(1)\right)}}+\xi^{\prime}$
(7)

Comparing equation D.7.1 and equation D.7.1, we can see that

  * •

For model averaging, the overlapped features $\bm{\mu}_{v,2}$ and
$\bm{\mu}_{s,3}$ (corresponding to $\bm{x}_{v,2}$ and $\bm{x}_{s,3}$) have
coefficients amplified by 2 in $\bar{\Phi}+\tilde{\Phi}$, and further
amplified twice in $\bar{\bm{w}}+\tilde{\bm{w}}$. This results in coefficients
of the overlapped feature becoming 4 in
$(\bar{\bm{w}}+\tilde{\bm{w}})^{\top}\bm{x}(\bar{\Phi}+\tilde{\Phi}$.

  * •

For model ensemble, i.e.,
$\bar{\bm{w}}^{\top}\bm{x}\bar{\Phi}+\tilde{\bm{w}}^{\top}\tilde{\bm{x}}\Phi$,
the coefficients of the overlapped feature are 2.

####  D.7.2 The theoretical condition of WSE outperforming OSE

Recall Proposition 2 that we have

$\displaystyle\mathcal{A}_{\mbox{ood}}(f_{\mbox{wse}})=F_{p}\left(\frac{(1-p)(\tilde{n}_{s}+\bar{n}_{s}+2n_{so})+(\tilde{n}_{v}+\bar{n}_{v}+2n_{vo})}{\sqrt{\tilde{n}_{s}+\bar{n}_{s}+14n_{so}}}\right),$
$\displaystyle\mathcal{A}_{\mbox{ood}}(f_{\mbox{ose}})=F_{p}\left(\frac{(1-p)(\tilde{n}_{s}+\bar{n}_{s})+(\tilde{n}_{v}+\bar{n}_{v})}{\sqrt{\tilde{n}_{s}+\bar{n}_{s}+2n_{so}}}\right).$

A direct consequence of Proposition 2 is as follows, which illustrates when
model averaging can be more effective than model ensemble:

######  Proposition 7.

Consider the models in Definition 2, suppose Assumption 1 and 2 hold, there
are infinite ID and OOD samples. Suppose the number of features that
$\bar{\Phi}$ and $\tilde{\Phi}$ learn are the same, i.e.,
$\bar{n}_{v}=\tilde{n}_{v}\doteq n_{v},\quad\bar{n}_{s}=\tilde{n}_{s}\doteq
n_{s}$ and denote $\rho_{s}\doteq n_{so}/n_{s},\rho_{v}\doteq n_{vo}/n_{v}$.
Omitting small constants involving $\epsilon$, we have
$\mathcal{A}_{\mbox{ood}}(f_{\mbox{wse}})>\mathcal{A}_{\mbox{ood}}(f_{\mbox{ose}})$
when $\frac{\rho_{v}}{\rho_{s}}>\frac{3(1-p)n_{s}}{n_{v}},$ and
$\mathcal{A}_{\mbox{ood}}(f_{\mbox{wse}})\leq\mathcal{A}_{\mbox{ood}}(f_{\mbox{ose}})$
when $\frac{\rho_{v}}{\rho_{s}}\leq\frac{3(1-p)n_{s}}{n_{v}}.$

As shown in Appendix D.7.1, the coefficient of an overlapped feature in model
averaging is 4, and The coefficient of an overlapped feature in model ensemble
is 2. If more $\bar{\Phi}$ and $\tilde{\Phi}$ learns more overlapped invariant
features, the model averaging would put more weight on the invariant features,
leading to better OOD performance.

Figure 16: (a)
$\mathcal{A}_{\mbox{ood}}(f_{\mbox{wse}})-\mathcal{A}_{\mbox{ood}}(\bar{f})$
on Example 4, (b)
$\mathcal{A}_{\mbox{ood}}(f_{\mbox{wse}})-\mathcal{A}_{\mbox{ood}}(f_{\mbox{ose}})$
on Example 4,

In Figure 4 (c) and (d), we illustrate
$\mathcal{A}_{\mbox{ood}}(f_{\mbox{wse}})-\mathcal{A}_{\mbox{ood}}(\bar{f})$
and
$\mathcal{A}_{\mbox{ood}}(f_{\mbox{wse}})-\mathcal{A}_{\mbox{ood}}(f_{\mbox{ose}})$
on the following example:

######  Example 4.

Consider both models learn the same number of features, i.e., fixing
$\bar{n}_{v}=\tilde{n}_{v}=10$ and $\bar{n}_{s}=\tilde{n}_{s}=20$, vary
$n_{vo}=0,1,...,5$ and $n_{so}=0,1,...,5$.

We can see that $f_{\mbox{wse}}$ achieves larger OOD improvement over
$f_{\mbox{ose}}$ when two individual models learns more overlapped invariant
features (e.g., larger $n_{vo}$) and less overlapped spurious features (e.g.,
smaller $n_{so}$). In Appendix D.7.2, we provide conditions when
$f_{\mbox{wse}}$ outperforms $f_{\mbox{ose}}$, discuss why this can happen
easily in real-world datasets, provide some primary experimental results.

Why does it easily happen in OOD on many real-world applications? Recall that
there are totally $d_{v}$ invariant features and $d_{s}$ spurious features. It
is a common believe that spurious features are high-dimensional and invariant
features are low-dimensional, i.e., $d_{s}\gg d_{v}$ [5, 51]. Since the
spurious features are high dimensional and [3, 63] indicate that different
models can learn different (limited size) subsets of features, the overlap
ratio of spurious feature $\rho_{s}$ is relatively low. On the other hand,
there are a small number of invariant features and recent studies [52, 44, 31]
show that models always learn some invariant features for the fine-tuned task
during ERM fine-tuning regardless of the presence of spurious features, so we
conjecture that the overlapped ratio of invariant feature $\rho_{v}$ is
relatively higher.

However, we recognize that our discussion regarding the overlap ratio of
invariant spurious features being larger than spurious features is not
supported by rigorous proof, but rather it remains a conjecture. Further
research in this area is necessary to provide more conclusive evidence and
establish a solid foundation for this claim. In the next part, we will conduct
experiments to provide some initial support for this conjecture.

####  D.7.3 Empirical Verification

It is very difficult to directly empirically verified Proposition 7 because

  * •

For real-world datasets, it is hard to identify whether and how much a model
relies on invariant or spurious features. Verifying Proposition 7 needs to
estimate how much two models relies on the same feature.

  * •

For synthetic datasets, such as CMNIST [5], there is no feasible pre-trained
models available. On the other hand, weight space ensemble needs to be
conducted on models close to pre-trained models.

In this part, we design a primary experiment to get around the above
obstacles. Consider the ensemble of two models: pre-trained CLIP ($\bar{f}$)
and the CLIP fine-tuned on ImageNet ($\tilde{f}$).

First, we use ImageNet variants (ImageNet-V2, ImageNet-Sketch, ImageNet-A,
ImageNet-R, ObjectNet) for OOD performance evaluation. Recall that ImageNet
variants share the same invariant features with ImageNet. Also recent studies
[52, 44, 31] show that ERM fine-tuned models always learn some invariant
features for the fine-tuned task regardless of the presence of spurious
features. So $\tilde{f}$ learns the invariant features for ImageNet variants.
At the same time, the pre-trained CLIP $\bar{f}$ can stably perform zero-shot
classification on ImageNet and its variants, indicating that $\bar{f}$ also
learns good invariant features for ImageNet variants. According to the
previous discussion, $\bar{f}$ and $\tilde{f}$ have some overlapped invariant
features for ImageNet variants, leading to better weight space ensemble than
output space ensemble on ImageNet variants (shown in Figure 17(Left)).

We then evaluate the OSE and WSE on three other distinct datasets, i.e.,
Places365, StanfordCars, DTD and Food101 (refer as PSDF datasets). These tasks
have different label space with ImageNets, and contains different invariant
features with ImageNet. Then in this case, the model $\tilde{f}$ fine-tuned on
the ImageNet learns little invariant for PSDF datasets. So overlap invariant
features used by the pre-trained model $\bar{f}$ and fine-tuned $\tilde{f}$
are rather limited, indicating $\rho_{v}$ is close to zero. Then according to
Proposition 7, WSE would be no better than OSE. This is consistent with the
results in Figure 17(right).

Figure 17: Comparison of model ensemble and averaging. Left) OOD performance
on ImageNet variants, Right) OOD performance on PSDF (Places365, StanfordCars,
DTD and Food101).

###  D.8 Illustrating the over-confidence.

In Section 4, we use $\lambda$ to characterize the over-confidence of
$f_{\lambda}=(\lambda\bm{w},\lambda\Phi)$. Specifically, we have

$\displaystyle f_{\lambda}(\bm{x})=\lambda^{2}\bm{w}^{\top}\bm{x}\Phi.$

Denote $q:=\bm{w}^{\top}\bm{x}\Phi$, which is a 3-dimensional vector for a
3-class classification problem. Consider an example, i.e., $q=[2,1,1]$. Recall
that $q(k)$ is the $k$-th element of $q$ for $k=1,2,3.$. The predicted
probability for the first class when $\lambda=1$ is

$\displaystyle\text{Probability of class
1}=\frac{\exp(q(1))}{\exp(q(1))+\exp(q(2))\exp(q(3))}=\frac{\exp(2)}{\exp(2)+\exp(1)+\exp(1)}=0.576.$

When the predicted class of $f_{\lambda}$ would be the same for $\lambda>1$
and $\lambda=1$. Whereas, when $\lambda>1$, the predicted probability for the
largest class would be amplified, e.g., when $\lambda=\sqrt{5}$

$\displaystyle\text{Probability of class 1}=$
$\displaystyle\frac{\exp(\lambda^{2}q(1))}{\exp(\lambda^{2}q(1))+\exp(\lambda^{2}q(2))\exp(\lambda^{2}q(3))}$
$\displaystyle=$
$\displaystyle\frac{\exp(2\lambda^{2})}{\exp(2\lambda^{2})+\exp(\lambda^{2})+\exp(\lambda^{2})}$
$\displaystyle=$ $\displaystyle 0.99$

So we can see that a larger $\lambda$ won’t change the predicted class, but
would make $f_{\lambda}$ more confident.

##  Appendix E More experimental details and results on BANG

###  E.1 Details on ImageNet Variants

Figure 18: Datasets on ImageNet and its variants. For each dataset, we pick 4
samples of the class lemon and show illustrative images from each dataset. The
dataset descriptions are similar to that of [60].

Details for ImageNet variants:

  * •

ImageNet-V2(IN-V2): A recreated version of the ImageNet test set, but with a
different set of data distribution.

  * •

ImageNet-R(IN-R): Renditions of 200 ImageNet classes resulting in 30,000
images.

  * •

ImageNet Sketch(IN-S): Sketch style images of the same categories as ImageNet,
with a total of 50000 images.

  * •

ObjectNet(ON): Objects in this dataset are captured in cluttered and natural
environments at unusual poses.

  * •

ImageNet-A(IN-A): This dataset consists of naturally occurring images that are
misclassified by a ResNet-50 model for 200 ImageNet classes.

###  E.2 Details of Places365, StanfordCars, DTD and Food101 (PSDF)

  * •

Places365 ([65]): A scene recognition dataset. In this paper, we use the
validation set from the Places365-standard, which is composed of 36,000
validation images from 365 scene classes.

  * •

StanfordCars ([32]): This dataset contains 196 classes of cars. Classes are
typically at the level of Make, Model, Year, ex. 2012 Tesla Model S or 2012
BMW M3 coupe. In this paper, we evaluate models on the test set, comprising
8,041 images.

  * •

Describable Textures Dataset (DTD) ([17]): DTD is a texture database,
organized according to a list of 47 categories inspired from human perception
such as banded, dotted and gauzy. In the paper, we use the test set with 40
images per class.

  * •

Food101 ([9]): This dataset consists of 101 food categories. In the paper, we
use the test set with 250 test images for each class.

###  E.3 Details on calculating the confidence

Consider a $K$-class classification problem. Denote the $l$th element of the
output as $\mbox{Prob}_{l}$, indicating the probability the model assigns to
the $l$th class. We have $\sum_{l=1}^{K}\mbox{Prob}_{l}=1$. The confidence is
defined as as:

$\mbox{Confidence}=\max\left(\\{\mbox{Prob}_{l}\\}_{l=1}^{K}\right).$

###  E.4 Experimental Details

We use the CLIP model ViT-B/16[45]. We fine-tune the pre-trained model on
ImageNet. We use the AdamW optimizer with the default PyTorch AdamW
hyperparameters and choose 512 as batch size. We use a learning rate of
$3\times 10^{-5}$, gradient clipping at global norm 1 and fine-tune for a
total of 10 epochs. The settings mentioned above are the same with [60]. For
our method BANG, we try four smoothing for LS (label smoothing): 0.05, 0.10,
0.15 and 0.20. We adopt 0.10 in our reported results in Table 2. Further
results in Table 7 show that BANG is relatively insensitive to the hyper-
parameter. We do not tune the hyper-parameters of Mixup [62]. We use the
default hyperparamter as MMPreTrain[18].

(a) The vanilla fine-tuned model

(b) The vanilla fine-tuned model calibrated on in the ID dataset [33].

(c) The model fine-tuned with label smoothing

(d) The model fine-tuned with Mixup

(e) The model fine-tuned with both Mixup and LS

Figure 19:  Comparison of confidence and accuracy between zero-shot and the
model finetuned with different methods. In the figure, the ID dataset is the
ImageNet dataset, which is represented by $\triangledown$. the five OOD
datasets are: $\circ$ for ImageNetA, $\square$ for ImageNetR, $+$ for
ImageNetSketch, $\diamondsuit$ for ImageNetV2 and $\times$ for ObjectNet.

###  E.5 More Results on BANG and Discussions

Mixup and label smoothing can alleviate the over-confidence of the fine-tuned
model Compare Figure 19(c), 19(d), 19(e) with Figure 19(a), we can see that
imposing Mixup and LS during fine-tuning can alleviate the over-confidence of
the fine-tuned model on both ID (ImageNet, denoted by $\triangledown$ in the
figure) and OOD datasets, which is consistent with existing results [41].

Comparison with Calibrated Ensemble [33]. [33] calibrates the fine-tuned model
on the ID dataset by searching for a temperature $T$ of the softmax. Figure
19(b) shows that the confidence of the calibrated fine-tuned model
approximately equals its accuracy on the ID dataset (ImageNet). However, such
model is still highly over-confidence in OOD datasets, e.g., the confidence is
over 0.6 while the accuracy is lower than 0.4 on ImageNetA (denoted by
$\circ$), which is consistent with the findings in [39] and also the
discussions in the Section 4.2 of [39]. So the scaling issue shown in
Proposition 4 still exists in OOD datasets. Notably, Calibrated Ensemble
itself [33] can not be directly applied on model averaging: Model averaging
merges the parameters of each layer. However, calibrated Ensemble only tunes
the temperature of the softmax, which does not affect the lower layers,
indicating that the layers other than the output layer can still suffer from
scaling issues.

Comparison between Mixup with other data augmentations We also compare Mixup
with other data augmentations. We fine-tune the CLIP on ImageNet with flip,
rotate, and color augmentation, respectively. We then performance weight
averaging on these fine-tuned model with the pre-trained model as [60] does.
Table 6 shows that flip, rotate, and color augmentation can not enhance the
performance of model averaging. Figure 20 also shows that these augmentation
methods can not alleviate the over-confidence of the fine-tuned model.

BANG is relatively insensitive to hyper-parameters. Table 7 shows the
performance of BANG with different hyper-parameters of label smoothing. BANG
is relatively insensitive to such hyper-parameters, e.g., the average OOD
performance of BANG(Mixup+LS) all remains at about 64.9% for the four hyper-
parameters.

Methods | Model Averaging | IN | IN-V2 | IN-R | IN-A | IN-S | ObjectNet | Avg OOD
---|---|---|---|---|---|---|---|---
Zero-shot [60] | No | 68.3 | 61.9 | 77.6 | 49.8 | 48.2 | 53.0 | 58.1
Fine-tuning [60] | No | 81.3 | 70.9 | 65.6 | 36.7 | 46.3 | 49.6 | 53.8
Flip | No | 81.3 | 70.5 | 63.1 | 36.8 | 44.6 | 51.4 | 53.3
Rotate | No | 81.4 | 70.7 | 65.2 | 35.6 | 45.3 | 49.5 | 53.3
Color | No | 81.4 | 71.5 | 65.3 | 37.3 | 46.7 | 50.4 | 54.2
Mixup | No | 83.0 | 72.7 | 66.4 | 43.7 | 48.8 | 52.4 | 56.8
Flip | Yes | 81.8 | 72.7 | 78.2 | 52.9 | 53.6 | 58.4 | 63.1
Rotate | Yes | 81.7 | 72.8 | 78.8 | 52.7 | 53.7 | 57.3 | 63.1
Color | Yes | 81.7 | 72.9 | 78.5 | 53.2 | 54.2 | 58.2 | 63.4
Mixup | Yes | 81.5 | 73.0 | 79.5 | 57.9 | 54.5 | 58.7 | 64.7

Table 6: Results of fine-tuning CLIP VIT-B/16 with flip, color, and rotation
data augmentation on ImageNet.

(a) With rotate augmentation

(b) With flip augmentation

(c) With color augmentation

Figure 20:  Comparison of confidence and accuracy between zero-shot and the
model finetuned with different data augmentation. In the figure,
$\triangledown$ refers to ImageNet dataset, $\circ$ for ImageNetA, $\square$
for ImageNetR, $+$ for ImageNetSketch, $\diamondsuit$ for ImageNetV2 and
$\times$ for ObjectNet.

Methods | Model Averaging | IN(ImageNet) | IN-V2 | IN-R | IN-A | IN-Sketch | ObjectNet | Avg OOD
---|---|---|---|---|---|---|---|---
Zero-shot [60] | No | 68.3 | 61.9 | 77.6 | 49.8 | 48.2 | 53.0 | 58.1
Fine-tuning [60] | No | 81.3 | 70.9 | 65.6 | 36.7 | 46.3 | 49.6 | 53.8
Fine-tuning(LS(0.05)) | No | 82.0 | 71.5 | 62.8 | 37.7 | 45.5 | 50.6 | 53.6
Fine-tuning(LS(0.10)) | No | 82.0 | 72.3 | 63.3 | 38.3 | 46.5 | 51.1 | 54.3
Fine-tuning(LS(0.15)) | No | 82.1 | 72.1 | 63.3 | 38.0 | 46.6 | 50.7 | 54.1
Fine-tuning(LS(0.20)) | No | 82.1 | 72.1 | 62.8 | 36.9 | 46.2 | 50.5 | 53.7
Fine-tuning(Mixup) | No | 83.0 | 72.7 | 66.4 | 43.7 | 48.8 | 52.4 | 56.8
Fine-tuning(Mixup + LS(0.05)) | No | 83.0 | 73.2 | 65.9 | 43.9 | 48.5 | 52.3 | 56.7
Fine-tuning(Mixup + LS(0.10)) | No | 82.7 | 73.0 | 66.4 | 43.3 | 48.6 | 52.4 | 56.8
Fine-tuning(Mixup + LS(0.15)) | No | 82.9 | 72.7 | 65.8 | 43.6 | 48.5 | 52.2 | 56.6
Fine-tuning(Mixup + LS(0.20)) | No | 82.9 | 73.2 | 66.4 | 44.6 | 48.5 | 52.4 | 57.0
WiSE-FT [60] | Yes | 81.7 | 72.8 | 78.7 | 52.2 | 53.9 | 57.3 | 63.0
BANG(LS(0.05)) | Yes | 82.2 | 73.0 | 78.1 | 54.7 | 53.8 | 58.3 | 63.6
BANG(LS(0.10)) | Yes | 82.1 | 73.3 | 78.2 | 55.2 | 53.7 | 58.9 | 63.9
BANG(LS(0.15)) | Yes | 82.0 | 73.2 | 78.1 | 55.0 | 53.4 | 58.9 | 63.7
BANG(LS(0.20)) | Yes | 81.7 | 73.1 | 77.9 | 54.2 | 53.6 | 58.6 | 63.4
BANG(Mixup) | Yes | 81.5 | 73.0 | 79.5 | 57.9 | 54.5 | 58.7 | 64.7
BANG(Mixup + LS(0.05)) | Yes | 81.6 | 73.1 | 79.7 | 58.2 | 54.8 | 58.9 | 64.9
BANG(Mixup + LS(0.10)) | Yes | 81.5 | 73.0 | 79.8 | 57.9 | 54.8 | 59.0 | 64.9
BANG(Mixup + LS(0.15)) | Yes | 81.7 | 72.9 | 79.6 | 57.7 | 54.6 | 59.1 | 64.8
BANG(Mixup + LS(0.20)) | Yes | 81.6 | 73.1 | 79.9 | 57.8 | 54.8 | 59.0 | 64.9

Table 7: Results of BANG with CLIP-B/16. We show different hyper-parameters of
label smoothing. Mixup use the default hyper-parameter of MMPreTrain[18].

###  E.6 WiSE-FT benefits significantly from better calibration

In Section 4, we theoretically show that model WSE can suffer from the
imbalance issue where two individual models have different scaling. This can
happen if one model is much more confident than the other. Unfortunately, we
observe that the popular method, WiSE-FT suffers from this issue.
Specifically, WiSE-FT averages the pre-trained model with the fine-tuned
model. In Section 4, we show that the fine-tuned model is high-overconfident
compared with the pre-trained model. We propose BANG, which averages the pre-
trained model with the model fine-tuned with Label Smoothing (LS) or MixUp.
Since LS and MixUp can also improve the fine-tuned performance, we conduct the
following experiment to isolate the effect of better calibration from better
fine-tuned performance.

Scale the fine-tuned model during weight space ensemble. A straightforward
method to alleviate over-confidence is to tune the temperature of the softmax
of the fine-tuned model [33]. However, this method can not be directly applied
to WiSE-FT since WiSE-FT averages model weights instead of ensemble the
outputs. We first apply a direct adaptation of [33] to WiSE-FT: divide the
weights in the last layer $\bm{w}$ by a scalar (temperature), which is
equivalent to softmax tempering. However, recall the model averaging
$(\bar{\bm{w}}+\tilde{\bm{w}})^{\top}\bm{x}(\bar{\Phi}+\tilde{\Phi})$ also
suffer from the imbalance issue of $\Phi$. Specifically, Proposition 4 shows
that the averaged feature can be biased towards $\tilde{\Phi}$ if the scaling
of $\tilde{\Phi}$ is larger than $\bar{\Phi}$. So merely adjusting the weight
of the classifier $\bm{w}$ cannot alleviate this bias. The experiment result
(Exp 1) in Table 8 also shows that merely re-scaling the classifier can hardly
improve WiSE-FT. In practice, we use a transformer (VIT-B/16) with 12 block
layers and 1 linear layer. Denoting the parameter of the 12 blocks as $\theta$
and the parameter of the linear layer as $\bm{w}$, we conduct three
experiments during the averaging process for Wise-FT (introducing $\alpha\leq
0.5$). We obtain the averaged model $(\hat{\theta},\hat{w})$ as follows

  * •

(Exp 1) Re-scale the classifier of FM (fine-tuned, $\tilde{\theta}$,
$\tilde{w}$) model during averaging, i.e.,
$\hat{\theta}=0.5(\bar{\theta}+\tilde{\theta})$ and
$\hat{w}=({1-\alpha})\bar{w}+{\alpha}\tilde{w}$.

  * •

(Exp 2) Re-scale whole network of FM as,
$\hat{\theta}=({1-\alpha})\bar{\theta}+{\alpha}\tilde{\theta}$ and
$\hat{w}=({1-\alpha})\bar{w}+{\alpha}\tilde{w}$.

  * •

(Exp 3) Use two independent coefficients, $\alpha_{1}$ to re-scale the first 6
blocks (denoted as $\theta_{1}$), and $\alpha_{2}$ to re-scale last 6 blocks
(denoted as $\theta_{2}$) and the linear layer $w$, i.e.,
$\hat{\theta}_{1}=({1-\alpha_{1}})\bar{\theta}_{1}+{\alpha_{1}}\tilde{\theta}_{1}$,
$[\hat{\theta}_{2},\hat{w}]=({1-\alpha_{2}})[\bar{\theta}_{1},\bar{w}]+{\alpha_{2}}[\tilde{\theta}_{2},\tilde{w}]$.

  * •

(Exp 4) Use three independent coefficients ($\alpha_{1}$, $\alpha_{2}$,
$\alpha_{3}$) re-scale the first 6 blocks (denoted as $\theta_{1}$) , last 6
blocks (denoted as $\theta_{2}$) and the linear layer $w$, i.e.,
$\hat{\theta}_{1}=({1-\alpha_{1}})\bar{\theta}_{1}+{\alpha_{1}}\tilde{\theta}_{1}$,
$\hat{\theta}_{2}=({1-\alpha_{2}})\bar{\theta}_{1}+{\alpha_{2}}\tilde{\theta}_{2}$,
and $\hat{w}=({1-\alpha_{3}})\bar{w}+{\alpha_{3}}\tilde{w}$.

We search for the best $\alpha$ among 0.2-0.5 (with interval 0.1) for each ood
dataset. The results in Table 8 shows can merely scaling the fine-tuned model
to alleviate its over-confidence can significantly improvement the performance
of WiSE-FT.

WiSE-FT | Exp 1 | Exp 2 | Exp 3 | Exp 4 | BANG
---|---|---|---|---|---
63.0% | 63.0% | 64.1% | 64.5% | 64.8% | 64.9%
Table 8: WiSE-FT can benefit significantly from better calibration by scaling
the fine-tuned model. (Exp 1)-(Exp 4) are described in Appendix E.6.

BANG can correct more samples on which the fine-tuned model make mistakes. In
this part, We denote the pre-trained model as PM, fine-tuned model as FM, and
averaged model as AM. We divide each dataset into four groups of samples
according to whether the PM and FM make correct predictions, respectively. We
further use T/F to denote whether a model makes correct predictions, i.e., T
for True and F for False. For example, we use PM(T)-FM(F) to denote the group
of samples on which the predictions of PM and FM AM are correct and wrong,
respectively. We visualize the average margin of fine-tuned and pre-trained
models on four groups, i.e., PM(T)-FM(T), PM(T)-FM(F), PM(F)-FM(T), and
PM(F)-FM(F). The margin is the difference between the probability assigned to
the correct class and the maximum probability among the wrong classes, i.e.,

$\displaystyle\mbox{Margin}=\mbox{Prob}_{l}-\max_{k\neq l}\mbox{Prob}(k)$

where $l$ is the it the true class, $k=1,..K$, and $\mbox{Prob}(k)$ is the
probability that a assign to the class $k$. Averaging models with negative and
positive margins can potentially correct mistakes. Figure 21 (Left) and
(Right) visualize the the margins of pre-trained and fine-tuned models on each
group of each datasets for Wise-ft and BANG. In Wise-ft, the fine-tuned model
exhibits significantly negative margins in the PM(T)-FM(F) group.
Specifically, $\mbox{Margin}(\mbox{PM})+\mbox{Margin}(\mbox{FM})$ on the group
PM(T)-FM(F) is negative for WiSE-FT on Figure 21(Left), indicating dominance
of fine-tuned models in WiSE-FT even fine-tuned make mistakes. This also
explains that why in some datasets, e.g., IN-R and IN-A, ImproveContri(TF+FT)
is negative as shown in Figure 7. However, in BANG,
$\mbox{Margin}(\mbox{PM})+\mbox{Margin}(\mbox{FM})$ on the group PM(T)-FM(F)
is positive on average as shown in Figure 21(Right), suggesting that BANG is
capable of correcting more mistakes within the PM(T)-FM(F) group. Table 9
shows that ratio (versus the entire dataset) of samples where model averaging
can correct the prediction in the PM(T)-FM(F) group. Specifically, let $N_{1}$
denote the number of samples where WiSE-FT can correct the prediction in the
PM(T)-FM(F) group and $N_{2}$ denote the total sample size in the dataset,
Table 9 compares the $N_{1}/N_{2}$ (averaged over 5 OOD datasets) of WiSE-FT
and BANG. Table 9 shows BANG can correct substantially more mistakes made by
the fine-tuned model.