# ERM++: An Improved Baseline for Domain Generalization

Piotr Teterwak1, Kuniaki Saito1,Theodoros Tsiligkaridis2, Kate Saenko1,3,
Bryan A. Plummer1
1Boston University, 2MIT Lincoln Laboratory, 3Meta AI
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Multi-source Domain Generalization (DG) measures a classifier’s ability to
generalize to new distributions of data it was not trained on, given several
training domains. While several multi-source DG methods have been proposed,
they incur additional complexity during training by using domain labels.
Recent work has shown that a well-tuned Empirical Risk Minimization (ERM)
training procedure, that is simply minimizing the empirical risk on the source
domains, can outperform most existing DG methods. We identify several key
candidate techniques to further improve ERM performance, such as better
utilization of training data, model parameter selection, and weight-space
regularization. We call the resulting method ERM++, and show it significantly
improves the performance of DG on five multi-source datasets by over 5%
compared to standard ERM, and beats state-of-the-art despite being less
computationally expensive. Additionally, we demonstrate the efficacy of ERM++
on the WILDS-FMOW dataset, a challenging DG benchmark. We hope that ERM++
becomes a strong baseline for future DG research. Code is released at
https://github.com/piotr-teterwak/erm_plusplus.

Figure 1: ERM++: We tackle the task of Multi-Source Domain Generalization,
where a model is trained on several source domains and evaluated on a
different target domain. We do this by improving the classic, and already
strong, ERM [20] algorithm with known methodologies. We verify our method on a
diverse set of domain shifts, and show that it improves over the best reported
numbers in the literature.

##  1 Introduction

Domain Generalization (DG) is a crucial problem in the field of machine
learning, as it addresses the challenge of building models that perform well
on unseen (target) data distributions, without using target data to update the
model [7, 39, 64]. This is important in many real-world applications, where
the distribution of data may vary between settings, and it is not always
feasible to collect and label a large amount of data for each new domain.
Similarly, it is not always known a-priori how the distribution on which the
model is deployed differs from the training distribution. In multi-source
domain generalization, each training sample is labelled as being part of one
of several domains. Many advanced methods leverage domain membership
explicitly. For example, DANN [18] uses an adversarial loss to match feature
distributions across source domains. Adaptive Risk Minimization [62] meta-
learns parameters which adapt a model to newly seen distribution shift. Yet,
recently DomainBed [20] holistically evaluated methods and found that ERM
(Empirical Risk Minimization), outperforms most prior work for DG in a setting
where hyper-parameters are tuned. This is all the more impressive since ERM
only leverages domain labels in a very weak way; by oversampling minority
domains to balance domain sizes in the training data. Advanced techniques do
not beat ERM [20] despite strong inductive biases and additional complexities
(and hyper-parameters to tune).

In this paper, our goal is to revisit the framework used to benchmark
multisource domain generalization problems to ensure that we maximize the
performance of baseline methods. As illustrated in Figure 1, our new baseline,
ERM++, is able to outperform the state-of-the-art without the need for domain
labels, architecture changes or complex training strategies. Instead, we
critically evaluate the components of the training pipeline along three major
themes. First, we explore how the training data is being used, including
training length and checkpoint selection. Second, we consider how we
initialize network parameters such as the selection of pretraining network and
whether or not to fine-tune or freeze layers. Third, we investigate weight-
space regularization methods that are often used to help avoid overfitting to
the training data.

Revisiting and improving baselines to address shortcomings or incorporating
new training techniques can provide new insights into the state of the
research on the topic. For example, SimSiam [12] showed that a simple siamese
network can perform competitively on self-supervised learning by incorporating
a stop-gradient function. Beyer  _et al_. [5] show that a few simple
techniques, such as longer training or increased augmentation strength,
outperform all prior work in knowledge distillation. Wightman  _et al_. [55]
show that techniques such as longer training can substantially improve
ImageNet [14] performance. These works helped provide new insights into their
respective tasks, as we aim to do in our work for domain generalization.

Through a careful evaluation of the training framework used to compare DG
methods across six diverse domains, we are able to make several interesting
observations. For example, we find that improved performance on ImageNet [14]
does not necessitate a gain in generalization ability. We also find that many
of the hyperparameters such as training time used by many methods (such as
DomainBed [20]) result in evaluating models before they have converged. To
address this, we utilized an adaptive training procedure that would
automatically determine the sufficient training length for a model to obtain
the best performance. Compared to the state-of-the-art DG methods such as MIRO
[10] and DIWA [42], our approach is able to obtain a 1% gain across five
datasets used by DomainBed [20], while also reducing the required training
compute by 50% compared with MIRO and 95% compared to DIWA due to reduced need
for hyperparameter tuning. Critically, although we also show that using the
techniques we identified boosts MIRO and DIWA’s performance, the improved DIWA
is unable to outperform ERM++. This helps highlight the need for our work.

Algorithm 1 ERM++: Components of ERM++ are annotated in the algorithm
comments. We run this training loop in two passes, the first to set training
length by using a validation set split from the source domains. In the second
pass we train on the combination of train and val data.

$ModelWeights\leftarrow AugmixWeights$ $\triangleright$ Strong init

while $steps\neq LongTrainSteps$ do $\triangleright$ Long Train

$X,Y\leftarrow$ next(FullDataIterator) $\triangleright$ Full Data

if $steps\leq 500$ then

Update linear classifier $\triangleright$ Warmstart

else

Update linear classifier and backbone

end if

if $steps\geq 600$ then

$ModelWeightsAvg\leftarrow$ Update $\triangleright$ Weight Reg.

$ModelWeightsAvgBN\leftarrow$ BN stats $\triangleright$ UBN

end if

end while

##  2 Related Works

In this work, we focus on improving ERM [20] for DG, however here we review
existing methods for multi-source DG for completeness.
Domain-invariant feature learning: In multi-source domain generalization, it
is common to leverage the domain labels to learn domain-invariant features.
CORAL [46] aligns second-order statistics of different domains. DANN [18] uses
an adversarial loss to match feature distributions across source domains.
However, these approaches need to acquire a domain label for each sample,
which is sometimes expensive to annotate. Furthermore, using domain knowledge
to learn domain-invariant features can learn to ignore signals which can be
important for new domains, as evidenced by the strong performance of ERM [20].
In fact, Vedantam et al. [50] find low correlation between low source-target
discrepancy and good DG performance.
Domain-Aware Data Augmentation: Data augmentation is a common tool to
diversify training data and expand the training domain [64, 22, 63, 60]. For
example, Inter-domain mixup [60] blends the images of different domains, and
augmentation with style transfer can further diversify training images [63],
though it is expensive. Instead of relying on data augmentation techniques
during training on sources, we propose to employ all training samples from the
source, including validation data, which expands knowledge about the task. We
also propose to use backbones pretrained with strong domain-agnostic
augmentation such as Augmix [22], which mixes different synthetic
augmentations in pixel space.
Ensembling: Deep ensembles are effective for domain generalization [3, 17].
However, they are computationally inefficient, needing to run inference
through many models. It has been recently shown that averaging model weights
can approximate an ensemble. This can either be from multiple fine-tuning runs
[56, 42] or from different points within a single training trajectory [57, 9,
24]. We choose to leverage the ensembles from a single training trajectory for
its effectiveness, and do not find further improvement from averaging from
multiple trajectories.
Preventing Catastrophic Forgetting: Several recent approaches aim to leverage
generalizable features from a model pre-trained on large-scale data. Adapting
such a model to the downstream task without forgetting its generalizable
representations is the key to achieving generalization. Wortsman et al. [57]
interpolate between the pre-trained and adapted model. Kumar et al. [30, 61]
mitigate feature distortion by pre-training a linear probe first before fine-
tuning the backbone, warmstarting the fine-tuning with a good initialization.
MIRO [10] maximizes the mutual information in feature space between the fine-
tuned and pre-trained networks. Our approach utilizes warmstart and confirms
its effectiveness in diverse settings.

Our approach does not use explicit domain labels, as in the domain invariant
methods presented above. Instead, we start with ERM [20], aka just training on
the source, and build on top of it with general methods.

##  3 Revisiting training procedures to create ERM++ for Domain Generalization

We study the problem of Multi-Source Domain Generalization for classification.
We train a model on training data consisting of multiple domains and evaluate
it on data from unseen domains. More formally, let us consider training
domains $d\in\\{d_{1},...,d_{n}\\}$. A training dataset is constructed using
all sample, label pairs in all training domains
$D=\\{(X^{d_{1}},Y^{d_{1}})...(X^{d_{n}},Y^{d_{n}})\\}$. After training
classifier $f$ on $D$, it is tested on a held-out testing domain $d_{test}$.
As stated in previous sections, approaches utilizing invariance of the domain
or regularization of features can complicate the training. Instead we perform
simple empirical risk minimization (ERM), formalized as minimizing the average
loss over all samples $\frac{1}{n}\sum_{i\in D}\ell(x_{i},y_{i})$, and shown
to be successful on diverse tasks [42].

Our goal is to investigate the general training components that go into
creating an ERM model to help ensure we have maximized its performance. These
components include how to effectively use the source data (Section 3.1),
considerations when selecting and using pretrained weights (Section 3.2), and
weight-space regularization methods that help prevent overfitting to the
source domains (Section 3.3). We refer to our new stronger baseline as ERM++,
and we summarize the procedure we found to be most effective in Algorithm 1.
As our experiments will show, our training procedures and can also be used to
improve the performance of any DG method.

###  3.1 Improved Data Utilization

A key component of training any neural network is utilizing the (often
limited) training data effectively. A common practice in the domain
generalization literature is to split source datasets into (often 80%/20%)
train/validation sets under a fixed number of iterations for each dataset
(_e.g_., [20, 9, 42, 3]). The validation data is used to set hyperparameters
and perform checkpoint (no. training steps) selection. This approach has two
major drawbacks. First, by creating a separate validation set we are
sacrificing a significant portion of our labeled data. Second, by training
under a fixed (relatively small) number of iterations we ignore the varying
convergence rates of different models, which may result in a model
underperforming its true ability.

Inspired by the training procedures in metric learning literature (_e.g_.,
[37, 38, 47, 48, 52]), we reformulate the training pipeline to take into
account method convergence rates and to utilize the entire dataset when
training the final model for deployment. Specifically, we explore a two-stage
training procedure, where in the first stage we use the same train/validation
splits as in prior work, and in the second stage we train our model for
deployment using the entire (train+validation) dataset.

To accomplish this, when setting hyperparameters in the first stage, we
include a new parameter $\phi$ that sets the training length (Early Stopping).
Once we have set all the hyperparameters (including $\phi$), we train our
deployment model using the full dataset as noted earlier, selecting the final
checkpoint as our model. More concretely, we continue training until we no
longer observe significant performance gains, which we refer to as Long
Training (LT). Note that this needs only be performed once per model. In this
uses training labels more efficiently by training on the Full-Dataset (FD).

###  3.2 Leveraging Pretrained Model Weights

Most domain generalization methods do not train a model from scratch, but
rather transfer the weights of an existing model, typically pretrained on
ImageNet [14]. There are three main decisions that we explore further:
selecting what model weights to transfer (Section 3.2.1), determining what
weights to fine-tune or keep frozen (Section 3.2.2), and how to initialize any
new weights (_e.g_., to recognize the categories of your dataset) in your
network (Section 3.2.3).

####  3.2.1 Model Weight Selection

Recent work has shown that better ImageNet models have better domain
generalization properties for both single-source and multi-source DG [26, 1].
However, this has been explored in the context of varying model size.
Therefore, performance gains can be either from a.) improved pre-training
dataset (upstream) performance resulting in improved DG or b.) larger models
resulting in improved DG performance, regardless of upstream performance.
These also disregard the needs of some applications, such as computational
requirements (larger models necessitate more resources) or restrictions on
architectures due to a shared encoder for a multitask problem. Thus, we
explore the effect of different initializations for the same model
architecture, specifically a ResNet-50 [21]. We describe them in more detail
below:

  * •

TorchVision Model Weights: This is the standard ImageNet pretrained
initialization present in TorchVision. It was trained with weak augmentations
for 90 epochs.

  * •

AugMix trained network: AugMix [22] is a method used to improve model
consistency using augmentations without training the model on data which is
too different from the test data. AugMix takes two augmented views of an image
and mixes them in pixel space. Then the model is trained to produce consistent
output between two AugMix augmentations and the clean image. Furthermore,
split Batchnorm is used as introduced in [58], i.e. learn separate Batchnorm
for clean and augmented images. The model is trained for 200 epochs.

  * •

ResNet A1: ResNet A1 initializes weights from the training recipe presented in
[55]. The model is heavily tuned to find training settings which result in
very strong ImageNet performance. Examples include training for 600 epochs,
the LAMB optimizer, strong augmentations, and a binary cross-entropy.

  * •

Meal V2 : MealV2 [45] is a highly performant ensemble, distilled into a
ResNet-50. In particular, a SeNet-154 [23] (81.23% ImageNet Top-1) and
ResNet-152 (81.02% ImageNet Top-1) are distilled into ResNet-50.

Each of these models has different ImageNet validation accuracies, ranging
from 76.13% (TorchVision weights) to 80.7% (Meal-V2 [45]). However, as our
experiments will show, simply swapping out the standard initialization for the
strongest ImageNet model does not result in the best performance. We
empirically find the strongest of these, Augmix [22], and refer to it as
Strong init.

####  3.2.2 Finetuning or Freezing Model Weights

It has been shown that what parameters to update during fine-tuning a pre-
trained model, and when, can have substantial effects on downstream
performance. Surgical fine-tuning [31] shows that only updating some blocks
results in improved performance, but that different datasets require the
unfreezing of different blocks, making it unsuitable for a general DG training
procedure (as is our goal). Most domain generalization methods will fine-tune
most layer weights, with the exception of BatchNorm parameters, which are
sometimes kept frozen. In our experiments we compare the effect freezing or
finetuning the BatchNorm parameters has on performance, and refer to
unfreezing them (UBN). The remaining layer weights we finetune as in prior
work.

####  3.2.3 Initializing New Layer Weights

Initializing new layer weights is typically accomplished by giving the new
layer weights a random initialization and then training them on the target
datasets. However, a recurring observation made by many researchers over the
years is that your model may suffer from catastrophic forgetting of the pre-
trained features due to the noisy gradients from the newly initialized layer
[19, 21, 30, 42]. To address this, researchers would begin training by
Warmstart (WS) [25, 61] (also commonly referred to as warmup), where the new
layer weights are trained with all pretrained weights kept frozen for a few
epochs. After this short training cycle, new and old layer weights are
finetuned together (sometimes except for BatchNorm layers, as discussed in
Section 3.2.2).

###  3.3 Weight-Space Regularization

Averaging model parameter iterates has a long history within machine learning
[3, 9, 24, 43, 56, 57, 42, 35], and improves generalization by converging to
flatter minima [24]. Methods can roughly be divided into those which average
within a single trajectory [3, 24, 9], or those between different trajectories
originating from a single parent [35, 56, 42]. Because the different model
parameters averaged can be interpreted as independent members of an ensemble
[56], most prior work takes care to ensure the averaged models are
sufficiently diverse and each member has strong performance. This is
accomplished by cyclic learning rates [24] or searching ranges over which to
average [9]. Most recently, Arpit et al. [3] revisit a simple method for
parameter averaging where simply all iterates are averaged(MPA). We verify
that this works in combination with other techniques present in ERM++. In a
departure from most of the other improvements explored (wrt using domain
labels), we also experiment with training domain experts to induce model
diversity(SMPA), but find that this does not result in improved performance
over within-trajectory averaging.

###  3.4 ERM++ Computational Cost

ERM++ induces less training cost overhead compared to competing methods. ERM
[20], DIWA [42], and MIRO [10] all use expensive hyper-parameter searches,
while we simply use reasonable default ones. For example, MIRO [10] searches
over 4 $\lambda$ regularization weight parameters to obtain SOTA results, and
DIWA [42] averages 20-60 independent runs. While ERM++ does use longer
training for improved performance, it exceeds SOTA even with standard training
lengths (see Table 3, Experiment 8). Overall, without long training, ERM++
achieves SOTA accuracy with 50% of the training compute of MIRO and 5% of the
compute of DIWA [42], while retaining the same inference overhead. Long
training increases this cost by 4x, which could be reasonable to pay for
improved performance.

##  4 Experimental Settings

| OfficeHome | PACS | DomainNet | TerraIncognita | VLCS | Avg.
---|---|---|---|---|---|---
MMD [34] | 66.3$\pm 0.1$ | 84.7$\pm 0.5$ | 23.4$\pm 9.5$ | 42.2$\pm 1.6$ | 77.5$\pm 0.9$ | 58.8
Mixstyle [64] | 60.4$\pm 0.3$ | 85.2$\pm 0.3$ | 34.0$\pm 0.1$ | 44.0$\pm 0.7$ | 77.9$\pm 0.5$ | 60.3
GroupDRO [44] | 66.0$\pm 0.7$ | 84.4$\pm 0.8$ | 33.3$\pm 0.2$ | 43.2$\pm 1.1$ | 76.7$\pm 0.6$ | 60.7
IRM [2] | 64.3$\pm 2.2$ | 83.5$\pm 0.8$ | 33.9$\pm 2.8$ | 47.6$\pm 0.8$ | 78.5$\pm 0.5$ | 61.6
CDANN [36] | 65.8$\pm 1.3$ | 82.6$\pm 0.9$ | 38.3$\pm 0.3$ | 45.8$\pm 1.6$ | 77.5$\pm 0.1$ | 62.0
DANN [18] | 65.9$\pm 0.6$ | 83.6$\pm 0.4$ | 38.3$\pm 0.1$ | 46.7$\pm 0.5$ | 78.6$\pm 0.4$ | 62.6
MTL [6] | 66.4$\pm 0.5$ | 84.6$\pm 0.5$ | 40.6$\pm 0.1$ | 45.6$\pm 1.2$ | 77.2$\pm 0.4$ | 62.9
Mixup [59, 60, 53] | 68.1$\pm 0.3$ | 84.6$\pm 0.6$ | 39.2$\pm 0.1$ | 47.9$\pm 0.8$ | 77.4$\pm 0.6$ | 63.4
MLDG [32] | 66.8$\pm 0.6$ | 84.9$\pm 1.0$ | 41.2$\pm 0.1$ | 47.7$\pm 0.9$ | 77.2$\pm 0.4$ | 63.6
ERM [49] | 67.6$\pm 0.2$ | 84.2$\pm 0.1$ | 44.0$\pm 0.1$ | 47.8$\pm 0.6$ | 77.3$\pm 0.1$ | 64.2
SagNet [40] | 68.1$\pm 0.1$ | 86.3$\pm 0.2$ | 40.3$\pm 0.1$ | 48.6$\pm 1.0$ | 77.8$\pm 0.5$ | 64.2
SelfReg [27] | 67.9$\pm 0.7$ | 85.6$\pm 0.4$ | 42.8$\pm 0.0$ | 47.0$\pm 0.3$ | 77.8$\pm 0.9$ | 64.2
CORAL [46] | 68.7$\pm 0.3$ | 86.2$\pm 0.3$ | 41.5$\pm 0.1$ | 47.6$\pm 1.0$ | 78.8$\pm 0.6$ | 64.5
mDSDI [8] | 69.2$\pm 0.4$ | 86.2$\pm 0.2$ | 42.8$\pm 0.1$ | 48.1$\pm 1.4$ | 79.0$\pm 0.3$ | 65.1
ERM + MIRO [10] | 70.5$\pm 0.4$ | 85.4$\pm 0.4$ | 44.3$\pm 0.2$ | 50.4$\pm 1.1$ | 79.0$\pm 0.0$ | 65.9
ERM + SWAD [9] | 70.6$\pm 0.2$ | 88.1$\pm 0.1$ | 46.5$\pm 0.1$ | 50.0$\pm 0.3$ | 79.1$\pm 0.1$ | 66.9
CORAL + SWAD [9, 46] | 71.3$\pm 0.1$ | 88.3$\pm 0.1$ | 46.8$\pm 0.0$ | 51.0$\pm 0.1$ | 78.9$\pm 0.1$ | 67.3
DIWA [42] | 72.8 | 89.0 | 47.7 | 51.9 | 78.6 | 68.0
ERM + MIRO + SWAD [9, 10] | 72.4$\pm 0.1$ | 88.4$\pm 0.1$ | 47.0$\pm 0.0$ | 52.9$\pm 0.2$ | 79.6$\pm 0.2$ | 68.1
ERM++ (Ours) | 74.7$\pm 0.0$ | 89.8$\pm 0.3$ | 50.8$\pm 0.0$ | 51.2$\pm 0.3$ | 78.0$\pm 0.1$ | 68.9
Table 1: Comparison to recent methods: Performance of recent methods as
reported by [10]. ERM outperforms almost all prior work, especially when
combined with techniques such as SWAD and MIRO. ERM++ outperforms all prior
work on average. DIWA does not report confidence intervals.

We benchmark our methods on a diverse set of classification datasets used for
evaluating multi-source DG:
OfficeHome [51] is a 65-way classification problem depicting everyday objects
from 4 domains: art, clipart, product, and real, with a total of 15,588
samples.
DomainNet [41] is 345-way object classification problem from 6 domains:
clipart, infograph, painting, quickdraw, real, and sketch. With a total of
586,575 samples, it is larger than most of the other evaluated datasets in
both samples and classes.
PACS [33] is a 7-way object classification problem from 4 domains: art,
cartoon, photo, and sketch, with 9,991 samples. It helps verify our method in
smaller-scale settings.
VLCS [16] is a 5-way classification problem from 4 domains: Caltech101,
LabelMe, SUN09, and VOC2007. There are 10,729 samples. VLCS is a good test for
close OOD; the member datasets are all real photos. The distribution shifts
are subtle and simulate real-life scenarios well.
TerraIncognita [4] is a 10-way classification problem of animals in wildlife
cameras, where the 4 domains are different locations. There are 24,788
samples. This represents a realistic use-case where generalization is indeed
critical.
Wilds-FMOW [29, 13] is a 62-way land-use classification problem, with
satellites from 5 regions as different domains. There are 141,696 samples.
Wilds-FMOW is a realistic problem different from the above and not focused on
objects which helps validate the broad applicability of ERM++.

We follow the DomainBed training procedure and add additional components from
ERM++. In particular, we use the default hyper-parameters from DomainBed [20],
_e.g_., a batch size of 32 (per-domain), a learning rate of 5e-5, a ResNet
dropout value of 0, and a weight decay of 0. Unless we specify that the “Long
Training” component is added, we train models for 15000 steps on DomainNet
(following SWAD[9]) and 5000 steps for other datasets, which corresponds to a
variable number of epochs dependent on dataset size. If Long Training is used,
we extend training by 4x. We train on all source domains except for one,
validate the model on held-out data from the sources every 300 steps, and
evaluate on the held-out domain.

##  5 Results

Table 1 compares ERM++ to prior work, where we outperform the state-of-the-art
across five DomainBed datasets by an average of 1%. The single largest gain
was on DomainNet (3% gain), with OfficeHome and PACS obtaining still
substantial gains of 1.5-2%. Table 2 demonstrates our training procedure’s
ability to generalize, where we combine our approach with the two highest
performing methods in prior work (DIWA [42] and MIRO [10]). We find that our
approach is able to boost the performance of both methods by around 1%.
However, one key observation is that our ERM++ model does not compose well
with DIWA (_i.e_., the combination is on par with ERM++ alone). This helps
demonstrate the importance of our work, as strong training procedures as used
by ERM++ model can affect the ranking of compared methods. We provide a
detailed analysis of each component of our training procedures below.

| OH | PA | VL | DN | TI | Avg
---|---|---|---|---|---|---
MIRO + SWAD [10] | 72.4 | 88.4 | 79.6 | 47.0 | 52.9 | 68.1
DIWA [42] | 72.8 | 89.0 | 78.6 | 47.7 | 52.9 | 68.0
ERM++ | 74.7 | 89.8 | 78.0 | 50.8 | 51.2 | 68.9
DIWA [42] \+ ERM++ | 74.5 | 90.0 | 78.1 | 50.1 | 51.4 | 68.8
MIRO + ERM++ [10] | 76.3 | 88.8 | 77.9 | 50.4 | 53.4 | 69.4
Table 2: We combine ERM++ with MIRO [10] and DIWA[42] DIWA slightly degrade performance while MIRO substantially improves performance.  ERM++ Components | OfficeHome | PACS | VLCS | DomainNet | TerraInc | Avg.
---|---|---|---|---|---|---
# | MPA | FD | LT | WS | ES | S. Init | UBN | 15K | 10K | 11K | 590K | 25K
1 | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | 67.1 | 85.1 | 76.9 | 44.1 | 45.2 | 63.7
2 | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | 70.2 | 85.7 | 78.5 | 46.4 | 49.4 | 66.0
3 | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✓ | 71.5 | 87.3 | 77.4 | 46.8 | 49.8 | 66.5
4 | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✓ | 71.7 | 88.7 | 76.9 | 48.3 | 49.6 | 67.0
5 | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ | ✓ | 72.6 | 88.8 | 77.0 | 48.6 | 49.3 | 67.3
6 | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | 72.6 | 88.8 | 78.7 | 48.6 | 49.2 | 67.6
7 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 74.7 | 89.8 | 78.0 | 50.8 | 51.2 | 68.9
8 | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | ✓ | 74.6 | 87.9 | 78.6 | 49.8 | 51.1 | 68.4
9 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | 74.7 | 90.1 | 78.6 | 49.9 | 49.0 | 68.3
Table 3:  We present the overall ablation for ERM++. (1) ERM [20] baseline
with unfrozen BN. (2) MPA: Model parameter averaging, which uniformly improves
results. (3) FD: training on the full data. (4) LT: Training for 4x longer,
which ensures convergence improves performance by an additional half percent.
(5) WS: Warm-starting the classification layer especially improves OfficeHome,
but also helps minimize overfitting (Figure 2). (6) ES: Splitting off
validation data to find a training length yields substantial gains. (7)
S.Init: Initializing the initial parameters to those trained with AugMix
brings performance to state of the art. (8) Removing LT from (7) still results
in state-of-the-art performance with half of the training cost of MIRO. (9)
UBN: When we freeze the BN parameters, we see that performance substantially
degrades.

###  5.1 Data Utilization

Using the full data (FD): The most common ERM [20] implementation splits off
80% of the source domains for training, and keeps the remaining 20% for hyper-
parameter validation and checkpoint selection. By comparing Table 3 in
experiments 2 and 3, we show that training on the full data improves over
checkpoint selection on a validation set on all datasets except for VLCS.
Early Stopping (ES) below helps us recover VLCS performance.

Long training (LT): Prior work has shown that training to proper convergence
can have large impacts on transfer learning performance [11]. To explore this
setting for DG, we extended training by 4x for each dataset. In other words,
DomainNet models are trained for 60K steps while the other datasets are
trained for 20K steps. This training length is one where we observe source
validation accuracies start to saturate for most datasets (see supplementary).
We present the results in Table 3, experiment 4. We find that training for
longer, on average, increases performance by 0.5%.

Early Stopping (ES): Although the training pieces presented so far improve DG
performance on the datasets considered on average, one consistent pattern is
that VLCS performance degrades in experiments number 3 (Full-Data), 4 (Long
training). This suggests that VLCS is a dataset which is prone to overfitting.
We observe that this is true even on a validation set constructed from the
source domains. Therefore, we propose an additional step where we use 20%
validation splits in order to search for the proper number of training steps,
and then retrain using the full data. In Table 3, Experiment 6, we see this
dramatically improves performance on VLCS without affecting other datasets.

###  5.2 Pretrained Model Weight Usage

Warmstart (WS): In Table 3, we compare to training using a random
initialization for the new classification layer (Experiment 4) or by using
Warmstart (Experiment 5). We find WS provides a small but consistent boost on
average across datasets. We find this is likely due to a decrease in
overfitting to the source domains. For example, in Figure 2, we show
accuracies plotted across fine-tuning steps for models with and without warm-
start for several domains of TerraIncognita. Without Warmstart, performance
quickly plateaus and in some cases, _e.g_., location 100 and location 43,
performance even decreases. This kind of performance decrease is not benign;
it is impossible to detect without access to the test data. Therefore, for
reliable deployments of systems which generalize well, training procedures
which do not overfit to source domains are important. We verify that WS has a
regularization effect by measuring the L2 distance of the final model from
initialization (the pre-trained model) and find that the trained weights were
more than twice as far without using WS (58.1 with and 122.5 w/o).

Figure 2: Oracle test performances : We plot the top-1 accuracy on held-out
test domains of TerraIncognita as a function of fine-tuning epochs with and
without warmstart. Warmstart substantially decreases overfitting to the source
domains.  Figure 3: Unfreezing Batchnorm: Here we show the test curves of the
fine-tuning on the held-out painting domain of DomainNet. With frozen
BatchNorm, the initial training is faster but it overfits.

Unfreezing the Batchnorm (UBN): BatchNorm is commonly frozen in current DG
recipes for reasons not well justified. However, we find that frozen batch
normalization leads to quick over-fitting in the long-training regime. In
Figure 3 we can see that frozen batch normalization results in overfitting. In
contrast, without frozen batch normalization this is not an issue. As seen in
Table 3, Experiment 9, this freezing BN also results in lower performance. It
can therefore be concluded that unfrozen BatchNorm, gives an effective
regularization effect by randomizing shifting and scaling of features.

| OfficeHome | PACS | VLCS | DomainNet | TerraIncognita | Average | ImageNet Accuracy
---|---|---|---|---|---|---|---
TorchVision Weights | 72.2 | 85.9 | 78.5 | 46.9 | 49.7 | 66.6 | 76.1
AugMix Trained Weights [22] | 74.6 | 87.9 | 78.6 | 49.8 | 51.0 | 68.4 | 79.0
Meal V2 [45] | 75.5 | 86.7 | 79.1 | 49.5 | 50.9 | 68.3 | 80.7
ResNet A1 [55] | 70.8 | 82.8 | 77.7 | 43.0 | 37.3 | 62.3 | 80.4
Table 4: Top-1 Accuracy with different ResNet-50 initialization: We
investigate initialization weights from different pre-training procedures. The
differences between different initializations are very substantial, up to
about 6%. Interestingly, improved ImageNet accuracy does not strongly
correlate with improved performance. In fact, the strongest initialization is
from AugMix pretrained weights, with an ImageNet validation accuracy 1.7% less
than the strongest model. Additionally, MealV2 is a distilled model from a
very strong ensemble, where the student is initialized to AugMix weights. The
distillation process doesn’t improve generalization performance overall,
improving over AugMix only in domains which resemble ImageNet. This suggests
that the distillation process effectively matches the student to the teacher
over the data used in the distillation process, at the price of function
smoothness away from the distillation data.

Stronger initializations (S. Init): One of the key components of the standard
DG training scheme is initializing the model parameters with a pre-trained
model. The effect of the strong initialization for our model is shown in Table
3, experiment 7, where we achieve a 1% boost an average. However, selecting a
model takes care. Table 4 compares ResNet-50 models of varying ImageNet
performance described in Section 3.2.1. We summarize our findings below:

  * •

Stronger ImageNet performance does not necessarily correspond to better DG
performance. In particular, both the ResNet-50 A1 and Meal V2 weights achieves
much better ImageNet Top-1 Accuracy than the standard TorchVision weights, but
achieve worse DG performance. However, the overall consistency of the AugMix
weights across all 5 datasets makes it a reasonable choice.

  * •

Model Distillation, which strongly improves source accuracy, does not increase
overall DG performance. Meal-V2 is a distillation of the ensemble if two very
strong ImageNet models into a ResNet-50. Interestingly, the student in Meal-V2
is initialized with the same AugMix trained network as we do in our
experiments. Therefore, the differences in performance can be strictly
attributed to the effects of model distillation. Looking at the results in
more detail, as in Table 5, we can see that performance on ImageNet-like
domains improves while performance on other domains suffers. This suggests
that the distillation process effectively matches the student to the teacher
over the data used in the distillation process, at the price of function
smoothness away from the distillation data.

  * •

AugMix is a model trained with generalization to synthetic corruptions as a
goal and results in a very strong DG performance. Therefore, while ImageNet
Top-1 accuracy is not a good indicator for DG performance, investigating the
correlation between synthetic corruption performance and DG performance is
promising.

| P | C | I | R | Q | S | Av
---|---|---|---|---|---|---|---
Aug[22] | 57.3 | 68.8 | 25.6 | 70.2 | 17.1 | 59.8 | 49.8
MV2[45] | 57.3 | 68.5 | 25.4 | 70.9 | 16.1 | 59.0 | 49.5
Table 5: Model distillation’s effect on domain generalization: We look at the
per-domain accuracy on DomainNet, comparing Augmix training (Aug) and MealV2
(MV2). MealV2 is a method used to distill a large ensemble into a student
ResNet-50, where the student is initialized to AugMix weights. The held-out
domains considered are (P)ainting, (C)lipart, (I)nfo, (R)eal, (Q)uickdraw, and
(S)ketch. We can see that the distillation process, while dramatically
improving ImageNet performance, only slightly changes generalization
performance. In particular, generalization gets slightly worse for all domains
except for (R)eal, which is the most similar to ImageNet. This is surprising,
since it has been shown that both ensembles [3] and larger model [1] improve
domain generalization performance. The distillation process seems to match the
teacher function poorly on OOD data.

###  5.3 Weight Space Regularization

Generalist Model Parameter Averaging (MPA): We confirm that regularizing model
parameters by averaging iterates is an important tool in improving domain
generalization performance; in Table 3 (Experiments 1 and 2) we compare models
trained with and without parameter averaging across timesteps. Specifically,
we average the parameters of all training steps after an initial burn-in
period of 100 steps. We confirm that such model parameter averaging
consistently and substantially improves domain generalization.
Specialist Model Parameter Averaging (SMPA): We also explored a setting where
instead of averaging model weights, we attempt to include diversity between
the models being averaged as this has been shown to boost performance [42].
Following [35], we first train a generalist model on all source domains for 5
epochs, then train specialist models for 5 epochs, before averaging
parameters. Results on the DomainNet dataset are reported in Table 6. Although
averaging specialists improves over ERM, it does not improve over averaging
model iterates of a generalist.

| P | I | Q | S | R | C | Av
---|---|---|---|---|---|---|---
ERM [20] | 51.1 | 21.2 | 13.9 | 52.0 | 63.7 | 63.0 | 44.1
SMPA | 52.9 | 27.2 | 14.3 | 51.3 | 65.6 | 65.2 | 46.1
MPA | 55.2 | 24.0 | 16.7 | 57.4 | 67.0 | 67.49 | 48.0
Table 6: Weight Space Regularization: We show experiments different types of
parameter averaging for weight regularization on DomainNet. SMPA is a
specialized model parameter averaging, where we average parameters of domain
specialists, while MPA averages parameters within a single training
trajectory. While both MPA and SMPA outperform ERM, MPA outperforms SMPA.

###  5.4 Generalizing Beyond Web-scraped Datasets

We have demonstrated that ERM++ is a highly effective recipe for DG on several
datasets: OfficeHome, PACS, DomainNet, and TerraIncognita. These datasets are
diverse and represent a strong evaluation of ERM++. However, [15] show that on
datasets not consisting of web-scraped data, the correlation between ImageNet
performance and transfer performance is quite weak. To verify that this is not
the case for ERM++, we perform an ablation study on WILDS-FMOW, a land-use
classification dataset, and see that the ERM++ substantially improves over ERM
(Table 7).

| R0 | R1 | R2 | R3 | R4 | Av.
---|---|---|---|---|---|---
ERM [20] | 34.9 | 47.1 | 38.6 | 43.8 | 53.8 | 43.6
ERM++ | 41.5 | 50.3 | 40.5 | 50.4 | 57.7 | 48.1
\- Strong Init | 39.2 | 50.1 | 39.5 | 49.5 | 58.7 | 47.4
\- WS | 41.3 | 50.4 | 41.0 | 50.6 | 59.3 | 48.5
\- UBN | 39.6 | 49.1 | 38.9 | 49.1 | 58.1 | 47.0
Table 7: WILDS-FMOW Top-1 Accuracy: We show that ERM++ outperforms ERM on this
on the challenging WILDS-FMOW classification dataset. We also ablate several
components of ERM++. UBN (Unfrozen Batch Norm) and Strong Init (from Augmix)
improve performance, while surprisingly WS (warmstart) decreases performance
in this particular scenario. We emphasize that ERM++ overall improves over
ERM[20].

##  6 Conclusion

This paper develops a strong baseline, ERM++, that can be used to improve the
performance of DG models. By identifying several techniques for enhancing ERM,
our approach achieves significant gains in DG performance, reporting a 1%
average boost over the state-of-the-art on the challenging DomainBed
evaluation datasets and demonstrating efficacy in realistic deployment
scenarios on WILDS-FMOW. We find that ERM++ can also boost the performance of
state-of-the-art methods, but that ERM++ alone may still outperform them. Our
results highlight the importance of improving the training procedure for
better DG performance and provide a strong baseline for future research. ERM++
opens up opportunities for exploring additional techniques to further improve
DG performance.

## Acknowledgment

DISTRIBUTION STATEMENT A. Approved for public release. Distribution is
unlimited. This material is based upon work supported by the Under Secretary
of Defense for Research and Engineering under Air Force Contract No.
FA8702-15-D-0001. Any opinions, findings, conclusions or recommendations
expressed in this material are those of the author(s) and do not necessarily
reflect the views of the Under Secretary of Defense for Research and
Engineering.

##  Appendix A Additional Results

###  A.1 Per-dataset details

In Tables 8 (OfficeHome), 9 (DomainNet), 10 (VLCS), 11 (TerraIncognita), 12
(PACS), we expand results for the datasets and report accuracies for each
held-out domain. We compare ERM++ with reported performances of ERM [20], DIWA
[42], SWAD, [9], and MIRO [10]. ERM + SWAD + MIRO and DIWA are the current
SOTA for ResNet-50 models for this set of datasets. Overall trends include
ERM++ being especially effective at sketch-like domains, indicating a lowered
texture bias. On the sketch and clipart domains in DomainNet, ERM++
outperforms prior best performance by over 4%. When we additionally combine
MIRO with ERM++, we see much improved performance on OfficeHome and
TerraIncognita without much affecting the performance on the other datasets.

| art | clipart | product | real | avg
---|---|---|---|---|---
ERM [20] | 63.1 | 51.9 | 77.2 | 78.1 | 67.6
ERM + SWAD [9] | 66.1 | 57.7 | 78.4 | 80.2 | 70.6
DIWA [42] | 69.2 | 59 | 81.7 | 82.2 | 72.8
ERM + MIRO + SWAD [10] | - | - | - | - | 72.4
ERM++ | 70.7 | 62.2 | 81.8 | 84.0 | 74.7
ERM++ + MIRO | 74.0 | 61.5 | 83.8 | 85.7 | 76.3
Table 8: OfficeHome: Per-domain top-1 accuracy against reported results of recent top-performing methods SWAD, DIWA, and MIRO. [10] does not report per-domain performance for MIRO, so we only show average for that case. DIWA doesn’t report standard errors. ERM++ not only greatly increases performance relative to SWAD, DIWA, and MIRO but also reduce variance between runs. The largest gains are on the held-out domain with the largest domain shift(clipart), illustrating the ability of ERM++ to improve performance on difficult DG tasks. Figure 4: OfficeHome:Samples from the OfficeHome dataset, from each domain and selected classes. The dataset focuses on household objects. The domain shifts are in low-level style mostly, and there is little spatial bias.  | painting | clipart | info | real | quickdraw | sketch | avg
---|---|---|---|---|---|---|---
ERM [20] | 50.1 | 63.0 | 21.2 | 63.7 | 13.9 | 52.9 | 44.0
ERM + SWAD [9] | 53.5 | 66.0 | 22.4 | 65.8 | 16.1 | 55.5 | 46.5
DIWA [42] | 55.4 | 66.2 | 23.3 | 68.7 | 16.5 | 56 | 47.7
ERM + MIRO + SWAD [10] | - | - | - | - | - | - | 47.0
ERM++ | 58.4 | 71.5 | 26.2 | 70.7 | 17.3 | 60.5 | 50.8
ERM++ + MIRO | 58.5 | 71.0 | 26.5 | 71.1 | 15.9 | 59.5 | 50.4
Table 9: DomainNet: Per-domain top-1 accuracy against reported results of recent top-performing methods SWAD, DIWA, and MIRO. [10] does not per-domain performance for MIRO, so we only show average for that case. DIWA doesn’t report standard errors. ERM++ not only greatly increases performance relative to SWAD, DIWA, and MIRO but also reduce variance between runs. Similar to results on OfficeHome (Table 8), the largest performance gains(of larger than 4%) are on domains very different from the source domain(clipart and sketch). This suggests ERM++ is less sensitive to texture bias than ERM [20]. The bias of MIRO to the pre-trained weights manifests in slightly higher performance on close to ImageNet domains like real when combined with ERM++, at the slight expense of performance on other domains.  Figure 5: DomainNet: Samples from the DomainNet dataset. While the real domain is quite similar to what one might expect in ImageNet, the distribution shifts are quite substantial in other domains. Quickdraw and Infograph are particularly challenging, so the 1-3% gains of ERM++ on these domains is meaningful (Table 9). While most domains contain primarily shifts in low level statistics (for example, real to painting), Infograph also has many non-centered objects.  | caltech101 | labelme | sun09 | voc2007 | avg
---|---|---|---|---|---
ERM [20] | 97.7 | 64.3 | 73.4 | 74.6 | 77.3
ERM + SWAD [9] | 98.8 | 63.3 | 75.3 | 79.2 | 79.1
DIWA [42] | 98.9 | 62.4 | 73.9 | 78.9 | 78.6
ERM + MIRO + SWAD [9] | - | - | - | - | 79.6
ERM++ | 98.7 | 63.2 | 71.6 | 78.7 | 78.0
ERM++ + MIRO | 99.0 | 62.4 | 71.8 | 78.3 | 77.9
Table 10: VLCS: Per-domain top-1 accuracy against reported results of recent top-performing methods SWAD, DIWA, and MIRO. [10] does not per-domain performance for MIRO, so we only show average for that case. DIWA doesn’t report standard errors. Although overall performance on VLCS is lower than competing methods, we can see that this drop primarily comes from lower performance on sun09. Furthermore, there are many ambiguous images in the LabelMe domain (see Figure 10), raising questions about the usefulness of trying to train on this domain.  Figure 6: VLCS: The low-level statistics are quite similar between domains, however spatial biases differ between domains. Caltetch objects are quite centered, while other domains do not have this trait. For example the LabelMe domain has cars along the side of the image, and there are many chairs in the VOC2007 domain. Furthermore, in some cases the size of the objects differs dramatically. Finally, there are many ambiguous images in the LabelMe domain (see Figure 10), raising questions about the usefulness of trying to train on this domain.  | Location 100 | Location 38 | Location 43 | Location 46 | Average
---|---|---|---|---|---
ERM [20] | 54.3 | 42.5 | 55.6 | 38.8 | 47.8
ERM + SWAD [9] | 55.4 | 44.9 | 59.7 | 39.9 | 50.0
DIWA [42] | 57.2 | 50.1 | 60.3 | 39.8 | 51.9
ERM + MIRO + SWAD [10] | - | - | - | - | 52.9
ERM++ | 48.3 | 50.7 | 61.8 | 43.9 | 51.2
ERM++ + MIRO | 60.81 | 48.8 | 61.1 | 42.7 | 53.4
Table 11: TerraIncognita: Per-domain top-1 accuracy against reported results of recent top-performing methods SWAD, DIWA, and MIRO. [10] does not per-domain performance for MIRO, so we only show average for that case. DIWA doesn’t report standard errors. ERM++ outperforms other methods on 3 out of 4 held out domains despite slighly underperforming on average. However, we point out that ERM++ w/MIRO outperforms both DIWA and MIRO, and improves ERM++ by a further 2%.  Figure 7: TerraIncognita: Samples from the TerraIncognita dataset, from each domain and selected classes. The background stays consistent, and the animal object frequently takes up a small portion of the frame. At night the images are black-and-white. This dataset matches realistic deployment scenarios well.  | art_painting | cartoon | photo | sketch | avg
---|---|---|---|---|---
ERM [20] | 84.7 | 80.8 | 97.2 | 79.3 | 84.2
ERM + SWAD [9] | 89.3 | 83.4 | 97.3 | 82.5 | 88.1
DIWA [42] | 90.6 | 83.4 | 98.2 | 83.8 | 89
ERM + MIRO + SWAD [10] | - | - | - | - | 88.4
ERM++ | 90.6 | 83.7 | 98.1 | 86.6 | 89.8
ERM++ + MIRO | 90.2 | 83.8 | 98.6 | 82.4 | 88.8
Table 12: PACS: Per-domain top-1 accuracy against reported results of recent
top-performing methods SWAD, DIWA, and MIRO. [10] does not per-domain
performance for MIRO, so we only show average for that case. DIWA doesn’t
report standard errors. ERM++ leads to substantial improvement over prior
work. As in other dataset (OfficeHome, DomainNet), large performance gains are
made on the sketch domain.  Figure 8: PACS: Samples from the PACS dataset,
from each domain and selected classes. The subjects tend to be centered, and
the sketches are more realistic than the quickdraw setting in DomainNet.
Though the domians are similar to that of DomainNet, PACS has fewer than 10000
samples compared to 586000 of DomainNet. Therefore PACS tests the capabilities
of ERM++ on smaller data. Figure 9: FMoW:Samples from the TerraIncognita
dataset, from each domain and selected classes. The images differ in region
but also in resolution and scale. The distribution shift between FMoW and the
pretraining data is large, therefore FmoW represents the ability of ERM++ to
perform on non web-scraped data (see Section 5.4 of the main paper).  Figure
10: Sample from LabelMe Domain in VLCS: Is this a dog, person, or chair? Many
samples in the LabelMe domain of VLCS are ambigrous but assigned a label (in
this case, dog). This raises questions about the usefulness of training on
this domain.

###  A.2 Standard Errors for ERM++ ablation

In Table 13, we reproduce Table 3 from the main paper which shows the impact
of the pieces of ERM++, but add standard errors, in addition to the mean of 3
trials. We can see that not only does each component of ERM++ contribute to
the improvement of the mean performance but also the final ERM++ (Experiment
7) has the smallest standard errors on three out of five datasets.

ERM++ Components | OfficeHome | PACS | VLCS | DomainNet | TerraInc | Avg.
---|---|---|---|---|---|---
# | MPA | FD | LT | WS | ES | S. Init | UBN | 15K | 10K | 11K | 590K | 25K
1 | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | 67.1${\pm 0.2}$ | 85.1${\pm 0.3}$ | 76.9${\pm 0.6}$ | 44.1${\pm 0.15}$ | 45.2${\pm 0.6}$ | 63.7
2 | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | 70.2${\pm 0.3}$ | 85.7${\pm 0.2}$ | 78.5${\pm 0.3}$ | 46.4${\pm 0.0}$ | 49.4${\pm 0.4}$ | 66.0
3 | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✓ | 71.5${\pm 0.1}$ | 87.3${\pm 0.2}$ | 77.4${\pm 0.1}$ | 46.8${\pm 0.0}$ | 49.8${\pm 0.5}$ | 66.5
4 | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✓ | 71.7${\pm 0.1}$ | 88.7${\pm 0.2}$ | 76.9${\pm 0.1}$ | 48.3${\pm 0.0}$ | 49.6${\pm 0.4}$ | 67.0
5 | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ | ✓ | 72.6${\pm 0.1}$ | 88.8${\pm 0.1}$ | 77.0${\pm 0.1}$ | 48.6${\pm 0.0}$ | 49.3${\pm 0.3}$ | 67.3
6 | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | 72.6${\pm 0.1}$ | 88.8${\pm 0.1}$ | 78.7${\pm 0.0}$ | 48.6${\pm 0.0}$ | 49.2${\pm 0.3}$ | 67.6
7 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 74.7${\pm 0.0}$ | 89.8${\pm 0.3}$ | 78.0${\pm 0.1}$ | 50.8${\pm 0.0}$ | 51.2${\pm 0.3}$ | 68.9
8 | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | ✓ | 74.6${\pm 0.1}$ | 87.9${\pm 0.2}$ | 78.6${\pm 0.1}$ | 49.8${\pm 0.0}$ | 51.1${\pm 0.8}$ | 68.4
9 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | 74.7${\pm 0.2}$ | 90.1${\pm 0.0}$ | 78.6${\pm 0.1}$ | 49.9${\pm 0.0}$ | 49.0${\pm 0.4}$ | 68.3
Table 13:  We present the overall ablation for ERM++, with standard errors.
The full ERM++ (Experiment 7), achieves the lowest standard errors on 3 out of
5 datasets, indicating its components decrease variance.. (1) ERM [20]
baseline with unfrozen BN. (2) MPA: Model parameter averaging, which uniformly
improves results. (3) FD: training on the full data. (4) LT: Training for 4x
longer, which ensures convergence improves performance by an additional half
percent. (5) WS: Warm-starting the classification layer especially improves
OfficeHome, but also helps minimize overfitting (Figure 2 of main paper). (6)
ES: Splitting off validation data to find a training length yields substantial
gains. (7) S.Init: Initializing the initial parameters to those trained with
AugMix brings performance to state of the art. (8) Removing LT from (7) still
results in state-of-the-art performance with half of the training cost of
MIRO. (9) UBN: When we freeze the BN parameters, we see that performance
substantially degrades.

###  A.3 Validation-Set Accuracy Curves

In Figures 11,12,13,14, and 15, we provide source-validation accuracies for
each of the 5 datasets, for the number of steps corresponding to long
training, which is 20000 steps for most datasets except for the larger
DomainNet, which is 60000 steps. As one can see, at this point, validation
accuracy is saturated for most domains in most datasets, so this training
length is reasonable. Prior training lengths are denoted as red vertical lines
in these figures, and one can see that for many datasets this is not a
sufficient training length. As we describe in Section 5.1 of the main paper,
this improves performance by 0.5% on average.

Figure 11: OfficeHome: Source validation accuracies. The validation accuracy
saturates by 20000 steps, which corresponds to number of steps in Long
Training(Section 5.1 of the main paper). Training length used in prior works
is denoted as a red line, and the training is not yet converged.  Figure 12:
PACS: Source validation accuracies. The validation accuracy saturates by 20000
steps, which corresponds to number of steps in Long Training(Section 5.1 of
the main paper).Training length used in prior works is denoted as a red line,
and the training is not yet converged.  Figure 13: DomainNet: Source
validation accuracies. The validation accuracy saturates by 60000 steps, which
corresponds to number of steps in Long Training(Section 5.1 of the main
paper). Training length used in prior works is denoted as a red line, and the
training is not yet converged.

Figure 14: VLCS: Source validation accuracies. The validation accuracy
saturates by 20000 steps, which corresponds to number of steps in Long
Training(Section 5.1 of the main paper).Training length used in prior works is
denoted as a red line. In the case of VLCS, it seems like longer training is
not so helpful, and this is reflected in our ablations (Table 13)

.

Figure 15: TerraIncognita: Source validation accuracies. The validation
accuracy saturates by 20000 steps, which corresponds to number of steps in
Long Training(Section 5.1 of the main paper). Training length used in prior
works is denoted as a red line, and the training is not yet converged.

##  Appendix B Dataset Visualizations

In Figures 4 (OfficeHome), 5 (DomainNet), 6 (VLCS), 7 (TerraIncognita), 8
(PACS), 9 (FMoW) we show samples of a few classes from each of the datasets,
and each domain. As one can see, both the datasets and distribution shifts are
quite diverse, highlighting the flexibility of our method. We present some key
attributes of the datasets below.

OfficeHome [51] Figure 4. This dataset focuses on household objects. The
domain shifts are in low-level style mostly, and there is little spatial bias.

DomainNet [41] Figure 5. While the real domain is quite similar to what one
might expect in ImageNet, the distribution shifts are quite substantial in
other domains. Quickdraw and Infograph are particularly challenging, so the
1-3% gains of ERM++ on these domains is meaningful (Table 9).

VLCS [16]: Figure 6. Low-level statistics are quite similar between domains in
this dataset, however spatial biases differ between domains. For example,
Caltetch objects are quite centered, while other domains do not have this
trait. For example the LabelMe domain has cars along the side of the image,
and there are many chairs in the VOC2007 domain. Furthermore, in some cases
the size of the objects differs dramatically. Lastly, there are many ambiguous
images in the LabelMe domain (see Figure 10), raising questions about the
validity of trying to improve performance on this dataset.

TerraIncognita [4]: Figure 7 The background stays consistent, and the animal
object frequently takes up a small portion of the frame. At night the images
are black-and-white. This is a very realistic dataset, on which is good to
test.

PACS [33] Figure 8. The subjects tend to be centered, and the sketches are
more realistic than the quickdraw setting in DomainNet. Though the domains are
similar to that of DomainNet, PACS has fewer than 10000 samples compared to
586000 of DomainNet. Therefore PACS tests the capabilities of ERM++ on smaller
data.

FMoW: Figure 9. The images differ in region but also in resolution and scale.
The distribution shift between FMoW and the pretraining data is large,
therefore FmoW represents the ability of ERM++ to perform on non web-scraped
data (see Section 5.4 of the main paper).

##  Appendix C Runtime Comparisons

As discussed in the main paper Section 3.4; ERM++ achieves higher predictive
performance than competing methods MIRO [10] and DIWA [42] despite lower
computational cost for training. The reason is reduced cost of hyper-parameter
search; we use fixed hyper-parameters, borrowed from the DomainBed framework,
(see Section D.2 for more details ) while DIWA averages 20-60 models and MIRO
search for 4 $\lambda$ weight regularization values in each experiment.
Assuming the worst case scenario of training two full passes (one on
validation data for number of training steps for Early Stopping, and one on
full training data with validation data folded in Full Data), and the same
number of training steps as MIRO; ERM++ costs $\frac{1}{2}$ that of MIRO while
obtaining better performance. In particular, this configuration represents
Experiment 8 in Table 3 of the main paper.

For each forward step MIRO there is an additional forward pass of the data
through the model which is absent in ERM++. On the other hand, ERM++ does take
a forward pass through the running average model to update batch normalization
statistics, which is not done in former methods. This means that each forward
pass is compute-equivalent for ERM++ and MIRO, for a given architecture.

##  Appendix D Reproducibility

We provide code in a zip file along with this supplementary, and will open-
source the code upon acceptance.

###  D.1 Infrastructure

We train on a heterogeneous cluster, primarily on NVIDIA A6000 GPU’s. Each
experiment is conducted on a single GPU with 4 CPUs. A single run could range
from 12-48 hours, depending on number of steps trained.

###  D.2 Training details

We follow the DomainBed [20] training procedure and add additional components
from ERM++. In particular, we use the default hyper-parameters from DomainBed
[20], _e.g_., a batch size of 32 (per-domain), a learning rate of 5e-5, a
ResNet dropout value of 0, and a weight decay of 0. We use the ADAM optimizer
[28] optimizer with $\beta$ and $\epsilon$ values set default values from
Pytorch 1.12. Unless we specify that the “Long Training” component is added,
we train models for 15000 steps on DomainNet (following SWAD[9]) and 5000
steps for other datasets, which corresponds to a variable number of epochs
dependent on dataset size. If Long Training is used, we extend training by 4x.
We train on all source domains except for one, validate the model on held-out
data from the sources every 300 steps(20% of the source data), and evaluate on
the held-out domain. If using Full Data we retrain using the full data. We use
the same data augmentation techniques as ERM [20]. We use the ResNet-50
architechture in all experiments.

Model Parameter Averaging details: If we use Model Parameter Averaging( MPA),
we begin to keep a running average at the 100th step. If we additionally use
warm-start, we only optimize the classification head for the first 500 steps,
and start MPA 100 steps after that. For the Specialist Model Parameter
Averaging(SMPA) experiments (Table 6 of main paper), we first train a
generalist model for 15000 steps , then train an independent model for each
domain for another 1500 steps. At the end, we average parameters and re-
compute batch norm running statistics. This recomputing of BN stats makes sure
the averaged model has accurately computed batch norm statistics which may not
be a simple average of experts, due to the non-linearity of neural nets.

Batch Normalization details: With unfrozen batch normalization( UBN), we
update the evaluation model BN statistics by averaging the model iterates
first (from MPA), then then forward propagating the current batch at each step
through the evaluation model. In this way, the BN running statistics and model
used for inference match.

Sources of pre-trained weights: We use torchvision 0.13.1 for vanilla
ResNet-50 initialization. For augmix and ResNet-A1 initialized weights, we
leverage TIMM [54] 111 Augmix Weights :https://github.com/rwightman/pytorch-
image-models/releases/download/v0.1-weights/resnet50_ram-a26f946b.pth 222
ResNet-A1 Weights :https://github.com/rwightman/pytorch-image-
models/releases/download/v0.1-rsb-weights/resnet50_a1_0-14fe96d1.pth .

A note on hyper-parameter search: In this work, we focus on methodological
improvements that do not depend on expensive hyper-parameter tuning, and as a
result we use default learning rate, weight decay, etc. We demonstrate state-
of-the-art performance despite this, and greatly reduce the computational cost
of training as a result. However, we believe there is substantial headroom for
improvement with further hyper-parameter tuning.

MIRO Implementation: We directly follow the MIRO implementation and borrow the
lambda weights values from [10] when we combine MIRO with ERM++ in Table 2 of
the main paper. ERM++ substantially improves the performance of MIRO.

DIWA Implementation: We follow a simplified version of the DIWA [42] algorithm
due to computational reasons; we average the parameters of the three seeds of
ERM++. The authors of DIWA show that about half of the performance boost comes
from the first few models averaged (Figure 4 of [42]), therefore this is a
reasonable approximation of the method. It is interesting that DIWA reduces
performance of ERM++, but that ERM++ w/DIWA is still improved over DIWA as
reported in [42].

## References

  * [1] Simone Angarano, Mauro Martini, Francesco Salvetti, Vittorio Mazzia, and Marcello Chiaberge.  Back-to-bones: Rediscovering the role of backbones in domain generalization.  arXiv preprint arXiv:2209.01121, 2022.
  * [2] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz.  Invariant risk minimization.  arXiv preprint arXiv:1907.02893, 2019.
  * [3] Devansh Arpit, Huan Wang, Yingbo Zhou, and Caiming Xiong.  Ensemble of averages: Improving model selection and boosting performance in domain generalization.  arXiv preprint arXiv:2110.10832, 2021.
  * [4] Sara Beery, Grant Van Horn, and Pietro Perona.  Recognition in terra incognita.  In Proceedings of the European conference on computer vision (ECCV), pages 456–473, 2018.
  * [5] Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov.  Knowledge distillation: A good teacher is patient and consistent.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10925–10934, 2022.
  * [6] Gilles Blanchard, Aniket Anand Deshmukh, Ürun Dogan, Gyemin Lee, and Clayton Scott.  Domain generalization by marginal transfer learning.  The Journal of Machine Learning Research, 22(1):46–100, 2021.
  * [7] Gilles Blanchard, Gyemin Lee, and Clayton Scott.  Generalizing from several related classification tasks to a new unlabeled sample.  Advances in neural information processing systems, 24, 2011.
  * [8] Manh-Ha Bui, Toan Tran, Anh Tran, and Dinh Phung.  Exploiting domain-specific features to enhance domain generalization.  Advances in Neural Information Processing Systems, 34:21189–21201, 2021.
  * [9] Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park.  Swad: Domain generalization by seeking flat minima.  Advances in Neural Information Processing Systems, 34:22405–22418, 2021.
  * [10] Junbum Cha, Kyungjae Lee, Sungrae Park, and Sanghyuk Chun.  Domain generalization by mutual-information regularization with pre-trained models.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXIII, pages 440–457. Springer, 2022.
  * [11] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  * [12] Xinlei Chen and Kaiming He.  Exploring simple siamese representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15750–15758, June 2021.
  * [13] Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee.  Functional map of the world.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
  * [14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [15] Alex Fang, Simon Kornblith, and Ludwig Schmidt.  Does progress on imagenet transfer to real-world datasets?  arXiv preprint arXiv:2301.04644, 2023.
  * [16] Chen Fang, Ye Xu, and Daniel N Rockmore.  Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1657–1664, 2013.
  * [17] Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan.  Deep ensembles: A loss landscape perspective.  arXiv preprint arXiv:1912.02757, 2019.
  * [18] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky.  Domain-adversarial training of neural networks.  The journal of machine learning research, 17(1):2096–2030, 2016\.
  * [19] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He.  Diverse weight averaging for out-of-distribution generalization.  arXiv preprint arXiv:1706.02677, 2017.
  * [20] Ishaan Gulrajani and David Lopez-Paz.  In search of lost domain generalization.  arXiv preprint arXiv:2007.01434, 2020.
  * [21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [22] Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan.  AugMix: A simple data processing method to improve robustness and uncertainty.  Proceedings of the International Conference on Learning Representations (ICLR), 2020.
  * [23] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132–7141, 2018.
  * [24] Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson.  Averaging weights leads to wider optima and better generalization.  arXiv preprint arXiv:1803.05407, 2018.
  * [25] Fahdi Kanavati and Masayuki Tsuneki.  Partial transfusion: on the expressive influence of trainable batch norm parameters for transfer learning.  In Medical Imaging with Deep Learning, pages 338–353. PMLR, 2021\.
  * [26] Donghyun Kim, Kaihong Wang, Stan Sclaroff, and Kate Saenko.  A broad study of pre-training for domain generalization and adaptation.  arXiv preprint arXiv:2203.11819, 2022.
  * [27] Daehee Kim, Youngjun Yoo, Seunghyun Park, Jinkyu Kim, and Jaekoo Lee.  Selfreg: Self-supervised contrastive regularization for domain generalization.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9619–9628, 2021.
  * [28] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [29] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al.  Wilds: A benchmark of in-the-wild distribution shifts.  In International Conference on Machine Learning, pages 5637–5664. PMLR, 2021.
  * [30] Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang.  Fine-tuning can distort pretrained features and underperform out-of-distribution.  arXiv preprint arXiv:2202.10054, 2022.
  * [31] Yoonho Lee, Annie S Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn.  Surgical fine-tuning improves adaptation to distribution shifts.  arXiv preprint arXiv:2210.11466, 2022.
  * [32] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales.  Learning to generalize: Meta-learning for domain generalization.  In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
  * [33] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales.  Deeper, broader and artier domain generalization.  In Proceedings of the IEEE international conference on computer vision, pages 5542–5550, 2017.
  * [34] Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot.  Domain generalization with adversarial feature learning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5400–5409, 2018.
  * [35] Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A Smith, and Luke Zettlemoyer.  Branch-train-merge: Embarrassingly parallel training of expert language models.  arXiv preprint arXiv:2208.03306, 2022.
  * [36] Ya Li, Mingming Gong, Xinmei Tian, Tongliang Liu, and Dacheng Tao.  Domain generalization via conditional invariant representations.  In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
  * [37] Samarth Mishra, Zhongping Zhang, Yuan Shen, Ranjitha Kumar, Venkatesh Saligrama, and Bryan A. Plummer.  Effectively leveraging attributes for visual similarity.  In The IEEE International Conference on Computer Vision (ICCV), 2021\.
  * [38] Yair Movshovitz-Attias, Alexander Toshev, Thomas K. Leung, Sergey Ioffe, and Saurabh Singh.  No fuss distance metric learning using proxies.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
  * [39] Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf.  Domain generalization via invariant feature representation.  In International conference on machine learning, pages 10–18. PMLR, 2013.
  * [40] Hyeonseob Nam, HyunJae Lee, Jongchan Park, Wonjun Yoon, and Donggeun Yoo.  Reducing domain gap by reducing style bias.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8690–8699, 2021.
  * [41] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang.  Moment matching for multi-source domain adaptation.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 1406–1415, 2019.
  * [42] Alexandre Rame, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy, Patrick Gallinari, and Matthieu Cord.  Diverse weight averaging for out-of-distribution generalization.  arXiv preprint arXiv:2205.09739, 2022.
  * [43] David Ruppert.  Efficient estimations from a slowly convergent robbins-monro process.  Technical report, Cornell University Operations Research and Industrial Engineering, 1988.
  * [44] Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang.  Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization.  arXiv preprint arXiv:1911.08731, 2019.
  * [45] Zhiqiang Shen and Marios Savvides.  Meal v2: Boosting vanilla resnet-50 to 80%+ top-1 accuracy on imagenet without tricks.  arXiv preprint arXiv:2009.08453, 2020.
  * [46] Baochen Sun and Kate Saenko.  Deep coral: Correlation alignment for deep domain adaptation.  In Computer Vision–ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III 14, pages 443–450. Springer, 2016.
  * [47] Reuben Tan, Mariya I Vasileva, Kate Saenko, and Bryan A Plummer.  Learning similarity conditions without explicit supervision.  In The IEEE International Conference on Computer Vision (ICCV), pages 10373–10382, 2019.
  * [48] Eu Wern Teh, Terrance DeVries, and Graham W. Taylor.  Proxynca++: Revisiting and revitalizing proxy neighborhood component analysis.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, The European Conference on Computer Vision (ECCV), 2020.
  * [49] Vladimir N Vapnik.  An overview of statistical learning theory.  IEEE transactions on neural networks, 10(5):988–999, 1999.
  * [50] Ramakrishna Vedantam, David Lopez-Paz, and David J Schwab.  An empirical investigation of domain generalization with empirical risk minimizers.  Advances in Neural Information Processing Systems, 34:28131–28143, 2021.
  * [51] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan.  Deep hashing network for unsupervised domain adaptation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5018–5027, 2017.
  * [52] Xun Wang, Haozhi Zhang, Weilin Huang, and Matthew R Scott.  Cross-batch memory for embedding learning.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [53] Yufei Wang, Haoliang Li, and Alex C Kot.  Heterogeneous domain generalization via domain mixup.  In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3622–3626. IEEE, 2020.
  * [54] Ross Wightman.  Pytorch image models.  https://github.com/rwightman/pytorch-image-models, 2019.
  * [55] Ross Wightman, Hugo Touvron, and Hervé Jégou.  Resnet strikes back: An improved training procedure in timm.  arXiv preprint arXiv:2110.00476, 2021.
  * [56] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al.  Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.  In International Conference on Machine Learning, pages 23965–23998. PMLR, 2022.
  * [57] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al.  Robust fine-tuning of zero-shot models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7959–7971, 2022.
  * [58] Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan L Yuille, and Quoc V Le.  Adversarial examples improve image recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 819–828, 2020.
  * [59] Minghao Xu, Jian Zhang, Bingbing Ni, Teng Li, Chengjie Wang, Qi Tian, and Wenjun Zhang.  Adversarial domain adaptation with domain mixup.  In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 6502–6509, 2020.
  * [60] Shen Yan, Huan Song, Nanxiang Li, Lincan Zou, and Liu Ren.  Improve unsupervised domain adaptation with mixup training.  arXiv preprint arXiv:2001.00677, 2020.
  * [61] Andrew Zhai and Hao-Yu Wu.  Classification is a strong baseline for deep metric learning.  arXiv preprint arXiv:1811.12649, 2018.
  * [62] Marvin Zhang, Henrik Marklund, Nikita Dhawan, Abhishek Gupta, Sergey Levine, and Chelsea Finn.  Adaptive risk minimization: Learning to adapt to domain shift.  Advances in Neural Information Processing Systems, 34:23664–23678, 2021.
  * [63] Zhun Zhong, Yuyang Zhao, Gim Hee Lee, and Nicu Sebe.  Adversarial style augmentation for domain generalized urban-scene segmentation.  arXiv preprint arXiv:2207.04892, 2022.
  * [64] Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang.  Domain generalization with mixstyle.  arXiv preprint arXiv:2104.02008, 2021.