# Aggregation of disentanglement: Reconsidering domain variations in domain
generalization

Daoan Zhang111footnotemark: 1 Mingkai Chen211footnotemark: 1 Chenming Li1
Lingyun Huang3 Jianguo Zhang100footnotetext: Equal contribution
1 Southern University of Science and Technology 2 Stony Brook University
3 PingAn Technology

###### Abstract

This research addresses the challenge of Domain Generalization (DG) in machine
learning, which aims to improve the generalization of models across multiple
domains. While previous methods focused on generating domain-invariant
features, we propose a new approach that incorporates domain variations that
contain useful classification-aware information for downstream tasks. Our
proposed method, called Domain Disentanglement Network (DDN), decouples input
images into Domain Expert Features and noise. The domain expert features are
located in a learned latent space where images in each domain can be
classified independently, allowing the implicit use of task-specific domain
variations. In addition, we introduce a novel contrastive learning method to
guide the domain expert features to form a more balanced and separable feature
space. Our experimental results on widely used benchmarks, including PACS,
VLCS, OfficeHome, DomainNet, and TerraIncognita, show that our method
outperforms recently proposed alternatives, demonstrating its competitive
performance.

**footnotetext: These authors contributed equally to this work

##  1 Introduction

In the field of deep learning[51, 52, 50, 62], it is widely assumed that the
training and test data conform to an independent and identical
distribution[49, 61]. However, in practical applications, it is often
necessary for the models to perform well on out-of-distribution data, in
addition to in-distribution data. This presents a formidable challenge for the
models, as their ability to generalize is put to test. To address this
challenge, domain generalization (DG) has been proposed as a crucial task for
enhancing the model’s generalization performance [30].

The main idea behind DG is that during the training phase, only the source
domains are accessible to the model, and the test data from target domains are
not visible [4, 36]. Traditional DG methods aim to extract domain-invariant
features across the source domains [1, 29, 45]. Specifically, the input data
are transformed into a domain-invariant latent space, where the domain-
invariant features can be effectively extracted. During inference, the test
data from the unseen target domains are also projected onto the same latent
space for downstream tasks such as classification. It is noteworthy that the
domain-invariant features are those that are shared by all source domains

Figure 1: Domains differ from each other. Some features that benefit
classification tasks in one domain have no presence in other domains. For
example, skin texture features in domain 1 (left) help in identifying the
elephant, and in domain 2 (middle), no texture features are present for
classification, while in domain 3 (right), texture features impose lots of
noise on the whole picture, making the classification tasks more challenging.

In this paper, we argue that such invariant features learned from source
domains are not necessarily the best for classification, _i.e_., the best
classification-aware features. Previous methods [6, 38] have already
introduced the domain specific features but they fail to consider whether the
features is useful to downstream tasks. While we demonstrate the task-specific
domain variations should have the following properties: (1) Domain variations
should be specific for each domain, and (2) effective for downstream tasks. An
illustrative example is presented in Fig. 1 to emphasize these properties. In
domain 1, domain-specific features such as skin textures and surrounding
context features play a crucial role in identifying an elephant, while in
domain 2, these features are not available, and shape features are the
principal cue. Domain 3 demonstrates a scenario in which texture features can
have an adverse impact on elephant identification.

Moreover, from the view of unseen target domain, merely producing domain-
invariant features from all source domains is inadequate for classification,
because (1) the generated domain-invariant information from the source domains
may not be suitable for the target domain and (2) domain variations in the
target domain may also aid in classification.

Taking into account the task-specific domain variations, current domain
generalization works can be classified into three categories: (a) data
manipulation methods [48, 58, 64] that can diversify domain variations while
maintaining domain-invariant features, (b) representation learning methods
[11, 31, 38] that aim to produce better domain-invariant representation from
existing data, and (c) learning strategies [20, 39, 60] that integrate other
general machine learning paradigms into domain generalization.

Practically, we propose the adoption of task-specific domain variations in two
key areas:

(1) Disentanglement: At training time, for each source domain, the mapped
features of the images should contain as much information as possible that is
helpful for classification, _i.e_., the proposed features can be disentangled
into the domain invariant and the task-specific domain variations.

(2) Aggregation: At the time of testing, the features mapped from the target
test domains may also contain task-specific domain variations. While the task-
specific domain variations from target domains can be seen as an aggregation
(_e.g_., ideally linear combination) of the task-specific domain variations
from training source domains.

To address these concerns, we introduce a new paradigm for DG classification
tasks called the Domain Disentanglement Network (DDN). To maintain task-
specific domain variations for each domain, we design a special Domain Expert
Classifier (DEC) for every domain as a domain specific ERM to reduce the
domain specific noise. This classifier consists of a domain-specific
classifier and a domain-specific projection head. Additionally, we develop a
new contrastive approach known as Domain Prototype Contrastive Learning (DPCL)
to disentangle domain invariant features and task-specific domain variations,
respectively. This contrastive approach effectively elicits the discrepancies
and similarities among source domains and maps features from the target domain
to the aggregation of features from the source domains.

Specifically, our main contributions can be summarized as follows:

  * •

We propose a new perspective that emphasizes the importance of task-specific
domain variations in domain generalization. We argue that to achieve effective
generalization, all domain variations must be fully considered during
training, and the task-specific domain variations from unseen target domains
should be decoupled into the task-specific domain variations generated from
accessible source domains.

  * •

We introduce an end-to-end paradigm for classification tasks in DG that
consists of two parts. In the first part, images are sent into the domain
expert classifier (DEC), which classifies the images and maintains task-
specific domain variations. In the second part, then the domain prototype
contrastive Learning (DPCL) is a reinforcement to learn task-specific domain
variations from source domains.

  * •

We demonstrate the effectiveness of our proposed approach by achieving state-
of-the-art performance on multiple standard benchmarks, including PACS, VLCS,
OfficeHome, TerraIncognita, and DomainNet. The experimental results validates
the significance of our proposed perspective and the end-to-end paradigm for
domain generalization.

##  2 Related Work

###  2.1 Domain Generalization

The domain generalization methods have been developed with the primary
objective of learning a universal representation across all source domains.
These methods have been broadly categorized into three groups: (1) Data-
centered methods [17, 34, 43], which include data augmentation approaches
aimed at presenting complex and diverse samples to the model. In addition,
data generation techniques [26, 58] have been implemented that utilize
generative methods to produce pseudo OOD images for further training. (2)
Learning-strategy centered methods, which incorporate several strategies such
as ensemble strategy [10, 28], aimed at effectively combining multiple models
to improve the generalization of the model, and gradient-manipulation methods
[40, 41] that use gradient information to enhance the domain invariant
representation. (3) Representation learning methods [6, 16, 19, 33], which aim
to generate domain invariant features that can minimize the representation
discrepancy among domains, thus facilitating better generalization of the
model. While we focus on the specific task-aware domain variations which is
expected to enhance the overall classification accuracy and improve the
model’s performance in real-world scenarios.

###  2.2 Contrastive Learning

Contrastive learning [32] is a common strategy to enhance the similarity
between two instances in a positive pair and the dissimilarity in a negative
pair.

To achieve this, various contrastive learning algorithms have been proposed,
such as SimCLR [8], which calculates the co-occurrence of multiple augmented
views; MoCo [15], which utilizes a memory bank mechanism to handle a larger
number of negative pairs during training; SimSiam [9] and BYOL [13], which
employ stop gradient operation to learn representations without a memory bank
or a large batch size; DenseCL [56], utilizes dense alignments between
semantic features; PixPro [57], which leverages pixel-level pretext tasks and
pixel-wise consistency to enhance downstream tasks; And PixContrast [55],
which expands contrastive learning to supervised learning.

The domain generalization problem can also be considered as an dense
contrastive learning problem. In dense prediction tasks such as segmentation,
regional semantic features from different images are labeled with the same
tag, while in the domain generalization problem, image features with different
tags but in the same domain can be classified in the same domain category. Our
proposed method introduces a prototype-based contrastive learning approach to
better handle out-of-domain (OOD) tasks in domain generalization problems.

##  3 Domain Variations in DG

###  3.1 Formulation

We consider the classification task in DG as a mixture of minimizing the
empirical risk of classification in each domain and measuring the decoupled
weights of target domains over each source domain. We follow the leave-one-
domain setting and assume that the source domains are $1,2,...,S$, and a
single target domain is $T$, we then define $C_{m}$ as the domain invariant
features, where $m$ is the $m^{th}$ class in a total of $M$ classes; $D_{T}$
as the task-specific domain variations over all the data in the target domain;
$D_{S}$ as the task-specific domain variations over the source domains. From
domain $s$, we sample $N$ data points which are defined as
${\\{(x_{i}^{s},y_{i}^{s})\\}}_{i=1}^{N}$.

###  3.2 Disentangle and Aggregate Domain Variations

Figure 2: Left: The relationships between source and target domains.; Right:
The disentangled task-specific domain variations and domain expert features of
target domains.

Taking the classification as an example, we first conceptually decompose the
generated information from an image $I^{s}_{m}$ from domain $s$, class $m$
into three component vectors:

$I^{s}_{m}=C_{m}+D_{s}+N^{s}_{m},$ (1)

where $C_{m}$ denotes the domain invariant features, $D_{s}$ denotes the task-
specific domain variations, and $N^{n}_{m}$ denotes the useless information
for classification.

The Venn diagram depicted in Fig. 2 (left) provides a visual representation of
our conceptual decomposition. Specifically, $I^{1}_{1}$ and $I^{2}_{1}$ are
instances of class 1 sampled from domains 1 and 2, respectively. As such, they
share the same domain invariant features $C_{1}$. However, $I^{1}_{1}$ and
$I^{2}_{1}$ also contain task-specific domain variations $D_{1}$ and $D_{2}$,
respectively, as well as noisy information $N_{1}^{1}$ and $N_{1}^{2}$. In the
roght of the graph, for an image $I_{1}^{k}$ sampled from an unseen target
domain $k$, the task-specific domain variations can be expressed as a linear
combination of $D_{1}$ and $D_{2}$, which is an ideal scenario. With this
hypothesis, we introduce the Domain Expert Features $R^{s}_{m}$, depicted in
Fig. 2 (right), which combine domain invariant features with task-specific
domain variations, and are deemed sufficient for classification purposes. The
feature is defined as:

$R^{s}_{m}=C_{m}+D_{s},$ (2)

In training, we propose to generate domain expert features for each source
domain. In inference, as the unseen target domain is hard to analyze for the
trained model, we propose to use the existing task-specific domain variations
to represent the task-specific domain variations from the target domain which
can be seen as an aggregation of the disentangled domain expert features. We
first denote the relationship between the task-specific domain variations from
source domains and the target domain as:

$D_{T}=\sum_{s=1}^{S}{w_{Ts}D_{s}},$ (3)

Then,

$C_{m}+D_{T}=C_{m}+\sum_{s=1}^{S}{w_{Ts}D_{s}},$ (4)

where $w_{T}=[w_{T1},...,w_{Ts}]$ is the domain-specific weights of decoupled
target task-specific domains variations over the source task-specific domain
variations, which lies on
$w_{Ts}\in\mathbb{R},{\left\|w_{T}\right\|}_{1}=1,w_{Ts}\geq 0$. Thus,
according to Eq. 2, Eq. 4 can be modified to:

$R_{m}^{T}=\sum_{s=1}^{S}{w_{Ts}}R_{m}^{s},$ (5)

Thus,
$R^{T}=\sum_{m=1}^{M}{R^{T}_{m}}=\sum_{m=1}^{M}\sum_{s=1}^{S}R_{m}^{s}=\sum_{s=1}^{S}R^{s}$.

As the key point of DG problems is to reduce the discrepancy between the
source domains features and target domain features [48], which can be written
as:

$\mathcal{L_{P}}(E(x;\theta_{E});\theta_{P})=\frac{1}{S}\sum_{s=1}^{S}{\textit{d}(R^{s},R^{T})},$
(6)

where $E(\cdot;\theta_{E})$ is defined as the backbone of the model,
$\mathcal{L_{P}}$ is the loss of domain awareness head,
$\textit{d}(\cdot,\cdot)$ is a distance to measure the discrepancy between the
features. According to our assumption, the domain expert features $R^{s}$ are
diverse from each other, but how to map the features into an appropriate
latent space that can better demonstrate target features $R^{T}$ tends to be
explored.

To better estimate the discrepancy, following [2], we utilize the commonly
used metric for distributions: Wasserstein-1 distance as the measurement.
Thus, Eq. 6 can be written as:

$\displaystyle\mathcal{L_{P}}(E(x;\theta_{E});\theta_{P})$ (7)
$\displaystyle=\frac{1}{S}\sum_{s=1}^{S}\left(\underset{x\sim
R^{s}}{\mathbb{E}}[f_{s}(x)]-\underset{x\sim
R^{T}}{\mathbb{E}}[f_{s}(x)]\right),$

where $f_{s}(\cdot)$ is the 1-Lipschitz function $f:X\to\mathbb{R}$ which can
be implemented by a neural network [2] for every source domain.

We introduce Eq. 5 to the penalty Eq. 7, it turns to:

$\displaystyle\mathcal{L_{P}}(E(x;\theta_{E});\theta_{P})=\frac{1}{S}\sum_{s=1}^{S}\left(\underset{x\sim
R_{s}}{\mathbb{E}}[\gamma_{s}^{T}F(E(x))]\right)$ (8)

Where $F(E(x))={[f_{1}(E(x)),...,f_{S}(E(x))]}^{T}$ is the concatenation of
the 1-Lipschitz functions for every source domain, and $\gamma_{s}^{T}$ is
denoted as:

$\gamma_{s}=\left\\{\begin{aligned}
1-w_{ss^{{}^{\prime}}},&&s=s^{{}^{\prime}}\\\ -w_{ss^{{}^{\prime}}},&&s\neq
s^{{}^{\prime}}\end{aligned}\right.$ (9)

where $s^{{}^{\prime}}=1,...,S$, $w_{ss^{{}^{\prime}}}$ is the weight of the
feature from $s$ over the feature from $s^{{}^{\prime}}$.

As we assume that $N$ cases are sampled from each source domain, the Eq. 8
turns out to be:

$\displaystyle\mathcal{L_{P}}(E(x;\theta_{E});\theta_{P})\simeq\frac{1}{SN}\sum_{s=1}^{S}\left[\sum_{n=1}^{N}{\gamma_{s}^{T}}F(E(x))\right]$
(10)

Notice that, we have $w_{ss^{{}^{\prime}}}\in[0,1]$, and according to Eq. 9
and 10, to minimize the loss term
$\mathcal{L_{P}}(E(x;\theta_{E});\theta_{P})$, for every data from each
domain, when $s=s^{{}^{\prime}}$, the function $f_{s}(E(x))$ should be
minimized, and when $s\neq s^{{}^{\prime}}$, the function $f_{s}(E(x))$ should
be maximized. This means the distance between features from different domains
should be pulled away, and the distance between features from the same domain
should be narrowed. Practically, we design the Domain Prototype Contrastive
Learning to modify the distances.

##  4 Domain Disentanglement Network (DDN)

Figure 3: Right: DDN Network. Image features from different domains are sent
into different domain expert classifiers based on domain labels. The relations
between the domain expert features are calculated through the Domain Prototype
Contrastive Learning strategy. Left: Schematic Diagram of Domain Prototype
Contrastive Learning. Black lines mean pushing in-domain features near the
prototype of the domain, and red lines mean pushing features from other
domains far away from the prototype of the domain.

To satisfy the requirements, we designed a novel network called Domain
Disentanglement Network (DDN). Compared to traditional methods, the network
considers and decouples both domain invariant and task-specific domain
variations for classification when in training (Fig. 3. left), and in the
inference period, classification-aware features from unseen target domain are
identified and aggregated according to learned source domains (Fig. 5) and are
classified through the corresponding classifiers.

###  4.1 Domain Prototype Contrastive Learning (DPCL)

Motivations. Based on Eq. 10, we propound a newly designed contrastive
learning-based method called Domain Prototype Contrastive Learning (DPCL).
This method aims to modify the discrepancies between domain expert features
when training and decouple domain expert features from the target domain when
inference (see Fig. 3. right). We notice that in the inference, the model has
to calculate the aggregation weights over domain expert features of each
source domain for target test features. The commonly used paradigm, memory
bank [15], can store the domain expert features from source domains. However,
due to the discrepancies of different classes, features from the same domain
can be various from each other, and the stored memory is hard to converge.
Therefore, the memory bank mechanism fails to learn sufficient memory on DG
tasks. To relieve the problem, instead of storing the domain specific
features, we proposed a domain specific projection head to build the prototype
for calculating the weights.

Whole Structure. In our method, we generate the prototype dynamically, and we
use a tiny network to aggregate prototypes separately for each domain in the
batch (see Fig. 3. left). This can relieve the discrepancy between the
features as well as converge faster.

Thus, when we sample $N$ cases from one source domain $s^{+}$, our DCPL loss
is designed as follows:

$\displaystyle\mathcal{L_{P}}(E(x;\theta_{E});\theta_{P},s^{+})$ (11)
$\displaystyle=-log\frac{exp[{\sum^{N}_{n=1}cos(E(x^{s^{+}}_{n},q^{s^{+}}))}]}{\sum_{s=1}^{S}exp[{\sum_{n=1}^{N}}cos(E(x^{s}_{n}),q^{s^{+}})]}$

where $cos(\cdot,\cdot)$ denotes cosine similarities between two embeddings.
$x^{s^{+}}_{n}$ denotes the $n^{th}$ case sampled from the chosen $s^{+}$
domain, and $q^{s^{+}}$ is designed as:

$q^{s^{+}}=\frac{1}{n}\sum_{n=1}^{N}P^{s^{+}}(E(x^{s{{}^{+}}}_{n}))$ (12)

where $P^{s^{+}}$ denotes the projection head for domain $s^{+}$. Thus, with
the DPCL module and the domain expert classifier, When training with source
domains, both the classification-aware domain invariant and variant features
can be disentangled from the source domains for more precise classification.
When in the inference, the test domain features can be represented as the
aggregated domain expert features from source domains to better understand the
unseen domains,

Difference between DPCL and existing contrastive DG paradigms.

The key differences can be concluded as follows:

(1) Different solutions for contrastive learning: Traditional contrastive
paradigms like SelfReg [20] and PCL [60] are classification-based contrastive
learning while we propose a domain-based prototype contrastive learning
paradigm. As shown in Fig. 4, classification-based contrastive learning tries
to map a uniform latent space for classification but fails to consider the
information from domains. Also, in contrastive learning, too difficult sample
pairs from different domains can hamper the model generalization [42]. In our
proposed domain-based prototype contrastive learning, the domain prototype
provides domain-specific information for the classifier, and we can avoid
facing the problem of difficult samples.

(2) Different purposes of using contrastive learning: Conventional contrastive
paradigm aims to obtain better domain invariant features to generalize the
model. However, we focus on rather than domain invariant features but to
generate more uniform representations of task-specific domain variations to
better represent the domain information for unknown test domains.

Figure 4: The sketch map of DDN and traditional contrastive learning methods
in DG

###  4.2 Domain Expert Classifier (DEC)

We design a specific classifier DEC for every input domain. Each classifier
consists of a domain-aware classifier as a ERM model, a project head for
generating prototypes, and a skip connection for DPCL to calculate distance.
When we take a look at Eq. 1, the noise $N^{s}_{m}$ in each domain can be
removed by training the per-domain ERM classifiers. Thus, for each per-domain
ERM classifier, the input domain expert features can be seen as the
$C_{m}+D_{s}$ in Eq. 1. With the combination of all the per-domain ERM
classifier, the backbone can generate all the domain expert features for
different domains. Thus, the total classification loss is:

$\mathcal{L}_{Y}(x,y)=\frac{1}{SNM}\sum_{s=1}^{S}\sum_{n=1}^{N}\sum_{m=1}^{M}y_{n,m}^{s}logY^{s}_{m}[E(x^{s}_{n})]$
(13)

where $Y^{s}_{m}$ denotes the $m^{th}$ entry of the output of domain $s$.

The structure of the domain expert classifier is proposed in Fig. 3, left. We
use several MLP layers as classifiers and project heads.

###  4.3 Overall Strategy

Thus, the loss in our methods is designed to be:

$\mathcal{L}(x,y,t)=\mathcal{L}_{Y}(x,y)+\lambda\mathcal{L_{P}}(x,t)$ (14)

where $x$ is the input image, $y$ is the classification ground truth label,
and $t$ is the domain ground truth label. $\lambda$ is the loss weight. In the
training period, all domain-specific classifiers are trained simultaneously,
and the domain prototype contrastive learning is also processed to maintain
the classification-aware features.

In the inference period, as shown in Fig. 5, the domain expert features
generated from the target domain are sent into all domain expert classifiers,
and the prototypes from each source domain are calculated for measuring the
weights for each domain expert classifier. Then the weights are imposed on the
classification result of each classifier respectively. In the end, we
calculate the average result of all the classifiers to be the final decision.

Figure 5: The sketch map of inference. Images are sent into the backbone to
achieve the domain expert features, and then the features are fed into each
domain expert classifier. The classifier weights are calculated via the
distance between the prototype and the features. Afterward, the result can be
obtained by aggregating the weighted result of each classifier.

##  5 Experiments

###  5.1 Implementation Details

Datasets. To evaluate our proposed methods, we choose five larger datasets in
DomainBed [14]: PACS [24], VLCS [44], OfficeHome [47], DomainNet [33] and
TerraIncognita [3]. The PACS dataset consists of 10k images from four
different domains. VLCS dataset consists of four domains and five classes.
OfficeHome dataset is sampled from four domains with around 15500 images of 65
categories. DomainNet contains 0.6 million images of 345 classes sampled from
six domains. TerraIncognita consists of 24788 images, ten classes, and four
domains.

Baselines. Our experiments are based on the open-source code SWAD [7], and
DomainBed [14] benchmarks. Other training strategies and data augmentation we
use remains the same with SWAD [7] and DomainBed [14]. Follow [14], for PACS,
VLCS, OfficeHome, and TerraIncognita; we train our model for 5000 iterations.
For DomainNet, we train our model for 15000 iterations. We use the leave-one-
out setting, _i.e_.; a specific single domain is used as a test domain and the
others as a training domain.

Hyperparameter search and model selection. For DomainBed baseline, we conduct
a random search of 20 trials over the hyperparameter distribution for each
algorithm and test domain. Results are averaged over three seeds. We set the
search distribution of hyparameter $\lambda$ as $\\{1,5,10,20,30\\}$ .
Specifically, the model selection strategy we utilize is the ’train-domain
validation (oracle)’: the validation set follows the same distribution as the
test domain. For the SWAD baseline, we follow the initial settings in SWAD [7]
to implement the experiments, and the model selection is not required.

Model architectures. The pretrained model we used is Resnet-50 which is
pretrained on ImageNet. We do not use any additional information or training
loops compared to those who use GAN or complex data augmentations. Besides the
Resnet-50 backbone, the proposed classifier architecture is a one-layer MLP,
and the projection head is a two-layer MLP. We used cosine similarity loss to
calculate the distance in DPCL.

###  5.2 Qualitative Results

We compared our methods with current state-of-the-art methods. For a fair
comparison, all methods utilize Resnet-50 pre-trained on ImageNet. The first
set of comparative experiments is based on DomainBed [14] as is shown in
Table. 1, both results in most single domains, and on average of our model
prove that our model can effectively generalize on unseen domains. Notice
that, Our methods do not use any additional training loop like GAN or
adversarial training, which is not that feasible in practice. Additionally,
our method achieves better results compared to the previous methods [20],
[53], which are based on contrastive learning.

The second set of comparative experiments is based on SWAD [7], which is
presented in Table. 2. SWAD is a solid baseline that can be concatenated with
several DG methods. In the table, we compare our method to the current state-
of-the-art contrastive learning based method PCL [60] on PACS and VLCS
datasets. Our method achieves over a 1% increase on both of the datasets.

Moreover, we observe that our method outperforms other methods on two kinds of
target domains: The first kind of domains is the domains with adequate task-
specific domain variations, like the SUN09 (S) and VOC2007 (V) domains which
are sampled from the real-world in VLCS dataset. The second is the kinds of
domains with little or even no task-specific domain variations, like the Arts
(A) in PACS and Sketch (S) in PACS. While for those domains that have
unmatched task-specific domain variations, _e.g_., background information, to
the semantic objectives, like Cartoon (C) in PACS and Caltech101 (C) in VLCS,
our methods fail to achieve the best performance but still achieve competitive
performance. These phenomenons indicate that, when our model predicts an
image, it not only considers the domain invariant information but also
considers task-specific domain variations. That is why our model can
outperform many complex models that only focus on domain invariant
information.

Algorithm | VLCS | PACS | OfficeHome | TerraInc | DomainNet | Average
---|---|---|---|---|---|---
ERM [46] | 77.6 $\pm$ 0.3 | 86.7 $\pm$ 0.3 | 66.4 $\pm$ 0.5 | 53.0 $\pm$ 0.3 | 41.3 $\pm$ 0.1 | 65.0
IRM [1] | 76.9 $\pm$ 0.6 | 84.5 $\pm$ 1.1 | 63.0 $\pm$ 2.7 | 50.5 $\pm$ 0.7 | 28.0 $\pm$ 5.1 | 60.6
GroupDRO [37] | 77.4 $\pm$ 0.5 | 87.1 $\pm$ 0.1 | 66.2 $\pm$ 0.6 | 52.4 $\pm$ 0.1 | 33.4 $\pm$ 0.3 | 63.3
Mixup [59] | 78.1 $\pm$ 0.3 | 86.8 $\pm$ 0.3 | 68.0 $\pm$ 0.2 |  54.4 $\pm$ 0.3 | 39.6 $\pm$ 0.1 | 65.4
MLDG [23] | 77.5 $\pm$ 0.1 | 86.8 $\pm$ 0.4 | 66.6 $\pm$ 0.3 | 52.0 $\pm$ 0.1 | 41.6 $\pm$ 0.1 | 64.9
CORAL [40] | 77.7 $\pm$ 0.2 | 87.1 $\pm$ 0.5 | 68.4 $\pm$ 0.2 | 52.8 $\pm$ 0.2 | 41.8 $\pm$ 0.1 | 65.6
MMD [25] | 77.9 $\pm$ 0.1 | 87.2 $\pm$ 0.1 | 66.2 $\pm$ 0.3 | 52.0 $\pm$ 0.4 | 23.5 $\pm$ 9.4 | 61.4
DANN [11] | 79.7 $\pm$ 0.5 | 85.2 $\pm$ 0.2 | 65.3 $\pm$ 0.8 | 50.6 $\pm$ 0.4 | 38.3 $\pm$ 0.1 | 63.8
CDANN [27] | 79.9 $\pm$ 0.2 | 85.8 $\pm$ 0.8 | 65.3 $\pm$ 0.5 | 50.8 $\pm$ 0.6 | 38.5 $\pm$ 0.2 | 64.1
MTL [5] | 77.7 $\pm$ 0.5 | 86.7 $\pm$ 0.2 | 66.5 $\pm$ 0.4 | 52.2 $\pm$ 0.4 | 40.8 $\pm$ 0.1 | 64.8
SagNet [31] | 77.6 $\pm$ 0.1 | 86.4 $\pm$ 0.4 | 67.5 $\pm$ 0.2 | 52.5 $\pm$ 0.4 | 40.8 $\pm$ 0.2 | 65.0
ARM [63] | 77.8 $\pm$ 0.3 | 85.8 $\pm$ 0.2 | 64.8 $\pm$ 0.4 | 51.2 $\pm$ 0.5 | 36.0 $\pm$ 0.2 | 63.1
VREx [22] | 78.1 $\pm$ 0.2 | 87.2 $\pm$ 0.6 | 65.7 $\pm$ 0.3 | 51.4 $\pm$ 0.5 | 30.1 $\pm$ 3.7 | 62.5
RSC [18] | 77.8 $\pm$ 0.6 | 86.2 $\pm$ 0.5 | 66.5 $\pm$ 0.6 | 52.1 $\pm$ 0.2 | 38.9 $\pm$ 0.6 | 64.3
SelfReg [20] | 77.5 $\pm$ 0.0 | 86.5 $\pm$ 0.3 |  69.4 $\pm$ 0.2 | 51.0 $\pm$ 0.4 |  44.6 $\pm$ 0.1 | 65.8
DA-CORAL [10] | 78.5 $\pm$ 0.4 | 84.5 $\pm$ 0.7 | 68.9 $\pm$ 0.4 | 48.1 $\pm$ 0.3 |  43.9 $\pm$ 0.3 | 64.7
mDSDI [6] |  79.0 $\pm$ 0.3 | 86.2 $\pm$ 0.2 | 69.2 $\pm$ 0.4 | 48.1 $\pm$ 1.4 | 42.8 $\pm$ 0.1 | 65.0
ITL-Net [12] | 78.9 $\pm$ 0.7 | 86.4 $\pm$ 0.5 | 69.3 $\pm$ 0.4 | 51.0 $\pm$ 2.1 | 41.6 $\pm$ 0.3 | 65.5
Fishr [35] | 78.2 $\pm$ 0.2 | 86.9 $\pm$ 0.2 | 68.2 $\pm$ 0.2 |  53.6 $\pm$ 0.4 | 41.8 $\pm$ 0.2 | 65.7
DDN |  80.3 $\pm$ 0.2 |  87.2 $\pm$ 0.2 |  69.5 $\pm$ 0.2 | 50.4 $\pm$ 0.5 | 43.4 $\pm$ 0.1 | 66.2
Table 1: DomainBed accuracies (%). We utilize DomainBed [14] as the backbone. We average over 3 seeds and report standard deviations.  Algorithm | A | C | P | S | Avg. | C | L | S | V | Avg.
---|---|---|---|---|---|---|---|---|---|---
SWAD [7] | 89.3 | 83.4 | 97.3 | 82.5 | 88.1 | 98.4 | 63.6 | 72.0 | 74.5 | 77.1
SWAD + PCL [60] | 90.2 | 83.9 | 98.1 | 82.6 | 88.7 | 99.0 | 63.6 | 73.8 | 75.6 | 78.0
SWAD + DDN | 90.0 | 83.2 | 97.9 | 84.0 | 88.8 | 97.9 | 62.9 | 80.1 | 78.0 | 79.7
Table 2: SWAD accuracies (%). We also utilize SWAD [7] as the backbone on PACS
and VLCS datasets.

###  5.3 Visualization Results

We visualize the t-SNE results on several related works and our method. Notice
that the visualization is based on the training set of the PACS dataset. We
argue that in DG tasks, t-SNE presented on the unseen test dataset has
artificial bias due to the selection of the dataset. Thus we focus on the
uniformity and alignment measurement in self-supervised learning mentioned in
[54], as both DG and self-supervised model need to generalize to various
domains. In [54], the authors argue that (1) alignment (closeness) of features
from positive pairs and (2) uniformity of the induced distribution of the
features on the hypersphere can be seen as a measurement of representation
quality.

As shown in Fig. 6, from the perspective of alignment, our method is much
tighter within class compared to ERM [46], and SelfReg [20]. Also, The
uniformity of SelfReg and our method surpass the ERM methods, which is why
SelfReg performs better than ERM and worse than ours. The t-SNE results prove
that our domain expert features can aggregate compact clusters simultaneously
and hold a uniform distribution across source domains. Therefore, our model
can achieve a good classification result while proficiently mapping the
features from target domains to the domain expert space formed by the known
source domains.

Figure 6: From left to right, we propose the t-SNE visualizations of ERM [46],
SelfReg [20], and our method. Our approach is tighter within classes and more
uniform across classes.

###  5.4 Discrepancies among the Domains

Figure 7: Discrepancies between domains on PACS and VLCS.

In our methods, we disentangle the features from the source domains and
aggregate the domain expert features for unseen domains. The natural
distribution divergence can be measured by counting the weights among each
domain. The divergence is presented in Fig. 7. We randomly choose 128 cases
from target test domains and utilize the best model to evaluate the images and
access the weights.

For PACS, Arts (A) and Sketch (S) overlap greatly as both lack rich background
(can be seen as the main part of task-specific domain variations) information,
and our model may focus on the shape prior for classification. For VLCS,
Caltech101 (C) and SUN97 (S) remain a high overlap. When the test domain is C,
the model has a bigger chance to count on the classifier of S. However, the
background of C is always irrelevant to the semantic objects, which may
mislead the classifier to a wrong classification. Conversely, When the test
domain is S, the model has a bigger chance to count on the classifier of C. As
task-specific domain variations in C are misleading, the C classifier would
focus on the foreground. Thus, the test S domain can achieve an impressive
result.

###  5.5 Ablation Study

Ablation study on DPCL. We first analyze the effectiveness of the domain
prototype contrastive learning strategy. When we mute the strategy, the model
turns out to be an ensemble learning form. We utilize proxy-anchor loss [21]
and memory bank mechanism [15] to test the validity of our method. The result
is shown in Table. 3. The result proves that domain prototype contrastive
learning is an indispensable module.

Algorithm | A | C | P | S | Avg.
---|---|---|---|---|---
w/o DPCL | 87.3 | 81.1 | 97.2 | 82.4 | 87.0
\+ proxy-anchor | 86.4 | 79.9 | 96.7 | 81.6 | 86.2
\+ memory bank | 82.3 | 74.8 | 93.2 | 76.9 | 81.8
w/o shared classifier | 85.3 | 74.9 | 96.0 | 79.7 | 84.0
Ours | 87.4 | 80.6 | 97.1 | 83.8 | 87.2
Table 3: Ablation study on domain prototype contrastive learning strategy and
domain expert classifiers. The performance is much worse if we drop or change
these main components.

Ablation study on shared domain expert classifiers. Each domain should have an
independent classifier, thus can better fit the matched domain expert
features. We conduct the ablation study on the separated and shared
classifier. The results are shown in Table. 3 proves that the shared
classifier failed to be an expert classifier for all the domains. Therefore,
the use of a separated classifier can better generate classification-aware
features.

Ablation study on batch size.

Due to the sensitivity of contrastive learning to batch size, we also ablate
the batch size. The result is presented in Fig. 8. With the increase in batch
size from 8 to 64, we can see that the performance proliferates.

##  6 Limitations

Although we present a fundamental model for the DG problem and achieve
impressive results, there is an irreparable gap between target and source
domains. It is still challenging to disentangle features for some far
different domains from known source domains. However, this concern can be
further relieved by adding more domains in training to let the model see more
task-specific domain variations.

Figure 8: Ablation study of batch size on PACS.

##  7 Conclusion

We proposed a new perspective to utilize and disentangle domain expert
features which consist of domain invariant and task-specific domain
variations. In inference, our model effectively aggregates the knowledge from
known domains to represent the target domain. We build a novel and robust
network for the classification task in DG. Specially, we designed a
contrastive learning based paradigm to address task-specific domain variations
from the test domain. Our method outperforms the current methods and can
improve further if there are more source domains.

## References

  * [1] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz.  Invariant risk minimization.  arXiv preprint arXiv:1907.02893, 2019.
  * [2] Martin Arjovsky, Soumith Chintala, and Léon Bottou.  Wasserstein generative adversarial networks.  In International conference on machine learning, pages 214–223. PMLR, 2017.
  * [3] Sara Beery, Grant Van Horn, and Pietro Perona.  Recognition in terra incognita.  In Proceedings of the European conference on computer vision (ECCV), pages 456–473, 2018.
  * [4] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan.  A theory of learning from different domains.  Machine learning, 79(1):151–175, 2010.
  * [5] Gilles Blanchard, Aniket Anand Deshmukh, Ürun Dogan, Gyemin Lee, and Clayton Scott.  Domain generalization by marginal transfer learning.  The Journal of Machine Learning Research, 22(1):46–100, 2021.
  * [6] Manh-Ha Bui, Toan Tran, Anh Tran, and Dinh Phung.  Exploiting domain-specific features to enhance domain generalization.  Advances in Neural Information Processing Systems, 34:21189–21201, 2021.
  * [7] Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park.  Swad: Domain generalization by seeking flat minima.  Advances in Neural Information Processing Systems, 34:22405–22418, 2021.
  * [8] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  * [9] Xinlei Chen and Kaiming He.  Exploring simple siamese representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15750–15758, 2021.
  * [10] Abhimanyu Dubey, Vignesh Ramanathan, Alex Pentland, and Dhruv Mahajan.  Adaptive methods for real-world domain generalization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14340–14349, 2021.
  * [11] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky.  Domain-adversarial training of neural networks.  The journal of machine learning research, 17(1):2096–2030, 2016\.
  * [12] Boyan Gao, Henry Gouk, Yongxin Yang, and Timothy Hospedales.  Loss function learning for domain generalization by implicit gradient.  In International Conference on Machine Learning, pages 7002–7016. PMLR, 2022.
  * [13] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent-a new approach to self-supervised learning.  Advances in neural information processing systems, 33:21271–21284, 2020.
  * [14] Ishaan Gulrajani and David Lopez-Paz.  In search of lost domain generalization.  arXiv preprint arXiv:2007.01434, 2020.
  * [15] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729–9738, 2020.
  * [16] Shoubo Hu, Kun Zhang, Zhitang Chen, and Laiwan Chan.  Domain generalization via multidomain discriminant analysis.  In Uncertainty in Artificial Intelligence, pages 292–302. PMLR, 2020.
  * [17] Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu.  Fsdr: Frequency space domain randomization for domain generalization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6891–6902, 2021.
  * [18] Zeyi Huang, Haohan Wang, Eric P Xing, and Dong Huang.  Self-challenging improves cross-domain generalization.  In European Conference on Computer Vision, pages 124–140. Springer, 2020.
  * [19] Yunpei Jia, Jie Zhang, Shiguang Shan, and Xilin Chen.  Single-side domain generalization for face anti-spoofing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8484–8493, 2020.
  * [20] Daehee Kim, Youngjun Yoo, Seunghyun Park, Jinkyu Kim, and Jaekoo Lee.  Selfreg: Self-supervised contrastive regularization for domain generalization.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9619–9628, 2021.
  * [21] Sungyeon Kim, Dongwon Kim, Minsu Cho, and Suha Kwak.  Proxy anchor loss for deep metric learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3238–3247, 2020.
  * [22] David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville.  Out-of-distribution generalization via risk extrapolation (rex).  In International Conference on Machine Learning, pages 5815–5826. PMLR, 2021.
  * [23] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales.  Learning to generalize: Meta-learning for domain generalization.  In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
  * [24] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales.  Deeper, broader and artier domain generalization.  In Proceedings of the IEEE international conference on computer vision, pages 5542–5550, 2017.
  * [25] Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot.  Domain generalization with adversarial feature learning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5400–5409, 2018.
  * [26] Lei Li, Ke Gao, Juan Cao, Ziyao Huang, Yepeng Weng, Xiaoyue Mi, Zhengze Yu, Xiaoya Li, and Boyang Xia.  Progressive domain expansion network for single domain generalization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 224–233, 2021.
  * [27] Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao.  Deep domain generalization via conditional invariant adversarial networks.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 624–639, 2018.
  * [28] Massimiliano Mancini, Samuel Rota Bulo, Barbara Caputo, and Elisa Ricci.  Best sources forward: domain generalization through source-specific nets.  In 2018 25th IEEE international conference on image processing (ICIP), pages 1353–1357. IEEE, 2018.
  * [29] Saeid Motiian, Marco Piccirilli, Donald A Adjeroh, and Gianfranco Doretto.  Unified deep supervised domain adaptation and generalization.  In Proceedings of the IEEE international conference on computer vision, pages 5715–5725, 2017.
  * [30] Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf.  Domain generalization via invariant feature representation.  In International Conference on Machine Learning, pages 10–18. PMLR, 2013.
  * [31] Hyeonseob Nam, HyunJae Lee, Jongchan Park, Wonjun Yoon, and Donggeun Yoo.  Reducing domain gap by reducing style bias.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8690–8699, 2021.
  * [32] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  arXiv preprint arXiv:1807.03748, 2018.
  * [33] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang.  Moment matching for multi-source domain adaptation.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 1406–1415, 2019.
  * [34] Aayush Prakash, Shaad Boochoon, Mark Brophy, David Acuna, Eric Cameracci, Gavriel State, Omer Shapira, and Stan Birchfield.  Structured domain randomization: Bridging the reality gap by context-aware synthetic data.  In 2019 International Conference on Robotics and Automation (ICRA), pages 7249–7255. IEEE, 2019.
  * [35] Alexandre Rame, Corentin Dancette, and Matthieu Cord.  Fishr: Invariant gradient variances for out-of-distribution generalization.  In International Conference on Machine Learning, pages 18347–18377. PMLR, 2022.
  * [36] Philip N Sabes and Michael I Jordan.  Advances in neural information processing systems.  In In G. Tesauro & D. Touretzky & T. Leed (Eds.), Advances in Neural Information Processing Systems. Citeseer, 1995.
  * [37] Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang.  Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization.  arXiv preprint arXiv:1911.08731, 2019.
  * [38] Seonguk Seo, Yumin Suh, Dongwan Kim, Geeho Kim, Jongwoo Han, and Bohyung Han.  Learning to optimize domain specific normalization for domain generalization.  In European Conference on Computer Vision, pages 68–83. Springer, 2020.
  * [39] Yuge Shi, Jeffrey Seely, Philip HS Torr, N Siddharth, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve.  Gradient matching for domain generalization.  arXiv preprint arXiv:2104.09937, 2021.
  * [40] Baochen Sun and Kate Saenko.  Deep coral: Correlation alignment for deep domain adaptation.  In European conference on computer vision, pages 443–450. Springer, 2016.
  * [41] Chris Xing Tian, Haoliang Li, Xiaofei Xie, Yang Liu, and Shiqi Wang.  Neuron coverage-guided domain generalization.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022\.
  * [42] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola.  What makes for good views for contrastive learning?  Advances in Neural Information Processing Systems, 33:6827–6839, 2020.
  * [43] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel.  Domain randomization for transferring deep neural networks from simulation to the real world.  In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 23–30. IEEE, 2017.
  * [44] Antonio Torralba and Alexei A Efros.  Unbiased look at dataset bias.  In CVPR 2011, pages 1521–1528. IEEE, 2011.
  * [45] Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell.  Deep domain confusion: Maximizing for domain invariance.  arXiv preprint arXiv:1412.3474, 2014.
  * [46] Vladimir N Vapnik.  An overview of statistical learning theory.  IEEE transactions on neural networks, 10(5):988–999, 1999.
  * [47] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan.  Deep hashing network for unsupervised domain adaptation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5018–5027, 2017.
  * [48] Riccardo Volpi, Hongseok Namkoong, Ozan Sener, John C Duchi, Vittorio Murino, and Silvio Savarese.  Generalizing to unseen domains via adversarial data augmentation.  Advances in neural information processing systems, 31, 2018.
  * [49] Likang Wang.  Dionysus: Recovering scene structures by dividing into semantic pieces.
  * [50] Likang Wang and Lei Chen.  Ftso: Effective nas via first topology second operator.  arXiv preprint arXiv:2303.12948, 2023.
  * [51] Likang Wang, Yue Gong, Xinjun Ma, Qirui Wang, Kaixuan Zhou, and Lei Chen.  Is-mvsnet: Importance sampling-based mvsnet.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXII, pages 668–683. Springer, 2022.
  * [52] Likang Wang, Yue Gong, Qirui Wang, Kaixuan Zhou, and Lei Chen.  Flora: dual-frequency loss-compensated real-time monocular 3d video reconstruction.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 1, 2023.
  * [53] Shujun Wang, Lequan Yu, Caizi Li, Chi-Wing Fu, and Pheng-Ann Heng.  Learning from extrinsic and intrinsic supervisions for domain generalization.  In European Conference on Computer Vision, pages 159–176. Springer, 2020.
  * [54] Tongzhou Wang and Phillip Isola.  Understanding contrastive representation learning through alignment and uniformity on the hypersphere.  In International Conference on Machine Learning, pages 9929–9939. PMLR, 2020.
  * [55] Wenguan Wang, Tianfei Zhou, Fisher Yu, Jifeng Dai, Ender Konukoglu, and Luc Van Gool.  Exploring cross-image pixel contrast for semantic segmentation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7303–7313, 2021.
  * [56] Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li.  Dense contrastive learning for self-supervised visual pre-training.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3024–3033, 2021.
  * [57] Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu.  Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16684–16693, 2021.
  * [58] Qinwei Xu, Ruipeng Zhang, Ya Zhang, Yanfeng Wang, and Qi Tian.  A fourier-based framework for domain generalization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14383–14392, 2021.
  * [59] Shen Yan, Huan Song, Nanxiang Li, Lincan Zou, and Liu Ren.  Improve unsupervised domain adaptation with mixup training.  arXiv preprint arXiv:2001.00677, 2020.
  * [60] Xufeng Yao, Yang Bai, Xinyun Zhang, Yuechen Zhang, Qi Sun, Ran Chen, Ruiyu Li, and Bei Yu.  Pcl: Proxy-based contrastive learning for domain generalization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7097–7107, 2022.
  * [61] Dingyi Zeng, Wanlong Liu, Wenyu Chen, Li Zhou, Malu Zhang, and Hong Qu.  Substructure aware graph neural networks.  In Proc. of AAAI, 2023.
  * [62] Dingyi Zeng, Li Zhou, Wanlong Liu, Hong Qu, and Wenyu Chen.  A simple graph neural network via layer sniffer.  In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5687–5691. IEEE, 2022.
  * [63] Marvin Zhang, Henrik Marklund, Nikita Dhawan, Abhishek Gupta, Sergey Levine, and Chelsea Finn.  Adaptive risk minimization: Learning to adapt to domain shift.  Advances in Neural Information Processing Systems, 34:23664–23678, 2021.
  * [64] Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang.  Domain generalization with mixstyle.  arXiv preprint arXiv:2104.02008, 2021.