# Generalizing to Unseen Domains: A Survey on Domain Generalization

Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, , Wang Lu,
Yiqiang Chen, , Wenjun Zeng, , Philip S. Yu, J. Wang, C. Lan, C. Liu, W. Zeng,
and T. Qin are with Microsoft Research Asia, Beijing, China.
E-mail<EMAIL_ADDRESS>Y. Ouyang
is with Central University of Finance and Economics, China. W. Lu and Y. Chen
are with Institute of Computing Technology, Chinese Academy of Sciences,
Beijing, China. P. Yu is with University of Illinois at Chicago, and Institute
for Data Science, Tsinghua University. Email<EMAIL_ADDRESS>
###### Abstract

Machine learning systems generally assume that the training and testing
distributions are the same. To this end, a key requirement is to develop
models that can generalize to unseen distributions. Domain generalization
(DG), i.e., out-of-distribution generalization, has attracted increasing
interests in recent years. Domain generalization deals with a challenging
setting where one or several different but related domain(s) are given, and
the goal is to learn a model that can generalize to an _unseen_ test domain.
Great progress has been made in the area of domain generalization for years.
This paper presents the first review of recent advances in this area. First,
we provide a formal definition of domain generalization and discuss several
related fields. We then thoroughly review the theories related to domain
generalization and carefully analyze the theory behind generalization. We
categorize recent algorithms into three classes: data manipulation,
representation learning, and learning strategy, and present several popular
algorithms in detail for each category. Third, we introduce the commonly used
datasets, applications, and our open-sourced codebase for fair evaluation.
Finally, we summarize existing literature and present some potential research
topics for the future.

###### Index Terms:

Domain generalization, Domain adaptation, Transfer learning, Out-of-
distribution generalization

##  1 Introduction

Machine learning (ML) has achieved remarkable success in various areas, such
as computer vision, natural language processing, and healthcare. The goal of
ML is to design a model that can learn general and predictive knowledge from
training data, and then apply the model to new (test) data. Traditional ML
models are trained based on the _i.i.d._ assumption that training and testing
data are identically and independently distributed. However, this assumption
does not always hold in reality. When the probability distributions of
training data and testing data are different, the performance of ML models
often deteriorates due to domain distribution gaps [1]. Collecting the data of
all possible domains to train ML models is expensive and even prohibitively
impossible. Therefore, enhancing the _generalization_ ability of ML models is
important in both industry and academic fields.

There are many generalization-related research topics such as domain
adaptation, meta-learning, transfer learning, covariate shift, and so on. In
recent years, _Domain generalization (DG)_ has received much attention. As
shown in Figure 1, the goal of domain generalization is to learn a model from
one or several different but related domains (i.e., diverse training datasets)
that will generalize well on _unseen_ testing domains. For instance, given a
training set consisting of images coming from sketches, cartoon images and
paintings, domain generalization requires to train a good machine learning
model that has minimum prediction error in classifying images coming from
natural images or photos, which are clearly having distinguished distributions
from the images in training set. Over the past years, domain generalization
has made significant progress in various areas such as computer vision and
natural language processing. Despite the progress, there has not been a survey
in this area that comprehensively introduces and summarizes its main ideas,
learning algorithms and other related problems to provide research insights
for the future.

In this paper, we present the first survey on domain generalization to
introduce its recent advances, with special focus on its formulations,
theories, algorithms, research areas, datasets, applications, and future
research directions. We hope that this survey can provide a comprehensive
review for interested researchers and inspire more research in this and
related areas.

Figure 1: Examples from the dataset PACS [2] for domain generalization. The
training set is composed of images belonging to domains of sketch, cartoon,
and art paintings. DG aims to learn a generalized model that performs well on
the unseen target domain of photos.

This paper is a heavily extended version of our previously accepted short
paper at IJCAI-21 survey track (6 pages, included in the appendix file).
Compared to the short paper, this version makes the following extensions:

  * •

We present the theory analysis on domain generalization and the related domain
adaptation.

  * •

We substantially extend the methodology by adding new categories: e.g.,
causality-inspired methods, generative modeling for feature disentanglement,
invariant risk minimization, gradient operation-based methods, and other
learning strategies to comprehensively summarize these DG methods.

  * •

For all the categories, we broaden the analysis of methods by including more
related algorithms, comparisons, and discussion. And we also include more
recent papers (over 30% of new work).

  * •

We extend the scope of datasets and applications, and we also explore
evaluation standards to domain generalization. Finally, we build an open-
sourced codebase for DG research named _DeepDG_
111https://github.com/jindongwang/transferlearning/tree/master/code/DeepDG and
conduct some analysis of the results on public datasets.

There are several survey papers after the conference version of our paper, and
they are significantly different from ours. Concurrently, Zhou et al. [3] also
wrote a survey on DG, while their focus is on DG in computer vision field. A
more recent survey paper is on out-of-distribution (OOD) generalization by
Shen et al. [4]. Their work focused on causality and stable neural networks. A
related survey paper [5] is for OOD detection instead of building a working
algorithm that can be applied to any unseen environments.

This paper is organized as follows. We formulate the problem of domain
generalization and discuss its relationship with existing research areas in
Section 2. Section 3 presents the related theories in domain generalization.
In Section 4, we describe some representative DG methods in detail. In Section
5, we show some new DG research areas extended from the traditional setting.
Section 6 presents the applications and Section 7 introduces the benchmark
datasets for DG. We summarize the insights from existing work and present some
possible future directions in Section 8. Finally, we conclude this paper in
Section 9.

##  2 Background

###  2.1 Formalization of Domain Generalization

In this section, we introduce the notations and definitions used in this
paper.

######  Definition 1 (Domain).

Let $\mathcal{X}$ denote a nonempty input space and $\mathcal{Y}$ an output
space. A domain is composed of data that are sampled from a distribution. We
denote it as $\mathcal{S}=\\{(\mathbf{x}_{i},y_{i})\\}^{n}_{i=1}\sim P_{XY}$,
where $\mathbf{x}\in\mathcal{X}\subset\mathbb{R}^{d}$,
$y\in\mathcal{Y}\subset\mathbb{R}$ denotes the label, and $P_{XY}$ denotes the
joint distribution of the input sample and output label. $X$ and $Y$ denote
the corresponding random variables.

######  Definition 2 (Domain generalization).

As shown in Figure 2, in domain generalization, we are given $M$ training
(source) domains $\mathcal{S}_{train}=\\{\mathcal{S}^{i}\mid i=1,\cdots,M\\}$
where $\mathcal{S}^{i}=\\{(\mathbf{x}^{i}_{j},y^{i}_{j})\\}^{n_{i}}_{j=1}$
denotes the $i$-th domain. The joint distributions between each pair of
domains are different: $P^{i}_{XY}\neq P^{j}_{XY},1\leq i\neq j\leq M$. The
goal of domain generalization is to learn a robust and generalizable
predictive function $h:\mathcal{X}\to\mathcal{Y}$ from the $M$ training
domains to achieve a minimum prediction error on an _unseen_ test domain
$\mathcal{S}_{test}$ (i.e., $\mathcal{S}_{test}$ cannot be accessed in
training and $P^{test}_{XY}\neq P^{i}_{XY}\text{ for }i\in\\{1,\cdots,M\\}$):

$\min_{h}\,\mathbb{E}_{(\mathbf{x},y)\in\mathcal{S}_{test}}[\ell(h(\mathbf{x}),y)],$
(1)

where $\mathbb{E}$ is the expectation and $\ell(\cdot,\cdot)$ is the loss
function.

Figure 2: Illustration of domain generalization. Adapted from [6].

We list the frequently used notations in TABLE I.

TABLE I: Notations used in this paper.

Notation | Description | Notation | Description
---|---|---|---
$\mathbf{x},y$ | Instance/label | $\ell(\cdot,\cdot)$ | Loss function
$\mathcal{X},\mathcal{Y}$ | Feature/label space | $h$ | Predictive function
$\mathcal{S}$ | Domain | $g,f$ | Feature extractor/classifier
$P(\cdot)$ | Distribution | $\epsilon$ | Error (risk)
$\mathbb{E}[\cdot]$ | Expectation | $\theta$ | Model parameter
$M$ | Number of source domain | $n_{i}$ | Data size of source domain $i$

###  2.2 Related Research Areas

There are several research fields closely related to domain generalization,
including but not limited to: transfer learning, domain adaptation, multi-task
learning, multiple domain learning, meta-learning, lifelong learning, and
zero-shot learning. We summarize their differences with domain generalization
in TABLE II and briefly describe them in the following.

Multi-task learning [7] jointly optimizes models on several related tasks. By
sharing representations between these tasks, we could enable the model to
generalize better on the original task. Note that multi-task learning does not
aim to enhance the generalization to a new (unseen) task. Particularly, multi-
domain learning is a kind of multi-task learning, which trains on multiple
related domains to learn good models for each original domain [8] instead of
new test domains.

Transfer learning [9, 10, 11] trains a model on a source task and aims to
enhance the performance of the model on a different but related target
domain/task. Pretraining-finetuning is the commonly used strategy for transfer
learning where the source and target domains have different tasks and target
domain is accessed in training. In DG, the target domain cannot be accessed
and the training and test tasks are often the same while they have different
distributions.

TABLE II: Comparison between domain generalization and some related learning paradigms. Learning paradigm | Training data | Test data | Condition | Test access
---|---|---|---|---
Multi-task learning | $\mathcal{S}^{1},\cdots,\mathcal{S}^{n}$ | $\mathcal{S}^{1},\cdots,\mathcal{S}^{n}$ | $\mathcal{Y}^{i}\neq\mathcal{Y}^{j},1\leq i\neq j\leq n$ | $\checkmark$
Transfer learning |  $\mathcal{S}^{src}$, $\mathcal{S}^{tar}$ | $\mathcal{S}^{tar}$ | $\mathcal{Y}^{src}\neq\mathcal{Y}^{tar}$ | $\checkmark$
Domain adaptation | $\mathcal{S}^{src},\mathcal{S}^{tar}$ | $\mathcal{S}^{tar}$ | $P(\mathcal{X}^{src})\neq P(\mathcal{X}^{tar})$ | $\checkmark$
Meta-learning | $\mathcal{S}^{1},\cdots,\mathcal{S}^{n}$ | $\mathcal{S}^{n+1}$ | $\mathcal{Y}^{i}\neq\mathcal{Y}^{j},1\leq i\neq j\leq n+1$ | $\checkmark$
Lifelong learning | $\mathcal{S}^{1},\cdots,\mathcal{S}^{n}$ | $\mathcal{S}^{1},\cdots,\mathcal{S}^{n}$ |  $\mathcal{S}^{i}$ arrives sequentially | $\checkmark$
Zero-shot learning | $\mathcal{S}^{1},\cdots,\mathcal{S}^{n}$ | $\mathcal{S}^{n+1}$ | $\mathcal{Y}^{n+1}\neq\mathcal{Y}^{i},1\leq i\leq n$ | $\times$
Domain generalization | $\mathcal{S}^{1},\cdots,\mathcal{S}^{n}$ | $\mathcal{S}^{n+1}$ | $P(\mathcal{S}^{i})\neq P(\mathcal{S}^{j}),1\leq i\neq j\leq n+1$ | $\times$

Domain adaptation (DA) [12, 13] is also popular in recent years. DA aims to
maximize the performance on a given target domain using existing training
source domain(s). The difference between DA and DG is that DA has access to
the target domain data while DG cannot see them during training. This makes DG
more challenging than DA but more realistic and favorable in practical
applications.

Meta-learning [14, 15, 16] aims to learn the learning algorithm itself by
learning from previous experience or tasks, i.e., learning-to-learn. While the
learning tasks are different in meta-learning, the learning tasks are the same
in domain generalization. Meta-learning is a general learning strategy that
can be used for DG [17, 18, 19, 20] by simulating the meta-train and meta-test
tasks in training domains to enhance the performance of DG.

Lifelong Learning [21], or continual learning, cares about the learning
ability among multiple sequential domains/tasks. It requires the model to
continually learn over time by accommodating new knowledge while retaining
previously learned experiences. This is also different from DG since it can
access the target domain in each time step, and it does not explicitly handle
the different distributions across domains.

Zero-shot learning [22, 23] aims at learning models from seen classes and
classify samples whose categories are unseen in training. In contrast, domain
generalization in general studies the problem where training and testing data
are from the same classes but with different distributions.

##  3 Theory

In this section, we review some theories related to domain generalization.
Since domain adaptation is closely related to DG, we begin with the theory of
domain adaptation.

###  3.1 Domain Adaptation

For a binary classification problem, we denote the true labeling functions on
the source domain as ${h^{*}}^{s}:\mathcal{X}\rightarrow[0,1]$ 222When the
output is in $(0,1)$, it means the probability of $y=1$. and that on the
target domain as ${h^{*}}^{t}$. Let $h:\mathcal{X}\rightarrow[0,1]$ be any
classifier from a hypothesis space $\mathcal{H}$. The classification
difference on the source domain between two classifiers $h$ and $h^{\prime}$
can then be measured by

$\displaystyle\epsilon^{s}(h,h^{\prime})=\mathbb{E}_{\mathbf{x}\sim
P^{s}_{X}}[h(\mathbf{x})\neq
h^{\prime}(\mathbf{x})]=\mathbb{E}_{\mathbf{x}\sim
P^{s}_{X}}[|h(\mathbf{x})-h^{\prime}(\mathbf{x})|],$ (2)

and similarly we can define $\epsilon^{t}$ when taking $\mathbf{x}\sim
P^{t}_{X}$ in the expectation. We define
$\epsilon^{s}(h):=\epsilon^{s}(h,{h^{*}}^{s})$ and
$\epsilon^{t}(h):=\epsilon^{t}(h,{h^{*}}^{t})$ as the risk of classifier $h$
on the source and target domains, respectively.

The goal of DG/DA is to minimize the target risk $\epsilon^{t}(h)$, but it is
not accessible since we do not have any information on ${h^{*}}^{t}$. So
people seek to bound the target risk $\epsilon^{t}(h)$ using the tractable
source risk $\epsilon^{s}(h)$. Ben-David et al. [24] (Thm. 1) give a bound
relating the two risks:

$\displaystyle\epsilon^{t}(h)\leq{}$
$\displaystyle\epsilon^{s}(h)+2d_{1}(P^{s}_{X},P^{t}_{X})$ (3)
$\displaystyle{}+\min_{P_{X}\in\\{P^{s}_{X},P^{t}_{X}\\}}\mathbb{E}_{\mathbf{x}\sim
P_{X}}[|{h^{*}}^{s}(\mathbf{x})-{h^{*}}^{t}(\mathbf{x})|],$ (4)

where
$d_{1}(P^{s}_{X},P^{t}_{X}):=\sup_{\mathcal{A}\in\mathscr{X}}|P^{s}_{X}[\mathcal{A}]-P^{t}_{X}[\mathcal{A}]|$
is the _total variation_ between the two distributions, and $\mathscr{X}$
denotes the sigma-field on $\mathcal{X}$. The second term on the r.h.s
measures the difference of cross-domain distributions, and the third term
represents the difference in the labeling functions.

However, the total variation is a strong distance (i.e., it tends to be very
large) that may loosen the bound (4), and is hard to estimate using finite
samples. To address this, Ben-David et al. [24] developed another bound ([24],
Thm. 2; [25], Thm. 1):

$\displaystyle\epsilon^{t}(h)\leq{}$
$\displaystyle\epsilon^{s}(h)+d_{\mathcal{H}\Delta\mathcal{H}}(P^{s}_{X},P^{t}_{X})+\lambda_{\mathcal{H}},$
(5)

where the $\mathcal{H}\Delta\mathcal{H}$-divergence is defined as
$d_{\mathcal{H}\Delta\mathcal{H}}(P^{s}_{X},P^{t}_{X}):=\sup_{h,h^{\prime}\in\mathcal{H}}|\epsilon^{s}(h,h^{\prime})-\epsilon^{t}(h,h^{\prime})|$,
replacing the total variation $d_{1}$ to measure the distribution difference,
and the ideal joint risk
$\lambda_{\mathcal{H}}:=\inf_{h\in\mathcal{H}}\left[\epsilon^{s}(h)+\epsilon^{t}(h)\right]$
measures the complexity of $\mathcal{H}$ for the prediction tasks on the two
domains. $\mathcal{H}\Delta\mathcal{H}$-divergence has a better finite-sample
guarantee, leading to a non-asymptotic bound:

######  Theorem 1 (Domain adaptation error bound (non-asymptotic) [24] (Thm.
2)).

Let $d$ be the Vapnik–Chervonenkis (VC) dimension [26] of $\mathcal{H}$, and
$\mathcal{U}^{s}$ and $\mathcal{U}^{t}$ be unlabeled samples of size $n$ from
the two domains. Then for any $h\in\mathcal{H}$ and $\delta\in(0,1)$, the
following inequality holds with probability at least $1-\delta$:

$\displaystyle\epsilon^{t}(h)\leq{}$
$\displaystyle\epsilon^{s}(h)+{\hat{d}}_{\mathcal{H}\Delta\mathcal{H}}(\mathcal{U}^{s},\mathcal{U}^{t})+\lambda_{\mathcal{H}}$
(6) $\displaystyle{}+4\sqrt{\frac{2d\log(2n)+\log(2/\delta)}{n}},$ (7)

where
${\hat{d}}_{\mathcal{H}\Delta\mathcal{H}}(\mathcal{U}^{s},\mathcal{U}^{t})$ is
the estimate of $d_{\mathcal{H}\Delta\mathcal{H}}(P^{s}_{X},P^{t}_{X})$ on the
two sets of finite data samples.

In the above bounds, the domain distribution difference
$d(P^{s}_{X},P^{t}_{X})$ is not controllable, but one may learn a
representation function $g:\mathcal{X}\to\mathcal{Z}$ that maps the original
input data $\mathbf{x}$ to some representation space $\mathcal{Z}$, so that
the representation distributions of the two domains become closer. This
direction of DA is thus called DA based on domain-invariant representation
(DA-DIR). The theory of domain-invariant representations has since inspired
many DA/DG methods, which can be seen in Section 4.2.

###  3.2 Domain Generalization

####  3.2.1 Average risk estimation error bound

The first line of domain generalization theory considers the case where the
target domain is totally unknown (not even unsupervised data), and measures
the average risk over all possible target domains. Assume that all possible
target distributions follow an underlying hyper-distribution $\mathcal{P}$ on
$(\mathbf{x},y)$ distributions: $P^{t}_{XY}\sim\mathcal{P}$, and that the
source distributions also follow the same hyper-distribution:
$P^{1}_{XY},\cdots,P^{M}_{XY}\sim\mathcal{P}$. For generalization to any
possible target domain, the classifier to be learned in this case also
includes the domain information $P_{X}$ into its input, so prediction is in
the form $y=h(P_{X},\mathbf{x})$ on the domain with distribution $P_{XY}$. For
such a classifier $h$, its average risk over all possible target domains is
then given by:

$\displaystyle\mathcal{E}(h):=\mathbb{E}_{P_{XY}\sim\mathcal{P}}\mathbb{E}_{(\mathbf{x},y)\sim
P_{XY}}[\ell(h(P_{X},\mathbf{x}),y)],$ (8)

where $\ell$ is a loss function on $\mathcal{Y}$. Exactly evaluating the
expectations is impossible, but we can estimate it using finite
domains/distributions following $\mathcal{P}$, and finite $(\mathbf{x},y)$
samples following each distribution. As we have assumed
$P^{1}_{XY},\cdots,P^{M}_{XY}\sim\mathcal{P}$, the source domains and
supervised data could serve for this estimation:

$\displaystyle{\hat{\mathcal{E}}}(h):=\frac{1}{M}\sum_{i=1}^{M}\frac{1}{n^{i}}\sum_{j=1}^{n^{i}}\ell(h(\mathcal{U}^{i},\mathbf{x}^{i}_{j}),y^{i}_{j}),$
(9)

where we use the supervised dataset
$\mathcal{U}^{i}:=\\{\mathbf{x}^{i}_{j}\mid(\mathbf{x}^{i}_{j},y^{i}_{j})\in\mathcal{S}^{i}\\}$
from domain $i$ as an empirical estimation for $P^{i}_{X}$.

The first problem to consider is how well such an estimate approximates the
target $\mathcal{E}(h)$. This can be measured by the largest difference
between $\mathcal{E}(h)$ and ${\hat{\mathcal{E}}}(h)$ on some space of $h$. To
our knowledge, this is first analyzed by Blanchard et al. [6], where the space
of $h$ is taken as a reproducing kernel Hilbert space (RKHS). However,
different from common treatment, the classifier $h$ here also depends on the
distribution $P_{X}$, so the kernel defining the RKHS should be in the form
${\bar{k}}((P_{X}^{1},\mathbf{x}_{1}),(P_{X}^{2},\mathbf{x}_{2}))$. Blanchard
et al. [6] construct such a kernel using kernels $k_{X},k^{\prime}_{X}$ on
$\mathcal{X}$ and kernel $\kappa$ on the RKHS $\mathcal{H}_{k^{\prime}_{X}}$
of kernel $k^{\prime}_{X}$:
${\bar{k}}((P_{X}^{1},\mathbf{x}_{1}),(P_{X}^{2},\mathbf{x}_{2})):=\kappa(\Psi_{k^{\prime}_{X}}(P_{X}^{1}),\Psi_{k^{\prime}_{X}}(P_{X}^{2}))k_{X}(\mathbf{x}_{1},\mathbf{x}_{2})$,
where $\Psi_{k^{\prime}_{X}}(P_{X}):=\mathbb{E}_{\mathbf{x}\sim
P_{X}}[k^{\prime}_{X}(\mathbf{x},\cdot)]\in\mathcal{H}_{k^{\prime}_{X}}$ is
the kernel embedding of distribution $P_{X}$ via kernel $k^{\prime}$. The
result is given in the following theorem, which gives a bound on the largest
average risk estimation error within an origin-centered closed ball
$\mathcal{B}_{\mathcal{H}_{\bar{k}}}(r)$ of radius $r$ in the RKHS
$\mathcal{H}_{\bar{k}}$ of kernel ${\bar{k}}$, in a slightly simplified case
where $n^{1}=\cdots=n^{M}=:n$.

######  Theorem 2 (Average risk estimation error bound for binary
classification [6]).

Assume that the loss function $\ell$ is $L_{\ell}$-Lipschitz in its first
argument and is bounded by $B_{\ell}$. Assume also that the kernels
$k_{X},k^{\prime}_{X}$ and $\kappa$ are bounded by
$B_{k}^{2},B_{k^{\prime}}^{2}\geq 1$ and $B_{\kappa}^{2}$, respectively, and
the canonical feature map
$\Phi_{\kappa}:v\in\mathcal{H}_{k^{\prime}_{X}}\mapsto\kappa(v,\cdot)\in\mathcal{H}_{\kappa}$
of $\kappa$ is $L_{\kappa}$-Hölder of order $\alpha\in(0,1]$ on the closed
ball $\mathcal{B}_{\mathcal{H}_{k^{\prime}_{X}}}(B_{k^{\prime}})$ 333 This
means that for any
$u,v\in\mathcal{B}_{\mathcal{H}_{k^{\prime}_{X}}}(\mathcal{B}_{\mathcal{H}_{k^{\prime}}})$,
it holds that $\|\Phi_{\kappa}(u)-\Phi_{\kappa}(v)\|\leq
L_{\kappa}\|u-v\|^{\alpha}$, where the norms are of the respective RKHSs..
Then for any $r>0$ and $\delta\in(0,1)$, with probability at least $1-\delta$,
it holds that:

$\displaystyle\sup_{h\in\mathcal{B}_{\mathcal{H}_{\bar{k}}}(r)}\left|{\hat{\mathcal{E}}}(h)-\mathcal{E}(h)\right|\leq{}$
$\displaystyle C\bigg{(}B_{\ell}\sqrt{-M^{-1}\log\delta}$ (10)
$\displaystyle+rB_{k}L_{\ell}\Big{(}B_{k^{\prime}}L_{\kappa}\big{(}$
$\displaystyle
n^{-1}\log(M/\delta)\big{)}^{\alpha/2}+B_{\kappa}/\sqrt{M}\Big{)}\bigg{)},$
(11)

where $C$ is a constant.

The bound becomes larger in general if $(M,n)$ is replaced with $(1,Mn)$. It
indicates that using domain-wise datasets is better than just pooling them
into one mixed dataset, so the domain information plays a role. This result is
later extended in [27], and Deshmukh et al. [28] give a bound for multi-class
classification in a similar form.

for tree= grow=east, reversed=true, anchor=base west, parent anchor=east,
child anchor=west, base=left, font=, rectangle, draw, rounded
corners,align=left, minimum width=2.5em, inner xsep=4pt, inner ysep=1pt, ,
where level=1text width=5em,fill=blue!10, where level=2text
width=5em,font=,fill=pink!30, where level=3font=,yshift=0.26pt,fill=yellow!20,
[Domain
generalization
algorithms,fill=green!20 [Data
manipulation
(Sec. 4.1),text width=6em [Data augmentation,text width=8em [Domain
randomization [29, 30, 31, 32, 33, 34, 35, 36] ] [Adversarial data
augmentation [37, 38, 39] ] ] [Data generation,text width=7em [[40, 41, 42,
43, 44, 45, 35, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58] ] ] ]
[Representation
learning
(Sec. 4.2),text width=6em [Domain-invariant
representation learning,text width=8.5em [Kernel methods [6, 59, 60, 27, 61,
62, 63, 64, 65, 28] ] [Explicit feature alignment [66, 67, 68, 69, 70, 71, 72,
73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87] ] [Domain
adversarial learning [88, 89, 71, 90, 91, 92, 93, 47, 94, 95, 96, 97, 98, 99,
100, 85] ] [Invariant risk minimization [101, 102, 103, 104, 105, 106, 107,
108, 109] ] ] [Feature disentanglement, text width=10em [Multi-component
analysis [110, 111, 112, 113, 2, 114, 115, 116, 117, 118, 119, 120] ]
[Generative modeling [121, 122, 41, 123, 124, 125, 126, 127] ] [Causality-
inspired methods [81, 109, 128, 129, 130, 131, 132, 133] ] ] ] [Learning
strategy
(Sec. 4.3),text width=6em [Ensemble learning,text width=7em [[134, 135, 136,
8, 137, 138, 139]] ] [Meta-learning, text width=6em [[17, 18, 19, 140, 20,
141, 142, 143, 46, 144, 145, 146, 147]] ] [Gradient operation, text width=8em
[[148, 149, 150, 151]] ] [Distributionally robust optimization, text
width=14em [[152, 108, 153, 154, 155]] ] [Self-supervised learning, text
width=9em [[156, 157, 158, 159]] ] [Others, text width=2.5em [Random forest
[160] Shape-invariant [161] Lottery ticket hypothesis [162] Flat minima [163]
Metric learning [164] ] ] ] ]

Figure 3: Taxonomy of domain generalization methods.

####  3.2.2 Generalization risk bound

Another line of DG theory considers the risk on a specific target domain,
under the assumption of covariate shift (i.e., the labeling function $h^{*}$
or $P_{Y|X}$ is the same over all domains). This measurement is similar to
what is considered in domain adaptation theory in Section 3.1, so we adopt the
same definition for the source risks $\epsilon^{1},\cdots,\epsilon^{M}$ and
the target risk $\epsilon^{t}$. With the covariate shift assumption, each
domain is characterized by the distribution on $\mathcal{X}$. Albuquerque et
al. [96] then consider approximating the target domain distribution
$P^{t}_{X}$ within the convex hull of source domain distributions:
$\Lambda:=\\{\sum_{i=1}^{M}\pi_{i}P^{i}_{X}\mid\pi\in\Delta_{M}\\}$, where
$\Delta_{M}$ is the $(M-1)$-dimensional simplex so that each $\pi$ represents
a normalized mixing weights. Similar to the domain adaptation case,
distribution difference is measured by the $\mathcal{H}$-divergence to include
the influence of the classifier class.

######  Theorem 3 (Domain generalization error bound [96]).

Let
$\gamma:=\min_{\pi\in\Delta_{M}}d_{\mathcal{H}}(P^{t}_{X},\sum_{i=1}^{M}\pi_{i}P^{i}_{X})$
with minimizer $\pi^{*}$ 444The original presentation does not mention that
the $\pi$ is the minimizer, but the proof indicates so. be the distance of
$P^{t}_{X}$ from the convex hull $\Lambda$, and
$P^{*}_{X}:=\sum_{i=1}^{M}\pi^{*}_{i}P^{i}_{X}$ be the best approximator
within $\Lambda$. Let
$\rho:=\sup_{P^{\prime}_{X},P^{\prime\prime}_{X}\in\Lambda}d_{\mathcal{H}}(P^{\prime}_{X},P^{\prime\prime}_{X})$
be the diameter of $\Lambda$. Then it holds that

$\displaystyle\epsilon^{t}(h)\leq\sum_{i=1}^{M}\pi^{*}_{i}\epsilon^{i}(h)+\frac{\gamma+\rho}{2}+\lambda_{\mathcal{H},(P^{t}_{X},P^{*}_{X})},$
(12)

where $\lambda_{\mathcal{H},(P^{t}_{X},P^{*}_{X})}$ is the ideal joint risk
across the target domain and the domain with the best approximator
distribution $P^{*}_{X}$.

The result can be seen as the generalization of domain adaptation bounds in
Section 3.1 when there are multiple source domains. Again similar to the
domain adaptation case, this bound motivates domain generalization methods
based on domain invariant representation, which simultaneously minimize the
risks over all source domains corresponding to the first term of the bound, as
well as the representation distribution differences among source and target
domains in the hope to reduce $\gamma$ and $\rho$ on the representation space.
To sum up, the theory of generalization is an active research area and other
researchers also derived different DG theory bounds using informativeness
[165] and adversarial training [96, 95, 165, 28].

##  4 Methodology

In this section, we introduce existing domain generalization methods in
detail. As shown in Figure 3, we categorize them into three groups, namely:

  1. (1)

Data manipulation: This category of methods focuses on manipulating the inputs
to assist learning general representations. Along this line, there are two
kinds of popular techniques: a). _Data augmentation_ , which is mainly based
on augmentation, randomization, and transformation of input data; b). _Data
generation_ , which generates diverse samples to help generalization.

  2. (2)

Representation learning: This category of methods is the most popular in
domain generalization. There are two representative techniques: a). _Domain-
invariant representation learning_ , which performs kernel, adversarial
training, explicitly feature alignment between domains, or invariant risk
minimization to learn domain-invariant representations; b). _Feature
disentanglement_ , which tries to disentangle the features into domain-shared
or domain-specific parts for better generalization.

  3. (3)

Learning strategy: This category of methods focuses on exploiting the general
learning strategy to promote the generalization capability, which mainly
includes several kinds of methods: a). _Ensemble learning_ , which relies on
the power of ensemble to learn a unified and generalized predictive function;
b). _Meta-learning_ , which is based on the learning-to-learn mechanism to
learn general knowledge by constructing meta-learning tasks to simulate domain
shift; c). _Gradient operation_ , which tries to learn generalized
representations by directly operating on gradients; d). _Distributionally
robust optimization_ , which learns the worst-case distribution scenario of
training domains; e). _Self-supervised learning_ , which constructs pretext
tasks to learn generalized representations. Additionally, there are other
learning strategy that can be used for DG.

These three categories of approaches are conceptually different. They are
complementary to each other and can be combined towards higher performance. We
will describe the approaches for each category in detail hereafter.

###  4.1 Data Manipulation

We are always hungry for more training data in machine learning (ML). The
generalization performance of a ML model often relies on the quantity and
diversity of the training data. Given a limited set of training data, data
manipulation is one of the cheapest and simplest way to generate samples so as
to enhance the generalization capability of the model. The main objective for
data manipulation-based DG is to increase the diversity of existing training
data using different data manipulation methods. At the same time, the data
quantity is also increased. Although the theoretical insight for why data
augmentation or generation techniques can enhance the generalization ability
of a model, experiments by Adila and Kang [166] showed that the model tend to
make predictions for both OOD and in-distribution samples based on trivial
syntactic heuristics for NLP tasks.

We formulate the general learning objective of data manipulation-based DG as:

$\min_{h}\,\mathbb{E}_{\mathbf{x},y}[\ell(h(\mathbf{x}),y)]+\mathbb{E}_{\mathbf{x}^{\prime},y}[\ell(h(\mathbf{x}^{\prime}),y)],$
(13)

where $\mathbf{x}^{\prime}=\operatorname{mani}(\mathbf{x})$ denotes the
manipulated data using a function $\operatorname{mani}(\cdot)$. Based on the
difference on this function, we further categorize existing work into two
types: _data augmentation_ and _data generation_.

####  4.1.1 Data augmentation-based DG

Augmentation is one of the most useful techniques for training machine
learning models. Typical augmentation operations include flipping, rotation,
scaling, cropping, adding noise, and so on. They have been widely used in
supervised learning to enhance the generalization performance of a model by
reducing overfitting [167, 35]. Without exception, they can also be adopted
for DG where $\operatorname{mani}(\cdot)$ can be instantiated as these data
augmentation functions.

##### Domain randomization

Other than typical augmentation, domain randomization is an effective
technique for data augmentation. It is commonly done by generating new data
that can simulate complex environments based on the limited training samples.
Here, the $\operatorname{mani}(\cdot)$ function is implemented as several
manual transformations (commonly used in image data) such as: altering the
location and texture of objects, changing the number and shape of objects,
modifying the illumination and camera view, and adding different types of
random noise to the data. Tobin et al. [30] first used this method to generate
more training data from the simulated environment for generalization in the
real environment. Similar techniques were also used in [31, 32, 33, 29] to
strengthen the generalization capability of the models. Prakash et al. [34]
further took into account the structure of the scene when randomly placing
objects for data generation, which enables the neural network to learn to
utilize context when detecting objects. Peng et al. [56] proposed to not only
augment features, but also labels. It is easy to see that by randomization,
the diversity of samples can be increased. But randomization is often random,
indicating that there could be some useless randomizations that could be
further removed to improve the efficiency of the model.

##### Adversarial data augmentation

Adversarial data augmentation aims to guide the augmentation to optimize the
generalization capability, by enhancing the diversity of data while assuring
their reliability. Shankar et al. [37] used a Bayesian network to model
dependence between label, domain and input instance, and proposed CrossGrad, a
cautious data augmentation strategy that perturbs the input along the
direction of greatest domain change while changing the class label as little
as possible. Volpi et al. [38] proposed an iterative procedure that augments
the source dataset with examples from a fictitious target domain that is
“hard” under the current model, where adversarial examples are appended at
each iteration to enable adaptive data augmentation. Zhou et al. [39]
adversarially trained a transformation network for data augmentation instead
of directly updating the inputs by gradient ascent while they adopted the
regularization of weak and strong augmentation in [168, 36]. Adversarial data
augmentation often has certain optimization goals that can be used by the
network. However, its optimization process often involves adversarial
training, thus is difficult.

####  4.1.2 Data generation-based DG

Data generation is also a popular technique to generate diverse and rich data
to boost the generalization capability of a model. Here, the function
$\operatorname{mani}(\cdot)$ can be implemented using some generative models
such as Variational Auto-encoder (VAE) [169], and Generative Adversarial
Networks (GAN) [170]. In addition, it can also be implemented using the Mixup
[171] strategy.

Rahman et al. [40] used ComboGAN [172] to generate new data and then applied
domain discrepancy measure such as MMD [173] to minimize the distribution
divergence between real and generated images to help learn general
representations. Qiao et al. [41] leveraged adversarial training to create
“fictitious” yet “challenging” populations, where a Wasserstein Auto-Encoder
(WAE) [174] was used to help generate samples that preserve the semantic and
have large domain transportation. Zhou et al. [44] generated novel
distributions under semantic consistency and then maximized the difference
between source and the novel distributions. Somavarapu et al. [45] introduced
a simple transformation based on image stylization to explore cross-source
variability for better generalization, where AdaIN [175] was employed to
achieve fast stylization to arbitrary styles. Different from others, Li et al.
[54] used adversarial training to generate _domains_ instead of samples. These
methods are more complex since different generative models are involved and we
should pay attention to the model capacity and computing cost.

In addition to the above generative models, Mixup [171] is also a popular
technique for data generation. Mixup generates new data by performing linear
interpolation between any two instances and between their labels with a weight
sampled from a Beta distribution, which does not require to train generative
models. Recently, there are several methods using Mixup for DG, by either
performing Mixup in the original space [49, 50, 57] to generate new samples;
or in the feature space [51, 145] which does not explicitly generate raw
training samples. These methods achieved promising performance on popular
benchmarks while remaining conceptually and computationally simple.

###  4.2 Representation Learning

Representation learning has always been the focus of machine learning for
decades [176] and is also one of the keys to the success of domain
generalization. We decompose the prediction function $h$ as $h=f\circ g$,
where $g$ is a representation learning function and $f$ is the classifier
function. The goal of representation learning can be formulated as:

$\min_{f,g}\,\mathbb{E}_{\mathbf{x},y}\ell(f(g(\mathbf{x})),y)+\lambda\ell_{\operatorname{reg}},$
(14)

where $\ell_{\operatorname{reg}}$ denotes some regularization term and
$\lambda$ is the tradeoff parameter. Many methods are designed to better learn
the feature extraction function $g$ with corresponding
$\ell_{\operatorname{reg}}$. In this section, we categorize the existing
literature on representation learning into two main categories based on
different learning principles: _domain-invariant representation learning_ and
_feature disentanglement_.

####  4.2.1 Domain-invariant representation-based DG

The work of [177] theoretically proved that if the feature representations
remain invariant to different domains, the representations are general and
transferable to different domains (also refer to Section 3). Based on this
theory, a plethora of algorithms have been proposed for domain adaptation.
Similarly, for domain generalization, the goal is to reduce the representation
discrepancy between multiple source domains in a specific feature space to be
domain invariant so that the learned model can have a generalizable capability
to the unseen domain. Along this line, there are mainly four types of methods:
kernel-based methods, domain adversarial learning, explicit feature alignment,
and invariant risk minimization.

##### Kernel-based methods

Kernel-based method is one of the most classical learning paradigms in machine
learning. Kernel-based machine learning relies on the kernel function to
transform the original data into a high-dimensional feature space without ever
computing the coordinates of the data in that space, but by simply computing
the inner products between the samples of all pairs in the feature space. One
of the most representative kernel-based methods is Support Vector Machine
(SVM) [178]. For domain generalization, there are plenty of algorithms based
on kernel methods, where the representation learning function $g$ is
implemented as some feature map $\phi(\cdot)$ which is easily computed using
kernel function $k(\cdot,\cdot)$ such as RBF kernel and Laplacian kernel.

Blanchard et al. [6] first used kernel method for domain generalization and
extended it in [59]. They adopted the positive semi-definite kernel learning
to learn a domain-invariant kernel from the training data. Grubinger et al.
[60] adapted transfer component analysis (TCA) [179] to bridge the multi-
domain distance to be closer for DG. Similar to the core idea of TCA, Domain-
Invariant Component Analysis (DICA) [27] is one of the classic methods using
kernel for DG. The goal of DICA is to find a feature transformation kernel
$k(\cdot,\cdot)$ that minimizes the distribution discrepancy between all data
in feature space. Gan et al. [61] adopted a similar method as DICA and further
added attribute regularization. In contrast to DICA which deals with the
marginal distribution, Li et al. [62] learned a feature representation which
has domain-invariant class conditional distribution. Scatter component
analysis (SCA) [63] adopted Fisher’s discriminant analysis to minimize the
discrepancy of representations from the same class and the same domain, and
maximize the discrepancy of representations from the different classes and
different domains. Erfani et al. [64] proposed an Elliptical Summary
Randomisation (ESRand) that comprises of a randomised kernel and elliptical
data summarization. ESRand projected each domain into an ellipse to represent
the domain information and then used some similarity metric to compute the
distance. Hu et al. [65] proposed multi-domain discriminant analysis to
perform class-wise kernel learning for DG, which is more fine-grained. To sum
up, this category of methods is often highly related to other categories to
act as their divergence measures or theoretical support.

##### Domain adversarial learning

Domain-adversarial training is widely used for learning domain-invariant
features. Ganin and Lempitsky [88] and Ganin et al. [89] proposed Domain-
adversarial neural network (DANN) for domain adaptation, which adversarially
trains the generator and discriminator. The discriminator is trained to
distinguish the domains while the generator is trained to fool the
discriminator to learn domain invariant feature representations. Li et al.
[71] adopted such idea for DG. Gong et al. [90] used adversarial training by
gradually reducing the domain discrepancy in a manifold space. Li et al. [91]
proposed a conditional invariant adversarial network (CIAN) to learn class-
wise adversarial networks for DG. Similar ideas were also used in [92, 97,
98]. Jia et al. [99] used single-side adversarial learning and asymmetric
triplet loss to make sure only the real faces from different domains were
indistinguishable, but not for the fake ones. In addition to adversarial
domain classification, Zhao et al. [93] introduced additional entropy
regularization by minimizing the KL divergence between the conditional
distributions of different training domains to push the network to learn
domain-invariant features. Some other GAN-based methods [47, 95, 96] were also
proposed with theoretically guaranteed generalization bound.

##### Explicit feature alignment

This line of works aligns the features across source domains to learn domain-
invariant representations through explicit feature distribution alignment [71,
180, 181], or feature normalization [182, 183, 184, 69]. Motiian et al. [67]
introduced a cross-domain contrastive loss for representation learning, where
mapped domains are semantically aligned and yet maximally separated. Some
methods explicitly minimized the feature distribution divergence by minimizing
the maximum mean discrepancy (MMD) [185, 179, 186, 187], second order
correlation [188, 189, 190], both mean and variance (moment matching) [181],
Wasserstein distance [180], etc, of domains for either domain adaptation or
domain generalization. Zhou et al. [180] aligned the marginal distribution of
different source domains via optimal transport by minimizing the Wasserstein
distance to achieve domain-invariant feature space.

Moreover, there are some works that used feature normalization techniques to
enhance domain generalization capability [182, 183]. Pan et al. [182]
introduced Instance Normalization (IN) layers to CNNs to improve the
generalization capability of models. IN has been extensively investigated in
the field of image style transfer [191, 192, 175], where the style of an image
is reflected by the IN parameters, i.e., mean and variance of each feature
channel. Thus, IN layers [193] could be used to eliminate instance-specific
style discrepancy to enhance generalization [182]. However, IN is task
agnostic and may remove some discriminative information. In IBNNet, IN and
Batch Normalization (BN) are utilized in parallel to preserve some
discriminative information [182]. In [184], BN layers are replaced by Batch-
Instance Normalization (BIN) layers, which adaptively balance BN and IN for
each channel by selectively using BN and IN. Jin et al. [69, 70] proposed a
Style Normalization and Restitution (SNR) module to simultaneously ensure both
high generalization and discrimination capability of the networks. After the
style normalization by IN, a restitution step is performed to distill task-
relevant discriminative features from the residual (i.e., the difference
between the original feature and the style normalized feature) and add them
back to the network to ensure high discrimination. The idea of restitution is
extended to other alignment-based method to restorate helpful discriminative
information dropped by alignment [87]. Recently, Qi et al. [84] applied IN to
unsupervised DG where there are no labels in the training domains to acquire
invariant and transferable features. A combination of different normalization
techniques is presented in [85] to show that adaptively learning the
normalization technique can improve DG. This category of methods is more
flexible and can be applied to other kind of categories.

##### Invariant risk minimization (IRM)

Arjovsky et al. [101] considered another perspective on the domain-invariance
of representation for domain generalization. They did not seek to match the
representation distribution of all domains, but to enforce the optimal
classifier on top of the representation space to be the same across all
domains. The intuition is that the ideal representation for prediction is the
_cause_ of $y$, and the causal mechanism should not be affected by other
factors/mechanisms, thus is domain-invariant. Formally, IRM can be formulated
as:

$\displaystyle\min_{\begin{subarray}{c}g\in\mathcal{G},\\\
f\in\bigcap_{i=1}^{M}\arg\min_{f^{\prime}\in\mathcal{F}}\epsilon^{i}(f^{\prime}\circ
g)\end{subarray}}\sum_{i=1}^{M}\epsilon^{i}(f\circ g)$ (15)

for some function classes $\mathcal{F}$ of $g$ and $\mathcal{F}$ of $f$. The
constraint for $f$ embodies the desideratum that all domains share the same
representation-level classifier, and the objective function encourages $f$ and
$g$ to achieve a low source domain risk. However, this problem is hard to
solve as it involves an inner-level optimization problem in its constraint.
The authors then develop a surrogate problem to learn the feature extractor
$g$ that is much more practical:

$\displaystyle\min_{g\in\mathcal{G}}\sum_{i=1}^{M}\epsilon^{i}(g)+\lambda\left\|\left.\nabla_{f}\epsilon^{i}(f\circ
g)\right|_{f=1}\right\|^{2},$ (16)

where a dummy representation-level classifier $f=1$ is considered, and the
gradient norm term measures the optimality of this classifier. The work also
presents a generalization theory under a perhaps strong linear assumption,
that for plenty enough source domains, the ground-truth invariant classifier
can be identified.

IRM has gain notable visibility recently. There are some further theoretical
analyses on the success [107] and failure cases of IRM [105], and IRM has been
extended to other tasks including text classification [104] and reinforcement
learning [194]. The idea to pursue the invariance of optimal representation-
level classifier is also extended. Krueger et al. [108] promote this
invariance by minimizing the extrapolated risk among source domains, which
essentially minimizes the variance of source-domain risks. Mitrovic et al.
[109] aim to learn such a representation in a self-supervised setup, where the
second domain is constructed by data augmentation showing various semantic-
irrelevant variations. Recently, Ahuja et al. [106] found the invariance of
$f$ alone is not sufficient. They found IRM still fails if $g$ captures “fully
informative invariant features”, which makes $y$ independent of $x$ on all
domains. This is particularly the case for classification (vs. regression)
tasks. An information bottleneck regularization is hence introduced to
maintain only partially informative features.

####  4.2.2 Feature disentanglement-based DG

The goal of disentangled representation learning is to learn a function that
maps a sample to a feature vector that contains all the information about
different factors of variation, with each dimension (or a subset of
dimensions) containing information about only some factor(s). Disentanglement
based DG approaches in general decompose a feature representation into
understandable compositions/sub-features, with one feature being domain-
shared/invariant feature and the other domain-specific feature. The
optimization objective of disentanglement-based DG can be summarized as:

$\min_{g_{c},g_{s},f}\mathbb{E}_{\mathbf{x},y}\ell(f(g_{c}(\mathbf{x})),y)+\lambda\ell_{\mathrm{reg}}+\mu\ell_{\mathrm{recon}}([g_{c}(\mathbf{x}),g_{s}(\mathbf{x})],\mathbf{x}),$
(17)

where $g_{c}$ and $g_{s}$ denote the domain-shared and domain-specific feature
representations, respectively. $\lambda$ and $\mu$ are tradeoff parameters.
The loss $\ell_{\mathrm{reg}}$ is a regularization term that explicitly
encourages the separation of the domain shared and specific features and
$\ell_{\mathrm{recon}}$ denotes a reconstruction loss that prevents
information loss. Note that $[g_{c}(\mathbf{x}),g_{s}(\mathbf{x})]$ denotes
the combination of two kinds of features (which is not limited to
concatenation operation).

Based on the choice of network structures and implementation mechanisms,
disentanglement-based DG can mainly be categorized into three types: _multi-
component analysis_ , _generative modeling_ , and _causality-inspired
methods_.

##### Multi-component analysis

In multi-component analysis, the domain-shared and domain-specific features
are in general extracted using the domain-shared and domain-specific network
parameters. The method of UndoBias [111] started from a SVM model to maximize
interval classification on all training data for domain generalization. They
represented the parameters of the $i$-th domain as
$\mathbf{w}_{i}=\mathbf{w}_{0}+\Delta_{i}$, where $\mathbf{w}_{0}$ denotes the
domain-shared parameters and $\Delta_{i}$ denotes the domain-specific
parameters. Some other methods extented the idea of UndoBias from different
aspects. Niu et al. [112] proposed to use multi-view learning for domain
generalization. They proposed Multi-view DG (MVDG) to learn the combination of
exemplar SVMs under different views for robust generalization. Ding and Fu
[113] designed domain-specific networks for each domain and one shared domain-
invariant network for all domains to learn disentangled representations, where
low-rank reconstruction is adopted to align two types of networks in
structured low-rank fashion. Li et al. [2] extended the idea of UndoBias into
the neural network context and developed a low-rank parameterized CNN model
for end-to-end training. Zunino et al. [114] learned disentangled
representations through manually comparing the attention heat maps for certain
areas from different domains. There are also other works that adopt multi-
component analysis for disentanglement [115, 110, 116, 117, 119, 195, 196,
197, 120]. In general, multi-component analysis can be implemented in
different architectures and remains effective for representation
disentanglement.

##### Generative modeling

Generative models can be used for disentanglement from the perspective of data
generation process. This kind of methods tries to formulate the generative
mechanism of the samples from the domain-level, sample-level, and label-level.
Some works further disentangle the input into class-irrelevant features, which
contain the information related to specific instance [198]. The Domain-
invariant variational autoencoder (DIVA) [121] disentangled the features into
domain information, category information, and other information, which is
learned in the VAE framework. Peng et al. [122] disentangled the fine-grained
domain information and category information that are learned in VAEs. Qiao et
al. [41] also used VAE for disentanglement, where they proposed a Unified
Feature Disentanglement Network (UFDN) that treated both data domains and
image attributes of interest as latent factors to be disentangled. Similarly,
Zhang et al. [123] disentangled the semantic and variational part of the
samples. Similar spirits also include [125, 126]. Nam et al. [124] proposed to
disentangle the style and other information using generative models that their
method is both for domain adaptation and domain generalization. Generative
models can not only improve OOD performance, but can also be used for
generation tasks, which we believe is useful for many potential applications.

##### Causality-inspired methods

Causality is a finer description of variable relations beyond statistics
(joint distribution). Causal relation gives information of how the system will
behave under intervention, so it is naturally suitable for transfer learning
tasks, since domain shift can be seen as an intervention. Particularly, under
the causal consideration, the desired representation is the true cause of the
label (e.g., object shape), so that prediction will not be affected by
intervention on correlated but semantically irrelevant features (e.g.,
background, color, style). There are a number of works [199, 200, 201, 202]
that exploited causality for domain adaptation.

For domain generalization, He et al. [132] reweighted input samples in a way
to make the weighted correlation reflect causal effect. Zhang et al. [133]
took Fourier features as the causing factors of images, and enforce the
independence among these features. Using the additional data of object
identity (it is a more detailed label than the class label), [203] enforced
the conditional independence of the representation from domain index given the
same object. When such object label is unavailable, [81] further learned an
object feature based on labels in a separate stage. For single-source domain
generalization, [109, 128] used data augmentation to present information of
the causal factor. The augmentation operation is seen as producing outcomes
under intervention on irrelevant features, which is implemented based on
specific domain knowledge. There are also generative methods under the causal
consideration. Zhang et al. [129] explicitly modeled a manipulation variable
that causes domain shift, which may be unobserved. Liu et al. [130] leveraged
causal invariance for single-source generalization, i.e., the invariance of
the process of generating $(x,y)$ data based on the factors, which is
explained more general than inference invariance that existing methods
implicitly rely on. The two factors are allowed correlated which is more
realistic. They theoretically prove the identifiability of the causal factor
is possible and the identification benefits generalization. Sun et al. [131]
extended the method and theory to multiple source domains. With more
informative data, the irrelevant factor is also identifiable.

###  4.3 Learning Strategy

In addition to data manipulation and representation learning, DG was also
studied in general machine learning paradigms, which is divided into several
categories: _ensemble learning-based DG_ , _meta-learning-based DG_ ,
_gradient operation-based DG_ , _distributionally robust optimization-based
DG_ , _self-supervised learning-based DG_ , and _other strategies_.

####  4.3.1 Ensemble learning-based DG

Ensemble learning usually combines multiple models, such as classifiers or
experts, to enhance the power of models. For domain generalization, ensemble
learning exploits the relationship between multiple source domains by using
specific network architecture designs and training strategies to improve
generalization. They assume that any sample can be regarded as an integrated
sample of the multiple source domains, so the overall prediction result can be
seen as the superposition of the multiple domain networks.

Mancini et al. [134] proposed to use learnable weights for aggregating the
predictions from different source specific classifiers, where a domain
predictor is used to predict the probability that a sample belongs to each
domain (weights). Segù et al. [68] maintained domain-dependent batch
normalization (BN) statistics and BN parameters for different source domains
while all the other parameters were shared. In inference, the final prediction
was a linear combination of the domain-dependent models with the combination
weights inferred by measuring the distances between the instance normalization
statistics of the test sample and the accumulated population statistics of
each domain. The work of [135] proposed domain-specific layers of different
source domains and learning the linear aggregation of these layers to
represent a test sample. Zhou et al. [8] proposed Domain Adaptive Ensemble
Learning (DAEL), where a DAEL model is composed of a CNN feature extractor
shared across domains and multiple domain-specific classifier heads. Each
classifier is an expert to its own domain and a non-expert to others. DAEL
aims to learn these experts collaboratively, by teaching the non-experts with
the expert so as to encourage the ensemble to learn how to handle data from
unseen domains. There are also other works [138, 139]. Ensemble learning
remains a powerful tool for DG since ensemble allows more diversity of models
and features. However, one drawback of ensemble learning-based DG is maybe its
computational resources as we need more space and computations for training
and saving different models.

####  4.3.2 Meta-learning-based DG

The key idea of meta-learning is to learn a general model from multiple tasks
by either optimization-based methods [204], metric-based learning [205], or
model-based methods [206]. The idea of meta-learning has been exploited for
domain generalization. They divide the data form multi-source domains into
meta-train and meta-test sets to simulate domain shift. Denote $\theta$ the
model parameters to be learned, meta-learning can be formulated as:

$\begin{split}\theta^{\ast}&=\operatorname{Learn}(\mathcal{S}_{mte};\phi^{\ast})\\\
&=\operatorname{Learn}(\mathcal{S}_{mte};\operatorname{MetaLearn}(\mathcal{S}_{mtrn})),\end{split}$
(18)

where $\phi^{\ast}=\operatorname{MetaLearn}(\mathcal{S}_{mtrn})$ denotes the
meta-learned parameters from the meta-train set $\mathcal{S}_{mtrn}$ which is
then used to learn the model parameters $\theta^{\ast}$ on the meta-test set
$\mathcal{S}_{mte}$. The two functions $\operatorname{Learn}(\cdot)$ and
$\operatorname{MetaLearn}(\cdot)$ are to be designed and implemented by
different meta-learning algorithms, which corresponds to a bi-level
optimization problem. The gradient update can be formulated as:

$\theta=\theta-\alpha\frac{\partial(\ell(\mathcal{S}_{mte};\theta)+\beta\ell(\mathcal{S}_{mtrn};\phi))}{\partial\theta},$
(19)

where $\eta$ and $\beta$ are learning rates for outer and inner loops,
respectively.

Finn et al. [204] proposed Model-agnostic meta-learning (MAML). Inspired by
MAML, Li et al. [17] proposed MLDG (meta-learning for domain generalization)
to use the meta-learning strategy for DG. MLDG splits the data from the source
domains into meta-train and meta-test to simulate the domain shift situation
to learn general representations. Balaji et al. [18] proposed to learn a meta
regularizer (MetaReg) for the classifier. [19] proposed feature-critic
training for the feature extractor by designing a meta optimizer. Dou et al.
[140] used the similar idea of MLDG and additionally introduced two
complementary losses to explicitly regularize the semantic structure of
feature space. Du et al. [20] proposed an extended version of information
bottleneck named Meta Variational Information Bottleneck (MetaVIB). They
regularize the Kullback–Leibler (KL) divergence between distributions of
latent encoding of the samples that have the same category from different
domains and learn to generate weights by using stochastic neural networks.
Recently, some works also adopted meta-learning for semi-supervised DG or
discriminative DG [141, 142, 143, 46, 146]. Meta-learning is widely adopted in
DG research and it can be incorporated into several paradigms such as
disentanglement [195]. Meta-learning performs well on massive domains since
meta-learning can seek transferable knowledge from multiple tasks.

####  4.3.3 Gradient operation-based DG

Other than meta-learning and ensemble learning, several recent works consider
using gradient information to force the network to learn generalized
representations. Huang et al. [148] proposed a self-challenging training
algorithm that aims to learn general representations by manipulating
gradients. They iteratively discarded the dominant features activated on the
training data, and forced the network to activate remaining features that
correlate with labels. In this way, network can be forced to learn from more
bad cases which will improve generalization ability. Shi et al. [149] proposed
a gradient-matching scheme, where their assumption is that the gradient
direction of two domains should be the same to enhance common representation
learning. To this end, they proposed to maximize the gradient inner product
(GIP) to align the gradient direction across domains. With this operation, the
network can find weights such that the input-output correspondence is as close
as possible across domains. GIP can be formulated as:

$\mathcal{L}=\mathcal{L}_{\text{cls
}}\left(\mathcal{S}_{train};\theta\right)-\lambda\frac{2}{M(M-1)}\sum_{i,j}^{i\neq
j}G_{i}\cdot G_{j},$ (20)

where $G_{i}$ and $G_{j}$ are gradient for two domains that can be calulated
as $G=\mathbb{E}\frac{\partial\ell(x,y;\theta)}{\partial\theta}$. The
gradient-invariance was achieved by adding CORAL [189] loss between gradients
in [150], while Tian et al. [151] maximized the neuron coverage of DNN with
gradient similarity regularization between the original and the augmented
samples.

####  4.3.4 Distributionally robust optimization-based DG

The goal of distributionally robust optimization (DRO) [207] is to learn a
model at worst-case distribution scenario to hope it can generalize well to
the test data, which shares similar goal as DG. To optimize the worst-case
distribution scenario, Sagawa et al. [208] proposed a GroupDRO algorithm that
requires explicit group annotation of the samples. Such annotation was later
narrowed down to a small fraction of validation set in [153], where they
formulated a two-stage weighting framework. Other researchers reduced the
variance of training domain risks by risk extrapolation (VRex) [108] or
reducing class-conditioned Wasserstein DRO [155]. Recently, Koh et al. [152]
proposed the setting of subpopulation shift where they also applied DRO to
this problem. Particularly, Du et al. [209] proposed AdaRNN, a similar
algorithm to the spirit of DRO that did not require explicit group annotation;
instead, they learned the worst-case distribution scenario by solving an
optimization problem. To summarize, DRO focuses on the optimization process
that can also be leveraged in DG research.

####  4.3.5 Self-supervised learning-based DG

Self-supervised learning (SSL) is a recently popular learning paradigm that
builds self-supervised tasks from the large-scale unlabeled data [210].
Inspired by this, Carlucci et al. [156] introduced a self-supervision task of
solving jigsaw puzzles to learn generalized representations. Apart from
introducing new pretext tasks, contrastive learning is another popular
paradigm of self-supervised learning, which was adopted in several recent
works [158, 159, 157]. The core of contrastive learning is to perform
unsupervised learning between positive and negative pairs. Note that self-
supervised learning is a general paradigm that can be applied to any existing
DG methods, especially unsupervised DG where there are no labels in training
domains [84]. Another possible application of SSL-based DG is the pretraining
of multi-domain data that trains powerful pretraining models while also
handling domain shifts. However, a possible limitation of SSL-based DG maybe
its computational efficiency and requirement of computing resources.

####  4.3.6 Other learning strategy for DG

There are some other learning strategies for domain generalization. For
instance, metric learning was adopted in [164] to explore better pair-wise
distance for DG. Ryu et al. [160] used random forest to improve the
generalization ability of convolutional neural networks (CNN). They sampled
the triplets based on the probability mass function of the split results given
by random forest, which is used for updating the CNN parameters by triplet
loss. Other works [211, 212] adopted model calibration for DG, where they
argued that the calibrated performance has a close relationship with OOD
performance. Zhang et al. [162] followed the lottery ticket hyphothesis to
design network substructures for DG, while Narayanan et al. [161] focused on
the shape-invariant features. Additionally, Cha et al. [163] observed the flat
minima is important to DG and they designed a simple stochastic weight
averaging densely method to find the flat minima. Since DG is a general
learning problem, there will be more works that uses other strategy in the
future.

##  5 Other Domain Generalization Research Areas

Most of the existing literature on domain generalization adopt the basic
(traditional) definition of DG in Def. 2. There is some existing literature
that extends such setting to new scenarios to push the frontiers of DG (ref.
TABLE III). This section briefly discuss the existing DG research areas to
give the readers a brief overview of this problem.

TABLE III: Existing research areas of domain generalization Setting | Definition | Reference
---|---|---
Traditional domain generalization | Def. 2 | Most of this paper
Single-source domain generalization | Set $M=1$ in Def. 2 | [213, 99, 158, 54, 128, 56, 41, 213, 85, 58]
Semi-supervised domain generalization |  $\mathcal{S}_{train}$ is partially labeled | [168, 214]
Federated domain generalization |  $\mathcal{S}_{train}$ cannot broadcast to the server | [215, 216, 138]
Open domain generalization | $\mathcal{Y}_{train}\neq\mathcal{Y}_{test}$ | [57]
Unsupervised domain generalization |  $\mathcal{S}_{train}$ is totally unlabeled | [84]

###  5.1 Single-source Domain Generalization

Setting $M=1$ in Def. 2 gives single-source DG. Compared to traditional DG
($M>1$), single-source DG becomes more challenging since there are less
diversity in training domains. Thus, the key to this problem is to generate
novel domains using data generation techniques to increase the diversity and
informativeness of training data. Several methods designed different
generation strategies [213, 99, 158, 54, 128, 56, 41, 213, 85, 58] for single-
source DG in computer vision tasks. A recent work [209] studied this setting
in time series data where there is usually one unified dataset by using min-
max optimization. We expect more application areas can benefit from single-
source DG.

###  5.2 Semi-supervised Domain Generalization

Compared to traditional DG, semi-supervised DG does not require the full
labels of training domains. It is common to apply existing semi-supervised
learning algorithms such as FixMatch [217] and FlexMatch [218] to learn pseudo
labels for the unlabeled samples. For instance, two recent works adopted the
consistency regularization in semi-supervised learning [168, 214] for semi-
supervised DG. It can be seen that this setting is more general than
traditional DG and we expect there will be more works in this area.

###  5.3 Federated Learning with Domain Generalization

Privacy of machine learning is becoming increasingly critical. Does better OOD
performance induce better privacy protection? Mahajan et al. [219] firstly
studied this problem and their experiments showed that if the features are
stable, then the model is more robust to membership inference attack. Another
avenue for privacy protecting is decentralized learning, or federated learning
[220, 221] where models do not access the raw training data; instead, they
aggregate the parameters from different clients. Under this circumstance, the
key is to design better aggregation scheme through generalization techniques
[215, 216, 138]. Federated DG is more important in healthcare [222] and we
hope there could be more work in this area.

###  5.4 Other DG Settings

There are also other settings in DG, such as open domain generalization [57]
and unsupervised domain generalization [84]. Open DG shares the similar
setting of universal domain adaptation where the training and test label
spaces are not the same. Unsupervised DG assumes that all labels in the
training domains are not accessible. As the environment gets more general and
challenging, there will be other DG research areas aiming at solving certain
limitations.

##  6 Applications

In this section, we discuss the popular tasks/applications for domain
generalization (ref. Figure 4).

High generalization ability is desired in various vision tasks. Many works
investigate DG on classification. Some works also study DG for semantic
segmentation [90], action recognition [116, 144], face anti-spoofing [92],
person Re-ID [49, 87], street view recognition [41], video understanding
[112], and image compression [223]. Medical analysis [118] is one of the
important application areas for DG due to its nature of data scarcity and
existence of domain gaps, with the tasks of tissue segmentation [140],
Parkinson’s disease recognition [27], activity recognition [64], chest X-ray
recognition [81, 79], and EEG-based seizure detection [76].

Apart from those areas, DG is also useful in reinforcement learning of robot
control [51, 17] to generalize to an unseen environment. Some work used DG to
recognize speech utterance [37, 117], fault diagnosis [79, 75, 74], physics
[77], brain-computer interface [73].

for tree= grow=east, reversed=true, anchor=base west, parent anchor=east,
child anchor=west, base=left, font=, rectangle, draw, rounded
corners,align=left, minimum width=2.5em, inner xsep=4pt, inner ysep=1pt, ,
where level=1text width=5em,fill=blue!10, where level=2text
width=5em,font=,fill=pink!30, where level=3font=,yshift=0.26pt,fill=yellow!20,
[Domain
generalization
applications,fill=green!20 [Computer
vision,text width=5em [Image classification,text width=8em [e.g. [29, 2, 130]
] ] [Semantic segmentation,text width=9em [e.g. [90, 125] ] ] [Action
recognition,text width=8em [e.g. [116, 144] ] ] [Person Re-ID,text width=6em
[e.g. [49, 87] ] ] ] [Natural
language
processing,text width=5em [Sentiment classification,text width=9em [e.g. [98,
18] ] ] [Semantic parsing,text width=7em [e.g. [143] ] ] [Web page
classification,text width=9em [e.g. [47] ] ] ] [Reinforcement
learning,text width=6em [Robot control,text width=5em [e.g. [17, 51] ] ] ]
[Medical
analysis,text width=5em [Tissue segmentation,text width=8em [e.g. [140] ] ]
[Parkinson’s disease,text width=8em [e.g. [27] ] ] [Chest X-ray
recognition,text width=9em [e.g. [81, 79] ] ] [EEG-based seizure
detection,text width=11em [e.g. [76] ] ] ] [Other
applications,text width=5em [Human activity recognition,text width=11em [e.g.
[127] ] ] [Fault diagnosis,text width=6em [e.g. [79, 75, 74] ] ] [Time series
forecasting,text width=9em [e.g. [209] ] ] [Speech utterance recognition,text
width=11em [e.g. [37, 117] ] ] ] ]

Figure 4: Several applications of domain generalization.

In natural language processing, it is also common that training data comes
from different domains with different distributions and DG techniques are
helpful. Some work used domain generalization for sentiment classification on
the Amazon Review dataset [98, 18]. Others used DG for semantic parsing [143],
web page classification [47].

Moreover, DG techniques favor broad prospects in some applications, such as
financial analysis, weather prediction, and logistics. For instance, Du et al.
[209] tried to adopt DG to time series modeling. They first propose the
temporal covariate shift problem that widely exists in time series data, then,
they proposed an RNN-based model to solve this problem to align the hidden
representations between any pair of training data that are from different
domains. Their algorithm, the so-called AdaRNN, was applied to stock price
prediction, weather prediction, and electric power consumption. Another
example is [127], where they applied domain generalization to sensor-based
human activity recognition. In their application, the activity data from
different persons are from different distributions, resulting in severe model
collapse when applied to new users. To resolve such problem, they developed a
variational autoencoder-based network to learn the domain-invariant and
domain-specific modules, thus achieving the disentanglement.

In the future, we hope there can be more DG applications in other areas to
tackle with the distributional shift that widely exists in different
applications.

TABLE IV: Eighteen popular datasets for domain generalization. The last ten datasets are from WILDS [152]. Dataset | #Domain | #Class | #Sample | Description | Reference
---|---|---|---|---|---
Office-Caltech | 4 | 10 | 2,533 | Caltech, Amazon, Webcam, DSLR | [224]
Office-31 | 3 | 31 | 4,110 | Amazon, Webcam, DSLR | [224]
PACS | 4 | 7 | 9,991 | Art, Cartoon, Photos, Sketches | [2]
VLCS | 4 | 5 | 10,729 | Caltech101, LabelMe, SUN09, VOC2007 | [225]
Office-Home | 4 | 65 | 15,588 | Art, Clipart, Product, Real | [226]
Terra Incognita | 4 | 10 | 24,788 | Wild animal images taken at locations L100, L38, L43, L46 | [227]
Rotated MNIST | 6 | 10 | 70,000 | Digits rotated from $0^{\circ}$ to $90^{\circ}$ with an interval of $15^{\circ}$ | [66]
DomainNet | 6 | 345 | 586,575 | Clipart, Infograph, Painting, Quickdraw, Real, Sketch | [228]
iWildCam2020-wilds | 323 | 182 | 203,029 | Species classification across different camera traps | [229]
Camelyon17-wilds | 5 | 2 | 45,000 | Tumor identification across five different hospitals | [230]
RxRx1-wilds | 51 | 1,139 | 84,898 | Genetic perturbation classification across experimental batches | [231]
OGB-MolPCBA | 120,084 | 128 | 400,000 | Molecular property prediction across different scaffolds | [232]
GlobalWheat-wilds | 47 | bounding boxes | 6,515 | Wheat head detection across regions of the world | [233]
CivilComments-wilds | - | 2 | 450,000 | Toxicity classification across demographic identities | [234]
FMoW-wilds | 80 | 62 | 118,886 | Land use classification across different regions and years | [235]
PovertyMap-wilds | 46 | real value | 19,669 | Poverty mapping across different countries | [236]
Amazon-wilds | 3920 | 5 | 539,502 | Sentiment classification across different users | [237]
Py150-wilds | 8,421 | next token | 150,000 | Code completion across different codebases | [238, 239]

##  7 Datasets, Evaluation, and Benchmark

In this section, we summarize the existing common datasets and model selection
strategies for domain generalization. Then, we introduce the codebase, DeepDG,
and demonstrate some observations from experiment conducted via it.

###  7.1 Datasets

TABLE IV provides an overview of several popular datasets. Among them, PACS
[2], VLCS [225], and Office-Home [226] are three most popular datasets. For
large-scale evaluation, DomainNet [228] and Wilds [152] (_i.e.,_ a collection
of datasets in TABLE IV that ends with ‘-wilds’) are becoming increasing
popular.

Besides the datasets mentioned above, there exists some other datasets for
domain generalization with different tasks. The Graft-versus-host disease
(GvHD) dataset [240] is also popular and is used to test several methods [27,
6] for the flow cytometry problem. This dataset was collected from 30 patients
with sample sizes ranging from 1,000 to 10,000. This is a time series
classification dataset. Some works [90, 29, 70] applied domain generalization
to semantic segmentation, where CityScape [241] and GTA5 [242] datasets were
adopted as benchmark datasets. Some works applied domain generalization to
object detection, using the datasets of Cityscapes [241], GTA5 [242], Synthia
[243] for investigation [70]. Some other works used public datasets or
RandPerson [244] for person re-identification [183, 245, 69]. Some works [17,
51] used the OpenAI Gym [246] as the testbed to evaluate the performance of
algorithms in reinforcement learning problems such as Cart-Pole and mountain
car.

In addition to these widely used datasets, there are also other datasets used
in existing literature. The Parkinson’s telemonitoring dataset [247] is
popular for predicting the clinician’s motor and total UPDRS scoring of
Parkinson’s disease symptoms from voice measures. Some methods [27, 59, 60]
used the data from several people as the training domains to learn models that
generalize to unseen subjects.

It is worth noting that the datasets of domain generalization have some
overlaps with domain adaptation. For instance, Office-31, Office-Caltech,
Office-Home, and DomainNet are also widely used benchmarks for domain
adaptation. Therefore, most domain adaptation datasets can be used for domain
generalization benchmark in addition to those we discussed here. For example,
Amazon Review dataset [248] is widely used in domain adaptation. It has four
different domains on product review (DVDs, Kitchen appliance, Electronics and
Books), which can also be used for domain generalization.

###  7.2 Evaluation

To test domain generalization algorithm on a test domain, three strategies are
proposed [78], namely, _Test-domain validation set_ , _Leave-one-domain-out
cross-validation_ , and _Training-domain validation set_. Test-domain
validation set utilizes parts of the target domain as validations. Although it
can obtain the best performance in most circumstances for that validation and
testing share the same distribution, there is often no access to targets when
training, which means it cannot be adopted in real applications. Leave-one-
domain-out is another strategy to choose the final model when training data
contains multiple sources. It leaves one training source as the validation
while treating the others as the training part. Obviously, when only a single
source exists in the training data, it is no longer applicable. In addition,
due to different distributions among sources and targets, final results rely
heavily on the selections of validation, which makes final results unstable.
The most common strategy for domain generalization is Training-domain
validation set which is used in most existing work. In this strategy, each
source is split into two parts, the training part and the validation part. All
training parts are combined for training while all validation parts are
combined for selecting the best model. Since there still exists divergences
between the combined validation and the real unseen targets, this simple and
most popular strategy cannot achieve the best performance for some time. We
need to mention that there may exist other evaluation protocols for DG or OOD
such as Ye et al. [165] since designing effective evaluation protocols is
often consistent with the OOD performance. Currently, most of the works
adopted the train-domain validation strategy.

###  7.3 Benchmark

To test the performance of DG algorithms in a unified codebase, in this paper,
we develop a new codebase for DG, named _DeepDG_ [249, 250]. Compared to the
existing DomainBed [78], DeepDG simplifies the data loading and model
selection process, while also makes it possible to run all experiments in a
single machine. DeepDG splits the whole process into a data preparation part,
a model part, a core algorithm part, a program entry, and some other auxiliary
functions. Each part can be freely modified by users without affecting other
parts. Users can add their own algorithms or datasets to DeepDG and compare
with some state-of-the-art methods fairly. The current public version of
DeepDG is only for image classification and we offer supports for Office-31,
PACS, VLCS, and Office-Home datasets. Currently, nine state-of-the-art methods
are implemented under the same environment, and it covers all three groups,
including Data manipulation (Mixup [171]), Representation learning (DDC [185],
DANN [88], CORAL [188]), and Learning strategy (MLDG [17], RSC [148], GroupDRO
[154], ANDMask [251]).

We conduct some experiments on the two most popular image classification
datasets, PACS and Office-Home, with DeepDG and TABLE V shows the results.
ResNet-18 is used as the base feature network. Training-domain validation set
is used for selecting final models, and $20\%$ of sources are for validation
while the others are for training. From TABLE V, we observe more insightful
conclusions. (1) The baseline method, ERM, has achieved acceptable results on
both datasets. Some methods, such as DANN and ANDMask, even have worse
performance. (2) Simple data augmentation method, Mixup, cannot obtain
remarkable results. (3) CORAL has slight improvements on both datasets
compared to ERM, which is consistent with results offered by DomainBed [78].
(4) RSC, a learning strategy, achieves the best performance on both datasets,
but the improvements are unremarkable compared to ERM. The results indicate
the benefits of domain generalization in different tasks.

TABLE V: Benchmark results for PACS and Office-Home with DeepDG.

Dataset | PACS | Office-Home
---|---|---
Method | A | C | P | S | AVG | A | C | P | R | AVG
ERM | 77.0 | 74.5 | 95.5 | 77.8 | 81.2 | 58.6 | 47.9 | 72.2 | 73.0 | 62.9
DANN [89] | 78.7 | 75.3 | 94.0 | 77.8 | 81.4 | 57.7 | 44.4 | 71.9 | 72.5 | 61.6
CORAL [189] | 77.7 | 77.0 | 92.6 | 80.5 | 82.0 | 58.7 | 48.7 | 72.3 | 73.6 | 63.3
Mixup [171] | 79.1 | 73.4 | 94.4 | 76.7 | 80.9 | 55.7 | 47.9 | 71.9 | 72.8 | 62.1
RSC [148] | 79.7 | 76.1 | 95.6 | 76.6 | 82.0 | 58.9 | 49.2 | 72.5 | 74.2 | 63.7
GroupDRO [208] | 76.0 | 76.0 | 91.2 | 79.0 | 80.6 | 57.6 | 48.7 | 71.5 | 73.1 | 62.7
ANDMask [251] | 76.2 | 73.8 | 91.6 | 78.0 | 79.9 | 56.7 | 45.9 | 70.6 | 73.2 | 61.6

##  8 Discussion

In this section, we summarize the existing methods and then present several
challenges that warrant attention in the future research.

###  8.1 Summary of Existing Literature

The quantity and diversity of training data are of great importance to a
model’s generalization ability. Many methods aim to enrich the training data
using the aforementioned data manipulation methods to achieve good
performance. However, one issue of the data manipulation methods is that there
is a lack of theoretical guarantee of the unbound risk of generalization.
Therefore, it is important to develop theories for the manipulation-based
methods which could further guide the data generation designs without
violating ethical standards.

Compared to data manipulation, representation learning has theoretical support
in general [177, 6, 96]. Kernel-based methods are widely used in traditional
methods while deep learning-based methods play a leading role in recent years.
While domain adversarial training often achieves better performance in domain
adaptation, in DG, we did not see significant results improvements from these
adversarial methods. We think this is probably because the task is relatively
easy. For the explicit distribution matching, more and more works tend to
match the joint distributions rather than just match the marginal [6, 71] or
conditional [62] distributions. Thus, it is more feasible to perform dynamic
distribution matching [186, 187]. Both disentanglement and IRM methods have
good motivations for generalization, while more efficient training strategy
can be developed. There are several studies [252] that pointed out merely
learning domain-invariant features are not sufficient for generalization and
representation smoothness should also be considered.

For learning strategy, there is a trend that many works used meta-learning for
DG, where it requires to design better optimization strategies to utilize the
rich information of different domains. In addition to deep networks, there are
also some work [160] that used random forest for DG, and we hope more diverse
methods will come.

###  8.2 Future Research Challenges

We summarize some future research challenges for domain generalization.

####  8.2.1 Continuous domain generalization

For many real applications, a system consumes streaming data with non-
stationary statistics. In this case, it is of great importance to perform
continuous domain generalization that efficiently updates DG models to
overcome catastrophic forgetting and adapts to new data. While there are some
domain adaptation methods focusing on continuous learning [253], there are
only very few investigations on continuous DG [254] whenever this is favorable
in real scenarios.

####  8.2.2 Domain generalization to novel categories

The existing DG algorithms usually assume the label space for different
domains are the same. A more practical and general setting is to support the
generalization on new categories, i.e., both domain and task generalization.
This is conceptually similar to the goal of meta-learning and zero-shot
learning. Some work [72, 255] proposed zero-shot DG and we expect more work to
come in this area.

####  8.2.3 Interpretable domain generalization

Disentanglement-based DG methods decompose a feature into domain-
invariant/shared and domain-specific parts, which provide some interpretation
to DG. For other categories of methods, there is still a lack of deep
understanding of the semantics or characteristics of learned features in DG
models. Causality [130] may be one promising tool to understand domain
generalization networks and provide interpretations.

####  8.2.4 Large-scale pre-training/self-learning and DG

In recent years, we have witnessed the rapid development of large-scale pre-
training/self-learning, such as BERT [256], GPT-3 [257], and Wav2vec [258].
Pre-training on large-scale dataset and then finetuning the model to
downstream tasks could improve its performance, where pre-training is
beneficial to learn general representations. Therefore, how to design useful
and efficient DG methods to help large-scale pre-training/self-learning is
worth investigating.

####  8.2.5 Test-time Generalization

While DG focuses on the training phase, we can also request test-time
generalization in inference phase. This further bridges domain adaptation and
domain generalization since we can also use the inference unlabeled data for
adaptation. Very few recent works [259, 260] paid attention to this setting.
Compared to traditional DG, test-time generalization will allow more
flexibility in inference time, while it requires less computation and more
efficiency as there are often limited resources in inference-end devices.

####  8.2.6 Performance evaluation for DG

The recent work [78] pointed out that on several datasets, the performance of
some DG methods is almost the same as the baseline method (i.e., empirical
risk minimization). We do not take it as the full evidence that DG is not
useful in real applications. Instead, we argue that this might be due to the
inappropriate evaluation schemes in use today, or the domain gaps not being so
large. In more realistic situations such as person ReID where there are
obvious domain gaps [70], the improvement of DG is dramatic. Therefore, we
stay positive about the value of DG and hope researchers can also find more
suitable settings and datasets for the study.

##  9 Conclusion

Generalization has always been an important research topic in machine learning
research. In this paper, we review the domain generalization areas by
providing in-depth analysis of theories, existing methods, datasets,
benchmarks, and applications. Then, we thoroughly analyze the methods. Based
on our analysis, we provide several potential research challenges that could
be the directions of future research. We hope that this survey can provide
useful insights to researchers and inspire more progress in the future.

## References

  * Quiñonero-Candela et al. [2009] J. Quiñonero-Candela, M. Sugiyama, N. D. Lawrence, and A. Schwaighofer, _Dataset shift in machine learning_. Mit Press, 2009.
  * Li et al. [2017a] D. Li, Y. Yang, Y.-Z. Song, and T. M. Hospedales, “Deeper, broader and artier domain generalization,” in _ICCV_ , 2017, pp. 5542–5550.
  * Zhou et al. [2021a] K. Zhou, Z. Liu, Y. Qiao, T. Xiang, and C. C. Loy, “Domain generalization: A survey,” _arXiv preprint arXiv:2103.02503_ , 2021.
  * Shen et al. [2021] Z. Shen, J. Liu, Y. He, X. Zhang, R. Xu, H. Yu, and P. Cui, “Towards out-of-distribution generalization: A survey,” _arXiv preprint arXiv:2108.13624_ , 2021.
  * Yang et al. [2021] J. Yang, K. Zhou, Y. Li, and Z. Liu, “Generalized out-of-distribution detection: A survey,” _arXiv preprint arXiv:2110.11334_ , 2021.
  * Blanchard et al. [2021] G. Blanchard, A. A. Deshmukh, Ü. Dogan, G. Lee, and C. Scott, “Domain generalization by marginal transfer learning.” _J. Mach. Learn. Res._ , vol. 22, pp. 2–1, 2021.
  * Caruana [1997] R. Caruana, “Multitask learning,” _Machine learning_ , vol. 28, no. 1, pp. 41–75, 1997.
  * Zhou et al. [2020a] K. Zhou, Y. Yang, Y. Qiao, and T. Xiang, “Domain adaptive ensemble learning,” _arXiv preprint arXiv:2003.07325_ , 2020.
  * Pan and Yang [2010] S. Pan and Q. Yang, “A survey on transfer learning,” _IEEE TKDE_ , vol. 22, pp. 1345–1359, 2010.
  * Weiss et al. [2016] K. Weiss, T. M. Khoshgoftaar, and D. Wang, “A survey of transfer learning,” _Journal of Big data_ , vol. 3, no. 1, pp. 1–40, 2016.
  * Zhuang et al. [2020] F. Zhuang, Z. Qi, K. Duan, D. Xi, Y. Zhu, H. Zhu, H. Xiong, and Q. He, “A comprehensive survey on transfer learning,” _Proceedings of the IEEE_ , vol. 109, no. 1, pp. 43–76, 2020.
  * Wang and Deng [2018] M. Wang and W. Deng, “Deep visual domain adaptation: A survey,” _Neurocomputing_ , vol. 312, pp. 135–153, 2018.
  * Patel et al. [2015] V. M. Patel, R. Gopalan, R. Li, and R. Chellappa, “Visual domain adaptation: A survey of recent advances,” _IEEE signal processing magazine_ , vol. 32, no. 3, pp. 53–69, 2015.
  * Vilalta and Drissi [2002] R. Vilalta and Y. Drissi, “A perspective view and survey of meta-learning,” _Artif. Intell. Rev._ , vol. 18, no. 2, pp. 77–95, 2002.
  * Hospedales et al. [2020] T. Hospedales, A. Antoniou, P. Micaelli, and A. Storkey, “Meta-learning in neural networks: A survey,” _arXiv preprint arXiv:2004.05439_ , 2020.
  * Vanschoren [2018] J. Vanschoren, “Meta-learning: A survey,” _arXiv preprint arXiv:1810.03548_ , 2018.
  * Li et al. [2018a] D. Li, Y. Yang, Y.-Z. Song, and T. M. Hospedales, “Learning to generalize: Meta-learning for domain generalization,” in _AAAI_ , 2018.
  * Balaji et al. [2018] Y. Balaji, S. Sankaranarayanan, and R. Chellappa, “Metareg: Towards domain generalization using meta-regularization,” in _NeurIPS_ , 2018, pp. 998–1008.
  * Li et al. [2019a] Y. Li, Y. Yang, W. Zhou, and T. M. Hospedales, “Feature-critic networks for heterogeneous domain generalization,” in _ICML_ , 2019.
  * Du et al. [2020] Y. Du, J. Xu, H. Xiong, Q. Qiu, X. Zhen, C. G. M. Snoek, and L. Shao, “Learning to learn with variational information bottleneck for domain generalization,” in _ECCV_ , 2020.
  * Biesialska et al. [2020] M. Biesialska, K. Biesialska, and M. R. Costa-jussà, “Continual lifelong learning in natural language processing: A survey,” _arXiv preprint arXiv:2012.09823_ , 2020.
  * Wang et al. [2019] W. Wang, V. W. Zheng, H. Yu, and C. Miao, “A survey of zero-shot learning: Settings, methods, and applications,” _ACM Transactions on Intelligent Systems and Technology (TIST)_ , vol. 10, no. 2, pp. 1–37, 2019.
  * JI et al. [2019] Z. JI, H. WANG, Y. YU, and Y. PANG, “A decadal survey of zero-shot image classification,” _SCIENTIA SINICA Informationis_ , vol. 49, no. 10, pp. 1299–1320, 2019.
  * Ben-David et al. [2010] S. Ben-David, J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. W. Vaughan, “A theory of learning from different domains,” _Machine learning_ , vol. 79, no. 1-2, pp. 151–175, 2010.
  * Johansson et al. [2019] F. D. Johansson, D. Sontag, and R. Ranganath, “Support and invertibility in domain-invariant representations,” in _AISTAS_ , 2019, pp. 527–536.
  * Vapnik et al. [1994] V. Vapnik, E. Levin, and Y. L. Cun, “Measuring the vc-dimension of a learning machine,” _Neural computation_ , vol. 6, no. 5, pp. 851–876, 1994.
  * Muandet et al. [2013] K. Muandet, D. Balduzzi, and B. Schölkopf, “Domain generalization via invariant feature representation,” in _ICML_ , 2013, pp. 10–18.
  * Deshmukh et al. [2019] A. A. Deshmukh, Y. Lei, S. Sharma, U. Dogan, J. W. Cutler, and C. Scott, “A generalization error bound for multi-class domain generalization,” _arXiv:1905.10392_ , 2019.
  * Yue et al. [2019] X. Yue, Y. Zhang, S. Zhao, A. Sangiovanni-Vincentelli, K. Keutzer, and B. Gong, “Domain randomization and pyramid consistency: Simulation-to-real generalization without accessing target domain data,” in _ICCV_ , 2019, pp. 2100–2110.
  * Tobin et al. [2017] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, “Domain randomization for transferring deep neural networks from simulation to the real world,” in _IROS_ , 2017, pp. 23–30.
  * Peng et al. [2018] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel, “Sim-to-real transfer of robotic control with dynamics randomization,” in _ICRA_. IEEE, 2018, pp. 1–8.
  * Khirodkar et al. [2019] R. Khirodkar, D. Yoo, and K. Kitani, “Domain randomization for scene-specific car detection and pose estimation,” in _WACV_. IEEE, 2019, pp. 1932–1940.
  * Tremblay et al. [2018] J. Tremblay, A. Prakash, D. Acuna, M. Brophy, V. Jampani, C. Anil, T. To, E. Cameracci, S. Boochoon, and S. Birchfield, “Training deep networks with synthetic data: Bridging the reality gap by domain randomization,” in _CVPR Workshop_ , 2018.
  * Prakash et al. [2019] A. Prakash, S. Boochoon, M. Brophy, D. Acuna, E. Cameracci, G. State, O. Shapira, and S. Birchfield, “Structured domain randomization: Bridging the reality gap by context-aware synthetic data,” in _ICRA_. IEEE, 2019, pp. 7249–7255.
  * Nazari and Kovashka [2020] N. H. Nazari and A. Kovashka, “Domain generalization using shape representation,” in _ECCV_ , 2020, pp. 666–670.
  * Huang et al. [2021] J. Huang, D. Guan, A. Xiao, and S. Lu, “Fsdr: Frequency space domain randomization for domain generalization,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 6891–6902.
  * Shankar et al. [2018] S. Shankar, V. Piratla, S. Chakrabarti, S. Chaudhuri, P. Jyothi, and S. Sarawagi, “Generalizing across domains via cross-gradient training,” in _ICLR_ , 2018.
  * Volpi et al. [2018] R. Volpi, H. Namkoong, O. Sener, J. C. Duchi, V. Murino, and S. Savarese, “Generalizing to unseen domains via adversarial data augmentation,” in _NeurIPS_ , 2018, pp. 5334–5344.
  * Zhou et al. [2020b] K. Zhou, Y. Yang, T. Hospedales, and T. Xiang, “Deep domain-adversarial image generation for domain generalisation,” in _AAAI_ , 2020.
  * Rahman et al. [2019] M. M. Rahman, C. Fookes, M. Baktashmotlagh, and S. Sridharan, “Multi-component image translation for deep domain generalization,” in _WACV_. IEEE, 2019, pp. 579–588.
  * Qiao et al. [2020] F. Qiao, L. Zhao, and X. Peng, “Learning to learn single domain generalization,” in _CVPR_ , 2020, pp. 12 556–12 565.
  * Liu et al. [2018] A. H. Liu, Y.-C. Liu, Y.-Y. Yeh, and Y.-C. F. Wang, “A unified feature disentangler for multi-domain image translation and manipulation,” in _NeurIPS_ , 2018, pp. 2590–2599.
  * Truong et al. [2019a] T.-D. Truong, C. N. Duong, K. Luu, and M.-T. Tran, “Recognition in unseen domains: Domain generalization via universal non-volume preserving models,” _arXiv preprint:1905.13040_ , 2019.
  * Zhou et al. [2020c] K. Zhou, Y. Yang, T. M. Hospedales, and T. Xiang, “Learning to generate novel domains for domain generalization,” in _ECCV_ , 2020.
  * Somavarapu et al. [2020] N. Somavarapu, C.-Y. Ma, and Z. Kira, “Frustratingly simple domain generalization via image stylization,” _arXiv:2006.11207_ , 2020.
  * Zhao et al. [2021a] Y. Zhao, Z. Zhong, F. Yang, Z. Luo, Y. Lin, S. Li, and N. Sebe, “Learning to generalize unseen domains via memory-based multi-source meta-learning for person re-identification,” in _CVPR_ , 2021.
  * Garg et al. [2021] V. K. Garg, A. Kalai, K. Ligett, and Z. S. Wu, “Learn to expect the unexpected: Probably approximately correct domain generalization,” in _International Conference on Artificial Intelligence and Statistics_ , 2021\.
  * Carlucci et al. [2019a] F. M. Carlucci, P. Russo, T. Tommasi, and B. Caputo, “Hallucinating agnostic images to generalize across domains,” in _CVPR Workshop_ , 2019, pp. 0–0.
  * Wang et al. [2021a] W. Wang, S. Liao, F. Zhao, C. Kang, and L. Shao, “Domainmix: Learning generalizable person re-identification without human annotations,” in _BMCV_ , 2021.
  * Wang et al. [2020a] Y. Wang, H. Li, and A. C. Kot, “Heterogeneous domain generalization via domain mixup,” in _ICASSP_ , 2020, pp. 3622–3626.
  * Zhou et al. [2021b] K. Zhou, Y. Yang, Y. Qiao, and T. Xiang, “Domain generalization with mixstyle,” in _ICLR_ , 2021.
  * Truong et al. [2019b] T.-D. Truong, K. Luu, C.-N. Duong, N. Le, and M.-T. Tran, “Image alignment in unseen domains via domain deep generalization,” _arXiv preprint arXiv:1905.12028_ , 2019.
  * Robey et al. [2021] A. Robey, G. J. Pappas, and H. Hassani, “Model-based domain generalization,” _arXiv preprint arXiv:2102.11436_ , 2021.
  * Li et al. [2021a] L. Li, K. Gao, J. Cao, Z. Huang, Y. Weng, X. Mi, Z. Yu, X. Li, and B. Xia, “Progressive domain expansion network for single domain generalization,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 224–233.
  * Li et al. [2021b] P. Li, D. Li, W. Li, S. Gong, Y. Fu, and T. M. Hospedales, “A simple feature augmentation for domain generalization,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 8886–8895.
  * Peng et al. [2021] X. Peng, F. Qiao, and L. Zhao, “Out-of-domain generalization from a single source: A uncertainty quantification approach,” _arXiv preprint arXiv:2108.02888_ , 2021.
  * Shu et al. [2021] Y. Shu, Z. Cao, C. Wang, J. Wang, and M. Long, “Open domain generalization with domain-augmented meta-learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 9624–9633.
  * Wang et al. [2021b] Z. Wang, Y. Luo, R. Qiu, Z. Huang, and M. Baktashmotlagh, “Learning to diversify for single domain generalization,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 834–843.
  * Blanchard et al. [2017] G. Blanchard, A. A. Deshmukh, U. Dogan, G. Lee, and C. Scott, “Domain generalization by marginal transfer learning,” _arXiv preprint arXiv:1711.07910_ , 2017.
  * Grubinger et al. [2015] T. Grubinger, A. Birlutiu, H. Schöner, T. Natschläger, and T. Heskes, “Domain generalization based on transfer component analysis,” in _International Work-Conference on Artificial Neural Networks_. Springer, 2015, pp. 325–334.
  * Gan et al. [2016] C. Gan, T. Yang, and B. Gong, “Learning attributes equals multi-source domain generalization,” in _CVPR_ , 2016, pp. 87–97.
  * Li et al. [2018b] Y. Li, M. Gong, X. Tian, T. Liu, and D. Tao, “Domain generalization via conditional invariant representations,” in _AAAI_ , 2018.
  * Ghifary et al. [2016] M. Ghifary, D. Balduzzi, W. B. Kleijn, and M. Zhang, “Scatter component analysis: A unified framework for domain adaptation and domain generalization,” _IEEE TPAMI_ , vol. 39, no. 7, pp. 1414–1430, 2016.
  * Erfani et al. [2016] S. Erfani, M. Baktashmotlagh, M. Moshtaghi, V. Nguyen, C. Leckie, J. Bailey, and R. Kotagiri, “Robust domain generalisation by enforcing distribution invariance,” in _AAAI_ , 2016, pp. 1455–1461.
  * Hu et al. [2019] S. Hu, K. Zhang, Z. Chen, and L. Chan, “Domain generalization via multidomain discriminant analysis,” in _UAI_ , vol. 35, 2019.
  * Ghifary et al. [2015] M. Ghifary, W. Kleijn, M. Zhang, and D. Balduzzi, “Domain generalization for object recognition with multi-task autoencoders,” _ICCV_ , pp. 2551–2559, 2015.
  * Motiian et al. [2017] S. Motiian, M. Piccirilli, D. A. Adjeroh, and G. Doretto, “Unified deep supervised domain adaptation and generalization,” in _ICCV_ , 2017, pp. 5715–5725.
  * Segù et al. [2020] M. Segù, A. Tonioni, and F. Tombari, “Batch normalization embeddings for deep domain generalization,” _arXiv preprint arXiv:2011.12672_ , 2020.
  * Jin et al. [2020a] X. Jin, C. Lan, W. Zeng, Z. Chen, and L. Zhang, “Style normalization and restitution for generalizable person re-identification,” in _CVPR_ , 2020, pp. 3143–3152.
  * Jin et al. [2021] X. Jin, C. Lan, W. Zeng, and Z. Chen, “Style normalization and restitution for domain generalization and adaptation,” _arXiv preprint arXiv:2101.00588_ , 2021.
  * Li et al. [2018c] H. Li, S. J. Pan, S. Wang, and A. Kot, “Domain generalization with adversarial feature learning,” in _CVPR_ , 2018, pp. 5400–5409.
  * Maniyar et al. [2020] U. Maniyar, A. A. Deshmukh, U. Dogan, V. N. Balasubramanian _et al._ , “Zero shot domain generalization,” in _BMVC_ , 2020.
  * Han and Jeong [2021] D.-K. Han and J.-H. Jeong, “Domain generalization for session-independent brain-computer interface,” in _International Winter Conference on Brain-Computer Interface_. IEEE, 2021, pp. 1–5.
  * Zheng et al. [2020] H. Zheng, Y. Yang, J. Yin, Y. Li, R. Wang, and M. Xu, “Deep domain generalization combining a priori diagnosis knowledge toward cross-domain fault diagnosis of rolling bearing,” _IEEE Trans. on Instrumentation and Measurement_ , vol. 70, pp. 1–11, 2020.
  * Liao et al. [2020] Y. Liao, R. Huang, J. Li, Z. Chen, and W. Li, “Deep semisupervised domain generalization network for rotary machinery fault diagnosis under variable speed,” _IEEE Transactions on Instrumentation and Measurement_ , vol. 69, no. 10, pp. 8064–8075, 2020.
  * Ayodele et al. [2020] K. Ayodele, W. Ikezogwo, M. Komolafe, and P. Ogunbona, “Supervised domain generalization for integration of disparate scalp eeg datasets for automatic epileptic seizure detection,” _Computers in Biology and Medicine_ , vol. 120, p. 103757, 2020.
  * Chen et al. [2020a] E. Chen, T. S. Mathai, V. Sarode, H. Choset, and J. Galeotti, “A study of domain generalization on ultrasound-based multi-class segmentation of arteries, veins, ligaments, and nerves using transfer learning,” in _Machine Learning for Health (ML4H) at NeurIPS 2020_ , 2020.
  * Gulrajani and Lopez-Paz [2021] I. Gulrajani and D. Lopez-Paz, “In search of lost domain generalization,” in _ICLR_ , 2021.
  * Li et al. [2020a] X. Li, W. Zhang, H. Ma, Z. Luo, and X. Li, “Domain generalization in rotating machinery fault diagnostics using deep neural networks,” _Neurocomputing_ , vol. 403, pp. 409–420, 2020.
  * Liu et al. [2020] H. Liu, P. Song, and R. Ding, “Towards domain generalization in underwater object detection,” in _ICIP_. IEEE, 2020, pp. 1971–1975.
  * Mahajan et al. [2021a] D. Mahajan, S. Tople, and A. Sharma, “Domain generalization using causal matching,” in _ICML_ , 2021.
  * Seo et al. [2020] S. Seo, Y. Suh, D. Kim, J. Han, and B. Han, “Learning to optimize domain specific normalization for domain generalization,” in _ECCV_ , 2020.
  * Zhang et al. [2021a] W. Zhang, M. Ragab, and R. Sagarna, “Robust domain-free domain generalization with class-aware alignment,” in _ICASSP_ , 2021.
  * Qi et al. [2021] L. Qi, L. Wang, Y. Shi, and X. Geng, “Unsupervised domain generalization for person re-identification: A domain-specific adaptive framework,” _arXiv preprint arXiv:2111.15077_ , 2021.
  * Fan et al. [2021] X. Fan, Q. Wang, J. Ke, F. Yang, B. Gong, and M. Zhou, “Adversarially adaptive normalization for single domain generalization,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 8208–8217.
  * Planamente et al. [2022] M. Planamente, C. Plizzari, E. Alberti, and B. Caputo, “Domain generalization through audio-visual relative norm alignment in first person action recognition,” in _IEEE Winter Conference on Computer Vision (WACV)_ , 2022\.
  * Jin et al. [2020b] X. Jin, C. Lan, W. Zeng, and Z. Chen, “Feature alignment and restoration for domain generalization and adaptation,” in _NeurIPS_ , 2020.
  * Ganin and Lempitsky [2015] Y. Ganin and V. Lempitsky, “Unsupervised domain adaptation by backpropagation,” in _ICML_ , 2015, pp. 1180–1189.
  * Ganin et al. [2016] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky, “Domain-adversarial training of neural networks,” _J. Mach. Learn. Res._ , vol. 17, pp. 59:1–59:35, 2016.
  * Gong et al. [2019] R. Gong, W. Li, Y. Chen, and L. V. Gool, “Dlow: Domain flow for adaptation and generalization,” in _CVPR_ , 2019, pp. 2477–2486.
  * Li et al. [2018d] Y. Li, X. Tian, M. Gong, Y. Liu, T. Liu, K. Zhang, and D. Tao, “Deep domain generalization via conditional invariant adversarial networks,” in _ECCV_ , 2018, pp. 624–639.
  * Shao et al. [2019] R. Shao, X. Lan, J. Li, and P. C. Yuen, “Multi-adversarial discriminative deep domain generalization for face presentation attack detection,” in _CVPR_ , 2019, pp. 10 023–10 031.
  * Zhao et al. [2020] S. Zhao, M. Gong, T. Liu, H. Fu, and D. Tao, “Domain generalization via entropy regularization,” in _NeurIPS_ , vol. 33, 2020.
  * Matsuura and Harada [2020] T. Matsuura and T. Harada, “Domain generalization using a mixture of multiple latent domains,” in _AAAI_ , 2020.
  * Sicilia et al. [2021] A. Sicilia, X. Zhao, and S. J. Hwang, “Domain adversarial neural networks for domain generalization: When it works and how to improve,” _arXiv preprint arXiv:2102.03924_ , 2021.
  * Albuquerque et al. [2019] I. Albuquerque, J. Monteiro, T. H. Falk, and I. Mitliagkas, “Adversarial target-invariant representation learning for domain generalization,” _arXiv preprint arXiv:1911.00804_ , 2019.
  * Rahman et al. [2020] M. M. Rahman, C. Fookes, M. Baktashmotlagh, and S. Sridharan, “Correlation-aware adversarial domain adaptation and generalization,” _Pattern Recognition_ , vol. 100, p. 107124, 2020.
  * Wang et al. [2020b] Z. Wang, Q. Wang, C. Lv, X. Cao, and G. Fu, “Unseen target stance detection with adversarial domain generalization,” in _IJCNN_ , 2020, pp. 1–8.
  * Jia et al. [2020] Y. Jia, J. Zhang, S. Shan, and X. Chen, “Single-side domain generalization for face anti-spoofing,” in _CVPR_ , 2020, pp. 8484–8493.
  * Luo et al. [2021] J. Luo, J. Guo, W. Qiu, Z. Huang, and H. Hui, “Scale invariant domain generalization image recapture detection,” _arXiv preprint arXiv:2110.03496_ , 2021.
  * Arjovsky et al. [2019] M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz, “Invariant risk minimization,” _arXiv preprint arXiv:1907.02893_ , 2019.
  * Guo et al. [2021] R. Guo, P. Zhang, H. Liu, and E. Kiciman, “Out-of-distribution prediction with invariant risk minimization: The limitation and an effective fix,” _arXiv preprint arXiv:2101.07732_ , 2021.
  * Ahuja et al. [2020a] K. Ahuja, J. Wang, A. Dhurandhar, K. Shanmugam, and K. R. Varshney, “Empirical or invariant risk minimization? a sample complexity perspective,” _arXiv preprint arXiv:2010.16412_ , 2020.
  * Choe et al. [2020] Y. J. Choe, J. Ham, and K. Park, “An empirical study of invariant risk minimization,” _arXiv preprint arXiv:2004.05007_ , 2020.
  * Rosenfeld et al. [2021] E. Rosenfeld, P. Ravikumar, and A. Risteski, “The risks of invariant risk minimization,” in _ICLR_ , 2021.
  * Ahuja et al. [2021] K. Ahuja, E. Caballero, D. Zhang, Y. Bengio, I. Mitliagkas, and I. Rish, “Invariance principle meets information bottleneck for out-of-distribution generalization,” in _NeurIPS_ , 2021.
  * Ahuja et al. [2020b] K. Ahuja, K. Shanmugam, K. Varshney, and A. Dhurandhar, “Invariant risk minimization games,” in _ICML_ , 2020, pp. 145–155.
  * Krueger et al. [2021] D. Krueger, E. Caballero, J.-H. Jacobsen, A. Zhang, J. Binas, D. Zhang, R. Le Priol, and A. Courville, “Out-of-distribution generalization via risk extrapolation (rex),” in _International Conference on Machine Learning_. PMLR, 2021, pp. 5815–5826.
  * Mitrovic et al. [2021] J. Mitrovic, B. McWilliams, J. C. Walker, L. H. Buesing, and C. Blundell, “Representation learning via invariant causal mechanisms,” in _International Conference on Learning Representations_ , 2021.
  * Xu et al. [2014] Z. Xu, W. Li, L. Niu, and D. Xu, “Exploiting low-rank structure from latent domains for domain generalization,” in _ECCV_. Springer, 2014, pp. 628–643.
  * Khosla et al. [2012] A. Khosla, T. Zhou, T. Malisiewicz, A. A. Efros, and A. Torralba, “Undoing the damage of dataset bias,” in _ECCV_. Springer, 2012, pp. 158–171.
  * Niu et al. [2015] L. Niu, W. Li, and D. Xu, “Multi-view domain generalization for visual recognition,” in _ICCV_ , 2015, pp. 4193–4201.
  * Ding and Fu [2017] Z. Ding and Y. Fu, “Deep domain generalization with structured low-rank constraint,” _IEEE TIP_ , vol. 27, no. 1, pp. 304–313, 2017.
  * Zunino et al. [2021] A. Zunino, S. A. Bargal, R. Volpi, M. Sameki, J. Zhang, S. Sclaroff, V. Murino, and K. Saenko, “Explainable deep classification models for domain generalization,” in _CVPR_ , 2021.
  * Deshmukh et al. [2018] A. A. Deshmukh, A. Bansal, and A. Rastogi, “Domain2vec: Deep domain generalization,” _arXiv preprint arXiv:1807.02919_ , 2018.
  * Li et al. [2017b] W. Li, Z. Xu, D. Xu, D. Dai, and L. Van Gool, “Domain generalization and adaptation using low rank exemplar svms,” _TPAMI_ , vol. 40, no. 5, pp. 1114–1127, 2017.
  * Piratla et al. [2020] V. Piratla, P. Netrapalli, and S. Sarawagi, “Efficient domain generalization via common-specific low-rank decomposition,” in _ICML_ , 2020, pp. 7728–7738.
  * Hu et al. [2021] S. Hu, Z. Liao, J. Zhang, and Y. Xia, “Domain and content adaptive convolution for domain generalization in medical image segmentation,” _arXiv preprint arXiv:2109.05676_ , 2021.
  * Triantafillou et al. [2021] E. Triantafillou, H. Larochelle, R. Zemel, and V. Dumoulin, “Learning a universal template for few-shot dataset generalization,” _arXiv preprint arXiv:2105.07029_ , 2021.
  * Peng et al. [2020] X. Peng, Y. Li, and K. Saenko, “Domain2vec: Domain embedding for unsupervised domain adaptation,” in _European Conference on Computer Vision_ , 2020, pp. 756–774.
  * Ilse et al. [2020] M. Ilse, J. M. Tomczak, C. Louizos, and M. Welling, “Diva: Domain invariant variational autoencoders,” in _Proceedings of the Third Conference on Medical Imaging with Deep Learning_ , 2020.
  * Peng et al. [2019a] X. Peng, Z. Huang, X. Sun, and K. Saenko, “Domain agnostic learning with disentangled representations,” in _ICML_ , 2019.
  * Zhang et al. [2021b] H. Zhang, Y.-F. Zhang, W. Liu, A. Weller, B. Schölkopf, and E. P. Xing, “Towards principled disentanglement for domain generalization,” in _ICML2021 Machine Learning for Data Workshop_ , 2021.
  * Nam et al. [2021] H. Nam, H. Lee, J. Park, W. Yoon, and D. Yoo, “Reducing domain gap by reducing style bias,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 8690–8699.
  * Li et al. [2021c] D. Li, J. Yang, K. Kreis, A. Torralba, and S. Fidler, “Semantic segmentation with generative models: Semi-supervised learning and strong out-of-domain generalization,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 8300–8311.
  * Choi et al. [2021] S. Choi, S. Jung, H. Yun, J. T. Kim, S. Kim, and J. Choo, “Robustnet: Improving domain generalization in urban-scene segmentation via instance selective whitening,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 11 580–11 590.
  * Qian et al. [2021] H. Qian, S. J. Pan, C. Miao, H. Qian, S. Pan, and C. Miao, “Latent independent excitation for generalizable sensor-based cross-person activity recognition,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 35, no. 13, 2021, pp. 11 921–11 929.
  * Ouyang et al. [2021] C. Ouyang, C. Chen, S. Li, Z. Li, C. Qin, W. Bai, and D. Rueckert, “Causality-inspired single-source domain generalization for medical image segmentation,” _arXiv preprint arXiv:2111.12525_ , 2021.
  * Zhang et al. [2020] C. Zhang, K. Zhang, and Y. Li, “A causal view on robustness of neural networks,” in _Advances in Neural Information Processing Systems_ , 2020\.
  * Liu et al. [2021a] C. Liu, X. Sun, J. Wang, T. Li, T. Qin, W. Chen, and T.-Y. Liu, “Learning causal semantic representation for out-of-distribution prediction,” in _Advances in Neural Information Processing Systems_ , 2021.
  * Sun et al. [2021] X. Sun, B. Wu, X. Zheng, C. Liu, W. Chen, T. Qin, and T.-Y. Liu, “Recovering latent causal factor for generalization to distributional shifts,” in _Thirty-Fifth Conference on Neural Information Processing Systems_ , 2021\.
  * He et al. [2019] Y. He, Z. Shen, and P. Cui, “Towards non-i.i.d. image classification: A dataset and baselines,” _arXiv preprint arXiv:1906.02899_ , 2019.
  * Zhang et al. [2021c] X. Zhang, P. Cui, R. Xu, L. Zhou, Y. He, and Z. Shen, “Deep stable learning for out-of-distribution generalization,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 5372–5382.
  * Mancini et al. [2018a] M. Mancini, S. R. Bulò, B. Caputo, and E. Ricci, “Best sources forward: domain generalization through source-specific nets,” in _ICIP_ , 2018, pp. 1353–1357.
  * D’Innocente and Caputo [2018] A. D’Innocente and B. Caputo, “Domain generalization with domain-specific aggregation modules,” in _German Conference on Pattern Recognition_. Springer, 2018, pp. 187–198.
  * He et al. [2018] W. He, H. Zheng, and J. Lai, “Domain attention model for domain generalization in object detection,” in _PRCV_ , 2018, pp. 27–39.
  * Mancini et al. [2018b] M. Mancini, S. R. Bulo, B. Caputo, and E. Ricci, “Robust place categorization with deep domain generalization,” _IEEE Robotics and Automation Letters_ , vol. 3, no. 3, pp. 2093–2100, 2018.
  * Wu and Gong [2021] G. Wu and S. Gong, “Collaborative optimization and aggregation for decentralized domain generalization and adaptation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 6484–6493.
  * Dubey et al. [2021] A. Dubey, V. Ramanathan, A. Pentland, and D. Mahajan, “Adaptive methods for real-world domain generalization,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 14 340–14 349.
  * Dou et al. [2019] Q. Dou, D. C. de Castro, K. Kamnitsas, and B. Glocker, “Domain generalization via model-agnostic learning of semantic features,” in _NeurIPS_ , 2019.
  * Chen et al. [2022] K. Chen, D. Zhuang, and J. M. Chang, “Discriminative adversarial domain generalization with meta-learning based cross-domain validation,” _Neurocomputing_ , vol. 467, pp. 418–426, 2022.
  * Sharifi-Noghabi et al. [2020] H. Sharifi-Noghabi, H. Asghari, N. Mehrasa, and M. Ester, “Domain generalization via semi-supervised meta learning,” _arXiv preprint arXiv:2009.12658_ , 2020.
  * Wang et al. [2021c] B. Wang, M. Lapata, and I. Titov, “Meta-learning for domain generalization in semantic parsing,” in _NAACL_ , 2021.
  * Li et al. [2019b] D. Li, J. Zhang, Y. Yang, C. Liu, Y.-Z. Song, and T. M. Hospedales, “Episodic training for domain generalization,” in _CVPR_ , 2019, pp. 1446–1455.
  * Qiao and Peng [2021] F. Qiao and X. Peng, “Uncertainty-guided model generalization to unseen domains,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 6790–6800.
  * Zhao et al. [2021b] Y. Zhao, Z. Zhong, F. Yang, Z. Luo, Y. Lin, S. Li, and N. Sebe, “Learning to generalize unseen domains via memory-based multi-source meta-learning for person re-identification,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 6277–6286.
  * Kim et al. [2021a] J. Kim, J. Lee, J. Park, D. Min, and K. Sohn, “Self-balanced learning for domain generalization,” in _2021 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2021, pp. 779–783.
  * Huang et al. [2020] Z. Huang, H. Wang, E. P. Xing, and D. Huang, “Self-challenging improves cross-domain generalization,” in _ECCV_ , vol. 2, 2020.
  * Shi et al. [2021] Y. Shi, J. Seely, P. H. Torr, N. Siddharth, A. Hannun, N. Usunier, and G. Synnaeve, “Gradient matching for domain generalization,” _arXiv preprint arXiv:2104.09937_ , 2021.
  * Rame et al. [2021] A. Rame, C. Dancette, and M. Cord, “Fishr: Invariant gradient variances for out-of-distribution generalization,” _arXiv preprint arXiv:2109.02934_ , 2021\.
  * Tian et al. [2021] C. X. Tian, H. Li, X. Xie, Y. Liu, and S. Wang, “Neuron coverage-guided domain generalization,” _arXiv preprint arXiv:2103.00229_ , 2021.
  * Koh et al. [2021] P. W. Koh, S. Sagawa, H. Marklund, S. M. Xie, M. Zhang, A. Balsubramani, W. Hu, M. Yasunaga, R. L. Phillips, I. Gao _et al._ , “Wilds: A benchmark of in-the-wild distribution shifts,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 5637–5664.
  * Liu et al. [2021b] E. Z. Liu, B. Haghgoo, A. S. Chen, A. Raghunathan, P. W. Koh, S. Sagawa, P. Liang, and C. Finn, “Just train twice: Improving group robustness without training group information,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 6781–6792.
  * Sagawa et al. [2020a] S. Sagawa, P. W. Koh, T. B. Hashimoto, and P. Liang, “Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization,” in _International Conference on Learning Representations (ICLR)_ , 2020.
  * Wang et al. [2021d] J. Wang, Y. Li, L. Xie, and Y. Xie, “Class-conditioned domain generalization via wasserstein distributional robust optimization,” in _RobustML workshop at ICLR_ , 2021.
  * Carlucci et al. [2019b] F. M. Carlucci, A. D’Innocente, S. Bucci, B. Caputo, and T. Tommasi, “Domain generalization by solving jigsaw puzzles,” in _CVPR_ , 2019, pp. 2229–2238.
  * Jeon et al. [2021] S. Jeon, K. Hong, P. Lee, J. Lee, and H. Byun, “Feature stylization and domain-aware contrastive learning for domain generalization,” in _Proceedings of the 29th ACM International Conference on Multimedia_ , 2021, pp. 22–31.
  * Kim et al. [2021b] D. Kim, Y. Yoo, S. Park, J. Kim, and J. Lee, “Selfreg: Self-supervised contrastive regularization for domain generalization,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 9619–9628.
  * Li et al. [2021d] Z. Li, Z. Cui, S. Wang, Y. Qi, X. Ouyang, Q. Chen, Y. Yang, Z. Xue, D. Shen, and J.-Z. Cheng, “Domain generalization for mammography detection via multi-style and multi-view contrastive learning,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2021, pp. 98–108.
  * Ryu et al. [2019] J. Ryu, G. Kwon, M.-H. Yang, and J. Lim, “Generalized convolutional forest networks for domain generalization and visual recognition,” in _ICLR_ , 2019\.
  * Narayanan et al. [2021] M. Narayanan, V. Rajendran, and B. Kimia, “Shape-biased domain generalization via shock graph embeddings,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 1315–1325.
  * Zhang et al. [2021d] D. Zhang, K. Ahuja, Y. Xu, Y. Wang, and A. Courville, “Can subnetwork structure be the key to out-of-distribution generalization?” in _ICML_ , 2021\.
  * Cha et al. [2021] J. Cha, S. Chun, K. Lee, H.-C. Cho, S. Park, Y. Lee, and S. Park, “Swad: Domain generalization by seeking flat minima,” in _Thirty-Fifth Conference on Neural Information Processing Systems_ , 2021.
  * Faraki et al. [2021] M. Faraki, X. Yu, Y.-H. Tsai, Y. Suh, and M. Chandraker, “Cross-domain similarity learning for face recognition in unseen domains,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 15 292–15 301.
  * Ye et al. [2021] H. Ye, C. Xie, T. Cai, R. Li, Z. Li, and L. Wang, “Towards a theoretical framework of out-of-distribution generalization,” in _NeurIPS_ , 2021.
  * Adila and Kang [2021] D. Adila and D. Kang, “Understanding out-of-distribution: A perspective of data dynamics,” _arXiv preprint arXiv:2111.14730_ , 2021.
  * Shorten and Khoshgoftaar [2019] C. Shorten and T. M. Khoshgoftaar, “A survey on image data augmentation for deep learning,” _Journal of Big Data_ , vol. 6, no. 1, pp. 1–48, 2019.
  * Zhou et al. [2021c] K. Zhou, C. C. Loy, and Z. Liu, “Semi-supervised domain generalization with stochastic stylematch,” _arXiv preprint arXiv:2106.00592_ , 2021.
  * Kingma and Welling [2013] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” _arXiv:1312.6114_ , 2013.
  * Goodfellow et al. [2014] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial networks,” in _NIPS_ , 2014.
  * Zhang et al. [2018] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” in _ICLR_ , 2018.
  * Anoosheh et al. [2018] A. Anoosheh, E. Agustsson, R. Timofte, and L. Van Gool, “Combogan: Unrestrained scalability for image domain translation,” in _CVPR Workshop_ , 2018, pp. 783–790.
  * Gretton et al. [2012] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola, “A kernel two-sample test,” _The Journal of Machine Learning Research_ , vol. 13, no. 1, pp. 723–773, 2012.
  * Tolstikhin et al. [2017] I. Tolstikhin, O. Bousquet, S. Gelly, and B. Schoelkopf, “Wasserstein auto-encoders,” _arXiv preprint arXiv:1711.01558_ , 2017.
  * Huang and Belongie [2017] X. Huang and S. Belongie, “Arbitrary style transfer in real-time with adaptive instance normalization,” in _ICCV_ , 2017, pp. 1501–1510.
  * Bengio et al. [2013] Y. Bengio, A. Courville, and P. Vincent, “Representation learning: A review and new perspectives,” _IEEE TPAMI_ , vol. 35, no. 8, pp. 1798–1828, 2013\.
  * Ben-David et al. [2007] S. Ben-David, J. Blitzer, K. Crammer, F. Pereira _et al._ , “Analysis of representations for domain adaptation,” in _NIPS_ , vol. 19, 2007.
  * Cortes and Vapnik [1995] C. Cortes and V. Vapnik, “Support-vector networks,” _Machine learning_ , vol. 20, no. 3, pp. 273–297, 1995.
  * Pan et al. [2011] S. J. Pan, I. Tsang, J. T. Kwok, and Q. Yang, “Domain adaptation via transfer component analysis,” _IEEE TNN_ , vol. 22, pp. 199–210, 2011.
  * Zhou et al. [2020d] F. Zhou, Z. Jiang, C. Shui, B. Wang, and B. Chaib-draa, “Domain generalization with optimal transport and metric learning,” _ArXiv_ , vol. abs/2007.10573, 2020.
  * Peng et al. [2019b] X. Peng, Q. Bai, X. Xia, Z. Huang, K. Saenko, and B. Wang, “Moment matching for multi-source domain adaptation,” in _ICCV_ , 2019, pp. 1406–1415.
  * Pan et al. [2018] X. Pan, P. Luo, J. Shi, and X. Tang, “Two at once: Enhancing learning and generalization capacities via ibn-net,” in _ECCV_ , 2018, pp. 464–479.
  * Jia et al. [2019] J. Jia, Q. Ruan, and T. M. Hospedales, “Frustratingly easy person re-identification: Generalizing person re-id in practice,” 2019.
  * Nam and Kim [2018] H. Nam and H.-E. Kim, “Batch-instance normalization for adaptively style-invariant neural networks,” in _NeurIPS_ , 2018, pp. 2558–2567.
  * Tzeng et al. [2014] E. Tzeng, J. Hoffman, N. Zhang, K. Saenko, and T. Darrell, “Deep domain confusion: Maximizing for domain invariance,” _arXiv preprint arXiv:1412.3474_ , 2014.
  * Wang et al. [2018] J. Wang, W. Feng, Y. Chen, H. Yu, M. Huang, and P. S. Yu, “Visual domain adaptation with manifold embedded distribution alignment,” in _ACMMM_ , 2018, pp. 402–410.
  * Wang et al. [2020c] J. Wang, Y. Chen, W. Feng, H. Yu, M. Huang, and Q. Yang, “Transfer learning with dynamic distribution adaptation,” _ACM TIST_ , vol. 11, no. 1, pp. 1–25, 2020.
  * Sun et al. [2016] B. Sun, J. Feng, and K. Saenko, “Return of frustratingly easy domain adaptation,” in _AAAI_ , 2016.
  * Sun and Saenko [2016] B. Sun and K. Saenko, “Deep coral: Correlation alignment for deep domain adaptation,” in _ECCV_ , 2016, pp. 443–450.
  * Peng and Saenko [2018] X. Peng and K. Saenko, “Synthetic to real adaptation with generative correlation alignment networks,” in _WACV_. IEEE, 2018, pp. 1982–1991.
  * Ulyanov et al. [2017] D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis,” in _CVPR_ , 2017, pp. 6924–6932.
  * Dumoulin et al. [2017] V. Dumoulin, J. Shlens, and M. Kudlur, “A learned representation for artistic style,” _ICLR_ , 2017.
  * Ulyanov et al. [2016] D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Instance normalization: The missing ingredient for fast stylization,” _arXiv preprint arXiv:1607.08022_ , 2016\.
  * Sonar et al. [2021] A. Sonar, V. Pacelli, and A. Majumdar, “Invariant policy optimization: Towards stronger generalization in reinforcement learning,” in _Learning for Dynamics and Control_. PMLR, 2021, pp. 21–33.
  * Bui et al. [2021] H. Bui, T. Tran, A. T. Tran, and D. Phung, “Exploiting domain-specific features to enhance domain generalization,” in _Thirty-Fifth Conference on Neural Information Processing Systems_ , 2021.
  * Kang and Nandakumar [2021] C. Kang and K. Nandakumar, “Dynamically decoding source domain knowledge for unseen domain generalization,” _arXiv preprint arXiv:2110.03027_ , 2021.
  * Liu et al. [2021c] C. Liu, L. Wang, K. Li, and Y. Fu, “Domain generalization via feature variation decorrelation,” in _Proceedings of the 29th ACM International Conference on Multimedia_ , 2021, pp. 1683–1691.
  * Wang et al. [2021e] Y. Wang, H. Li, L.-P. Chau, and A. C. Kot, “Variational disentanglement for domain generalization,” _arXiv preprint arXiv:2109.05826_ , 2021.
  * Schölkopf et al. [2012] B. Schölkopf, D. Janzing, J. Peters, E. Sgouritsa, K. Zhang, and J. M. Mooij, “On causal and anticausal learning,” in _International Conference on Machine Learning (ICML 2012)_. International Machine Learning Society, 2012, pp. 1255–1262.
  * Zhang et al. [2013] K. Zhang, B. Schölkopf, K. Muandet, and Z. Wang, “Domain adaptation under target and conditional shift,” in _International Conference on Machine Learning_ , 2013, pp. 819–827.
  * Zhang et al. [2015] K. Zhang, M. Gong, and B. Schölkopf, “Multi-source domain adaptation: A causal view,” in _Twenty-ninth AAAI conference on artificial intelligence_ , 2015.
  * Gong et al. [2018] M. Gong, K. Zhang, B. Huang, C. Glymour, D. Tao, and K. Batmanghelich, “Causal generative domain adaptation networks,” _arXiv preprint arXiv:1804.04333_ , 2018.
  * Heinze-Deml and Meinshausen [2019] C. Heinze-Deml and N. Meinshausen, “Conditional variance penalties and domain shift robustness,” _stat_ , vol. 1050, p. 13, 2019.
  * Finn et al. [2017] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in _ICML_ , 2017.
  * Snell et al. [2017] J. Snell, K. Swersky, and R. S. Zemel, “Prototypical networks for few-shot learning,” in _NeurIPS_ , 2017.
  * Santoro et al. [2016] A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap, “Meta-learning with memory-augmented neural networks,” in _ICML_ , 2016, pp. 1842–1850.
  * Rahimian and Mehrotra [2019] H. Rahimian and S. Mehrotra, “Distributionally robust optimization: A review,” _arXiv preprint arXiv:1908.05659_ , 2019.