1518Domain Adversarial Neural Networks for Domain Generalization: When It
Works and How to Improve

1012Anthony Sicilia1, Xingchen Zhao2, Seong Jae Hwang2 10121Intelligent
Systems Program, 2Department of Computer Science, University of Pittsburgh
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Theoretically, domain adaptation is a well-researched problem. Further, this
theory has been well-used in practice. In particular, we note the bound on
target error given by Ben-David et al. (2010) and the well-known domain-
aligning algorithm based on this work using Domain Adversarial Neural Networks
(DANN) presented by Ganin and Lempitsky (2015). Recently, multiple variants of
DANN have been proposed for the related problem of domain generalization, but
without much discussion of the original motivating bound. In this paper, we
investigate the validity of DANN in domain generalization from this
perspective. We investigate conditions under which application of DANN makes
sense and further consider DANN as a dynamic process during training. Our
investigation suggests that the application of DANN to domain generalization
may not be as straightforward as it seems. To address this, we design an
algorithmic extension to DANN in the domain generalization case. Our
experimentation validates both theory and algorithm.

## 1 Introduction

In general, in machine learning, we assume the training data for our learning
algorithm is well representative of the testing data. That is, we assume our
training data follows the same distribution as our testing data. Of primary
interest to this paper is the case where this assumption fails to hold: we
consider learning in the presence of multiple domains. We formalize the
multiple domain problem of interest as the case where (at train-time) we
observe $k$ domains referred to as sources which have distributions
$\mathbb{P}_{1},\mathbb{P}_{2},\ldots,\mathbb{P}_{k}$ over some space
$\mathcal{X}$. At test-time, we are evaluated on a distinct target domain
which has distribution $\mathbb{Q}$ over $\mathcal{X}$.111Notice, the
distribution shift we consider occurs over the feature space $\mathcal{X}$.
Often, this is called a co-variate shift assumption. Typically, we assume some
restriction on observation of the target domain at train-time. In the
literature, a large amount of work is concerned with the problem of Domain
Adaptation (DA) which assumes access to samples from $\mathbb{Q}$, but
restricts access to the labels of these samples. More recently, there has also
been an active investigation on the problem of Domain Generalization (DG),
which instead, assumes absolutely no access to the target domain. In spite of
these restrictions, in both cases, the goal is for our learning algorithm
trained on sources to perform well when evaluated on the target.

One popular approach to DA is the use of a Domain Adversarial Neural Network
(DANN) originally proposed by Ganin and Lempitsky (2015). Intuitively, this
approach attempts to align the source and target domains by learning feature
representations of both which are indiscernible by a domain discriminator
trained to distinguish between the two distributions. Without appealing to
theory, this seems like a sensible approach to DA. By accomplishing this
domain alignment, the neural network should still be adept at the learned task
when it is evaluated on the target domain at test-time. While DANN was
originally proposed for DA, the adoption of this reasoning has motivated
adaptations of this approach for DG (Matsuura and Harada, 2020, Albuquerque et
al., 2020, Li et al., 2018c; b). In fact, very early works in DG (Muandet et
al., 2013) are similarly motivated by the goal of domain-agnostic feature
representations.

Still, it is worth noting that the original proposal of DANN (Ganin and
Lempitsky, 2015) was grounded in theory. In particular, Ganin and Lempitsky
base their algorithm on the target-error bound given by Ben-David et al.
(2007; 2010). The bound explicitly motivates domain alignment, but
importantly, motivates alignment between the source and target. Counter to
this, DANN variants for DG generally align multiple source domains because no
access to target data is permitted. This shortcoming gives rise to the
question of primary interest to this paper: Is there a justification for
source alignment using DANN in DG?

Specifically, we are concerned with a target-error bound similar to those
provided by Ben-David et al. (2010). To answer this question, we appeal to a
recent theoretical proposal by Albuquerque et al. (2020) which uses a
reference object (i.e., the set of mixture distributions of the sources) to
derive a such a target-error bound in the domain generalization setting.
Building on this framework, we provide answers to two important
considerations:

  1. 1.

What additional reference objects (besides sets of mixture distributions)
satisfy the primary condition used to derive target-error bounds?

  2. 2.

How does the target-error bound behave as a dynamic quantity during the
training process?

Ultimately, answering these two questions allows us to formulate a novel
extension of the Domain Adversarial Neural Network. We validate experimentally
that this extension improves performance and otherwise agrees with our
theoretical expectations.

## 2 Domain Adversarial Neural Network (DANN)

In this section, we cover the necessary background on Domain Adversarial
Neural Networks (DANN). We first present the original bound on target-error in
the case of unsupervised DA (Ben-David et al., 2007; 2010) which motivates the
DANN algorithm proposed by Ganin and Lempitsky (2015). Following this, we
outline the key differences of a DANN variant proposed by Matsuura and Harada
(2020). Although this variant achieves state-of-the-art (DANN) performance in
DG, we point out the main concerns we have regarding the justification of this
approach.

### 2.1 In Domain Adaptation

As mentioned, we begin with a motivating result of Ben-David et al. (2010).
Intuitively, this result describes bounds on the target-error controlled, in
part, by a computable measure of divergence between distributions. While we
provide a more detailed exposition of the problem setup in Appendix A, we
begin by listing here the key terms to familiarize the reader.

##### Setup

For a binary hypothesis $h$, a distribution $\mathbb{P}$, and a binary
labeling function $f$ for $\mathbb{P}$, we define the error
$\mathcal{E}_{\mathbb{P}}(h)$ of $h$ on the distribution $\mathbb{P}$ in
accordance with Ben-David et al. (2007; 2010) as follows

$\small\mathcal{E}_{\mathbb{P}}(h)=\mathbf{E}_{x\sim\mathbb{P}}\left|h(x)-f(x)\right|=\mathbf{E}_{x\sim\mathbb{P}}\left[\mathbb{I}[h(x)\neq
f(x)]\right].$ (1)

This is our primary measure of the quality of a hypothesis when predicting on
a distribution $\mathbb{P}$. To measure differences in distribution, we use
the $\mathcal{H}$-divergence which is an adaptation of the
$\mathcal{A}$-distance (Kifer et al., 2004). In particular, given two
distributions $\mathbb{P}$, $\mathbb{Q}$ over a space $\mathcal{X}$ and a
corresponding hypothesis class
$\mathcal{H}\subseteq\\{h\mid\mathcal{X}\to\\{0,1\\}\\}$, the
$\mathcal{H}$-divergence (Ben-David et al., 2010) is defined

$\small
d_{\mathcal{H}}(\mathbb{P},\mathbb{Q})=2\sup\nolimits_{h\in\mathcal{H}}\left|\mathrm{Pr}_{\mathbb{P}}(I_{h})-\mathrm{Pr}_{\mathbb{Q}}(I_{h})\right|$
(2)

where $I_{h}=\\{x\in\mathcal{X}\mid h(x)=1\\}$. Generally, it is more useful
to consider the the $\mathcal{H}\Delta\mathcal{H}$-divergence, specifically,
where Ben-David et al. (2010) define the symmetric difference hypothesis class
$\mathcal{H}\Delta\mathcal{H}$ as the set of functions characteristic to
disagreements between hypotheses.222Specifically,
$g\in\mathcal{H}\Delta\mathcal{H}\quad\textrm{if and only if}\quad
g(x)=h_{1}(x)\oplus h_{2}(x)=|h_{1}(x)-h_{2}(x)|\quad\textrm{for}\quad
h_{1},h_{2}\in\mathcal{H}$ This special case of the $\mathcal{H}$-divergence
will be the measure of divergence in all considered bounds.

####  2.1.1 The Motivating Bound

We can now present the result of Ben-David et al. (2010) based on the triangle
inequality of classification error (Crammer et al., 2007, Ben-David et al.,
2007). This bound is the key motivation behind DANN (Ganin and Lempitsky,
2015). For proof and a discussion on sample complexity, see Appendix A.

######  Theorem 2.1 (modified from Ben-David et al. (2010); Theorem 2).

Let $\mathcal{X}$ be a space and $\mathcal{H}$ be a class of hypotheses
corresponding to this space. Suppose $\mathbb{P}$ and $\mathbb{Q}$ are
distributions over $\mathcal{X}$. Then for any $h\in\mathcal{H}$, the
following holds

$\small\mathcal{E}_{\mathbb{Q}}(h)\leq\lambda+\mathcal{E}_{\mathbb{P}}(h)+\tfrac{1}{2}d_{\mathcal{H}\Delta\mathcal{H}}(\mathbb{Q},\mathbb{P})$
(3)

with $\lambda$ the error of an ideal joint hypothesis for $\mathbb{Q}$,
$\mathbb{P}$.

This statement provides an upper bound on the target-error. Thus, minimizing
this upper bound is a good proxy for the minimization of the target-error
itself. While the first term $\lambda$ is a property of the dataset and
hypothesis class, the latter terms are more controllable. The source-error
$\mathcal{E}_{\mathbb{P}}(h)$ can be minimized as usual by Empirical Risk
Minimization (ERM). The divergence can be empirically computed using the
result of Ben-David et al. (2010). While we give this result in the Appendix
(Proposition A.1 and Proposition A.2, respectively), previous interpretation
by Ganin and Lempitsky (2015) suggests to minimize the divergence by learning
indiscernible representations of the distributions – i.e., aligning the
domains. As we describe in detail in the following, this may be accomplished
by maximizing the errors of a domain discriminator trained to distinguish the
distributions.

####  2.1.2 The DANN Algorithm

Ganin and Lempitsky (2015) separate the neural network used to learn the task
into a feature extractor network $r_{\theta}$ and task-specific network
$c_{\sigma}$, parameterized respectively by $\theta$ and $\sigma$. A binary
domain discriminator $d_{\mu}$ outputting probabilities is trained to
distinguish between the source and target distribution based on the
representation learned by $r_{\theta}$. Meanwhile, $r_{\theta}$ is trained to
learn a representation that is not only useful for the task at hand, but also
adept at “fooling” the domain discriminator (i.e., maximizing its errors). In
details, given a source distribution $\mathbb{P}$ and a target distribution
$\mathbb{Q}$, the domain adversarial training objective is described

$\small\begin{split}\min\nolimits_{\mu}\max\nolimits_{\theta}\mathbf{E}_{(x,x^{\prime})\sim(\mathbb{P},\mathbb{Q})}\
\mathcal{L}_{D}(\mu,\theta,x)+\mathcal{L}_{D}(\mu,\theta,x^{\prime})\end{split}$
(4)

where $d_{\mu}$ produces the probability $x\sim\mathbb{Q}$, $\mathbb{I}$ is
the indicator function, and

$\small\begin{split}-\mathcal{L}_{D}(\mu,\theta,x)=&\mathbb{I}[x\sim\mathbb{P}]\log(1-d_{\mu}\circ
r_{\theta}(x))\\\ &+\mathbb{I}[x\sim\mathbb{Q}]\log(d_{\mu}\circ
r_{\theta}(x)).\end{split}$ (5)

Therefore, $\mathcal{L}_{D}$ represents the binary cross-entropy loss for a
domain discriminator trained to distinguish $\mathbb{P}$ and $\mathbb{Q}$.
Combining this with a task-specific loss $\mathcal{L}_{T}$ we get the
formulation given by Ganin and Lempitsky (2015)

$\small\begin{split}\min\nolimits_{\sigma,\theta}&\max\nolimits_{\mu}\mathbf{E}_{x\sim\mathbb{P}}[\mathcal{L}_{T}(\sigma,\theta,x)]\\\
&\ \ \ \
-\lambda\mathbf{E}_{(x,x^{\prime})\sim(\mathbb{P},\mathbb{Q})}[\mathcal{L}_{D}(\mu,\theta,x)+\mathcal{L}_{D}(\mu,\theta,x^{\prime})]\end{split}$
(6)

where $\lambda$ is a trade-off parameter. The above is generally implemented
by simultaneous gradient descent. We remark that this is easily accomplished
by incorporating a Gradient Reversal Layer (GRL) between $r_{\theta}$ and
$d_{\mu}$ (Ganin and Lempitsky, 2015).

### 2.2 In Domain Generalization

Recent adaptions to the above formulation for DG have been proposed. Here, we
focus on the proposal of Matsuura and Harada (2020) since their empirical
results are one of the more competitive DG methods to date. In DG, since no
access to $\mathbb{Q}$ is given, one cannot actually compute $\mathcal{L}_{D}$
as described above – it assumes at least unlabeled examples from $\mathbb{Q}$.
Given this, Matsuura and Harada (2020) propose a modification operating on
multiple source distributions
$\mathbb{P}_{1},\mathbb{P}_{2},\ldots,\mathbb{P}_{k}$

$\small-\mathcal{L}_{SD}(\mu,\theta,x)=\sum\nolimits_{i}\mathbb{I}[x\sim\mathbb{P}_{i}]\log((d_{\mu}\circ
r_{\theta}(x))_{i})$ (7)

where now $d_{\mu}$ is a multi-class domain discriminator outputting
probabilities and trained to distinguish between sources. $\mathcal{L}_{SD}$
is the multi-class cross-entropy loss. We substitute this into Eq. (6):

$\small\begin{split}\min\nolimits_{\sigma,\theta}\max\nolimits_{\mu}\mathbf{E}_{(x_{j})_{j}\sim(\mathbb{P}_{j})_{j}}&\left[\sum\nolimits_{j}\mathcal{L}_{T}(\sigma,\theta,x_{j})\right.\\\
&-\lambda\left.\sum\nolimits_{j}\mathcal{L}_{SD}(\mu,\theta,x_{j})\right]\end{split}$
(8)

which gives a domain adversarial training objective aimed at aligning the
sources. Hereon, we often refer to this as a source-source DANN, rather than a
source-target DANN. On the surface, there seems to be no justification for the
source-source DANN. If we recall the interpretation of Theorem 2.1, there is
one key difference: rather than approximating the minimization of the source-
target divergence $d_{\mathcal{H}\Delta\mathcal{H}}(\mathbb{Q},\mathbb{P})$,
the objective in Eq. (8) approximates minimization of some combination of the
source-source divergences
$\\{d_{\mathcal{H}\Delta\mathcal{H}}(\mathbb{P}_{i},\mathbb{P}_{j})\\}_{i,j}$
which do not appear in the upper bound. If we look to recent literature,
preliminary theoretical work to motivate this modification of DANN does exist
(Albuquerque et al., 2020). In the next section, we start from this work in
the derivation of our own results.

## 3 Understanding Domain Alignment in Domain Generalization

Our discussion of source-source DANN for DG begins with the motivating target-
error bound proposed by Albuquerque et al. (2020). Originally, given a set of
source distributions $\\{\mathbb{P}_{i}\\}$, the bound uses the set of mixture
distributions having these sources as components – we refer to this set as
$\mathcal{M}$. Below, we consider a more general adaptation of this result.
Although the proof strategy is largely similar, we do provide a proof for this
re-statement in Appendix A.

######  Proposition 3.1.

(adapted from Albuquerque et al. (2020); Proposition 2) Let $\mathcal{X}$ be a
space and let $\mathcal{H}$ be a class of hypotheses corresponding to this
space. Let $\mathbb{Q}$ and the collection $\\{\mathbb{P}_{i}\\}_{i=1}^{k}$ be
distributions over $\mathcal{X}$ and let $\\{\varphi_{i}\\}_{i=1}^{k}$ be a
collection of non-negative coefficients with $\sum_{i}\varphi_{i}=1$. Let the
object $\mathcal{O}$ be a set of distributions such that for every
$\mathbb{S}\in\mathcal{O}$ the following holds

$\small\sum\nolimits_{i}\varphi_{i}d_{\mathcal{H}\Delta\mathcal{H}}(\mathbb{P}_{i},\mathbb{S})\leq\max\nolimits_{i,j}d_{\mathcal{H}\Delta\mathcal{H}}(\mathbb{P}_{i},\mathbb{P}_{j}).$
(9)

Then, for any $h\in\mathcal{H}$,

$\small\begin{split}\mathcal{E}_{\mathbb{Q}}(h)\leq\lambda_{\varphi}+\sum\nolimits_{i}\varphi_{i}\mathcal{E}_{\mathbb{P}_{i}}(h)&+\tfrac{1}{2}\min\nolimits_{\mathbb{S}\in\mathcal{O}}d_{\mathcal{H}\Delta\mathcal{H}}(\mathbb{S},\mathbb{Q})\\\
&+\tfrac{1}{2}\max\nolimits_{i,j}d_{\mathcal{H}\Delta\mathcal{H}}(\mathbb{P}_{i},\mathbb{P}_{j})\end{split}$
(10)

where $\lambda_{\varphi}=\sum_{i}\varphi_{i}\lambda_{i}$ and each
$\lambda_{i}$ is the error of an ideal joint hypothesis for $\mathbb{Q}$ and
$\mathbb{P}_{i}$.

As suggested in Albuquerque et al. (2020), interpreting this result provides a
reasonable motivation for the use of source-source DANN in DG. The second term
is a convex combination of the source errors where the coefficients are
assumed to be fixed, but arbitrary, replicating a natural data generation
process where amounts of data from each source are not assumed. Ben-David et
al. (2010) model data arising from multiple sources in this way and provide
generalization bounds as well. ERM on a mixture of the sources is appropriate
for controlling this term. When $\mathcal{O}$ is fixed as the set of mixtures
$\mathcal{M}$, Albuquerque et al. (2020) suggest the the third term
demonstrates the importance of diverse source distributions, so that the
unseen target $\mathbb{Q}$ might be “near” $\mathcal{M}$. We extend this
discussion later, showing how this term can change dynamically throughout the
training process. The final term is a maximum over the source-source
divergences. Application of the interpretation by Ganin and Lempitsky (2015) –
to approximate minimizing divergence by maximizing a domain discriminator’s
errors – motivates the suggestion of Matsuura and Harada (2020) to maximize
the errors of a multi-class (source-source) domain discriminator. A more
precise application might be to train all combinations of binary domain
discriminator, but as Albuquerque et al. (2020) point out, this leads to a
polynomial number of discriminators. As a practical surrogate, we opt to
employ the best empirical strategy to date (Matsuura and Harada, 2020). See
also Albuquerque et al. (2020) which instead uses a collection of one-versus-
all classifiers in place of a multi-class classifier. As neither method
precisely minimizes Eq. (10), we treat this as an implementation choice.

##### A Remark on Differences

As mentioned briefly, a reader familiar with the original statement of
Albuquerque et al. (2020) will notice two differences: 1) rather than limiting
consideration to the set of mixtures $\mathcal{M}$, this statement holds for
all sets $\mathcal{O}$ which satisfy Condition (9) and 2) $\lambda_{\varphi}$
is a different quantity for the ideal joint-error between $\mathbb{Q}$ and
$\\{\mathbb{P}_{i}\\}$ – we comment on this difference in Appendix A, but it
is largely unimportant, as both quantities are uncontrollable and usually
assumed to be small in the case of co-variate shift.

The first difference is of primary interest. Condition (9) may be considered
to be the key fact about $\mathcal{M}$ which allows the derivation of Eq.
(10). By identifying this, we open the possibility of considering more general
objects satisfying Condition (9). In the following, we demonstrate the
existence of such objects $\mathcal{O}$ and discuss the benefit they add.

### 3.1 Beyond Mixture Distributions

Consideration of general objects $\mathcal{O}$ which satisfy Condition (9) is
only useful if such objects exist (besides $\mathcal{M}$). The following
example provides proof. See Appendix B for an illustrative picture.

######  Example 3.1.

Let $\mathcal{X}$ be the real line $(-\infty,\infty)$ and let $\mathcal{H}$ be
the set of hypotheses $\\{h_{a}(.)\\}_{a\in\mathbb{R}}$ where $h_{a}(.)$ is
characteristic to the ray $(-\infty,a]$. Then, $\mathcal{H}\Delta\mathcal{H}$
is the set of hypotheses $\\{h_{a,b}(.)\\}_{(a,b)\in\mathbb{R}^{2}}$ where
$h_{a,b}(.)$ is characteristic to the interval $[a,b]$. Let $\mathbb{P}_{1}$
be the uniform distribution $\mathcal{U}(0,2)$, let $\mathbb{P}_{2}$ be
$\mathcal{U}(2,4)$, and let $\mathbb{S}$ be $\mathcal{U}(1,3)$. Then
$\mathbb{S}$ is not a mixture distribution of the components $\mathbb{P}_{1}$
and $\mathbb{P}_{2}$, but

$\small\begin{split}2&=\max\nolimits_{i,j}d_{\mathcal{H}\Delta\mathcal{H}}(\mathbb{P}_{i},\mathbb{P}_{j})\geq\sum\nolimits_{i}\varphi_{i}d_{\mathcal{H}\Delta\mathcal{H}}(\mathbb{P}_{i},\mathbb{S})\end{split}$
(11)

for all non-negative coefficients $\\{\varphi_{i}\\}_{i}$ which sum to 1.

In the context of this example, we might consider the object
$\mathcal{O}=\mathcal{M}+\\{\mathbb{S}\\}$ to quickly see that more than just
$\mathcal{M}$ can satisfy Condition (9). If $\mathbb{S}$ is a unique minimizer
of the third term in Eq. (10) and does not increase the final term, then using
$\mathcal{O}$ in place of $\mathcal{M}$ actually produces a strictly tighter
bound. Later we more generally expand on this and other benefits of
considering $\mathcal{O}\neq\mathcal{M}$.

Still, one simple example cannot fully justify the existence of useful
$\mathcal{O}\neq\mathcal{M}$. For a more general perspective, it is useful to
think of things geometrically. Albuquerque et al. (2020) often refer to
$\mathcal{M}$ as the convex-hull of the sources. In this same vein, we point
out that $d_{\mathcal{H}\Delta\mathcal{H}}$ is a pseudometric333In Appendix A,
we show the commonly used fact that $d_{\mathcal{H}\Delta\mathcal{H}}$
possesses a triangle-inequality. Symmetry and evaluation to $0$ for identical
distributions is easy to see. and therefore, shares most of the nice
properties required of metrics used in the vast mathematical literature on
metric spaces. Viewing a metric space as a topological space, it is common to
think of open balls as the “the fundamental unit” or “basis” of the metric
space. Loosely, borrowing this idea, we can define the (closed)
$\mathcal{H},\rho$-ball as below

$\small\mathcal{B}_{\rho}(\mathbb{P})=\\{\mathbb{S}\mid
d_{\mathcal{H}\Delta\mathcal{H}}(\mathbb{P},\mathbb{S})\leq\rho\\}.$ (12)

Using this object, the following result provides some useful information on
the types of objects $\mathcal{O}$ which satisfy Condition (9). We provide
proof in Appendix B.

######  Proposition 3.2.

Let $\mathcal{X}$ be a space and let $\mathcal{H}$ be a class of hypotheses
corresponding to this space. Let the collection
$\\{\mathbb{P}_{i}\\}_{i=1}^{k}$ be distributions over $\mathcal{X}$ and let
$\\{\varphi_{i}\\}_{i=1}^{k}$ be a collection of non-negative coefficients
with $\sum_{i}\varphi_{i}=1$. Now, set
$\rho=\max\nolimits_{u,v}d_{\mathcal{H}\Delta\mathcal{H}}(\mathbb{P}_{u},\mathbb{P}_{v})$.
We show three results,

  1. 1.

$\mathcal{M}\subseteq\bigcap\nolimits_{i}\mathcal{B}_{\rho}(\mathbb{P}_{i})$.

  2. 2.

If $\mathbb{S}\in\bigcap\nolimits_{i}\mathcal{B}_{\rho}(\mathbb{P}_{i})$, then
Condition (9) holds.

  3. 3.

If $\mathbb{S}\notin\bigcup\nolimits_{i}\mathcal{B}_{\rho}(\mathbb{P}_{i})$,
then Condition (9) fails to hold.

Statements 1 and 2 in conjunction show there are intuitive objects
$\mathcal{O}$ – i.e., $\bigcap\nolimits_{i}\mathcal{B}_{\rho}(\mathbb{P}_{i})$
– which both contain $\mathcal{M}$ and satisfy Condition (9). Statement 3
provides an intuitive boundary for $\mathcal{O}$. Comparison of $\mathcal{O}$
to the union and intersection of closed balls, respectively, provides
necessary and sufficient conditions for satisfying Condition (9). For a
helpful illustration, we direct the reader to Figure 1.

Figure 1: An informal visualization. Blue dots represent sources. Purple lines
define the boundaries of $\mathcal{M}$. Grey lines give the boundaries of the
closed $\mathcal{H},\rho$-balls around each source (defined in Proposition
3.2). Green colored areas define the boundary of
$\bigcap\nolimits_{i}\mathcal{B}_{\rho}(\mathbb{P}_{i})$. Distributions within
the yellow area may satisfy Condition (9). Distributions outside the yellow
area (red dots) do not satisfy Condition (9).

####  3.1.1 The Benefits of Looking Beyond Mixtures

While the above discussion is useful in its own right, a more careful
discussion of practical ramifications is needed.

First, we point out that different objects $\mathcal{O}$ can lead to
computationally tighter bounds in Eq. (10). For example, we prove
$\bigcap\nolimits_{i}\mathcal{B}_{\rho}(\mathbb{P}_{i})$ can lead to tighter
bounds than $\mathcal{M}$ in Appendix B. The proof follows a similar logic as
presented following Example 3.1. In fact, for Example 3.1, it is true that
$\bigcap\nolimits_{i}\mathcal{B}_{\rho}(\mathbb{P}_{i})$ contains
$\mathcal{M}+\\{\mathbb{S}\\}$, and thus, may reap the discussed benefit. But,
for DANN, our hypothesis will usually be a neural network. In this case,
tightness is often considered irrelevant because the large VC-Dimension of
neural networks Bartlett et al. (2019) is the dominant term in any bound on
error (i.e., using the PAC framework). Still, this conversation is not
complete without considering the recent success of PAC-Bayesian formulations
(e.g., Dziugaite and Roy (2017)) which provide much tighter bounds when the
hypothesis is a stochastic neural network. In Appendix A, we discuss a PAC-
Bayesian distribution psuedometric Germain et al. (2020) analogous to
$d_{\mathcal{H}\Delta\mathcal{H}}$. Because this psuedometric shares the
important properties of $d_{\mathcal{H}\Delta\mathcal{H}}$, these results are
easily re-framed in this more modern formulation as well – where tightness may
be a primary concern.

Second, we point out that a particular object $\mathcal{O}$ can be easier to
analyze. This fact will become evident as we develop an algorithmic extension
to DANN for DG. Ultimately, we find that the novel object
$\bigcap\nolimits_{i}\mathcal{B}_{\rho}(\mathbb{P}_{i})$ may be manipulated to
provide key motivating insights in algorithm design.

### 3.2 The $\mathcal{H}\Delta\mathcal{H}$-Divergence as a Dynamic Quantity

As mentioned, Albuquerque et al. (2020) interpret Proposition 3.1 as showing
the necessity of diverse source distributions to control the third term
$\min_{\mathbb{S}\in\mathcal{O}}d_{\mathcal{H}\Delta\mathcal{H}}(\mathbb{S},\mathbb{Q})$
when $\mathcal{O}=\mathcal{M}$. Logically, when distributions are
heterogeneous, $\mathcal{M}$ presumably contains more elements, and so, the
unseen target is more likely to be “close.” When
$\mathcal{O}=\bigcap\nolimits_{i}\mathcal{B}_{\rho}(\mathbb{P}_{i})$, this is
easier to see because the size of $\mathcal{O}$ is directly dependent on the
maximum divergence between the sources (by the definition of $\rho$). In
particular, reducing the maximum divergence and re-computing $\mathcal{O}$
could lead to removal of a unique minimizer for
$\min_{\mathbb{S}\in\mathcal{O}}d_{\mathcal{H}\Delta\mathcal{H}}(\mathbb{S},\mathbb{Q})$.444The
newly computed $\mathcal{O}$ will be a subset and this unique minimizer might
be absent in this subset, increasing the minimum. In the context of the DANN
algorithm, this is worrisome. Unfortunately, during training, the point of
using DANN is to effectively reduce the maximum divergence between sources –
we expect this divergence to be decreasing as the feature representations of
the source distributions are modified. In fact, under mild assumptions, we can
formally show that DANN acts like a contraction mapping, and therefore, can
only decrease the pairwise source-divergences. As a result, we expect
$\min_{\mathbb{S}\in\mathcal{O}}d_{\mathcal{H}\Delta\mathcal{H}}(\mathbb{S},\mathbb{Q})$
to increase as the changing object $\mathcal{O}$ shrinks during training.
Below we consider gradient descent on a smooth proxy of the
$\mathcal{H}\Delta\mathcal{H}$-Divergence in the simple, two-distribution
case. $r_{\theta}$ acts as the feature extractor effected by DANN. Proof is
provided in Appendix B.

######  Proposition 3.3.

Let $\mathfrak{D}$ be a space of empirical samples over $\mathcal{X}$. Let
$r_{\theta}:\mathcal{X}\to\mathcal{X}$ be a deterministic representation
function parameterized by the real vector $\theta\in\mathbb{R}^{m}$. Further,
denote by $r_{\theta}(\widehat{\mathbb{P}})$ the application of $r_{\theta}$
to every point of $\widehat{\mathbb{P}}\in\mathfrak{D}$. Fix
$\widehat{\mathbb{P}},\widehat{\mathbb{Q}}\in\mathfrak{D}$, let
$\mathcal{L}:\mathfrak{D}\times\mathfrak{D}\to[0,\infty)$. Define
$\ell(\theta)=\mathcal{L}(r_{\theta}(\widehat{\mathbb{P}}),r_{\theta}(\widehat{\mathbb{Q}}))$
and suppose it is differentiable with $K$-Lipschitz gradients. Further,
suppose $\theta^{*}$ is the unique local minimum of $\ell$ on a bounded subset
$\Omega\subset\mathbb{R}^{m}$. Then for $\theta\in\Omega$ such that
$\theta\neq\theta^{*}$, the function $\tau:\Omega\to\mathbb{R}^{m}$ defined
$\tau(\theta)=\theta-\gamma\nabla_{\theta}\ell(\theta)$ has the property

$\small\mathcal{L}(r_{\tau(\theta)}(\widehat{\mathbb{P}}),r_{\tau(\theta)}(\widehat{\mathbb{Q}}))\leq\beta_{\theta}\mathcal{L}(r_{\theta}(\widehat{\mathbb{P}}),r_{\theta}(\widehat{\mathbb{Q}}))$
(13)

for some constant $\beta_{\theta}$ dependent on $\theta$. In particular, for
all $\theta\in\Omega$, there is $\gamma$ so that $0<\beta_{\theta}<1$.

A key takeaway from the above is the presence of competing objectives during
training which require balance. While DANN reduces the source-divergences to
account for the final term in Eq. (10), we should also (somehow) consider the
diversity of our sources throughout training to account for the effected term
$\min_{\mathbb{S}\in\mathcal{O}}d_{\mathcal{H}\Delta\mathcal{H}}(\mathbb{S},\mathbb{Q})$.

## 4 An Algorithmic Extension to DANN

Motivated by the argument presented in Section 3.2, this section devises an
extension to DANN. While DANN acts to align domains, as noted, its success in
the context of domain generalization is also dependent on the heterogeneity of
the source distributions throughout the training process. Therefore, in an
attempt to balance these objectives, we propose an addition to source-source
DANN which acts to diversify the sources throughout the training.

##### Theoretical Motivation

We recall the intersection of closed balls
$\bigcap\nolimits_{i}\mathcal{B}_{\rho}(\mathbb{P}_{i})$. Taking
$\mathcal{O}=\bigcap\nolimits_{i}\mathcal{B}_{\rho}(\mathbb{P}_{i})$, this is
the main object of interest as it controls the size of the divergences in the
upper bound of Proposition 3.1. More specifically, we are concerned with the
quantity
$\min_{\mathbb{P}\in\bigcap\nolimits_{i}\mathcal{B}_{\rho}(\mathbb{P}_{i})}d_{\mathcal{H}\Delta\mathcal{H}}(\mathbb{P},\mathbb{Q})$.
Intuitively, if we want to reduce this quantity we should find some means to
increase $\rho$. One might propose to accomplish this by modifying our source
distributions – e.g., through data augmentation –, but clearly, modifying our
source distributions in an uncontrolled manner is not wise. This ignores the
structure of the space of distributions under consideration and whichever
distribution governs our sampling from this space – information that is, in
part, given by our sample of sources itself. In this sense, while increasing
$\rho$, we should preserve the structure of
$\bigcap\nolimits_{i}\mathcal{B}_{\rho}(\mathbb{P}_{i})$ as much as possible.
Proposition 4.1 identifies conditions we must satisfy if we wish to increase
$\rho$ by modifying our source distributions while guaranteeing that we do not
substantially modify $\bigcap\nolimits_{i}\mathcal{B}_{\rho}(\mathbb{P}_{i})$.
Proof is provided in Appendix B.

######  Proposition 4.1.

Let $\mathcal{X}$ be a space and let $\mathcal{H}$ be a class of hypotheses
corresponding to this space. Let $\mathfrak{D}$ be the space of distributions
over $\mathcal{X}$ and let the collection $\\{\mathbb{P}_{i}\\}_{i=1}^{k}$ and
the collection $\\{\mathbb{R}_{i}\\}_{i=1}^{k}$ be contained in
$\mathfrak{D}$. Now, consider the collection of mixture distributions
$\\{\mathbb{S}_{i}\\}_{i}$ defined so that for each set $A$,
$\mathrm{Pr}_{\mathbb{S}_{i}}(A)=\alpha\mathrm{Pr}_{\mathbb{P}_{i}}(A)+\beta\mathrm{Pr}_{\mathbb{R}_{i}}(A)$.
Further, set
$\rho=\max\nolimits_{i,j}d_{\mathcal{H}\Delta\mathcal{H}}(\mathbb{P}_{i},\mathbb{P}_{j})$
and
$\rho^{*}=\max\nolimits_{i,j}d_{\mathcal{H}\Delta\mathcal{H}}(\mathbb{S}_{i},\mathbb{S}_{j})$.
Then
$\bigcap\nolimits_{i}\mathcal{B}_{\rho}(\mathbb{P}_{i})\subseteq\bigcap\nolimits_{i}\mathcal{B}_{\rho^{*}}(\mathbb{S}_{i})$
whenever
$\rho^{*}-\beta\max_{i}d_{\mathcal{H}\Delta\mathcal{H}}(\mathbb{R}_{i},\mathbb{P}_{i})\geq\rho.$

The above statement suggests that if we want to diversify our training
distributions, we should train on a collection of modified source
distributions $\\{\mathbb{S}_{i}\\}_{i}$. The modified distributions are
mixture distributions whose components are pairs of our original source
distributions $\\{\mathbb{P}_{i}\\}_{i}$ and new auxiliary distributions
$\\{\mathbb{R}_{i}\\}_{i}$. The choice of $\\{\mathbb{R}_{i}\\}_{i}$ is
constrained to guarantee the new intersection
$\bigcap\nolimits_{i}\mathcal{B}_{\rho^{*}}(\mathbb{S}_{i})$ (with modified
sources) contains the original intersection
$\bigcap\nolimits_{i}\mathcal{B}_{\rho}(\mathbb{P}_{i})$.

##### Algorithm

Empirically speaking, our modified source samples
$\\{\hat{\mathbb{S}}_{i}\\}_{i}$ will be a mix of examples from the original
sources $\\{\mathbb{P}_{i}\\}_{i}$ and the auxiliary distributions
$\\{\mathbb{R}_{i}\\}_{i}$ – drawn from each proportionally to the mixture
weights $\alpha$ and $\beta$. We plan to generate samples from the auxiliary
distributions $\\{\mathbb{R}_{i}\\}_{i}$ and our interpretation of Proposition
4.1 suggests we should do so subject to the constraint below

$\small\max\nolimits_{i,j}d_{\mathcal{H}\Delta\mathcal{H}}(\mathbb{S}_{i},\mathbb{S}_{j})-\beta\max\nolimits_{i}d_{\mathcal{H}\Delta\mathcal{H}}(\mathbb{R}_{i},\mathbb{P}_{i})\geq\rho.$
(14)

Because $\rho$ is a property of our original dataset, it is independent of the
distributions $\\{\mathbb{R}_{i}\\}_{i}$. This suggests that we should
generate $\\{\hat{\mathbb{R}}_{i}\\}_{i}$ to maximize the left hand side.
Maximizing this requires: (Req.I) maximizing the largest divergence between
the new source samples $\\{\hat{\mathbb{S}}_{i}\\}_{i}$ and (Req.II)
minimizing the largest divergence between our auxiliary samples
$\\{\hat{\mathbb{R}}_{i}\\}_{i}$ and our original source samples
$\\{\hat{\mathbb{P}}_{i}\\}_{i}$. Algorithmically, we can coarsely approximate
these divergences, again appealing to the interpretation provided by Ben-David
et al. (2010) and Ganin and Lempitsky (2015): (Req.I) requires that our domain
discriminator make fewer errors when discriminating the new source samples
$\\{\hat{\mathbb{S}}_{i}\\}_{i}$ and (Req.II) requires that the auxiliary
samples $\\{\hat{\mathbb{R}}_{i}\\}_{i}$ and the original sources
$\\{\hat{\mathbb{P}}_{i}\\}_{i}$ be indiscernible by our domain discriminator.

To implement these requirements, we modify our dataset through gradient
descent. Suppose that $\hat{\mathbb{P}}_{i}$ is an empirical sample from the
distribution $\mathbb{P}_{i}$. We can alter data-points
$a_{i}\sim\hat{\mathbb{P}}_{i}$ to generate data-points
$b_{i}\sim\hat{\mathbb{R}}_{i}$ by setting $x^{0}_{i}=a_{i}$, iterating the
below update rule to minimize $\mathcal{L}_{SD}$ for $t$ steps

$\small\begin{split}x^{j}_{i}\leftarrow
x^{j-1}_{i}-\eta\nabla_{x}\mathcal{L}_{SD}(\mu,\theta,x^{j-1}_{i})\end{split}$
(15)

and then taking $b_{i}=x^{t}_{i}$. Importantly, we never modify the domain
labels. In context of Eq. (7) which defines $\mathcal{L}_{SD}$, we have

$\small\mathbb{I}[x^{j}_{i}\sim\mathbb{P}_{i}]=1\quad\text{for all}\quad
j\in[t].$ (16)

Then, our updates satisfy requirement (Req.I) because minimization of
$\mathcal{L}_{SD}$ approximates minimization of our domain discriminator’s
errors, and further, satisfy (Req.II) because $a_{i}$ and $b_{i}$ are
identically labeled, so minimization of the domain discriminator’s errors
suggests that these examples should be indiscernible.

While this update rule seemingly accomplishes our algorithmic goals, we must
recall the final upper bound we wish to minimize (see Eq. (10)). The first two
terms in this bound, $\lambda_{\varphi}$ and
$\sum_{i}\varphi_{i}\mathcal{E}_{\mathbb{P}_{i}}(h)$, relate to our
classification error – i.e., to the task-specific network $c_{\sigma}$. If our
generated distributions $\\{\hat{\mathbb{R}}_{i}\\}_{i}$ distort the
underlying class information, these terms may grow uncontrollably. To account
for this, we further modify the update rule of Eq. (15) to minimize the change
in the probability distribution output by the task classifier. We measure the
change caused by our updates using the loss $\mathcal{L}_{KL}$ – i.e., the KL-
Divergence (Kullback, 1997). This gives the modified update rule

$\small\begin{split}x^{j}_{i}\leftarrow&x^{j-1}_{i}-\eta\nabla_{x}\left[\mathcal{L}_{SD}(\mu,\theta,x^{j-1}_{i})\right.\\\
&\quad\left.+\mathcal{L}_{KL}(c_{\sigma}\circ
r_{\theta}(x^{0}_{i}),c_{\sigma}\circ
r_{\theta}(x^{j-1}_{i}))\right].\end{split}$ (17)

##### Interpretation

In totality, this algorithm may be seen as employing a style of adversarial
training where, rather than generating examples to fool a task classifier –
e.g., the single-source DG approach of Volpi et al. (2018) –, we instead
generate examples to exploit the weaknesses of the feature extractor
$r_{\theta}$ whose goal is to fool the domain discriminator. In this sense,
the generated examples can be interpreted as cooperating with the domain
discriminator. Hence, we refer to the technique as DANN with Cooperative
Examples, or DANNCE. For details on our implementation of DANNCE, including
pseudo-code, please refer to Appendix B and C.

## 5 Experimentation

In this section, we aim at addressing the primary point argued throughout this
paper: the application of DANN to DG can benefit from (algorithmic)
consideration of source diversity. To this end, our modus operandi is the
comparison to recent state-of-the-art methods using a source-source DANN, or
other domain alignment techniques, for domain generalization. See Appendix C
and code provided in supplement for all implementation details and additional
experiments.

##### Datasets and Hyper-Parameters

We evaluate our method on two multi-source DG datasets. (1) PACS (Li et al.,
2017) contains 4 different styles of images (Photo, Art, Cartoon, and Sketch)
with 7 common object categories. (2) Office-Home (Venkateswara et al., 2017)
also contains 4 different styles of images (Art, Clipart, Product, and
Real-W[orld]) with 65 common categories of daily objects. For both datasets,
we follow standard experimental setups. We use 1 domain as target and the
remaining 3 domains as sources. We report the average classification accuracy
of the unseen target over 3 runs, using the model state at the last epoch to
avoid peaking at the target. We select our hyper-parameters using leave-one-
source-out CV (Balaji et al., 2018); this again avoids using the target in any
way. Because some methods select parameters using a source train/val split, we
use only the training data of the standard splits for fairness. Other
parameters of our setup, unrelated to our own method, are selected based on
the environment of Matsuura and Harada (2020) (MMLD) – a SOTA source-source
DANN technique. For full details, see Appendix C.

##### Our Models

For the feature extractor $r_{\theta}$ we use AlexNet (Krizhevsky et al.,
2012) for PACS and ResNet-18 (He et al., 2016) for PACS and OfficeHome. Both
are pretrained on ImageNet with the last fully-connected (FC) layer removed.
For task classifier $c_{\sigma}$ and domain discriminator $d_{\mu}$ we use
only FC layers. For ERM (often called Vanilla or Deep All) only $r_{\theta}$
and $c_{\sigma}$ are used and the model is trained on a mixture of all
sources; this is a traditional DG baseline. For DANN, we add the domain
discriminator $d_{\mu}$ and additionally update $r_{\theta}$ with
$\mathcal{L}_{SD}$ (see Eq. (8)). Because we ultimately compare against DANN
as a baseline, we must ensure our implementation is state-of-the-art.
Therefore, we generally follow the implementation described by Matsuura and
Harada (2020), adding a commonly used Entropy Loss (Bengio et al., 1992, Shu
et al., 2018) and phasing-in the impact of $\mathcal{L}_{SD}$ on $r_{\theta}$
by setting $\lambda=2/(1+\exp(-\kappa\cdot p))-1$ in Eq. (8) with
$p=\textrm{epoch}/\textrm{max\\_epoch}$ and $\kappa=10$.

For our proposed method, DANNCE, we use the same baseline DANN, but update 50%
of the images (i.e., $\beta=0.5$) to cooperate with the domain discriminator
following Eq. (17). The number of update steps per image is 5 (i.e., $t=5$).

Table 1: PACS and OfficeHome Results in Accuracy (%). avg: Average of the
target domain accuracies. gain: avg gain over the respective ERM (if
reported).

PACS | art | cartoon | sketch | photo | avg | gain
---|---|---|---|---|---|---
AlexNet
ERM | 69.1 | 70.2 | 61.8 | 88.9 | 72.3 | -
MMLD | 68.5 | 72.2 | 66.3 | 89.3 | 74.1 | 1.8
MMLD-2 | 67.0 | 70.6 | 67.8 | 89.4 | 73.7 | 1.4
MMLD-3 | 69.3 | 72.8 | 66.4 | 89.0 | 74.4 | 2.1
ERM | 64.9 | 70.2 | 61.4 | 90.0 | 71.6 | -
G2DM | 66.6 | 73.4 | 66.2 | 88.1 | 73.6 | 2.0
ERM | 67.6 | 70.2 | 60.3 | 89.3 | 71.9 | -
DANN | 71.2 | 72.1 | 66.3 | 88.3 | 74.5 | 2.6
DANNCE | 70.9 | 72.0 | 67.9 | 89.6 | 75.1 | 3.2
ResNet-18
ERM | 77.0 | 75.9 | 69.2 | 96.0 | 79.5 | -
MMD-AAE | 75.2 | 72.7 | 64.2 | 96.0 | 77.0 | -2.5
CrossGRAD | 79.8 | 76.8 | 70.2 | 96.0 | 80.7 | 1.2
DDAIG | 84.2 | 78.1 | 74.7 | 95.3 | 83.1 | 3.6
ERM | 78.3 | 75.0 | 65.2 | 96.2 | 78.7 | -
MMLD-2 | 81.3 | 77.2 | 72.3 | 96.1 | 81.8 | 3.1
MMLD-3 | 79.6 | 76.8 | 71.2 | 95.9 | 80.9 | 3.1
G2DM | 77.8 | 75.5 | 77.6 | 93.8 | 81.2 | -
ERM | 78.1 | 75.6 | 66.0 | 95.4 | 78.7 | -
DANN | 80.2 | 77.6 | 70.0 | 95.4 | 80.8 | 2.0
DANNCE | 82.1 | 78.2 | 71.9 | 94.7 | 81.7 | 3.0
OfficeHome | art | clipart | product | real-w | avg | gain
ERM | 58.9 | 49.4 | 74.3 | 76.2 | 64.7 | -
MMD-AAE | 56.5 | 47.3 | 72.1 | 74.8 | 62.7 | -2.0
CrossGRAD | 58.4 | 49.4 | 73.9 | 75.8 | 64.4 | -0.4
DDAIG | 59.2 | 52.3 | 74.6 | 76.0 | 65.5 | 0.8
ERM | 60.0 | 49.0 | 75.4 | 76.8 | 65.3 | -
DANN | 61.6 | 48.9 | 75.8 | 76.2 | 65.6 | 0.3
DANNCE | 61.6 | 50.2 | 75.6 | 75.9 | 65.8 | 0.5

##### Experimental Baselines

As mentioned, we focus on comparison to other methods proposing domain
alignment for DG. Albuquerque et al. (2020) (G2DM) and Li et al. (2018b) (MMD-
AAE) propose variants of DANN,555MMD-AAE is based on the maximum-mean
discrepancy (Gretton et al., 2012) rather than $\mathcal{H}$-divergence. and
in particular, align domains by making updates to the feature extractor. As
noted, MMLD is the DANN setup most similar to our baseline DANN. For MMLD,
Matsuura and Harada (2020) additionally propose a source domain mixing
algorithm – we denote this by MMLD-$K$ with $K$ the number of domains after
re-clustering. Shankar et al. (2018) (CrossGRAD) and Zhou et al. (2020)
(DDAIG), contrary to our work, generate examples which maximize the domain
loss. Because, they do not update the feature extractor with the domain loss
$\mathcal{L}_{SD}$ as we do, this may actually be viewed as domain-alignment
by data generation (Liu et al., 2019). For MMD-AAE and CrossGRAD, we use
results reported by Zhou et al. (2020) because the original methods do not
test on our datasets.

##### Analysis of Performance

Generally, in DG, the comparison of performance is subjective across different
experimental setups – a problem highlighted by a recent commentary on the
experimental rigor of DG setups (Gulrajani and Lopez-Paz, 2020). As such, we
include reported results from other experimental setups, predominantly, to
show our DANN implementation is a competitive baseline. This much is visible
in Table 1. For 2 out of 3 setups, our DANN alone has higher overall accuracy
than any other method.

Our focus, then, is the validation of our main argument using our strong DANN
baseline. In this context, shown in Table 1, ablation of DANNCE reveals
substantial improvement upon the traditional source-source DANN in all PACS
setups and marginal improvement in the OfficeHome setup. This is somewhat
intuitive as OfficeHome has a staggering 65 categories to classify compared to
7 in PACS. Ultimately, the performance gains demonstrated by addition of
DANNCE agrees with our main argument: increasing diversity when aligning
domains can have practical benefits in DG.

Figure 2:  Domain Discriminator Loss of DANN and DANNCE on PACS. For each
target, we show the loss of its corresponding sources during training. Figure
3: Domain Discriminator Loss of DANNCE with 5 and 20 Steps of Image Updates.

##### Analysis of Loss Curves

To measure domain diversity, we use the loss of the domain discriminator
(averaged per epoch). This loss is used to proxy the $\mathcal{H}$-divergence
(an inverse relationship). A lower loss should then indicate more domain
diversity, and, has the benefit of dynamically measuring diversity during
training. Figure 2 shows the domain discriminator loss across epochs for our
implementations of DANN and DANNCE using AlexNet on PACS. We generally see
after epoch 15, the loss for DANNCE is lowest. Figure 3 further shows the
effect of increasing the number of steps per image update. This suggests that
increasing the number of updates has some control over the source domain
diversity as intended. Finally, in both Figures, epochs 10 to 24 show the
(inverted) smooth proxy for the domain divergence is increasing. This agrees
with the formal claim made in Prop. 3.3. Although the trend changes after
epoch 24, this is likely due to a decrease in $\gamma$ at this epoch, and
thus, does not necessarily disagree with our formal claim.

## 6 Related Works

Many works extend the theoretical framework of Ben-David et al. (2010) to
motivate new variants of DANN (Zhao et al., 2018, Schoenauer-Sebag et al.,
2019, Zhang et al., 2019). For DG, there is decidedly less work, but
throughout our text, we have attempted to compare to the most relevant (and
recent) – a bound proposed by Albuquerque et al. (2020).

Besides DANN and other domain-aligning algorithms mentioned in this text,
there are of course additional perspectives on DG. An early work in DG by
Muandet et al. (2013) proposes a kernel-based algorithm aimed at producing
domain-invariant features with a strong theoretical justification. More
recently, a common thread is the use of meta-learning (e.g. to simulate
domain-holdout) seen in Li et al. (2018a), Balaji et al. (2018), Dou et al.
(2019). Some authors, such as Wang et al. (2019) and Carlucci et al. (2019),
make additional assumptions on the domains to be seen, using this in algorithm
design. Lastly, some works focus on the neural network components themselves,
e.g., Li et al. (2017). These architecture changes can be very effective (see
Seo et al. (2019) for impressive results when modifying batch-normalization).
Related to our paper’s main point, we primarily focus on comparison to other
methods proposing domain alignment for DG, especially those which are, in some
sense, model agnostic (see Experimental Baselines).

## 7 Conclusion

In this work, we investigate the applicability of source-source DANN for
domain generalization. Our theoretical results and interpretation suggest a
complex relationship between the heterogeneity of the source domains and the
usual process of domain alignment. Motivated by this, we construct an
algorithmic extension for DANN which diversifies the sources via gradient-
based image updates. Our empirical results and analyses support our findings.

## References

  * Albuquerque et al. (2020) I. Albuquerque, J. Monteiro, T. H. Falk, and I. Mitliagkas.  Adversarial target-invariant representation learning for domain generalization.  _arXiv preprint arXiv:1911.00804_ , 2020.
  * Balaji et al. (2018) Y. Balaji, S. Sankaranarayanan, and R. Chellappa.  Metareg: Towards domain generalization using meta-regularization.  In _Advances in Neural Information Processing Systems_ , pages 998–1008, 2018.
  * Bartlett et al. (2019) P. L. Bartlett, N. Harvey, C. Liaw, and A. Mehrabian.  Nearly-tight vc-dimension and pseudodimension bounds for piecewise linear neural networks.  _Journal of Machine Learning Research_ , 20(63):1–17, 2019.
  * Ben-David et al. (2007) S. Ben-David, J. Blitzer, K. Crammer, and F. Pereira.  Analysis of representations for domain adaptation.  In _Advances in Neural Information Processing Systems_ , pages 137–144, 2007.
  * Ben-David et al. (2010) S. Ben-David, J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. W. Vaughan.  A theory of learning from different domains.  _Machine learning_ , 79(1-2):151–175, 2010.
  * Bengio et al. (1992) S. Bengio, Y. Bengio, J. Cloutier, and J. Gecsei.  On the optimization of a synaptic learning rule.  In _Preprints Conf. Optimality in Artificial and Biological Neural Networks_ , volume 2. Univ. of Texas, 1992.
  * Carlucci et al. (2019) F. M. Carlucci, A. D’Innocente, S. Bucci, B. Caputo, and T. Tommasi.  Domain generalization by solving jigsaw puzzles.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 2229–2238, 2019.
  * Crammer et al. (2007) K. Crammer, M. Kearns, and J. Wortman.  Learning from multiple sources.  In _Advances in Neural Information Processing Systems_ , pages 321–328, 2007.
  * Dou et al. (2019) Q. Dou, D. C. de Castro, K. Kamnitsas, and B. Glocker.  Domain generalization via model-agnostic learning of semantic features.  In _Advances in Neural Information Processing Systems_ , pages 6447–6458, 2019.
  * Dziugaite and Roy (2017) G. K. Dziugaite and D. M. Roy.  Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data.  _arXiv preprint arXiv:1703.11008_ , 2017.
  * Ganin and Lempitsky (2015) Y. Ganin and V. Lempitsky.  Unsupervised domain adaptation by backpropagation.  In _International Conference on Machine Learning_ , pages 1180–1189, 2015.
  * Germain et al. (2020) P. Germain, A. Habrard, F. Laviolette, and E. Morvant.  Pac-bayes and domain adaptation.  _Neurocomputing_ , 379:379–397, 2020.
  * Gretton et al. (2012) A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola.  A kernel two-sample test.  _The Journal of Machine Learning Research_ , 13(1):723–773, 2012.
  * Gulrajani and Lopez-Paz (2020) I. Gulrajani and D. Lopez-Paz.  In search of lost domain generalization, 2020.
  * He et al. (2016) K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , pages 770–778, 2016.
  * Kifer et al. (2004) D. Kifer, S. Ben-David, and J. Gehrke.  Detecting change in data streams.  In _VLDB_ , volume 4, pages 180–191, 2004.
  * Krizhevsky et al. (2012) A. Krizhevsky, I. Sutskever, and G. E. Hinton.  Imagenet classification with deep convolutional neural networks.  In _Advances in Neural Information Processing Systems_ , pages 1097–1105, 2012.
  * Kullback (1997) S. Kullback.  _Information theory and statistics_.  Courier Corporation, 1997.
  * Li et al. (2017) D. Li, Y. Yang, Y.-Z. Song, and T. M. Hospedales.  Deeper, broader and artier domain generalization.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 5542–5550, 2017.
  * Li et al. (2018a) D. Li, Y. Yang, Y.-Z. Song, and T. M. Hospedales.  Learning to generalize: Meta-learning for domain generalization.  In _Thirty-Second AAAI Conference on Artificial Intelligence_ , 2018a.
  * Li et al. (2018b) H. Li, S. Jialin Pan, S. Wang, and A. C. Kot.  Domain generalization with adversarial feature learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 5400–5409, 2018b.
  * Li et al. (2018c) Y. Li, X. Tian, M. Gong, Y. Liu, T. Liu, K. Zhang, and D. Tao.  Deep domain generalization via conditional invariant adversarial networks.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 624–639, 2018c.
  * Liu et al. (2019) H. Liu, M. Long, J. Wang, and M. Jordan.  Transferable adversarial training: A general approach to adapting deep classifiers.  In _International Conference on Machine Learning_ , pages 4013–4022, 2019.
  * Matsuura and Harada (2020) T. Matsuura and T. Harada.  Domain generalization using a mixture of multiple latent domains.  In _AAAI_ , 2020.
  * Muandet et al. (2013) K. Muandet, D. Balduzzi, and B. Schölkopf.  Domain generalization via invariant feature representation.  In _International Conference on Machine Learning_ , pages 10–18, 2013\.
  * Schoenauer-Sebag et al. (2019) A. Schoenauer-Sebag, L. Heinrich, M. Schoenauer, M. Sebag, L. F. Wu, and S. J. Altschuler.  Multi-domain adversarial learning.  In _International Conference on Learning Representation_ , 2019.
  * Seo et al. (2019) S. Seo, Y. Suh, D. Kim, J. Han, and B. Han.  Learning to optimize domain specific normalization with domain augmentation for domain generalization.  _arXiv preprint arXiv:1907.04275_ , 2019.
  * Shankar et al. (2018) S. Shankar, V. Piratla, S. Chakrabarti, S. Chaudhuri, P. Jyothi, and S. Sarawagi.  Generalizing across domains via cross-gradient training.  In _International Conference on Learning Representations_ , 2018.
  * Shu et al. (2018) R. Shu, H. H. Bui, H. Narui, and S. Ermon.  A dirt-t approach to unsupervised domain adaptation.  _arXiv preprint arXiv:1802.08735_ , 2018.
  * Venkateswara et al. (2017) H. Venkateswara, J. Eusebio, S. Chakraborty, and S. Panchanathan.  Deep hashing network for unsupervised domain adaptation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 5018–5027, 2017.
  * Volpi et al. (2018) R. Volpi, H. Namkoong, O. Sener, J. C. Duchi, V. Murino, and S. Savarese.  Generalizing to unseen domains via adversarial data augmentation.  In _Advances in Neural Information Processing Systems_ , pages 5334–5344, 2018.
  * Wang et al. (2019) H. Wang, Z. He, Z. L. Lipton, and E. P. Xing.  Learning robust representations by projecting superficial statistics out.  In _International Conference on Learning Representations_ , 2019.  URL https://openreview.net/forum?id=rJEjjoR9K7.
  * Zhang et al. (2019) Y. Zhang, T. Liu, M. Long, and M. Jordan.  Bridging theory and algorithm for domain adaptation.  In _International Conference on Machine Learning_ , pages 7404–7413, 2019.
  * Zhao et al. (2018) H. Zhao, S. Zhang, G. Wu, J. M. Moura, J. P. Costeira, and G. J. Gordon.  Adversarial multiple source domain adaptation.  In _Advances in Neural Information Processing Systems_ , pages 8559–8570, 2018.
  * Zhou et al. (2020) K. Zhou, Y. Yang, T. Hospedales, and T. Xiang.  Deep domain-adversarial image generation for domain generalisation.  _arXiv preprint arXiv:2003.06054_ , 2020.


* Willis and Kerswell, (2009) Willis, A. P. and Kerswell, R. R. (2009).  Turbulent dynamics of pipe flow captured in a reduced model: puff relaminarization and localized ’edge’s states.  Journal of Fluid Mechanics, 619:213–233.