# Information-theoretic regularization for Multi-source Domain Adaptation

Geon Yeong Park, Sang Wan Lee
KAIST
Daejeon, South Korea
{pky3436<EMAIL_ADDRESS>
###### Abstract

Adversarial learning strategy has demonstrated remarkable performance in
dealing with single-source Domain Adaptation (DA) problems, and it has
recently been applied to Multi-source DA (MDA) problems. Although most
existing MDA strategies rely on a multiple domain discriminator setting, its
effect on the latent space representations has been poorly understood. Here we
adopt an information-theoretic approach to identify and resolve the potential
adverse effect of the multiple domain discriminators on MDA: disintegration of
domain-discriminative information, limited computational scalability, and a
large variance in the gradient of the loss during training. We examine the
above issues by situating adversarial DA in the context of information
regularization. This also provides a theoretical justification for using a
single and unified domain discriminator. Based on this idea, we implement a
novel neural architecture called a Multi-source Information-regularized
Adaptation Networks (MIAN). Large-scale experiments demonstrate that MIAN,
despite its structural simplicity, reliably and significantly outperforms
other state-of-the-art methods.

##  1 Introduction

Although a large number of studies have demonstrated the ability of deep
learning to solve challenging tasks, the problems are mostly confined to a
similar type or a single domain. One remaining challenge is the problem known
as domain shift [16], where a direct transfer of information gleaned from a
single source domain to unseen target domains may lead to significant
performance impairment. Domain adaptation (DA) approaches aim to mitigate this
problem by learning to map data of both domains onto a common feature space.
Whereas several theoretical results [4, 55] and algorithms for DA [28, 30, 12]
have focused on the case in which only a single-source domain dataset is
given, we consider a more challenging and generalized problem of knowledge
transfer, referred to as Multi-source unsupervised DA (MDA). Following a
seminal theoretical result on MDA [3], many deep MDA approaches have been
proposed, mainly depend on the adversarial framework.

Most of existing adversarial MDA works [54, 56, 25, 58, 57, 53] have focused
on approximating all combinations of pairwise domain discrepancy between each
source and the target, which inevitably necessitates training of multiple
binary domain discriminators. While substantial technical advances have been
made in this regard, the pitfalls of using multiple domain discriminators have
not been fully studied. This paper focuses on the potential adverse effects of
using multiple domain discriminators on MDA in terms of both quantity and
quality. First, the domain-discriminative information is inevitably
distributed across multiple discriminators. For example, such discriminators
primarily focus on domain shift between each source and the target, while the
discrepancies between the source domains are neglected. Moreover, the multiple
source-target discriminator setting often makes it difficult to approximate
the combined $\mathcal{H}$-divergence between mixture of sources and the
target domain because each discriminator is deemed to utilize the samples only
from the corresponding source and target domain as inputs. Compared to a bound
using combined divergence, a bound based on pairwise divergence is not
sufficiently flexible to accommodate domain structures [3]. Second, the
computational load of the multiple domain discriminator setting rapidly
increases with the number of source domains ($\mathcal{O}(N)$), which
significantly limits scalability. Third, it could undermine the stability of
training, as earlier works solve multiple adversarial min-max problems.

To overcome such limitations, instead of relying on multiple pairwise domain
discrepancy, we constrain the mutual information between latent
representations and domain labels. The contribution of this study is
summarized as follows. First, we show that such mutual information
regularization is closely related to the explicit optimization of the
$\cal{H}$-divergence between the source and target domains. This affords the
theoretical insight that the conventional adversarial DA can be translated
into an information-theoretic regularization problem. Second, from these
theoretical findings we derive a new optimization problem for MDA: minimizing
adversarial loss over multiple domains with a single domain discriminator. The
algorithmic solution to this problem is called Multi-source Information
regularized Adaptation Networks (MIAN). Third, we show that our single domain
discriminator setting serves to penalize every pairwise combined domain
discrepancy between the given domain and the mixture of the others. Moreover,
by analyzing existing studies in terms of information regularization, we found
another negative effect of the multiple discriminators setting: significant
increase in the variance of the stochastic gradients.

Despite its structural simplicity, we demonstrated that MIAN works efficiently
across a wide variety of MDA scenarios, including the DIGITS-Five [37],
Office-31 [39], and Office-Home datasets [51]. Intriguingly, MIAN reliably and
significantly outperformed several state-of-the-art methods, including ones
that employ a domain discriminator separately for each source domain [54] and
that align the moments of deep feature distribution for every pairwise domain
[37].

##  2 Related works

Several DA methods have been used in attempt to learn domain-invariant
representations. Along with the increasing use of deep neural networks,
contemporary work focuses on matching deep latent representations from the
source domain with those from the target domain. Several measures have been
introduced to handle domain shift, such as maximum mean discrepancy (MMD) [29,
28], correlation distance [45, 46], and Wasserstein distance [8]. Recently,
adversarial DA methods [12, 50, 20, 41, 40] have become mainstream approaches
owing to the development of generative adversarial networks [15]. However, the
abovementioned single-source DA approaches inevitably sacrifice performance
for the sake of multi-source DA.

Some MDA studies [4, 3, 33, 19] have provided the theoretical background for
algorithm-level solutions. [4, 3] explore the extended upper bound of true
risk on unlabeled samples from the target domain with respect to a weighted
combination of multiple source domains. Following these theoretical studies,
MDA studies with shallow models [10, 9, 6] as well as with deep neural
networks [32, 37, 25] have been proposed. Recently, some adversarial MDA
methods have also been proposed. [54] implemented a k-way domain discriminator
and classifier to battle both domain and category shifts. [56] also used
multiple discriminators to optimize the average case generalization bounds.
[58] chose relevant source training samples for the DA by minimizing the
empirical Wasserstein distance between the source and target domains. Instead
of using separate encoders, domain discriminators or classifiers for each
source domain as in earlier works, our approach uses unified networks, thereby
improving reliability, resource-efficiency and scalability. To the best of our
knowledge, this is the first study to bridge the gap between MDA and
information regularization, and show that a single domain-discriminator is
sufficient for the adaptation.

Several existing MDA works have proposed methods to estimate the source domain
weights following [4, 3]. [33] assumed that the target hypothesis can be
approximated by a convex combination of the source hypotheses. [37, 56]
suggested ad-hoc schemes for domain weights based on the empirical risk of
each source domain. [25] computed a softmax-transformed weight vector using
the empirical Wasserstein-like measure instead of the empirical risks.
Compared to the proposed methods without robust theoretical justifications,
our analysis does not require any assumption or estimation for the domain
coefficients. In our framework, the representations are distilled to be
independent of the domain, thereby rendering the performance relatively
insensitive to explicit weighting strategies.

##  3 Theoretical insights

We first introduce the notations for the MDA problem in classification. A set
of source domains and the target domain are denoted by
$\left\\{D_{S_{i}}\right\\}_{i=1}^{N}$ and $D_{T}$, respectively. Let
$X_{S_{i}}=\left\\{\mathbf{x}_{S_{i}}^{j}\right\\}_{j=1}^{m}$ and
$Y_{S_{i}}=\left\\{\mathbf{y}_{S_{i}}^{j}\right\\}_{j=1}^{m}$ be a set of $m$
i.i.d. samples from $D_{S_{i}}$. Let
$X_{T}=\left\\{\mathbf{x}_{T}^{j}\right\\}_{j=1}^{m}\sim(D_{T}^{X})^{m}$ be
the set of $m$ i.i.d. samples generated from the marginal distribution
$D_{T}^{X}$. The domain label and its probability distribution are denoted by
$V$ and $P_{V}(\mathbf{v})$, where $\mathbf{v}\in\cal{V}$ and $\cal{V}$ is the
set of domain labels. In line with prior works [18, 13, 32, 14], domain label
can be generally treated as a stochastic latent random variable in our
framework. However, for simplicity, we take the empirical version of the true
distributions with given samples assuming that the domain labels for all
samples are known. The latent representation of the sample is given by $Z$,
and the encoder is defined as $F:\cal{X}\rightarrow\cal{Z}$, with $\cal{X}$
and $\cal{Z}$ representing data space and latent space, respectively.
Accordingly, $Z_{S_{i}}$ and $Z_{T}$ refer to the outputs of the encoder
$F(X_{S_{i}})$ and $F(X_{T})$, respectively. For notational simplicity, we
will omit the index $i$ from $D_{S_{i}}$, $X_{S_{i}}$ and $Z_{S_{i}}$ when
$N=1$. A classifier is defined as $C:{\cal{Z}}\rightarrow\cal{Y}$ where
$\cal{Y}$ is the class label space.

###  3.1 Problem formulation

For comparison with our formulation, we recast single-source DA as a
constrained optimization problem. The true risk $\epsilon_{T}(h)$ on unlabeled
samples from the target domain is bounded above the sum of three terms [3]:
(1) true risk $\epsilon_{S}(h)$ of hypothesis $h$ on the source domain; (2)
$\cal{H}$-divergence $d_{\cal{H}}(D_{S},D_{T})$ between a source and a target
domain distribution; and (3) the optimal joint risk $\lambda^{*}$.

######  Theorem 1 ([3]).

Let hypothesis class ${\cal{H}}$ be a set of binary classifiers
$h:{\cal{X}}\rightarrow\left\\{0,1\right\\}$. Then for the given domain
distributions $D_{S}$ and $D_{T}$,

$\forall{h}\in{\cal{H}},\epsilon_{T}(h)\leq\epsilon_{S}(h)+d_{\cal{H}}(D_{S},D_{T})+\lambda^{*},$
(1)

where
$d_{\cal{H}}(D_{S},D_{T})=2\underset{h\in\cal{H}}{sup}\Big{|}\underset{\mathbf{x}\sim
D_{S}^{X}}{\mathbb{E}}\big{[}\mathbb{I}(h(\mathbf{x})=1)\big{]}-\\\
\underset{\mathbf{x}\sim
D_{T}^{X}}{\mathbb{E}}\big{[}\mathbb{I}(h(\mathbf{x})=1)\big{]}\Big{|}$ and
$\mathbb{I}(a)$ is an indicator function whose value is $1$ if $a$ is true,
and $0$ otherwise.

The empirical $\cal{H}$-divergence $\hat{d}_{\cal{H}}(X_{S},X_{T})$ can be
computed as follows [3]:

######  Lemma 1.

$\begin{split}\hat{d}_{\cal{H}}(X_{S},X_{T})=2\Big{(}1-\min_{h\in\cal{H}}\Big{[}&\frac{1}{m}\sum_{\mathbf{x}\in
X_{S}}\mathbb{I}[h(\mathbf{x})=0]+\\\ &\frac{1}{m}\sum_{\mathbf{x}\in
X_{T}}\mathbb{I}[h(\mathbf{x})=1]\Big{]}\Big{)}\end{split}$ (2)

Following Lemma 1, a domain classifier $h:\cal{Z}\rightarrow\cal{V}$ can be
used to compute the empirical $\cal{H}$-divergence. Suppose the optimal joint
risk $\lambda^{*}$ is sufficiently small as assumed in most adversarial DA
studies [40, 7]. Thus, one can obtain the ideal encoder and classifier
minimizing the upper bound of $\epsilon_{T}(h)$ by solving the following min-
max problem:

$\begin{split}F^{*},C^{*}&=\arg\min_{F,C}L(F,C)+\beta\hat{d}_{\cal{H}}(Z_{S},Z_{T})\\\
&=\arg\min_{F,C}\max_{h\in\cal{H}}L(F,C)+\\\
&\frac{\beta}{m}\Big{(}\sum_{i:\mathbf{z}_{i}\in
Z_{S}}\mathbb{I}[h(\mathbf{z}_{i})=1]+\sum_{j:\mathbf{z}_{j}\in
Z_{T}}\mathbb{I}[h(\mathbf{z}_{j})=0]\Big{)},\end{split}$ (3)

where $L(F,C)$ is the loss function on samples from the source domain, $\beta$
is a Lagrangian multiplier, ${\cal{V}}=\\{0,1\\}$ such that each source
instance and target instance are labeled as $1$ and $0$, respectively, and $h$
is the binary domain classifier.

###  3.2 Information-regularized min-max problem for MDA

Intuitively, it is not highly desirable to adapt the learned representation in
the given domain to the other domains, particularly when the representation
itself is not sufficiently domain-independent. This motivates us to explore
ways to learn representations independent of domains. Inspired by a
contemporary fair model training study [38], the mutual information between
the latent representation and the domain label $I(Z;V)$ can be expressed as
follows:

######  Theorem 2.

Let $P_{Z}(\mathbf{z})$ be the distribution of $Z$ where
$\mathbf{z}\in{\cal{Z}}$. Let $h$ be a domain classifier
$h:\cal{Z}\rightarrow\cal{V}$, where $\cal{Z}$ is the feature space and
$\cal{V}$ is the set of domain labels. Let $h_{\mathbf{v}}(\mathbf{z})$ be a
conditional probability of $V$ where $\mathbf{v}\in\cal{V}$ given
$Z=\mathbf{z}$, defined by $h$. Then the following holds:

$\begin{split}I(Z;V)&=\max_{h_{\mathbf{v}}(\mathbf{z}):\sum_{\mathbf{v}\in\cal{V}}{h_{\mathbf{v}}(\mathbf{z})=1,\forall{\mathbf{z}}}}\\\
&\sum_{\mathbf{v}\in\cal{V}}{P_{V}(\mathbf{v})\mathbb{E}_{\mathbf{z}\sim
P_{Z\mid\mathbf{v}}}\big{[}\log{h_{\mathbf{v}}(\mathbf{z})}\big{]}}+H(V)\end{split}$
(4)

The detailed proof is provided in the [38] and Supplementary Material. We can
derive the empirical version of Theorem 2 as follows:

$\begin{split}\hat{I}(Z;V)&=\max_{h_{\mathbf{v}}(\mathbf{z}):\sum_{\mathbf{v}\in\cal{V}}{h_{\mathbf{v}}(\mathbf{z})=1,\forall{\mathbf{z}}}}\\\
&\frac{1}{M}\sum_{\mathbf{v}\in\cal{V}}\sum_{i:\mathbf{v}_{i}=\mathbf{v}}\log{h_{\mathbf{v}_{i}}(\mathbf{z}_{i})}+H(V),\end{split}$
(5)

where $M$ is the number of total representation samples, $i$ is the sample
index, and $\mathbf{v}_{i}$ is the corresponding domain label of the $i$th
sample. Using this equation, we combine our information-constrained objective
function and the results of Lemma 1. For binary classification
${\cal{V}}=\\{0,1\\}$ with $Z_{S}$ and $Z_{T}$ of equal size $M/2$, we propose
the following information-regularized minimax problem:

$\begin{split}F^{*},C^{*}&=\arg\min_{F,C}\ \ {L(F,C)+\beta\hat{I}(Z;V)}\\\
&=\arg\min_{F,C}\max_{h\in\cal{H}}\ \ L(F,C)+\\\
&\frac{\beta}{M}\big{[}\sum_{i:\mathbf{z}_{i}\in
Z_{S}}\log{h(\mathbf{z}_{i})}+\sum_{j:\mathbf{z}_{j}\in
Z_{T}}\log(1-{h(\mathbf{z}_{j})})\big{]},\end{split}$ (6)

where $\beta$ is a Lagrangian multiplier, $h(\mathbf{z}_{i})\triangleq
h_{\mathbf{v}_{i}=1}(\mathbf{z}_{i})$ and $1-h(\mathbf{z}_{i})\triangleq
h_{\mathbf{v}_{i}=0}(\mathbf{z}_{i})$, with $h(\mathbf{z}_{i})$ representing
the probability that $\mathbf{z}_{i}$ belongs to the source domain. This
setting automatically dismisses the condition
$\sum_{\mathbf{v}\in\cal{V}}h_{\mathbf{v}}(\mathbf{z})=1,\forall{\mathbf{z}}$.
Note that we have accommodated a simple situation in which the entropy $H(V)$
remains constant.

###  3.3 Advantages over other MDA methods

Integration of domain-discriminative information. The relationship between (3)
and (6) provides us a theoretical insights that the problem of minimizing
mutual information between the latent representation and the domain label is
closely related to minimizing the $\cal{H}$-divergence using the adversarial
learning scheme. This relationship clearly underlines the significance of
information regularization for MDA. Compared to the existing MDA approaches
[54, 56], which inevitably distribute domain-discriminative knowledge over $N$
different domain classifiers, the above objective function (6) enables us to
seamlessly integrate such information with the single-domain classifier $h$.
It will be further discussed in Section 4.

Variance of the gradient. Using a single domain discriminator also helps
reduce the variance of gradient. Large variances in the stochastic gradients
slow down the convergence, which leads to poor performance [22]. Herein, we
analyze the variances of the stochastic gradients of existing optimization
constraints. By excluding the weighted source combination strategy, we can
approximately express the optimization constraint of existing adversarial MDA
methods as sum of the information constraints:

$\begin{split}\sum_{k=1}^{N}{I}(Z_{k};U_{k})=\sum_{k=1}^{N}I_{k}+\sum_{k=1}^{N}H(U_{k}),\end{split}$
(7)

where

$\begin{split}I_{k}&=\max_{h_{\mathbf{u}}^{k}(\mathbf{z}):\sum_{\mathbf{u}\in\cal{U}}{h_{\mathbf{u}}^{k}(\mathbf{z})=1,\forall{\mathbf{z}}}}\\\
&\sum_{\mathbf{u}\in\cal{U}}{P_{U_{k}}(\mathbf{u})\mathbb{E}_{\mathbf{z}_{k}\sim
P_{Z_{k}\mid\mathbf{u}}}\big{[}\log{h^{k}_{\mathbf{u}}(\mathbf{z}_{k})}\big{]}},\end{split}$
(8)

$U_{k}$ is the $k$th domain label with ${\cal{U}}=\\{0,1\\}$,
$P_{Z_{k}\mid\mathbf{u}=0}(\cdot)=P_{Z\mid\mathbf{v}=N+1}(\cdot)$
corresponding to the target domain,
$P_{Z_{k}\mid\mathbf{u}=1}(\cdot)=P_{Z\mid\mathbf{v}=k}(\cdot)$ corresponding
to the $k$th source domain, and $h_{\mathbf{u}}^{k}(\mathbf{z}_{k})$ being the
conditional probability of $\mathbf{u}\in{\cal{U}}$ given $\mathbf{z}_{k}$
defined by the $k$th discriminator indicating that the sample is generated
from the $k$th source domain. Again, we treat the entropy $H(U_{k})$ as a
constant.

Given $M=m(N+1)$ samples with $m$ representing the number of samples per
domain, an empirical version of (7) is:

$\sum_{k=1}^{N}{\hat{I}}(Z_{k};U_{k})=\frac{1}{M}\sum_{k=1}^{N}\hat{I}_{k}+\sum_{k=1}^{N}H(U_{k}),$
(9)

where

$\begin{split}\hat{I}_{k}=\max_{h_{\mathbf{u}}^{k}(\mathbf{z}):\sum_{\mathbf{u}\in\cal{U}}{h_{\mathbf{u}}^{k}(\mathbf{z})=1,\forall{\mathbf{z}}}}\sum_{\mathbf{u}\in\cal{U}}\sum_{i:\mathbf{u}^{i}=\mathbf{u}}\log{h^{k}_{\mathbf{u}}(\mathbf{z}_{k}^{i})}.\end{split}$
(10)

For the sake of simplicity, we make simplifying assumptions that all
$Var[I_{k}]$ are approximately the same for all k and so are
$Cov[I_{k},I_{j}]$ for all pairs. Then the variance of (9) is given by:

$\begin{split}&Var\Big{[}\sum_{k=1}^{N}{\hat{I}}(Z_{k};U_{k})\Big{]}\\\
&=\frac{1}{M^{2}}\Big{(}\sum_{k=1}^{N}Var[\hat{I}_{k}]+2\sum_{k=1}^{N}\sum_{j=k}^{N}Cov[\hat{I}_{k},\hat{I}_{j}]\Big{)}\\\
&=\frac{1}{m^{2}}\Big{(}\frac{N}{(N+1)^{2}}Var[\hat{I}_{k}]+\frac{N(N-1)}{(N+1)^{2}}Cov[\hat{I}_{k},\hat{I}_{j}]\Big{)}.\end{split}$
(11)

As earlier works solve $N$ adversarial minimax problems, the covariance term
is additionally included and its contribution to the variance does not
decrease with increasing $N$. In other words, the covariance term may dominate
the variance of the gradients as the number of domain increases. In contrast,
the variance of our constraint (5) is inversely proportional to $(N+1)^{2}$.
Let $I_{m}$ be a shorthand for the maximization term except $\frac{1}{M}$ in
(5). Then the variance of (5) is given by:

$Var\Big{[}{\hat{I}}(Z;V)\Big{]}=\frac{1}{m^{2}(N+1)^{2}}\Big{(}Var[I_{m}]\Big{)}.$
(12)

It implies that our framework can significantly improve the stability of
stochastic gradient optimization compared to existing approaches, especially
when the model is deemed to learn from many domains.

###  3.4 Situating domain adaptation in context of information bottleneck
theory

In this Section, we bridge the gap between the existing adversarial DA method
and the information bottleneck (IB) theory [47, 48, 1]. [47] examined the
problem of learning an encoding $Z$ such that it is maximally informative
about the class $Y$ while being minimally informative about the sample $X$:

$\min_{P_{enc}(\mathbf{z}\mid\mathbf{x})}\beta I(Z;X)-I(Z;Y),$ (13)

where $\beta$ is a Lagrangian multiplier. Indeed, the role of the bottleneck
term $I(Z;X)$ matches our mutual information $I(Z;V)$ between the latent
representation and the domain label. We foster close collaboration between two
information bottleneck terms by incorporating those into $I(Z;X,V)$.

######  Theorem 3.

Let $P_{Z\mid\mathbf{x},\mathbf{v}}(\mathbf{z})$ be a conditional
probabilistic distribution of $Z$ where $\mathbf{z}\in{\cal{Z}}$, defined by
the encoder $F$, given a sample $\mathbf{x}\in\cal{X}$ and the domain label
$\mathbf{v}\in\cal{V}$. Let $R_{Z}(\mathbf{z})$ denotes a prior marginal
distribution of $Z$. Then the following inequality holds:

$\begin{split}&I(Z;X,V)\leq\mathbb{E}_{\mathbf{x,v}\sim
P_{X,V}}\big{[}D_{KL}[P_{Z\mid\mathbf{x},\mathbf{v}}\parallel
R_{Z}]\big{]}+H(V)\\\
&+\max_{h_{\mathbf{v}}(\mathbf{z}):\sum_{\mathbf{v}\in\cal{V}}{h_{\mathbf{v}}(\mathbf{z})=1},\forall{\mathbf{z}}}\sum_{\mathbf{v}\in\cal{V}}{P_{V}(\mathbf{v})\mathbb{E}_{P_{\mathbf{z}\sim
Z\mid\mathbf{v}}}\big{[}\log{h_{\mathbf{v}}(\mathbf{z})}\big{]}}\end{split}$
(14)

The proof of Theorem 3 uses the chain rule: $I(Z;X,V)=I(Z;V)+I(Z;X\mid V)$.
The detailed proof is provided in the Supplementary Material. Whereas the role
of $I(Z;X\mid V)$ is to purify the latent representation generated from the
given domain, $I(Z;V)$ serves as a proxy for regularization that aligns the
purified representations across different domains. Thus, the existing DA
approaches [31, 44] using variational information bottleneck [1] can be
reviewed as special cases for Theorem 3 with a single-source domain.

Figure 1: Proposed neural architecture for multi-source domain adaptation:
Multi-source Information-regularized Adaptation Network (MIAN). Multi-source
and target domain input data are fed into the encoder. We denote arbitrary
source domains as $S_{i}$ and $S_{j}$. The domain discriminator outputs a
logit vector, where each dimension corresponds to each domain.

##  4 Multi-source Information-regularized Adaptation Networks (MIAN)

In this Section, we provide the details of our proposed architecture, referred
to as a multi-source information-regularized adaptation network (MIAN). MIAN
addresses the information-constrained min–max problem for MDA (Section 3.2)
using the three subcomponents depicted in Figure 1: information
regularization, source classification, and Decaying Batch Spectral
Penalization (DBSP).

Information regularization. To estimate the empirical mutual information
$\hat{I}(Z;V)$ in (5), the domain classifier $h$ should be trained to minimize
softmax cross enropy. Let ${\cal{V}}=\left\\{1,2,...,N+1\right\\}$ and denote
$h(\mathbf{z})$ as $N+1$ dimensional vector of the conditional probability for
each domain given the sample $\mathbf{z}$. Let $\mathbbm{1}$ be a $N+1$
dimensional vector of all ones, and $\mathbbm{1}_{[k=\mathbf{v}]}$ be a $N+1$
dimensional vector whose $\mathbf{v}$th value is $1$ and $0$ otherwise. Given
$M=m(N+1)$ samples, the objective is:

$\min_{h}-\frac{1}{M}\sum_{\mathbf{v}\in\cal{V}}\sum_{i:\mathbf{v}_{i}=\mathbf{v}}\big{[}\mathbbm{1}_{[k=\mathbf{v}_{i}]}^{T}\log{h(\mathbf{z}_{i})}\big{]}.$
(15)

In this study, we slightly modify the softmax cross entropy (15) into multiple
binary cross entropy. Specifically, we explicitly minimize the conditional
probability of the remaining domains excepting the true $\mathbf{v}$th domain.
Let $\mathbbm{1}_{[k\neq\mathbf{v}]}$ be the flipped version of
$\mathbbm{1}_{[k=\mathbf{v}]}$. Then the modified objective function for the
domain discriminator is:

$\begin{split}\min_{h}-\frac{1}{M}\sum_{\mathbf{v}\in\cal{V}}\sum_{i:\mathbf{v}_{i}=\mathbf{v}}&\big{[}\mathbbm{1}_{[k=\mathbf{v}_{i}]}^{T}\log{h(\mathbf{z}_{i})}\\\
&+\mathbbm{1}_{[k\neq\mathbf{v}_{i}]}^{T}\log({\mathbbm{1}-h(\mathbf{z}_{i})})\big{]},\end{split}$
(16)

where the objective function for encoder training is to maximize (16). Our
objective function is also closely related to that of GAN [15], and we
experimentally found that using the variant objective function of GAN [34]
works slightly better.

Herein, we show that the objective (16) is closely related to optimizing (1)
an average of pairwise combined domain discrepancy between the given domain
and the mixture of the others $d_{\cal{H}}(\cal{V})$, and (2) an average of
every pairwise $\cal{H}$-divergence between each domain. Let each
$D_{\mathbf{v}}$ and $D_{\mathbf{v}^{c}}$ represent the $\mathbf{v}$th domain
and the mixture of the remaining $N$ domains with the same mixture weight
$\frac{1}{N}$, respectively. Then we can define $\cal{H}$-divergence as
${d}_{\cal{H}}(D_{\mathbf{v}},D_{\mathbf{v}^{c}})$, and an average of such
$\cal{H}$-divergence for every $\mathbf{v}$ as ${d}_{\cal{H}}(\cal{V})$.
Assume that the samples of size $m$, $Z_{\mathbf{v}}$ and
$Z_{\mathbf{v}^{c}}$, are generated from each $D_{\mathbf{v}}$ and
$D_{\mathbf{v}^{c}}$, where
$Z_{\mathbf{v}^{c}}=\bigcup_{\mathbf{v}^{\prime}\neq\mathbf{v}}Z_{\mathbf{v}^{\prime}}$
with $|Z_{\mathbf{v}^{\prime}}|=m/N$ for all $\mathbf{v}^{\prime}\in\cal{V}$.
Thus the domain label $\mathbf{v}_{j}\neq\mathbf{v}$ for every $j$th sample in
$Z_{\mathbf{v}^{c}}$. Then the empirical $\cal{H}$-divergence
$\hat{d}_{\cal{H}}(\cal{V})$ is defined as follows:

$\begin{split}\hat{d}_{\cal{H}}(\cal{V})&=\frac{1}{N+1}\sum_{\mathbf{v}\in\cal{V}}\hat{d}_{\cal{H}}(Z_{\mathbf{v}},Z_{\mathbf{v}^{c}})\\\
&=\frac{1}{N+1}\sum_{\mathbf{v}\in\cal{V}}2\Big{(}1-\min_{h\in\cal{H}}\Big{[}\frac{1}{m}\sum_{i:\mathbf{v}_{i}=\mathbf{v}}\mathbb{I}[h_{\mathbf{v}}(\mathbf{z}_{i})=0]\\\
&\qquad\qquad+\frac{1}{m}\sum_{j:\mathbf{v}_{j}\neq\mathbf{v}}\mathbb{I}[h_{\mathbf{v}}(\mathbf{z}_{j})=1]\Big{]}\Big{)},\end{split}$
(17)

where $\mathbb{I}[h_{\mathbf{v}}(\mathbf{z})=1]$ corresponds to the
$\mathbf{v}$th value of $N+1$ dimensional one-hot classification vector
$\mathbb{I}[h(\mathbf{z})]$, unlike the conditional probability vector
$h(\mathbf{z})$ in (16). Given the unified domain discriminator $h$ in the
inner minimization, we train $h$ to approximate $\hat{d}_{\cal{H}}(\cal{V})$
as follows:

$\begin{split}h^{*}&=\arg\max_{h\in\cal{H}}\frac{1}{M}\sum_{\mathbf{v}\in\cal{V}}\bigg{(}\sum_{i:\mathbf{v}_{i}=\mathbf{v}}\mathbb{I}[h_{\mathbf{v}}(\mathbf{z}_{i})=1]\\\
&\qquad\qquad\qquad\qquad+\sum_{j:\mathbf{v}_{j}\neq\mathbf{v}}\mathbb{I}[h_{\mathbf{v}}(\mathbf{z}_{j})=0]\bigg{)}\\\
&=\arg\min_{h\in\cal{H}}-\frac{1}{M}\sum_{\mathbf{v}\in\cal{V}}\sum_{i:\mathbf{v}_{i}=\mathbf{v}}\Big{(}\mathbbm{1}_{[k=\mathbf{v}_{i}]}^{T}\mathbb{I}[h(\mathbf{z}_{i})]\\\
&\qquad\qquad\qquad\qquad+\mathbbm{1}_{[k\neq\mathbf{v}_{i}]}^{T}\big{(}\mathbbm{1}-\mathbb{I}[h(\mathbf{z}_{i})]\big{)}\Big{)},\end{split}$
(18)

where the latter equality is obtained by rearranging the summation terms in
the first equality.

Based on the close relationship between (16) and (18), we can make the link
between information regularization and $\cal{H}$-divergence optimization given
multi-source domain; minimizing $\hat{d}_{\cal{H}}(\cal{V})$ is closely
related to implicit regularization of the mutual information between latent
representations and domain labels. Because the output classification vector
$\mathbb{I}[h(\mathbf{z})]$ often comes from the argmax operation, the
objective in (18) is not differentiable w.r.t. $\mathbf{z}$. However, our
framework has a differentiable objective for the discriminator as in (16).

There are two additional benefits of minimizing $d_{\cal{H}}(\cal{V})$. First,
it includes $\cal{H}$-divergence between the target and a mixture of sources
($\mathbf{v}=N+1$ in (17)). Note that it directly affects the upper bound of
the empirical risk on target samples (Theorem 5 in [3]). Moreover, the
synergistic penalization of other divergences ($\mathbf{v}\neq N+1$ in (17))
which implicitly include the domain discrepancy between the target and other
sources accelerates the adaptation. Second, ${d}_{\cal{H}}(\cal{V})$ lower-
bounds the average of every pairwise $\cal{H}$-divergence between each domain:

######  Lemma 2.

Let
$d_{\mathcal{H}}(\mathcal{V})=\frac{1}{N+1}\sum_{\mathbf{v}\in\mathcal{V}}d_{\mathcal{H}}(D_{\mathbf{v}},D_{\mathbf{v}^{c}})$.
Let $\mathcal{H}$ be a hypothesis class. Then,

$d_{\mathcal{H}}(\mathcal{V})\leq\frac{1}{N(N+1)}\sum_{\mathbf{v},\mathbf{u}\in{\mathcal{V}}}d_{\mathcal{H}}(D_{\mathbf{v}},D_{\mathbf{u}}).$
(19)

The detailed proof is provided in the appendix. It implies that not only the
domain shift between each source and the target domain, but also the domain
shift between each source domain can be indirectly penalized. Note that this
characteristic is known to be beneficial to MDA [25, 37]. Unlike our single
domain classifier setting, existing methods [25] require a number of about
$\mathcal{O}(N^{2})$ domain classifiers to approximate all pairwise
combinations of domain discrepancy. In this regard, there is no comparison
between the proposed method using a single domain classifier and existing
approaches in terms of resource efficiency.

Source classification. Along with learning domain-independent latent
representations illustrated in the above, we train the classifier with the
labeled source domain datasets. To minimize the empirical risk on source
domain, we use a generic softmax cross-entropy loss function with labeled
source domain samples as $L(F,C)$.

Decaying batch spectral penalization. Applying above information-theoretic
insights, we further describe a potential side effect of existing adversarial
DA methods. Information regularization may lead to overriding implicit entropy
minimization, particularly in the early stages of the training, impairing the
richness of latent feature representations. To prevent such a pathological
phenomenon, we introduce a new technique called Decaying Batch Spectral
Penalization (DBSP), which is intended to control the SVD entropy of the
feature space. Our version improves training efficiency compared to original
Batch Spectral Penalization [7]. We refer to this version of our model as
MIAN-$\gamma$. Since vanilla MIAN is sufficient to outperform other state-of-
the-art methods (Section 5), MIAN-$\gamma$ is further discussed in the
Supplementary Material.

##  5 Experiments

To assess the performance of MIAN, we ran a large-scale simulation using the
following benchmark datasets: Digits-Five, Office-31 and Office-Home. For a
fair comparison, we reproduced all the other baseline results using the same
backbone architecture and optimizer settings as the proposed method. For the
source-only and single-source DA standards, we introduce two MDA approaches
[54, 37]: ($1$) source-combined, i.e., all source-domains are incorporated
into a single source domain; ($2$) single-best, i.e., the best adaptation
performance on the target domain is reported. Owing to limited space, details
about simulation settings, used baseline models and datasets are presented in
the Supplementary Material.

Table 1: Accuracy ($\%$) on Digits-Five dataset. SYNTH denotes Synthetic Digits [11]. The baseline results for the Digits-Five dataset were taken from [37]. Standards | Models | MNIST-M | MNIST | USPS | SVHN | SYNTH | Avg
---|---|---|---|---|---|---|---
Source- combined | Source Only [17] | 63.70 | 92.30 | 90.71 | 71.51 | 83.44 | 80.33
DAN [28] | 67.87 | 97.50 | 93.49 | 67.80 | 86.93 | 82.72
DANN [12] | 70.81 | 97.90 | 93.47 | 68.50 | 87.37 | 83.61
Single-best | Source Only [17] | 63.37 | 90.50 | 88.71 | 63.54 | 82.44 | 77.71
DAN [28] | 63.78 | 96.31 | 94.24 | 62.45 | 85.43 | 80.44
DANN [12] | 71.30 | 97.60 | 92.33 | 63.48 | 85.34 | 82.01
JAN [30] | 65.88 | 97.21 | 95.42 | 75.27 | 86.55 | 84.07
ADDA [50] | 71.57 | 97.89 | 92.83 | 75.48 | 86.45 | 84.84
MEDA [52] | 71.31 | 96.47 | 97.01 | 78.45 | 84.62 | 85.60
| MCD [41] | 72.50 | 96.21 | 95.33 | 78.89 | 87.47 | 86.10
Multi- source | DCTN [54] | 70.53 | 96.23 | 92.81 | 77.61 | 86.77 | 84.79
M3SDA [37] | 69.76 | 98.58 | 95.23 | 78.56 | 87.56 | 86.13
M3SDA-$\beta$ [37] | 72.82 | 98.43 | 96.14 | 81.32 | 89.58 | 87.65
MIAN | 84.36 | 97.91 | 96.49 | 88.18 | 93.23 | 92.03
Table 2: Accuracy ($\%$) on Office-31 dataset. Standards | Models | Amazon | DSLR | Webcam | Avg
---|---|---|---|---|---
Single-best | Source Only [17] | 55.23$\pm$0.72 | 95.59$\pm$1.37 | 87.06$\pm$1.50 | 79.29
DAN [28] | 64.19$\pm$0.56 | 100.00$\pm$0.00 | 97.45$\pm$0.44 | 87.21
JAN [30] | 69.57$\pm$0.27 | 99.80$\pm$0.00 | 97.4$\pm$0.26 | 88.92
Source- combined | Source Only [17] | 60.80$\pm$2.00 | 92.68$\pm$0.31 | 86.91$\pm$2.37 | 80.13
DSBN [5] | 66.82$\pm$0.35 | 97.45$\pm$0.22 | 94.00$\pm$0.38 | 86.09
JAN [30] | 70.15$\pm$0.19 | 95.20$\pm$0.36 | 95.15$\pm$0.23 | 86.83
DANN [12] | 68.15$\pm$0.42 | 97.59$\pm$0.60 | 96.77$\pm$0.26 | 87.50
DAN [28] | 65.77$\pm$0.74 | 99.26$\pm$0.23 | 97.51$\pm$0.41 | 87.51
DANN+BSP [7] | 71.13$\pm$0.44 | 96.65$\pm$0.30 | 98.32$\pm$0.26 | 88.70
MCD [41] | 68.57$\pm$1.06 | 99.49$\pm$0.25 | 99.30$\pm$0.38 | 89.12
Multi- source | DCTN [54] | 62.74$\pm$0.50 | 99.44$\pm$0.25 | 97.92$\pm$0.29 | 86.70
M3SDA [37] | 67.19$\pm$0.22 | 99.34$\pm$0.19 | 98.04$\pm$0.21 | 88.19
M3SDA-$\beta$ [37] | 69.41$\pm$0.82 | 99.64$\pm$0.19 | 99.30$\pm$0.31 | 89.45
MIAN | 74.65$\pm$0.48 | 99.48$\pm$0.35 | 98.49$\pm$0.59 | 90.87
| MIAN-$\gamma$ | 76.17$\pm$0.24 | 99.22$\pm$0.35 | 98.39$\pm$0.76 | 91.26
Table 3: Accuracy ($\%$) on Office-Home dataset. Standards | Models | Art | Clipart | Product | Realworld | Avg
---|---|---|---|---|---|---
Source- combined | Source Only [17] | 64.58$\pm$0.68 | 52.32$\pm$0.63 | 77.63$\pm$0.23 | 80.70$\pm$0.81 | 68.81
DANN [12] | 64.26$\pm$0.59 | 58.01$\pm$1.55 | 76.44$\pm$0.47 | 78.80$\pm$0.49 | 69.38
DANN+BSP [7] | 66.10$\pm$0.27 | 61.03$\pm$0.39 | 78.13$\pm$0.31 | 79.92$\pm$0.13 | 71.29
DAN [28] | 68.28$\pm$0.45 | 57.92$\pm$0.65 | 78.45$\pm$0.05 | 81.93$\pm$0.35 | 71.64
MCD [41] | 67.84$\pm$0.38 | 59.91$\pm$0.55 | 79.21$\pm$0.61 | 80.93$\pm$0.18 | 71.97
Multi- source | M3SDA [37] | 66.22$\pm$0.52 | 58.55$\pm$0.62 | 79.45$\pm$0.52 | 81.35$\pm$0.19 | 71.39
DCTN [54] | 66.92$\pm$0.60 | 61.82$\pm$0.46 | 79.20$\pm$0.58 | 77.78$\pm$0.59 | 71.43
MIAN | 69.39$\pm$0.50 | 63.05$\pm$0.61 | 79.62$\pm$0.16 | 80.44$\pm$0.24 | 73.12
MIAN-$\gamma$ | 69.88$\pm$0.35 | 64.20$\pm$0.68 | 80.87$\pm$0.37 | 81.49$\pm$0.24 | 74.11

###  5.1 Simulation results

The classification accuracy for Digits-Five, Office-31, and Office-Home are
summarized in Tables 1, 2, and 3, respectively. We found that MIAN outperforms
most of other state-of-the-art single-source and multi-source DA methods by a
large margin. Note that our method demonstrated a significant improvement in
difficult task transfer with high domain shift, such as MNIST-M, Amazon or
Clipart, which is the key performance indicator of MDA.

###  5.2 Ablation study and Quantitative analyses

(a) Accuracy (MNIST-M)

(b) Accuracy (SVHN)

(c) Variance (MNIST-M)

(d) Variance (SVHN)

Figure 2: (a)$\sim$(b): Test accuracies for (a) MNIST-M and (b) SVHN as target
domains. (c)$\sim$(d): Variance of stochastic gradients after 1000 steps for
(c) MNIST-M and (d) SVHN as target domains in log scale. Less is better.

(a) Discrepancy

(b) Accuracy

(c) $\hat{d}_{\cal{H}}(\cal{V}$)

(d) $\hat{I}(Z;V)$

Figure 3: (a) Proxy $\cal{A}$-distance. (b)$\sim$(c) Ablation study on the
objective of domain discriminator. CEN stands for multi-class cross entropy
loss in (15), while BCE stands for binary-class cross entropy losses in (16).
(d) Empirical information $\hat{I}(Z;V)$. We treat $H(V)=\log|\cal{V}|$.

Design of domain discriminator. To quantify the extent to which performance
improvement is achieved by unifying the domain discriminators, we compared the
performances of the three different versions of MIAN (Figure 2(a), 2(b)). No
LS uses the objective function as in (16), and unlike [34]. Multi D employs as
many discriminators as the number of source domains which is analogous to the
existing approaches. For a fair comparison, all the other experimental
settings are fixed. The results illustrate that all the versions with the
unified discriminator reliably outperform Multi D in terms of both accuracy
and reliability. This suggests that unification of the domain discriminators
can substantially improves the task performance.

Variance of stochastic gradients. With respect to the above analysis, we
compared the variance of the stochastic gradients computed with different
available domain discriminators. We trained MIAN and Multi D using mini-
batches of samples. After the early stages of training, we computed the
gradients for the weights and biases of both the top and bottom layers of the
encoder on the full training set. Figures 2(c), 2(d) show that MIAN with the
unified discriminator yields exponentially lower variance of the gradients
compared to Multi D. Thus it is more feasible to use the unified discriminator
when a large number of domains are given.

Proxy $\cal{A}$-distance. To analyze the performance improvement in depth, we
measured Proxy $\cal{A}$-Distance (PAD) as an empirical approximation of
domain discrepancy [12]. Given the generalization error $\epsilon$ on
discriminating between the target and source samples, PAD is defined as
$\hat{d}_{\cal{A}}=2(1-2\epsilon)$. Figure 3(a) shows that MIAN yields lower
PAD between the source and target domain on average, potentially associated
with the modified objective of discriminator. To test this conjecture, we
conducted an ablation study on the objective of domain discriminator (Figure
3(b), 3(c)). All the other experimental settings were fixed except for using
the objective of the unified domain discriminator as (15), or (16). While both
cases help the adaptation, using (16) yields lower
$\hat{d}_{\cal{H}}(\cal{V})$ and higher test accuracy.

Estimation of mutual information. We measure the empirical mutual information
$\hat{I}(Z;V)$ with the assumption of $H(V)$ as a constant. Figure 3(d) shows
that MIAN yields the lowest $\hat{I}(Z;V)$, ensuring that the obtained
representation achieves low-level domain dependence. It empirically supports
the established bridge between adversarial DA and Information Bottleneck
theory in section 3.4.

##  6 Conclusion

In this paper, we have presented a unified information-regularization
framework for MDA. The proposed framework allows us to examine the existing
adversarial DA methods and motivated us to implement a novel neural
architecture for MDA. Specifically, we provided both theoretical arguments and
empirical evidence to justify potential pitfalls of using multiple
discriminators: disintegration of domain-discriminative knowledge, limited
computational efficiency and high variance in the objective. The proposed
model does not require complicated settings such as image generation,
pretraining, or multiple networks, which are often adopted in the existing MDA
methods [57, 58, 54, 56, 26].

## References

  * [1] Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy.  Deep variational information bottleneck.  arXiv preprint arXiv:1612.00410, 2016.
  * [2] Orly Alter, Patrick O Brown, and David Botstein.  Singular value decomposition for genome-wide expression data processing and modeling.  Proceedings of the National Academy of Sciences, 97(18):10101–10106, 2000.
  * [3] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan.  A theory of learning from different domains.  Machine learning, 79(1-2):151–175, 2010.
  * [4] John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman.  Learning bounds for domain adaptation.  In Advances in neural information processing systems, pages 129–136, 2008.
  * [5] Woong-Gi Chang, Tackgeun You, Seonguk Seo, Suha Kwak, and Bohyung Han.  Domain-specific batch normalization for unsupervised domain adaptation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7354–7362, 2019.
  * [6] Rita Chattopadhyay, Qian Sun, Wei Fan, Ian Davidson, Sethuraman Panchanathan, and Jieping Ye.  Multisource domain adaptation and its application to early detection of fatigue.  ACM Transactions on Knowledge Discovery from Data (TKDD), 6(4):1–26, 2012.
  * [7] Xinyang Chen, Sinan Wang, Mingsheng Long, and Jianmin Wang.  Transferability vs. discriminability: Batch spectral penalization for adversarial domain adaptation.  In International Conference on Machine Learning, pages 1081–1090, 2019.
  * [8] Nicolas Courty, Rémi Flamary, Amaury Habrard, and Alain Rakotomamonjy.  Joint distribution optimal transportation for domain adaptation.  In Advances in Neural Information Processing Systems, pages 3730–3739, 2017.
  * [9] Lixin Duan, Dong Xu, and Shih-Fu Chang.  Exploiting web images for event recognition in consumer videos: A multiple source domain adaptation approach.  In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 1338–1345. IEEE, 2012.
  * [10] Lixin Duan, Dong Xu, and Ivor Wai-Hung Tsang.  Domain adaptation from multiple sources: A domain-dependent regularization approach.  IEEE Transactions on neural networks and learning systems, 23(3):504–518, 2012.
  * [11] Yaroslav Ganin and Victor Lempitsky.  Unsupervised domain adaptation by backpropagation.  arXiv preprint arXiv:1409.7495, 2014.
  * [12] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky.  Domain-adversarial training of neural networks.  The Journal of Machine Learning Research, 17(1):2096–2030, 2016\.
  * [13] Boqing Gong, Kristen Grauman, and Fei Sha.  Reshaping visual datasets for domain adaptation.  In Advances in Neural Information Processing Systems, pages 1286–1294, 2013.
  * [14] Rui Gong, Wen Li, Yuhua Chen, and Luc Van Gool.  Dlow: Domain flow for adaptation and generalization.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2477–2486, 2019.
  * [15] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Advances in neural information processing systems, pages 2672–2680, 2014.
  * [16] Arthur Gretton, Alex Smola, Jiayuan Huang, Marcel Schmittfull, Karsten Borgwardt, and Bernhard Schölkopf.  Covariate shift by kernel mean matching.  Dataset shift in machine learning, 3(4):5, 2009.
  * [17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [18] Judy Hoffman, Brian Kulis, Trevor Darrell, and Kate Saenko.  Discovering latent domains for multisource domain adaptation.  In European Conference on Computer Vision, pages 702–715. Springer, 2012.
  * [19] Judy Hoffman, Mehryar Mohri, and Ningshan Zhang.  Algorithms and theory for multiple-source adaptation.  In Advances in Neural Information Processing Systems, pages 8246–8256, 2018.
  * [20] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei A Efros, and Trevor Darrell.  Cycada: Cycle-consistent adversarial domain adaptation.  arXiv preprint arXiv:1711.03213, 2017.
  * [21] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  arXiv preprint arXiv:1502.03167, 2015.
  * [22] Rie Johnson and Tong Zhang.  Accelerating stochastic gradient descent using predictive variance reduction.  In Advances in neural information processing systems, pages 315–323, 2013.
  * [23] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [24] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324, 1998.
  * [25] Yitong Li, David E Carlson, et al.  Extracting relationships by multi-domain matching.  In Advances in Neural Information Processing Systems, pages 6798–6809, 2018.
  * [26] Chuang Lin, Sicheng Zhao, Lei Meng, and Tat-Seng Chua.  Multi-source domain adaptation for visual sentiment classification.  arXiv preprint arXiv:2001.03886, 2020.
  * [27] Hong Liu, Mingsheng Long, Jianmin Wang, and Michael Jordan.  Transferable adversarial training: A general approach to adapting deep classifiers.  In International Conference on Machine Learning, pages 4013–4022, 2019.
  * [28] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I Jordan.  Learning transferable features with deep adaptation networks.  arXiv preprint arXiv:1502.02791, 2015.
  * [29] Mingsheng Long, Jianmin Wang, Guiguang Ding, Jiaguang Sun, and Philip S Yu.  Transfer joint matching for unsupervised domain adaptation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1410–1417, 2014.
  * [30] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan.  Deep transfer learning with joint adaptation networks.  In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 2208–2217. JMLR. org, 2017.
  * [31] Yawei Luo, Ping Liu, Tao Guan, Junqing Yu, and Yi Yang.  Significance-aware information bottleneck for domain adaptive semantic segmentation.  In Proceedings of the IEEE International Conference on Computer Vision, pages 6778–6787, 2019.
  * [32] Massimiliano Mancini, Lorenzo Porzi, Samuel Rota Bulò, Barbara Caputo, and Elisa Ricci.  Boosting domain adaptation by discovering latent domains.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3771–3780, 2018.
  * [33] Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh.  Domain adaptation with multiple sources.  In Advances in neural information processing systems, pages 1041–1048, 2009.
  * [34] Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley.  Least squares generative adversarial networks.  In Proceedings of the IEEE international conference on computer vision, pages 2794–2802, 2017.
  * [35] Zak Murez, Soheil Kolouri, David Kriegman, Ravi Ramamoorthi, and Kyungnam Kim.  Image to image translation for domain adaptation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4500–4509, 2018.
  * [36] Paul K Newton and Stephen A DeSalvo.  The shannon entropy of sudoku matrices.  Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 466(2119):1957–1975, 2010.
  * [37] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang.  Moment matching for multi-source domain adaptation.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1406–1415, 2019.
  * [38] Yuji Roh, Kangwook Lee, Steven Euijong Whang, and Changho Suh.  Fr-train: A mutual information-based approach to fair and robust training.  arXiv preprint arXiv:2002.10234, 2020.
  * [39] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell.  Adapting visual category models to new domains.  In European conference on computer vision, pages 213–226. Springer, 2010.
  * [40] Kuniaki Saito, Yoshitaka Ushiku, Tatsuya Harada, and Kate Saenko.  Adversarial dropout regularization.  arXiv preprint arXiv:1711.01575, 2017.
  * [41] Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada.  Maximum classifier discrepancy for unsupervised domain adaptation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3723–3732, 2018.
  * [42] Swami Sankaranarayanan, Yogesh Balaji, Carlos D Castillo, and Rama Chellappa.  Generate to adapt: Aligning domains using generative adversarial networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8503–8512, 2018.
  * [43] Swami Sankaranarayanan, Yogesh Balaji, Arpit Jain, Ser Nam Lim, and Rama Chellappa.  Learning from synthetic data: Addressing domain shift for semantic segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3752–3761, 2018.
  * [44] Yuxuan Song, Lantao Yu, Zhangjie Cao, Zhiming Zhou, Jian Shen, Shuo Shao, Weinan Zhang, and Yong Yu.  Improving unsupervised domain adaptation with variational information bottleneck.  arXiv preprint arXiv:1911.09310, 2019.
  * [45] Baochen Sun, Jiashi Feng, and Kate Saenko.  Return of frustratingly easy domain adaptation.  In Thirtieth AAAI Conference on Artificial Intelligence, 2016.
  * [46] Baochen Sun and Kate Saenko.  Deep coral: Correlation alignment for deep domain adaptation.  In European conference on computer vision, pages 443–450. Springer, 2016.
  * [47] Naftali Tishby, Fernando C Pereira, and William Bialek.  The information bottleneck method.  arXiv preprint physics/0004057, 2000.
  * [48] Naftali Tishby and Noga Zaslavsky.  Deep learning and the information bottleneck principle.  In 2015 IEEE Information Theory Workshop (ITW), pages 1–5. IEEE, 2015.
  * [49] Yi-Hsuan Tsai, Wei-Chih Hung, Samuel Schulter, Kihyuk Sohn, Ming-Hsuan Yang, and Manmohan Chandraker.  Learning to adapt structured output space for semantic segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7472–7481, 2018.
  * [50] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell.  Adversarial discriminative domain adaptation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7167–7176, 2017.
  * [51] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan.  Deep hashing network for unsupervised domain adaptation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5018–5027, 2017.
  * [52] Jindong Wang, Wenjie Feng, Yiqiang Chen, Han Yu, Meiyu Huang, and Philip S Yu.  Visual domain adaptation with manifold embedded distribution alignment.  In Proceedings of the 26th ACM international conference on Multimedia, pages 402–410, 2018.
  * [53] Junfeng Wen, Russell Greiner, and Dale Schuurmans.  Domain aggregation networks for multi-source domain adaptation.  arXiv preprint arXiv:1909.05352, 2019.
  * [54] Ruijia Xu, Ziliang Chen, Wangmeng Zuo, Junjie Yan, and Liang Lin.  Deep cocktail network: Multi-source unsupervised domain adaptation with category shift.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3964–3973, 2018.
  * [55] Han Zhao, Remi Tachet des Combes, Kun Zhang, and Geoffrey J Gordon.  On learning invariant representation for domain adaptation.  arXiv preprint arXiv:1901.09453, 2019.
  * [56] Han Zhao, Shanghang Zhang, Guanhang Wu, José MF Moura, Joao P Costeira, and Geoffrey J Gordon.  Adversarial multiple source domain adaptation.  In Advances in neural information processing systems, pages 8559–8570, 2018.
  * [57] Sicheng Zhao, Bo Li, Xiangyu Yue, Yang Gu, Pengfei Xu, Runbo Hu, Hua Chai, and Kurt Keutzer.  Multi-source domain adaptation for semantic segmentation.  In Advances in Neural Information Processing Systems, pages 7285–7298, 2019.
  * [58] Sicheng Zhao, Guangzhi Wang, Shanghang Zhang, Yang Gu, Yaxian Li, Zhichao Song, Pengfei Xu, Runbo Hu, Hua Chai, and Kurt Keutzer.  Multi-source distilling domain adaptation.  arXiv preprint arXiv:1911.11554, 2019.

##  Appendix A Pseudocode

Due to the limited space, we provide the algorithm of MIAN in this Section.
Details about training-dependent scaling of $\beta_{t}$ are in Section E.

Input: mini-batch size for each domain $m$, Number of source domains $N$,
Training iteration $T$. $M=m(N+1)$, Set of domain labels
${\cal{V}}=\\{1,\dots,N+1\\}$.

Output: Transferable Encoder $F$, Classifier $C$

for  _$t\leftarrow 1$ to $T$_ do

$X=\left\\{\mathbf{x}_{i}\right\\}_{i=1}^{M}=X_{S_{1}}\bigcup\dots\bigcup
X_{S_{N}}\bigcup X_{T}$

$Y=\left\\{\mathbf{y}_{i}\right\\}_{i=1}^{mN}=Y_{S_{1}}\bigcup\dots\bigcup
Y_{S_{N}}$

Encode latent representation $\mathbf{z}_{i}=F(\mathbf{x}_{i})$

// Inner maximization

Optimize discriminator $h$ by the objective $L(h)$ in (16) using gradient
descent.

// Outer minimization

$L(F,C)=-\frac{1}{mN}\sum_{\mathbf{y}\in\cal{Y}}\sum_{i:\mathbf{y}_{i}=\mathbf{y}}\big{[}\mathbbm{1}_{[k=\mathbf{y}_{i}]}^{T}\log{\mathbf{\hat{y}}_{i}}\big{]}$

$\beta_{t}=\beta_{0}\cdot 2\big{(}1-\frac{1}{1+exp{(-\sigma\cdot
t/T)}}\big{)}$

$L(F)=L(F,C)-\beta_{t}L(h)$ ;

Optimize encoder $F$ by the objective $L(F)$ using gradient descent.

Optimize classifier $C$ by the objective $L(F,C)$ using gradient descent.

Algorithm 1 Multi-source Information-regularized Adaptation Networks (MIAN)

##  Appendix B Proofs

In this Section, we present the detailed proofs for Theorems 2, 3 and Lemma 2,
explained in the main paper. Following [38], we provide a proof of Theorem 2
below for the sake of completeness.

###  B.1 Proof of Theorem 2

######  Theorem 2.

Let $P_{Z}(\mathbf{z})$ be the distribution of $Z$ where
$\mathbf{z}\in{\cal{Z}}$. Let $h$ be a domain classifier
$h:\cal{Z}\rightarrow\cal{V}$, where $\cal{Z}$ is the feature space and
$\cal{V}$ is the set of domain labels. Let $h_{\mathbf{v}}(Z)$ be a
conditional probability of $V$ where $\mathbf{v}\in\cal{V}$ given
$Z=\mathbf{z}$, defined by $h$. Then the following holds:

$\begin{split}I(Z;V)&=\max_{h_{\mathbf{v}}(\mathbf{z}):\sum_{\mathbf{v}\in\cal{V}}{h_{\mathbf{v}}(\mathbf{z})=1,\forall{\mathbf{z}}}}\\\
&\sum_{\mathbf{v}\in\cal{V}}{P_{V}(\mathbf{v})\mathbb{E}_{\mathbf{z}\sim
P_{Z\mid\mathbf{v}}}\big{[}\log{h_{\mathbf{v}}(\mathbf{z})}\big{]}}+H(V)\end{split}$
(20)

###### Proof.

By definition,

$\begin{split}I(Z;V)&=D_{KL}\big{(}P(Z,V)\parallel{P(Z)P(V)}\big{)}\\\
&=\underset{\mathbf{v}\in\cal{V}}{\sum}P_{V}(\mathbf{v})\mathbb{E}_{\mathbf{z}\sim
P_{Z\mid\mathbf{v}}}\Big{[}\log\frac{P_{Z,V}(\mathbf{z},\mathbf{v})}{P_{Z}(\mathbf{z})}\Big{]}+H(V)\end{split}$
(21)

Let us constrain the term inside the log by
$h_{\mathbf{v}}(\mathbf{z})=\frac{P_{Z,V}(\mathbf{z},\mathbf{v})}{P_{Z}(\mathbf{z})}$
where $h_{\mathbf{v}}(\mathbf{z})$ represents the conditional probability of
$V=\mathbf{v}$ for any $\mathbf{v}\in\cal{V}$ given $Z=\mathbf{z}$. Then we
have: $\sum_{\mathbf{v}\in\cal{V}}h_{\mathbf{v}}(\mathbf{z})=1$ for all
possible values of $\mathbf{z}$ according to the law of total probability. Let
$\mathbf{h}$ denote the collection of $h_{\mathbf{v}}(\mathbf{z})$ for all
possible values of $\mathbf{v}$ and $\mathbf{z}$, and $\boldsymbol{\lambda}$
be the collection of $\lambda_{\mathbf{z}}$ for all values of $\mathbf{z}$.
Then, we can construct the Lagrangian function by incorporating the constraint
$\sum_{\mathbf{v}\in\cal{V}}h_{\mathbf{v}}(\mathbf{z})=1$ as follows:

$\begin{split}L{(\mathbf{h},\boldsymbol{\lambda})}&=\underset{\mathbf{v}\in\cal{V}}{\sum}P_{V}(\mathbf{v})\mathbb{E}_{\mathbf{z}\sim
P_{Z\mid\mathbf{v}}}\Big{[}log\big{(}h_{\mathbf{v}}(\mathbf{z})\big{)}\Big{]}+H(V)\\\
&+\underset{\mathbf{z}\in\cal{Z}}{\sum}\lambda_{\mathbf{z}}\Big{(}1-\underset{\mathbf{v}\in\cal{V}}{\sum}h_{\mathbf{v}}(\mathbf{z})\Big{)}\end{split}$
(22)

We can use the following KKT conditions:

$\centering\frac{\partial{L(\mathbf{h},\boldsymbol{\lambda})}}{\partial{h_{\mathbf{v}}(\mathbf{z})}}=P_{V}(\mathbf{v})\frac{P_{Z\mid\mathbf{v}}(\mathbf{z})}{h^{*}_{\mathbf{v}}(\mathbf{z})}-\lambda^{*}_{\mathbf{z}}=0,\
\forall(\mathbf{z},\mathbf{v})\in{\cal{Z}\times\cal{V}}\@add@centering$ (23)
$1-\underset{\mathbf{v}\in\cal{V}}{\sum}h^{*}_{\mathbf{v}}(\mathbf{z})=0,\quad\forall\mathbf{z}\in{\cal{Z}}$
(24)

Solving the two equations, we have
$1-\underset{\mathbf{v}\in\cal{V}}{\sum}\frac{P_{V}(\mathbf{v})P_{Z\mid\mathbf{v}}(\mathbf{z})}{\lambda^{*}_{\mathbf{z}}}=0$
such that $\lambda_{\mathbf{z}}^{*}=P_{Z}(\mathbf{z})$ for all $\mathbf{z}$.
Then for all the possible values of $\mathbf{z}$,

$\begin{split}h^{*}_{\mathbf{v}}(\mathbf{z})&=\frac{P_{Z,V}(\mathbf{z},\mathbf{v})}{P_{Z}(\mathbf{z})}\\\
&=P_{V\mid\mathbf{z}}(\mathbf{v}),\end{split}$ (25)

where the given $h^{*}_{\mathbf{v}}(\mathbf{z})$ is same as the term inside
log in (21). Thus, the optimal solution of concave Lagrangian function (22)
obtained by $h^{*}_{\mathbf{v}}(\mathbf{z})$ is equal to the mutual
information in (21). The substitution of $h^{*}_{\mathbf{v}}(\mathbf{z})$ into
(21) completes the proof. ∎

Our framework can further be applied to segmentation problems because it
provides a new perspective on pixel space [42, 43, 35] and segmentation space
[49] adaptation. The generator in pixel space and segmentation space
adaptation learns to transform images or segmentation results from one domain
to another. In the context of information regularization, we can view these
approaches as limiting information $I(\hat{X};V)$ between the generated output
$\hat{X}$ and the domain label $V$, which is accomplished by involving the
encoder for pixel-level generation. This alleviates the domain shift in a raw
pixel level. Note that one can choose between limiting the feature-level or
pixel-level mutual information. These different regularization terms may be
complementary to each other depending on the given task.

###  B.2 Proof of Theorem 3

######  Theorem 3.

Let $P_{Z\mid\mathbf{x},\mathbf{v}}(\mathbf{z})$ be a conditional
probabilistic distribution of $Z$ where $\mathbf{z}\in{\cal{Z}}$, defined by
the encoder $F$, given a sample $\mathbf{x}\in\cal{X}$ and the domain label
$\mathbf{v}\in\cal{V}$. Let $R_{Z}(\mathbf{z})$ denotes a prior marginal
distribution of $Z$. Then the following inequality holds:

$\begin{split}&I(Z;X,V)\leq\mathbb{E}_{\mathbf{x,v}\sim
P_{X,V}}\big{[}D_{KL}[P_{Z\mid\mathbf{x},\mathbf{v}}\parallel
R_{Z}]\big{]}+H(V)\\\
&+\max_{h_{\mathbf{v}}(\mathbf{z}):\sum_{\mathbf{v}\in\cal{V}}{h_{\mathbf{v}}(\mathbf{z})=1},\forall{\mathbf{z}}}\sum_{\mathbf{v}\in\cal{V}}{P_{V}(\mathbf{v})\mathbb{E}_{P_{\mathbf{z}\sim
Z\mid\mathbf{v}}}\big{[}\log{h_{\mathbf{v}}(\mathbf{z})}\big{]}}\end{split}$
(26)

###### Proof.

Based on the chain rule for mutual information,

$\begin{split}&I(Z;X,V)\\\ &=I(Z;V)+I(Z;X\mid V)\\\ &=H(V)+I(Z;X\mid V)\\\
&+\max_{h_{\mathbf{v}}(\mathbf{z}):\sum_{\mathbf{v}\in\cal{V}}{h_{\mathbf{v}}(\mathbf{z})=1,\forall{\mathbf{z}}}}\sum_{\mathbf{v}\in\cal{V}}{P_{V}(\mathbf{v})\mathbb{E}_{\mathbf{z}\sim
P_{Z\mid\mathbf{v}}}\big{[}\log{h_{\mathbf{v}}(\mathbf{z})}\big{]}},\end{split}$
(27)

where the latter equality is given by Theorem 2. Then,

$\begin{split}&I(Z;X\mid V)\\\ &=\mathbb{E}_{\mathbf{v}\sim
P_{V}}\Big{[}\mathbb{E}_{\mathbf{z,x}\sim
P_{Z,X\mid\mathbf{v}}}\Big{[}\log\frac{P_{Z,X\mid\mathbf{v}}(\mathbf{z,x})}{P_{Z\mid\mathbf{v}}(\mathbf{z})P_{X\mid\mathbf{v}}(\mathbf{x})}\Big{]}\Big{]}\\\
&=\mathbb{E}_{\mathbf{x,v}\sim P_{X,V}}\Big{[}\mathbb{E}_{\mathbf{z}\sim
P_{Z\mid\mathbf{x,v}}}\Big{[}\log\frac{P_{Z\mid\mathbf{x,v}}(\mathbf{z})}{P_{Z\mid\mathbf{v}}(\mathbf{z})}\Big{]}\Big{]}\\\
&=\mathbb{E}_{\mathbf{x,v}\sim P_{X,V}}\Big{[}\mathbb{E}_{\mathbf{z}\sim
P_{Z\mid\mathbf{x,v}}}\big{[}\log
P_{Z\mid\mathbf{x,v}}(\mathbf{z})\big{]}\Big{]}\\\
&\qquad\qquad-\mathbb{E}_{\mathbf{v}\sim
P_{V}}\Big{[}\mathbb{E}_{\mathbf{z}\sim P_{Z\mid\mathbf{v}}}\big{[}\log
P_{Z\mid\mathbf{v}}(\mathbf{z})\big{]}\Big{]}\\\
&\leq\mathbb{E}_{\mathbf{x,v}\sim P_{X,V}}\Big{[}\mathbb{E}_{\mathbf{z}\sim
P_{Z\mid\mathbf{x,v}}}\big{[}\log
P_{Z\mid\mathbf{x,v}}(\mathbf{z})\big{]}\Big{]}\\\
&\qquad\qquad-\mathbb{E}_{\mathbf{v}\sim
P_{V}}\Big{[}\mathbb{E}_{\mathbf{z}\sim P_{Z\mid\mathbf{v}}}\big{[}\log
R_{Z}(\mathbf{z})\big{]}\Big{]}\\\ &=\mathbb{E}_{\mathbf{x,v}\sim
P_{X,V}}\Big{[}\mathbb{E}_{\mathbf{z}\sim
P_{Z\mid\mathbf{x,v}}}\Big{[}\log\frac{P_{Z\mid\mathbf{x,v}}(\mathbf{z})}{R_{Z}(\mathbf{z})}\Big{]}\Big{]}\\\
&=\mathbb{E}_{\mathbf{x,v}\sim
P_{X,V}}\Big{[}D_{KL}\big{[}P_{Z\mid\mathbf{x},\mathbf{v}}\parallel
R_{Z}\big{]}\Big{]}\end{split}$ (28)

The second equality is obtained by using
$P_{Z,X\mid\mathbf{v}}(\mathbf{z,x})=P_{X\mid\mathbf{v}}(\mathbf{x})P_{Z\mid\mathbf{x,v}}(\mathbf{z})$.
The inequality is obtained by using:

$D_{KL}[P_{Z\mid\mathbf{v}}\parallel R_{Z}]=\mathbb{E}_{\mathbf{z}\sim
P_{Z\mid\mathbf{v}}}\big{[}\log P_{Z\mid\mathbf{v}}(\mathbf{z})-\log
R_{Z}(\mathbf{z})\big{]},$ (29)

where $R_{Z}(\mathbf{z})$ is a variational approximation of the prior marginal
distribution of $Z$. The last equality is obtained from the definition of KL-
divergence. The substitution of (28) into (27) completes the proof. ∎

The existing DA work on semantic segmentation tasks [31, 44] can be explained
as the process of fostering close collaboration between the aforementioned
information bottleneck terms. The only difference between Theorem 3 for
${\cal{V}}=\left\\{0,1\right\\}$ and the objective function in [31] is that
[31] employed the shared encoding $P_{Z\mid\mathbf{x}}(\mathbf{z})$ instead of
$P_{Z\mid\mathbf{x},\mathbf{v}}(\mathbf{z})$, whereas some adversarial DA
approaches use the unshared one [50].

###  B.3 Proof of Lemma 2

######  Lemma 2.

Let
$d_{\mathcal{H}}(\mathcal{V})=\frac{1}{N+1}\sum_{\mathbf{v}\in\mathcal{V}}d_{\mathcal{H}}(D_{\mathbf{v}},D_{\mathbf{v}^{c}})$.
Let $\mathcal{H}$ be a hypothesis class. Then,

$d_{\mathcal{H}}(\mathcal{V})\leq\frac{1}{N(N+1)}\sum_{\mathbf{v},\mathbf{u}\in{\mathcal{V}}}d_{\mathcal{H}}(D_{\mathbf{v}},D_{\mathbf{u}}).$
(30)

###### Proof.

Let $\alpha=\frac{1}{N}$ represents the uniform domain weight for the mixture
of domain $D_{\mathbf{v}^{c}}$. Then,

$\begin{split}&d_{\mathcal{H}}(\mathcal{V})\\\
&=\frac{1}{N+1}\sum_{\mathbf{v}\in\mathcal{V}}d_{\mathcal{H}}(D_{\mathbf{v}},D_{\mathbf{v}^{c}})\\\
&=\frac{1}{N+1}\sum_{\mathbf{v}\in\mathcal{V}}2\sup_{h\in\mathcal{H}}\Big{|}\mathbb{E}_{\mathbf{x}\sim
P_{D_{\mathbf{v}}^{X}}}\big{[}\mathbb{I}\big{(}h(\mathbf{x}=1)\big{)}\big{]}\\\
&\qquad\qquad\qquad-\mathbb{E}_{\mathbf{x}\sim
P_{D_{\mathbf{v}^{c}}^{X}}}\big{[}\mathbb{I}\big{(}h(\mathbf{x}=1)\big{)}\big{]}\Big{|}\\\
&=\frac{1}{N+1}\sum_{\mathbf{v}\in\mathcal{V}}2\sup_{h\in\mathcal{H}}\bigg{|}\sum_{\mathbf{u}\in\mathcal{V}:\mathbf{u}\neq\mathbf{v}}\alpha\Big{(}\mathbb{E}_{\mathbf{x}\sim
P_{D_{\mathbf{v}}^{X}}}\big{[}\mathbb{I}\big{(}h(\mathbf{x}=1)\big{)}\big{]}\\\
&\qquad\qquad\qquad-\mathbb{E}_{\mathbf{x}\sim
P_{D_{\mathbf{u}}^{X}}}\big{[}\mathbb{I}\big{(}h(\mathbf{x}=1)\big{)}\big{]}\Big{)}\bigg{|}\\\
&\leq\frac{1}{N+1}\sum_{\mathbf{v}\in\mathcal{V}}\sum_{\mathbf{u}\in\mathcal{V}:\mathbf{u}\neq\mathbf{v}}\alpha\cdot
2\sup_{h\in\mathcal{H}}\bigg{|}\mathbb{E}_{\mathbf{x}\sim
P_{D_{\mathbf{v}}^{X}}}\big{[}\mathbb{I}\big{(}h(\mathbf{x}=1)\big{)}\big{]}\\\
&\qquad\qquad\qquad-\mathbb{E}_{\mathbf{x}\sim
P_{D_{\mathbf{u}}^{X}}}\big{[}\mathbb{I}\big{(}h(\mathbf{x}=1)\big{)}\big{]}\bigg{|}\\\
&=\frac{1}{N(N+1)}\sum_{\mathbf{v,u}\in\mathcal{V}}d_{\mathcal{H}}(D_{\mathbf{v}},D_{\mathbf{u}}),\end{split}$
(31)

where the inequality follows from the triangluar inequality and jensen’s
inequality.

∎

##  Appendix C Experimental setup

In this Section, we describe the datasets, network architecture and
hyperparameter configuration.

###  C.1 Datasets

We validate the Multi-source Information-regularized Adaptation Networks
(MIAN) with the following benchmark datasets: Digits-Five, Office-31 and
Office-Home. Every experiment is repeated four times and the average accuracy
in target domain is reported.

Digits-Five [37] dataset is a unified dataset including five different digit
datasets: MNIST [24], MNIST-M [11], Synthetic Digits [11], SVHN, and USPS.
Following the standard protocols of unsupervised MDA [54, 37], we used 25000
training images and 9000 test images sampled from a training and a testing
subset for each of MNIST, MNIST-M, SVHN, and Synthetic Digits. For USPS, all
the data is used owing to the small sample size. All the images are bilinearly
interpolated to $32\times 32$.

Office-31 [39] is a popular benchmark dataset including 31 categories of
objects in an office environment. Note that it is a more difficult problem
than Digits-Five, which includes 4652 images in total from the three domains:
Amazon, DSLR, and Webcam. All the images are interpolated to $224\times 224$
using bicubic filters.

Office-Home [51] is a challenging dataset that includes 65 categories of
objects in office and home environments. It includes 15,500 images in total
from the four domains: Artistic images (Art), Clip Art(Clipart), Product
images (Product), and Real-World images (Realworld). All the images are
interpolated to $224\times 224$ using bicubic filters.

###  C.2 Architectures

(a) Existing works

(b) Proposed model

Figure 4: Comparison of existing and proposed MDA models. (a) Existing
multiple-discriminator based methods align each pairwise source and target
domain but may fail due to the disintegration of domain-discriminative
knowledge. It also may suffer from unstable optimization and lack of resource-
efficiency. (b) Our proposed model mitigates suggested problems by unifying
domain discriminators.

Simulation setting For the Digits-Five dataset, we use the same network
architecture and optimizer setting as in [37]. For all the other experiments,
the results are based on ResNet-50, which is pre-trained on ImageNet. The
domain discriminator is implemented as a three-layer neural network. Detailed
architecture is shown in Figure 5.

(a) Encoder, domain discriminator, and classifier used in Digits-Five
experiments

(b) Encoder, domain discriminator, and classifier used in Office-31 and
Office-Home experiments

Figure 5: Network architectures. BN denotes Batch Normalization [21] and SVD
denotes differentiable SVD in PyTorch for MIAN-$\gamma$ (Section E)

We compare our method with the following state-of-the-art domain adaptation
methods: Deep Adaptation Network (DAN, [28]), Joint Adaptation Network (JAN,
[30]), Manifold Embedded Distribution Alignment (MEDA, [52]), Domain
Adversarial Neural Network (DANN, [12]), Domain-Specific Batch Normalization
(DSBN, [5]), Batch Spectral Penalization (BSP, [7]), Adversarial
Discriminative Domain Adaptation (ADDA, [50]), Maximum Classifier Discrepancy
(MCD, [41]), Deep Cocktail Network (DCTN, [54]), and Moment Matching for
Multi-Source Domain Adaptation (M3SDA, [37]).

Table 4: Experimental setup. The batch size for each domain is reported. Dataset | Optimization method | Learning rate | Momentum | Batch size | Iteration
---|---|---|---|---|---
Digits-Five | Adam | $2e^{-4}$ | (0.9, 0.99) | 128 | 50000
Office-31 | mini-batch SGD | $1e^{-3}$ | 0.9 | 16 | 25000
Office-Home | mini-batch SGD | $1e^{-3}$ | 0.9 | 16 | 25000

#### Hyperparameters

Details of the experimental setup are summarized in Table 4. Other state-of-
the-art adaptation models are trained based on the same setup except for these
cases: DCTN show poor performance with the learning rate shown in Table 4 for
both Office-31 and Office-Home datasets. Following the suggestion of the
original authors, $1e^{-5}$ is used as a learning rate with the Adam optimizer
[23]; MCD show poor performance for the Office-Home dataset with the learning
rate shown in Table 4. $1e^{-4}$ is selected as a learning rate. For both the
proposed and other baseline models, the learning rate of the classifier or
domain discriminator trained from the scratch is set to be 10 times of those
of ImageNet-pretrained weights, in Office-31 and Office-Home datasets. More
hyperparameter configurations are summarized in Table 5 (Section E)

##  Appendix D Additional results

Visualization of learned latent representations. We visualized domain-
independent representations extracted by the input layer of the classifier
with t-SNE (Figure 6). Before the adaptation process, the representations from
the target domain were isolated from the representations from each source
domain. However, after adaptation, the representations were well-aligned with
respect to the class of digits, as opposed to the domain.

Hyperparameter sensitivity. We conducted the analysis on hyperparameter
sensitivity with degree of regularization $\beta$. The target domain is set as
Amazon or Art, where the value $\beta_{0}$ changes from $0.1$ to $0.5$. The
accuracy is high when $\beta_{0}$ is approximately between 0.1 and 0.3. We
thus choose $\beta_{0}=0.2$ for Office-31, and $\beta_{0}=0.3$ for Office-
Home.

(a) Before adaptation

(b) After adaptation

Figure 6: t-SNE visualization (a) before and (b) after adaptation.
Representations from target domain (SVHN) are shown in red. Digit class labels
are shown with corresponding numbers. Figure 7: Analysis on hyperparameter
sensitivity.

##  Appendix E Decaying Batch Spectral Penalization

In this Section, we provides details on the Decaying Batch Spectral
Penalization (DBSP) which expands MIAN into MIAN-$\gamma$.

###  E.1 Backgrounds

There is little motivation for models to control the complex mutual dependence
to domains if reducing the entropy of representations is sufficient to
optimize the value of $I(Z;V)=H(Z)-H(Z\mid V)$. If so, such implicit entropy
minimization substantially reduce the upper bound of $I(Z;Y)$, potentially
leading to a increase in optimal joint risk $\lambda^{*}$. In other words, the
decrease in the entropy of representations may occur as the side effect of
$I(Z;V)$ regularization. Such unexpected side effect of information
regularization is highly intertwined with the hidden deterioration of
discriminability through adversarial training [7, 27].

Based on these insights, we employ the SVD-entropy $H_{SVD}(\mathbf{Z})$ [2]
of a representation matrix $\mathbf{Z}$ to assess the richness of the latent
representations during adaptation, since it is difficult to compute $H(Z)$.
Note that while $H_{SVD}(\mathbf{Z})$ is not precisely equivalent to $H(Z)$,
$H_{SVD}(\mathbf{Z})$ can be used as a proxy of the level of disorder of the
given matrix [36]. In future works, it would be interesting to evaluate the
temporal change in entropy with other metrics. We found that
$H_{SVD}(\mathbf{Z})$ indeed decreases significantly during adversarial
adaptation, suggesting that some eigenfeatures (or eigensamples) become
redundant and, thus, the inherent feature-richness diminishes (Figure 8(a)).
To preclude such deterioration, we employ Batch Spectral Penalization (BSP)
[7], which imposes a constraint on the largest singular value to solicit the
contribution of other eigenfeatures. The overall objective function in the
multi-domain setting is defined as:

$\underset{F,C}{min}\ \
{L(F,C)+\beta\hat{I}(Z;V)+\gamma\sum_{i=1}^{N+1}\sum_{j=1}^{k}s_{i,j}^{2}},$
(32)

where $\beta$ and $\gamma$ are Lagrangian multipliers and $s_{i,j}$ is the
$j$th singular value from the $i$th domain. We found that SVD entropy of
representations is severely deteriorated especially in the early stages of
training (Figure 8(a)), suggesting the possibility of over-regularization. The
noisy domain discriminative signals in the initial phase [12] may distort and
simplify the representations. To circumvent the impaired discriminability in
the early stages of the training, the discriminability should be prioritized
first with high $\gamma$ and low $\beta$, followed by a gradual decaying and
annealing in $\gamma$ and $\beta$, respectively, so that a sufficient level of
domain transferability is guaranteed. Based on our temporal analysis, we
introduce the training-dependent scaling of $\beta$ and $\gamma$ by modifying
the progressive training schedule [12]:

$\begin{split}\beta_{p}&=\beta_{0}\cdot 2\big{(}1-\frac{1}{1+exp{(-\sigma\cdot
p)}}\big{)}\\\ \gamma_{p}&=\gamma_{0}\cdot\big{(}\frac{2}{1+exp{(-\sigma\cdot
p)}}-1\big{)},\end{split}$ (33)

where $\beta_{0}$ and $\gamma_{0}$ are initial values, $\sigma$ is a decaying
parameter, and $p$ is the training progress from $0$ to $1$. We refer to this
version of our model as MIAN-$\gamma$. Note that MIAN only includes
annealing-$\beta$, excluding DBSP. For the proposed method, $\beta_{0}$ is
chosen from $\\{0.1,0.2,0.3,0.4,0.5\\}$ for Office-31 and Office-Home dataset,
while $\beta_{0}=1.0$ is fixed in Digits-Five. $\gamma_{0}$ is fixed to
$1e^{-4}$ following [7].

Table 5: Hyper parameters configuration. Annealing-$\beta$ is not adopted in the Digits-Five experiment. Decaying batch spectral penalization is not adopted in the MIAN. Dataset(Model) | $\beta_{0}$ | $\gamma_{0}$ | $\sigma$ | $k$
---|---|---|---|---
Digits-Five (MIAN) | 1.0 | N/A | N/A | N/A
Office-31 (MIAN) | 0.1 | N/A | 10.0 | N/A
Office-31 (MIAN-$\gamma$) | 0.2 | 0.0001 | 10.0 | 1
Office-Home (MIAN) | 0.3 | N/A | 10.0 | N/A
Office-Home (MIAN-$\gamma$) | 0.3 | 0.0001 | 10.0 | 1

###  E.2 Experiments

(a)

(b)

Figure 8: (a): SVD-entropy analysis. (Office-31; Source domain: DSLR) (b):
Comparisons between BSP and DBSP. (Office-31; DSLR $\rightarrow$ Amazon)

SVD-entropy. We evaluated the degree of compromise of SVD-entropy owing to
transfer learning. For this, DSLR was fixed as the source domain, and each
Webcam and Amazon target domain was used to simulate low
(DSLR$\rightarrow$Webcam; DW) and high domain (DSLR$\rightarrow$Amazon; DA)
shift conditions, respectively. SVD-entropy was applied to the representation
matrix extracted from ResNet-50 and MIAN (denoted as Adapt in Figure 8(a))
with constant $\beta=0.1$. For accurate assessment, we avoided using spectral
penalization. As depicted in the Figure 8(a), adversarial adaptation, or
information regularization, significantly decreases the SVD-entropy of both
the source and target domain representations, especially in the early stages
of training, indicating that the representations are simplified in terms of
feature-richness. Moreover, when comparing the Adapt_DA_source and
Adapt_DW_source conditions, we found that SVD-entropy decreases significantly
as the degree of domain shift increases.

We additionally conducted analyses on temporal changes of SVD entropy by
comparing BSP and decaying BSP (Figure 8(b)). SVD entropy gradually decreases
as the degree of compensation decreases in DBSP which leads to improved
transferability and accuracy. Thus DBSP can control the trade-off between the
richness of the feature representations and adversarial adaptation as the
training proceeds.

Ablation study of decaying spectral penalization. We performed an ablation
study to assess the contribution of the decaying spectral penalization and
annealing information regularization to DA performance (Table 6, 7). We found
that the prioritization of feature-richness in early stages (by controlling
$\beta$ and $\gamma$) significantly improves the performance. We also found
that the constant penalization schedule [7] is not reliable and sometimes
impedes transferability in the low domain shift condition (Webcam, DSLR in
Table 6). This implies that the conventional BSP may over-regularize the
transferability when the degree of domain shift and SVD-entropy decline are
relatively small.

Table 6: Ablation study of decaying batch spectral penalization and annealing information regularization (Office-31). For accurate assessment of extent to which performance improvement is caused by each strategies, $\gamma$ is fixed as $0$ in Annealing-$\beta$, and $\beta$ is fixed as $0.1$ in Decaying-$\gamma$. Results from Annealing-$\beta$ and Full version are reported in main paper as MIAN and MIAN-$\gamma$, respectively. Standards | Hyper parameters | Amazon | DSLR | Webcam | Avg
---|---|---|---|---|---
Baseline |  $\beta=0.1$ as a constant | 69.98 | 99.48 | 98.13 | 89.20
| Annealing-$\beta$
---
(MIAN)
$\beta_{0}=0.1,\sigma=10$ | 74.65 | 99.48 | 98.49 | 90.87
Decaying-$\gamma$ |  BSP: $\gamma=1e^{-4}$ as a constant | 74.73 | 98.65 | 96.24 | 89.87
DBSP: $\gamma_{0}=1e^{-4},\sigma=10$ | 75.01 | 99.68 | 98.10 | 90.93
| Full version
---
(MIAN-$\gamma$)
$\beta_{0}=0.1,\gamma_{0}=1e^{-4},\sigma=10$ | 76.17 | 99.22 | 98.39 | 91.26
Table 7: Accuracy ($\%$) on Office-Home dataset. Standards | Art | Clipart | Product | Realworld | Avg
---|---|---|---|---|---
MIAN | 69.39$\pm$0.50 | 63.05$\pm$0.61 | 79.62$\pm$0.16 | 80.44$\pm$0.24 | 73.12
MIAN-$\gamma$ | 69.88$\pm$0.35 | 64.20$\pm$0.68 | 80.87$\pm$0.37 | 81.49$\pm$0.24 | 74.11