# Your Classifier can Secretly Suffice
Multi-Source Domain Adaptation

Naveen Venkat Jogendra Nath Kundu Durgesh Kumar Singh
Ambareesh Revanur R. Venkatesh Babu
Video Analytics Lab, Indian Institute of Science, Bangalore
Corresponding author<EMAIL_ADDRESS>

###### Abstract

Multi-Source Domain Adaptation (MSDA) deals with the transfer of task
knowledge from multiple labeled source domains to an unlabeled target domain,
under a domain-shift. Existing methods aim to minimize this domain-shift using
auxiliary distribution alignment objectives. In this work, we present a
different perspective to MSDA wherein deep models are observed to implicitly
align the domains under label supervision. Thus, we aim to utilize implicit
alignment without additional training objectives to perform adaptation. To
this end, we use pseudo-labeled target samples and enforce a classifier
agreement on the pseudo-labels, a process called Self-supervised Implicit
Alignment (SImpAl). We find that SImpAl readily works even under category-
shift among the source domains. Further, we propose classifier agreement as a
cue to determine the training convergence, resulting in a simple training
algorithm. We provide a thorough evaluation of our approach on five
benchmarks, along with detailed insights into each component of our
approach111Project page: https://sites.google.com/view/simpal.

##  1 Introduction

The task of supervised learning for classification is based on the assumption
that the training data and the testing data are sampled from the same
distributions. Thus, supervised learning methods achieve state-of-the-art
results when evaluated on popular benchmarks such as ImageNet [46]. However,
when such models are deployed in real-world, they yield sub-optimal results
due to the inherent distribution-shift (domain-shift [55]) between the
training data and the real-world environment (_a.k.a_. the target domain).
While it is possible to obtain unlabeled samples from the target domain in
most cases, the huge costs of data annotation prohibit the creation of a
reliable labeled training dataset. To this end, Unsupervised Domain Adaptation
(DA) methods have been proposed that aim to transfer knowledge from a labeled
"source" dataset to an unlabeled "target" dataset under a domain-shift.

A popular strategy in Unsupervised DA is to learn the task-specific knowledge
using supervision from the labeled source dataset, while learning a domain-
invariant latent space where the features across the source and the target
domains align. Such an alignment is enforced using statistical discrepancy
minimization schemes [1, 12, 39, 43, 54] or via an adversarial objective [11,
30, 57, 61, 66], or by employing domain-specific transformations [6, 26, 44].
This alignment minimizes the domain-shift in the latent space, and improves
the target generalization. However, the performance of Single-Source Domain
Adaptation (SSDA) methods is usually determined by the choice of the source
dataset [24].

Figure 1: An illustration of the proposed concept for two-class
($c_{1},c_{2}$) classification, with two labeled source domains
($s_{1},s_{2}$) and one unlabeled target domain ($t$). Best viewed in color.
(a) Architecture. Several works employ a shared feature extractor ($f$), and
source-specific classifier modules. (b) Classical MSDA. Prior works employ
source-specific classifiers that learn distinct (domain-specific) decision
boundaries (denoted as $\text{DB}_{i}$). This results in a large discrepancy
in the classifier predictions (region shaded in yellow). Thus, an auxiliary
feature alignment loss is required for improving the classifier predictions.
(c) Our approach. We enforce classifier agreement on source-domain samples
leading to an implicit alignment of latent features. Further, imposing an
agreement on the pseudo-labeled target samples improves the generalization to
the target domain.

Recently, Multi-Source Domain Adaptation (MSDA) [35, 67] has garnered interest
wherein multiple labeled source domains are used to transfer the task
knowledge to the unlabeled target domain. A common approach [15, 43, 61] is to
learn a shared feature extractor, along with domain-specific classifier
modules (Fig. 1a), which yield an ensemble prediction for the target samples.
However, an additional challenge in MSDA is to tackle the domain-shift and
category-shift [61] between each pair of source-domains (Fig. 1b). To this
end, auxiliary losses are enforced encouraging the model to learn domain
invariant but class-discriminative representations. Ultimately, an appropriate
alignment of all the domains in the latent space [43] improves the
generalization on the target domain (Fig. 1b).

In this work, we approach the MSDA problem from a different perspective. Since
deep models are known to capture rich transferable representations [29, 38,
62], we ask, is an auxiliary feature alignment loss really necessary? The
motivation stems from the observation that deep models exhibit a strong
inductive bias to implicitly align the latent features under supervision. This
is demonstrated in Fig. 2. Following the prior approaches [43, 61], we train
domain-specific classifiers (Fig. 1b) and observe that the domains do not
align in the latent space (Fig. 2a), which calls for an explicit feature
alignment loss. However, when we enforce a classifier agreement on the class
label for each input instance (Fig. 2b), we find that the domains tend to
align, without requiring an explicit alignment loss.

This motivates us to further explore implicit alignment of latent features for
MSDA. We aim to leverage the labeled data from multiple source domains, and
the multi-classifier setup (Fig. 1a) employed in MSDA to perform alignment,
without incorporating auxiliary components such as a domain discriminator [61,
66]. In contrast to learning domain-specific classifier modules, we enforce an
agreement among the classifiers (Fig. 1c) to align the domains in the latent
space.

Since the target domain is unlabeled, we resort to the class labels predicted
by the model being trained (_a.k.a_. pseudo-labels [25]). The adaptation step
encourages the classifiers to agree upon these pseudo-labels which enables
alignment of the target features with the source features that have classifier
agreement owing to label supervision. Accordingly, we name the approach as
Self-supervised Implicit Alignment, abbreviated as SImpAl (pronounced
"simple"). We observe that even under category-shift, implicit alignment can
be leveraged to align the shared categories, without requiring additional
components (_e.g_. fine-grained alignment [5, 22, 42], adversarial
discriminator [61]) or cumbersome training strategies (_e.g_. to handle
arbitrary category-shifts [23, 61, 63]). We also find that classifier
agreement can be leveraged as a cue to determine adaptation convergence.

To summarize, we demonstrate successful MSDA by leveraging implicit alignment
exhibited by deep classifiers, corroborating the potential for designing
simple and effective adaptation algorithms. We conduct extensive evaluation of
our approach over five benchmark datasets, with two popular CNN backbone
models (ResNet-50, ResNet-101 [16]) and derive insights from the empirical
analysis.

Figure 2: t-SNE plot of the features at the pre-classifier space, showing the
feature distribution after learning a fully supervised model. Best viewed in
color. (a) Learning domain-specific classifiers. We train a model (with
ResNet-50 [16] backbone) with full label supervision from all the three
domains on Office-31 [47], while keeping the classifier heads unique to each
domain. Although we find that class discrimination is achieved, each domain
forms separate sub-clusters, and does not align in the latent space. (b)
Enforcing classifier agreement. Instead of learning domain-specific
classifiers, we enforce the classifiers to agree upon the labels for all the
samples. We observe that all the domains tend to align, without enforcing an
explicit alignment objective, even under a domain-shift. We aim to leverage
this inductive bias of deep models, to perform adaptation.

##  2 Related Work

Here, we briefly review the related works and refer the reader to [67] for an
extensive survey.

a) Single-Source Domain Adaptation (SSDA). Motivated by the seminal work by
Ben-David _et al_. [2, 3], a large number of SSDA methods [6, 10, 11, 12, 28,
29, 32] have been proposed, that aim to learn domain-agnostic but class-
discriminative representations. Inspired by the GAN framework [13], a popular
strategy is to employ adversarial learning [18, 20, 51, 52, 56, 57, 58] that
aims to confuse a domain-discriminator, thereby aligning the latent features
of the domains. Saito _et al_. [50] formulate an adversarial objective
employing classifier discrepancy. In contrast, we aim to study a simpler
approach which circumvents the training difficulties encountered in
adversarial learning paradigms. Recently, consistency based regularizers [8,
21, 36, 20] were proposed for domain adaptation. In our work, classifier
agreement can be interpreted as a form of consistency at the output space
which acts both as an implicit regularizer and as a means to perform latent
space alignment for adaptation.

b) Multi-Source Domain Adaptation (MSDA). Several methods [15, 43, 61, 68]
learn domain-specific classifier modules and obtain a weighted ensemble
prediction for the target samples, motivated by the distribution weighted
combining rule [17, 34, 35]. Zhe _et al_. [68] employ an alignment loss
between each source-target pair in domain-specific feature spaces. In
addition, Peng _et al_. [43] align each pair of source domains using kernel
based moment matching and also propose a variant based on adversarial learning
[50]. Xu _et al_. employ multiple domain discriminators to achieve latent
space alignment. In this work, we aim to explore a simple adaptation scheme
that leverages implicit alignment in deep models. As a result, our approach is
applicable even under category-shift among the source domains, while most
prior methods [15, 43, 68] consider only a shared category set.

c) Self-training methods. Pseudo-labeling [25] is a popular semi-supervised
learning approach where "pseudo" class labels are assigned to unlabeled
samples, typically using classifier confidence [7, 49, 61, 69, 70] or nearest
neighbor assignment [22, 40, 48, 65], while the model is retrained using such
samples. Confidence thresholding [27, 49, 61] is commonly applied to minimize
the noise in pseudo-labels. This introduces a sensitive threshold
hyperparameter, requiring labeled target samples or domain expertise for
precise tuning. Works such as Zou _et al_. [69, 70], Li _et al_. [27] and Chen
_et al_. [8] propose various regularizers to improve pseudo-label predictions.
Xu _et al_. [61] incorporate an adversarial alignment loss to mitigate the
performance degradation arising from noisy pseudo-labels. In contrast, we aim
to exploit classifier agreement to perform adaptation and improve the
reliability of pseudo-labels without incorporating additional hyperparameters.

##  3 Self-supervised Implicit Alignment (SImpAl)

Notations. Let $\mathcal{X}$ and $\mathcal{Y}$ denote the input and the output
spaces. We consider $n_{d}$ labeled source domain datasets
$\\{{\mathcal{D}_{s_{i}}}\\}_{i=1}^{n_{d}}$, where
${\mathcal{D}_{s_{i}}}=\\{(\mathbf{x}_{s_{i}}^{c_{j}},y_{s_{i}}^{c_{j}})\in\mathcal{X}\times\mathcal{Y}\\}$
and a single unlabeled target domain dataset
${\mathcal{D}_{t}}=\\{\mathbf{x}_{t}\in\mathcal{X}\\}$. Each source domain has
a label-set $\mathcal{C}_{s_{i}}$, and the target label-set is defined as
$\mathcal{C}=\cup_{i=1}^{n_{d}}\mathcal{C}_{s_{i}}$ with $n_{c}$ classes. We
learn a deep neural network model having a CNN based feature extractor
$f:\mathbb{R}^{224\times 224\times 3}\rightarrow\mathbb{R}^{256}$, and $n_{d}$
classifier modules $h:\mathbb{R}^{256}\rightarrow\mathbb{R}^{n_{d}\times
n_{c}}$. For convenience, we denote the output of the network as a matrix
$\mathbf{M}=h\circ f(\mathbf{x})$, where $\circ$ represents function
composition. $\mathbf{M}$ is obtained by stacking the logits produced by each
classifier (see Fig. 3).

Overview. As is conventional in the MSDA methods [15, 61], the multi-
classifier setup is treated as an ensemble of diverse classifiers, and the
class probabilities are obtained through a convex combination of each
classifier’s prediction. The model is first trained with the categorical
cross-entropy loss imposed on the combined data from all source domains. After
a "warm-start", we introduce pseudo-labeled target samples into the training
process. The adaptation is performed by enforcing the classifiers to agree on
these pseudo-labels. We now describe the approach in detail.

###  3.1 Warm-start with source domains

To adapt the network to the target domain, we use pseudo-labeled target
samples. Thus, we first aim to achieve a reliability in pseudo-labels by
training the model on all source domains. We call this as the warm-start
process, which is performed as follows.

a) Learning with source domains. For each source-domain instance
$\mathbf{x}_{s_{i}}^{c_{j}}$, we obtain the output matrix $\mathbf{M}=h\circ
f(\mathbf{x}_{s_{i}}^{c_{j}})$ (see Fig. 3) and define the class probability
vector $\mathbf{p}$ as a convex combination of the probabilities assigned by
each classifier,

$\mathbf{p}\leavevmode\nobreak\ =\leavevmode\nobreak\
\frac{1}{n_{d}}\sum_{i=1}^{n_{d}}\sigma(\mathbf{M}_{[i\cdot]})$ (1)

where, $\mathbf{M}_{[i\cdot]}$ represents the $i^{\text{th}}$ row vector of
the matrix $\mathbf{M}$ (_i.e_. the logits of the $i^{\text{th}}$ classifier),
and
$\sigma(\mathbf{v})_{[j]}=\exp{(\mathbf{v}_{[j]})}/\sum_{j^{\prime}=1}^{n_{c}}\exp{(\mathbf{v}_{[j^{\prime}]})}$
is the $\operatorname{softmax}$ function. Treating $\mathbf{p}$ as the class
probability vector, we minimize the categorical cross entropy loss ($l_{ce}$)
using the labeled source samples,

$l_{ce}(\mathbf{p},y_{s_{i}}^{c_{j}})\leavevmode\nobreak\ \leavevmode\nobreak\
=\leavevmode\nobreak\ \leavevmode\nobreak\
-\operatorname{log}(\mathbf{p}_{[j]})\leavevmode\nobreak\ \leavevmode\nobreak\
=\leavevmode\nobreak\ \leavevmode\nobreak\
-\operatorname{log}\Big{(}\frac{1}{n_{d}}\sum_{i=1}^{n_{d}}\sigma(\mathbf{M}_{[i\cdot]})_{[j]}\Big{)}\leavevmode\nobreak\
\leavevmode\nobreak\ \leq\leavevmode\nobreak\ \leavevmode\nobreak\
\frac{1}{n_{d}}\sum_{i=1}^{n_{d}}-\operatorname{log}(\sigma(\mathbf{M}_{[i\cdot]})_{[j]})$
(2)

The last term in Eq. 2 represents an upper bound for the categorical cross-
entropy loss of the ensemble, and is obtained by applying the Jensen’s
inequality for convex functions. We consider the formulation in Eq. 2 to drive
the classifiers to agree upon the label $y_{s_{i}}^{c_{j}}$ for
$\mathbf{x}_{s_{i}}^{c_{j}}$. Thus, the training objective is,

$\min_{f,h}\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\mathop{\mathbb{E}}_{\mathcal{D}\in\\{\mathcal{D}_{s_{i^{\prime}}}\\}_{i^{\prime}=1}^{n_{d}}}\leavevmode\nobreak\
\mathop{\mathbb{E}}_{(\mathbf{x}_{s_{i}}^{c_{j}},y_{s_{i}}^{c_{j}})\in\mathcal{D}}\leavevmode\nobreak\
\frac{1}{n_{d}}\sum_{i=1}^{n_{d}}-\operatorname{log}(\sigma(\mathbf{M}_{[i\cdot]})_{[j]})$
(3)

The objective in Eq. 3 is minimized by mini-batch stochastic optimization.
Each mini-batch contains an equal number of samples from each source domain.
In practice, each classifier is given a distinct random initialization, and is
trained with the same set of training samples at each mini-batch. Intuitively,
this process gradually enables a higher degree of similarity among the
classifiers (Fig. 1c) through an agreement in the predicted class labels for
source samples. Note that, both the feature extractor $f$ and the multi-
classifier module $h$ are shared across all source domains. This step provides
a warm-start to introduce pseudo-labeled target samples into the training.

Figure 3: Architecture of the proposed approach. The network contains a common
feature extractor $f$ having a CNN backbone followed by fully-connected
layers. The multi-classifier module $h$ contains $n_{d}$ classifiers.

b) Determining the convergence of warm-start. The next question we address is,
how to tell if a model is trained sufficiently for the target domain?
Intuitively, we would like to train the model until there is a saturation in
the target (pseudo-label) accuracy. However, with unlabeled target samples
measuring the pseudo-label accuracy is out of bounds. Thus, we propose the
classifier agreement as a criterion to determine the convergence. The
classifier agreement for an instance $\mathbf{x}$ is defined as,

$\displaystyle a(\mathbf{x},f,h)={\prod}_{i\neq
i^{\prime}}I(\leavevmode\nobreak\
\operatorname*{arg\,max}_{j\in\mathcal{C}}\mathbf{M}_{[ij]}\leavevmode\nobreak\
=\leavevmode\nobreak\
\operatorname*{arg\,max}_{j\in\mathcal{C}}\mathbf{M}_{[i^{\prime}j]}\leavevmode\nobreak\
)$ (4)

where $\mathbf{M}=h\circ f(\mathbf{x})$, and $I(\cdot)$ is the indicator
function that returns $1$ when the condition is true, else returns $0$.
Intuitively, when each classifier predicts the same class for a given sample
$\mathbf{x}$, we say that the classifiers "agree". Thus, $a(\mathbf{x},f,h)=1$
when classifiers agree, and $a(\mathbf{x},f,h)=0$ otherwise.

As we shall show in Sec. 4.2, the target pseudo-label accuracy is higher
whenever the classifiers agree [37, 64]. Thus, classifier agreement is used to
filter out target samples having a higher degree of noise in pseudo-labels.
Further, we estimate the fraction of target samples for which there is an
agreement in the class predictions among the classifiers. Thus, we define the
target agreement rate as,

$A({\mathcal{D}_{t}},f,h)=\frac{1}{|{\mathcal{D}_{t}}|}\sum_{\mathbf{x}_{t}\in{\mathcal{D}_{t}}}a(\mathbf{x}_{t},f,h)$
(5)

We hypothesize that the performance on target samples attains a saturation
when the agreement rate converges. Thus, we determine the warm-start interval
based on the convergence of $A(\cdot)$.

###  3.2 Introducing target data

After the warm-start, we introduce target samples into the training process.
The pseudo-labels are obtained from the classifier predictions as in Eq. 1,
_i.e_.
$y_{t}^{c_{j}}=\operatorname*{arg\,max}_{j^{\prime}}\frac{1}{n_{d}}\sum_{i=1}^{n_{d}}\sigma(\mathbf{M}_{[ij^{\prime}]})$.

We consider the following strategy for pseudo-labeling. To begin with, we
select only those target samples for which there is a classifier agreement,
since the labels are seen to be more accurate for such samples (verified in
Sec. 4.2). Thus, we obtain a subset
${\mathcal{D}_{t}}^{\prime}=\\{(\mathbf{x}_{t},y_{t}^{c_{j}})\leavevmode\nobreak\
|\leavevmode\nobreak\ \mathbf{x}_{t}\in{\mathcal{D}_{t}},\leavevmode\nobreak\
a(\mathbf{x}_{t},f,h)=1\\}$. Secondly, inspired by curriculum learning [4, 70]
we form an easy-to-hard sampling strategy for ${\mathcal{D}_{t}}^{\prime}$.
For this purpose, we obtain the average classifier margin as a weight for each
target instance,

$w(\mathbf{x}_{t},f,h)\leavevmode\nobreak\ =\leavevmode\nobreak\
\frac{1}{n_{d}}\sum_{i=1}^{n_{d}}(\mathbf{M}_{[ij]}-\mathbf{M}_{[ij^{\prime}]})$
(6)

where $j$ an $j^{\prime}$ correspond to the indices of the highest and the
second highest logit. Intuitively, $w$ measures a form of confidence in
prediction. Target samples that are farther from the decision boundaries
receive a higher $w$ (see Fig. 7c for the geometrical interpretation). We show
in Sec. 4.2 that, in general, samples with a higher $w$ are more likely to
possess correct pseudo-labels. Thus, target samples are sorted based on $w$,
and are fed to the training pipeline in the decreasing order of $w$. Finally,
the pseudo-labels are updated every $n_{e}$ epochs on
${\mathcal{D}_{t}}^{\prime}$. With this strategy, we formalize the training
objective for adaptation using the target samples as,

$\min_{f,h}\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\mathop{\mathbb{E}}_{(\mathbf{x}_{t},y_{t}^{c_{j}})\in{\mathcal{D}_{t}}^{\prime}}\leavevmode\nobreak\
\frac{1}{n_{d}}\sum_{i=1}^{n_{d}}-\operatorname{log}(\sigma(\mathbf{M}_{[i\cdot]})_{[j]})$
(7)

After introducing the target samples from ${\mathcal{D}_{t}}^{\prime}$, we
train on both source and target samples, in alternate mini-batches, _i.e_. we
minimize the objectives in Eq. 3 and Eq. 7 in alternate mini-batches. Finally,
the network is trained until the target agreement rate $A$ shows convergence.
This enables a simple and effective adaptation pipeline using implicit
alignment. The algorithm is given in Algo. 1.

Algorithm 1 SImpAl - Self-supervised Implicit Alignment

1:require: Source datasets $\\{{\mathcal{D}_{s_{i}}}\\}_{i=1}^{n_{d}}$, Target
dataset ${\mathcal{D}_{t}}$, Model $\\{f,h\\}$

2:

3:while $A({\mathcal{D}_{t}},f,h)$ has not converged do $\triangleright$ Warm-
start with source domains

4: Load a mini-batch of samples
$(\mathbf{x}_{s_{i}}^{c_{j}},y_{s_{i}}^{c_{j}})$ from each source dataset
${\mathcal{D}_{s_{i}}}$

5: Update $\\{f,h\\}$ using the objective in Eq. 3

6:

7:Obtain pseudo-labeled target subset
${\mathcal{D}_{t}}^{\prime}=\\{(\mathbf{x}_{t},y_{t}^{c_{j}})\leavevmode\nobreak\
|\leavevmode\nobreak\ \mathbf{x}_{t}\in\mathcal{D}_{t},\leavevmode\nobreak\
a(\mathbf{x}_{t},f,h)=1\\}$

8:Prepare ${\mathcal{D}_{t}}^{\prime}$ by sorting the samples in descending
order of $w(\mathbf{x}_{t},f,h)$ (as in Eq. 6)

9:

10:while $A({\mathcal{D}_{t}},f,h)$ has not converged do $\triangleright$
Introducing target samples

11:

12: Load a mini-batch of samples
$(\mathbf{x}_{s_{i}}^{c_{j}},y_{s_{i}}^{c_{j}})$ from each source dataset
${\mathcal{D}_{s_{i}}}$

13: Update $\\{f,h\\}$ using the objective in Eq. 3

14:

15: Load a mini-batch of samples $(\mathbf{x}_{t},y_{t}^{c_{j}})$ from pseudo-
labeled target subset ${\mathcal{D}_{t}}^{\prime}$

16: Update $\\{f,h\\}$ using the objective in Eq. 7

17:

18: if $n_{e}$ epochs on ${\mathcal{D}_{t}}^{\prime}$ are completed then
$\triangleright$ Periodically update pseudo-labels

19: Perform steps  5-6 to recompute ${\mathcal{D}_{t}}^{\prime}$

##  4 Experiments

We present the results of our approach on five standard benchmark datasets -
Office-Caltech, ImageCLEF, Office-31, Office-Home and the most challenging
large-scale benchmark, DomainNet.

a) Prior Arts. We compare against Deep Domain Confusion (DDC) [58], Deep
Adaptation Network (DAN) [29], Deep CORAL (D-CORAL) [54], Reverse Gradient
(RevGrad) [10], Residual Transfer Network (RTN) [32], Joint Adaptation Network
(JAN) [31], Maximum Classifier Discrepancy (MCD) [50], Manifold Embedded
Distribution Alignment (MEDA) [60], Adversarial Discriminative Domain
Adaptation (ADDA) [57], Deep Cocktail Network (DCTN) [61], Moment Matching
(M3SDA) [43] and Multiple Feature Space Adaptation Network (MFSAN) [68].
Specifically, DDC, RevGrad, ADDA, MCD, DCTN use an adversarial alignment
objective to perform adaptation, RTN learns a residual function to bridge the
distribution discrepancy, and DAN, MFSAN, D-CORAL, JAN, MEDA and M3SDA employ
a kernel based moment matching scheme to align the domains.

b) Evaluation. For ImageCLEF and Office-based datasets, we follow the
evaluation protocol in MFSAN [68], while for DomainNet, we follow the protocol
used in M3SDA [43]. Three types of baselines are considered - 1) Single Best
(SB) refers to the best single-source transfer results for the target domain,
2) Source Combine (SC) refers to the scenario where all sources are combined
into a single source domain to perform SSDA, 3) Multi-Source (MS) refers to
the MSDA methods. We report the multi-run statistics (mean and standard
deviation) obtained over three different runs.

c) Implementation Details. We implement our approach in PyTorch [41]. We use
the Adam [19] optimizer, with learning rate $10^{-5}$ and weight decay
$5\times 10^{-4}$ for stochastic optimization. The losses in Eq. 3 and Eq. 7
are alternatively optimized and the target agreement rate (Eq. 5) is
periodically monitored for convergence. We set $n_{e}=15\leavevmode\nobreak\
\text{epochs}$ as the update rate for the target pseudo-labels (line 12 in
Algo. 1). The total number of training iterations are decided based on the
convergence of the target agreement rate $A$ for each dataset. Following prior
MSDA approaches [68, 43], we use ResNet-50 (SImpAl50) and ResNet-101
(SImpAl101) [16] as the CNN backbone.

Table 1: Results on five standard benchmark datasets. ‘SB’ stands for Single
Best, ‘SC’ stands for Source Combined, and ‘MS’ denotes MSDA methods. The
results for prior baselines are reported from [43] and [68]. See Supplementary
for the full comparison table on DomainNet.

| A. Office-31
---
Method $\rightarrow$D $\rightarrow$W $\rightarrow$A Avg SB Source Only 99.3
96.7 62.5 86.2 DDC 98.2 95.0 67.4 86.9 DAN 99.5 96.8 66.7 87.7 D-CORAL 99.7
98.0 65.3 87.7 RevGrad 99.1 96.9 68.2 88.1 RTN 99.4 96.8 66.2 87.5 SC DAN 99.6
97.8 67.6 88.3 D-CORAL 99.3 98.0 67.1 88.1 RevGrad 99.7 98.1 67.6 88.5 MS DCTN
99.3 98.2 64.2 87.2 MFSAN 99.5 98.5 72.7 90.2 SImpAl50
99.2${}^{\pm\text{0.2}}$ 97.4${}^{\pm\text{0.1}}$ 70.6${}^{\pm\text{0.6}}$
89.0${}^{\pm\text{0.3}}$ SImpAl101 99.4${}^{\pm\text{0.2}}$
97.9${}^{\pm\text{0.2}}$ 71.2${}^{\pm\text{0.4}}$ 89.5${}^{\pm\text{0.3}}$
C. Office-Caltech
Method $\rightarrow$W $\rightarrow$D $\rightarrow$C $\rightarrow$A Avg SC
Source Only 99.0 98.3 87.8 86.1 92.8 DAN 99.3 98.2 89.7 94.8 95.5 MS Source
Only 99.1 98.2 85.4 88.7 92.9 DAN 99.5 99.1 89.2 91.6 94.8 DCTN 99.4 99.0 90.2
92.7 95.3 JAN 99.4 99.4 91.2 91.8 95.5 MEDA 99.3 99.2 91.4 92.9 95.7 MCD 99.5
99.1 91.5 92.1 95.6 M3SDA 99.4 99.2 91.5 94.1 96.1 SImpAl50
99.3${}^{\pm\text{0.1}}$ 99.8${}^{\pm\text{0.1}}$ 92.2${}^{\pm\text{0.1}}$
95.3${}^{\pm\text{0.2}}$ 96.7${}^{\pm\text{0.1}}$ SImpAl101
100${}^{\pm\text{0.0}}$ 100${}^{\pm\text{0.0}}$ 94.6${}^{\pm\text{0.2}}$
95.6${}^{\pm\text{0.3}}$ 97.5${}^{\pm\text{0.1}}$
| B. ImageCLEF
---
Method $\rightarrow$P $\rightarrow$C $\rightarrow$I Avg SB Source Only 74.8
91.5 83.9 83.4 DDC 74.6 91.1 85.7 83.8 DAN 75.0 93.3 86.2 84.8 D-CORAL 76.9
93.6 88.5 86.3 RevGrad 75.0 96.2 87.0 86.1 RTN 75.6 95.3 86.9 85.9 SC DAN 77.6
93.3 92.2 87.7 D-CORAL 77.1 93.6 91.7 87.5 RevGrad 77.9 93.7 91.8 87.8 MS DCTN
75.0 95.7 90.3 87.0 MFSAN 79.1 95.4 93.6 89.4 SImpAl50
77.5${}^{\pm\text{0.3}}$ 93.3${}^{\pm\text{0.3}}$ 91.0${}^{\pm\text{0.4}}$
87.3${}^{\pm\text{0.3}}$ SImpAl101 78.0${}^{\pm\text{0.5}}$
95.2${}^{\pm\text{0.5}}$ 91.7${}^{\pm\text{0.4}}$ 88.3${}^{\pm\text{0.5}}$
D. Office-Home
Method $\rightarrow$Ar $\rightarrow$Cl $\rightarrow$Pr $\rightarrow$Rw Avg SB
Source Only 65.3 49.6 79.7 75.4 67.5 DDC 64.1 50.8 78.2 75.0 67.0 DAN 68.2
56.5 80.3 75.9 70.2 D-CORAL 67.0 53.6 80.3 76.3 69.3 RevGrad 67.9 55.9 80.4
75.8 70.0 SC DAN 68.5 59.4 79.0 82.5 72.4 D-CORAL 68.1 58.6 79.5 82.7 72.2
RevGrad 68.4 59.1 79.5 82.7 72.4 MS MFSAN 72.1 62.0 80.3 81.8 74.1 SImpAl50
70.8${}^{\pm\text{0.2}}$ 56.3${}^{\pm\text{0.2}}$ 80.2${}^{\pm\text{0.3}}$
81.5${}^{\pm\text{0.3}}$ 72.2${}^{\pm\text{0.6}}$ SImpAl101
73.4${}^{\pm\text{0.4}}$ 62.4${}^{\pm\text{0.1}}$ 81.0${}^{\pm\text{0.2}}$
82.7${}^{\pm\text{0.2}}$ 74.8${}^{\pm\text{0.2}}$

E. DomainNet
---
Method $\rightarrow$Clp $\rightarrow$Inf $\rightarrow$Pnt $\rightarrow$Qdr
$\rightarrow$Rel $\rightarrow$Skt Avg MS M3SDA 57.2${}^{\pm\text{0.9}}$
24.2${}^{\pm\text{1.2}}$ 51.6${}^{\pm\text{0.4}}$ 5.2${}^{\pm\text{0.4}}$
61.6${}^{\pm\text{0.9}}$ 49.6${}^{\pm\text{0.5}}$ 41.5${}^{\pm\text{0.7}}$
SImpAl101 66.4${}^{\pm\text{0.8}}$ 26.5${}^{\pm\text{0.5}}$
56.6${}^{\pm\text{0.7}}$ 18.9${}^{\pm\text{0.8}}$ 68.0${}^{\pm\text{0.5}}$
55.5${}^{\pm\text{0.3}}$ 48.6${}^{\pm\text{0.6}}$

###  4.1 Results

We present the results in Table 1. The results for the prior baselines are
reported from [43] and [68]. Due to the limits of space, we present the full
comparison table for DomainNet in the Supplementary.

Office-31 [47] dataset has 4652 images across Amazon (A), DSLR (D) and Webcam
(W) domains having 31 object classes found in an office environment.
ImageCLEF222http://imageclef.org/2014/adaptation. dataset has been created by
selecting 12 shared classes among ImageNet (I) [46], Caltech-256 (C) [14],
Pascal-VOC 2012 (P) [9], with 600 images per domain. Office-Caltech [12]
dataset consists of 2533 images across 10 classes shared between Caltech-256
(C) and the three domains of Office-31 (A, D, W). Office-Home [59] is a more
challenging medium-scale dataset containing about 15588 images in 4 domains:
Art (Ar), Clipart (Cl), Product (Pr) and Real-World (Rw), sharing 65
categories of objects found in the office and home environments. DomainNet
[43] dataset is the largest and the most challenging benchmark, containing 6
diverse domains, with 345 classes, and around 0.6 million images.

###  4.2 Analysis

a) Implicit alignment of features. In Fig. 4a, we plot the t-SNE [33]
embeddings of the features at the pre-classifier space (output of $f$) for
SImpAl. Further, we calculate the Proxy-$\mathcal{A}$ distance [2] defined as
$\operatorname{dist}_{\mathcal{A}}=2(1-2\epsilon)$ where $\epsilon$ is the
generalization error of a domain discriminator. In Fig. 4b, we report the
$\operatorname{dist}_{\mathcal{A}}$ value across each source-target pair for 3
different models - 1) warm-start model, trained on the source domains, 2) the
model after adaptation using SImpAl, 3) an oracle model employing SImpAl,
where the target pseudo-labels are replaced by the ground-truth labels. This
shows that adaptation using SImpAl effectively reduces the distribution-shift
in the latent space. Further, we also demonstrate implicit alignment under
large domain-shifts (such as Quickdraw and Real-world domains on DomainNet),
which enables applications such as cross-domain image retrieval on an
unlabeled target domain. See Suppl. for further analysis on implicit
alignment.

b) Extension to category-shift. To present a more practical scenario for MSDA,
[61] introduced two category-shift settings - overlap and disjoint, where the
source domains contain overlapping label sets (_i.e_.
$\mathcal{C}_{s_{i}}\cap\mathcal{C}_{s_{i^{\prime}}}\neq\phi$, but
$\mathcal{C}_{s_{i}}\cap\mathcal{C}_{s_{i^{\prime}}}\neq\mathcal{C}_{s_{i}}\cup\mathcal{C}_{s_{i^{\prime}}}$)
and disjoint label-sets
($\mathcal{C}_{s_{i}}\cap\mathcal{C}_{s_{i^{\prime}}}=\phi$) respectively. In
such scenarios, it is vital to prevent mis-alignment of different classes
across the source domains to avoid negative transfer [42]. Furthermore, since
prior MSDA approaches learn domain-specific classifiers, they require separate
mechanisms to obtain class probabilities for the domain-specific and the
shared classes separately [61]. However, our approach remains unmodified under
the presence of category-shift; as such, each classifier learns all the target
classes, and the computation of the class probabilities (Eq. 1) remains
unchanged. Fig. 4c shows that category-shift is a challenging scenario where
all methods show performance degradation, however SImpAl is found to exhibit a
relatively lower degradation in the target performance. This is supported by
the observation that even under category-shift, only the shared classes align
as shown in Fig. 5. See Suppl. for further analysis.

Figure 4: (a) t-SNE. We show alignment of domains at the pre-classifier space
($f$-output). (b) Proxy $\mathcal{A}$-distance ($\downarrow$). The values of
$\operatorname{dist}_{\mathcal{A}}$ (Sec. 4.2a) are obtained between each
source-target pair for the corresponding models shown in (a). (c) Performance
drop under category-shift ($\downarrow$). Following [61], we compare SImpAl
against RevGrad [10], DAN [29], DCTN [61] in the Overlap and Disjoint
scenarios on Office-31 (A, D $\rightarrow$ W).  Figure 5: Pre-classifier space
t-SNE embeddings for category-shift scenarios on ImageCLEF (I, P $\rightarrow$
C). Two plots are shown: features of the source domains only, and, features of
all the domains. Best viewed in color. (a) Overlap. Here, 4 classes are shared
between the sources ($|\mathcal{C}_{s_{1}}\cap\mathcal{C}_{s_{2}}|=4$).
Observe that the shared classes align, while the domain-specific classes are
clustered separately. (b) Disjoint. In the absence of shared classes
($\mathcal{C}_{s_{1}}\cap\mathcal{C}_{s_{2}}=\phi$), class-wise alignment is
not observed among the sources, which is essential to avoid negative-transfer.
Note, in both cases, the target clusters align with the respective source
clusters (since $\mathcal{C}=\mathcal{C}_{s_{1}}\cup\mathcal{C}_{s_{2}}$).
This demonstrates implicit alignment under category-shift. See Suppl. for
wider trends.

c) Target Agreement Rate. Fig. 6a shows the trend in the target agreement rate
($A(\mathcal{D}_{t},f,h)$) and target performance as training proceeds. We
make two observations. Firstly, we find that $A$ increases during training,
indicating that the target samples migrate into the classifier agreement
region in the latent space ($\\{f(\mathbf{x}_{t})\leavevmode\nobreak\
|\leavevmode\nobreak\ a(\mathbf{x}_{t},f,h)=1\\}$). This migration is
necessary for a successful adaptation since the source domains inherently fall
in the classifier agreement region (due to the nature of the source training
for warm-start). Secondly, a correspondence between the convergence of the
target agreement rate and the target accuracy is seen, which validates our
hypothesis that $A$ can be used as a cue to determine the training
convergence. This result is of interest in Unsupervised Domain Adaptation
methods where the requirement of target labels has been the de-facto for model
selection.

d) Do the classifiers agree on correct pseudo-labels? We also calculate the
classifier agreement (and disagreement) for target samples that are pseudo-
labeled correctly. Notably, Fig. 7a demonstrates that the classifiers tend to
agree on an increasing number of target samples with correct pseudo-label
predictions. This motivates the periodic update of
${\mathcal{D}_{t}}^{\prime}$ (Lines  12-13 in Algo.  1), which captures an
increasing number of target samples with correct pseudo-labels, as the
adaptation proceeds.

e) How accurate are target pseudo-labels? As described in Sec. 3.2, we use
classifier agreement to select target samples (${\mathcal{D}_{t}}^{\prime}$)
with a higher pseudo-label accuracy. In Fig. 6b, we plot the accuracy of
pseudo-labels separately for target samples having classifier agreement
(_i.e_. $a(\mathbf{x}_{t},f,h)=1$) and disagreement (_i.e_.
$a(\mathbf{x}_{t},f,h)=0$). Clearly, pseudo-labels are more accurate (more
reliable) when the classifiers agree. Further, the accuracy on the target
samples with agreement, ${\mathcal{D}_{t}}^{\prime}$, is higher than the
accuracy on all target samples, $\mathcal{D}_{t}$ (orange curve in Fig. 6b).
Thus, the use of ${\mathcal{D}_{t}}^{\prime}$ with a higher accuracy in
pseudo-labels plays a key role in gradually improving the target performance.

Figure 6: (a) Target agreement rate $A(\mathcal{D}_{t},f,h)$. The target
agreement rate (bottom) can be used as a cue to determine training convergence
(top). (b) Pseudo-label accuracy. We observe a higher pseudo-label accuracy
for target samples with $a(\mathbf{x}_{t},f,h)=1$ (top, blue curve), as
compared to those with $a(\mathbf{x},f,h)=0$ (bottom).  Figure 7: (a)
Migration of target samples with correct pseudo-labels. During training, we
find that the fraction of target samples with correct pseudo-label predictions
increases in the agreement region. (b) Curriculum using
$w(\mathbf{x}_{t},f,h)$. When sorted in descending order based on $w$, the
target samples exhibit an easy-to-hard curriculum. (c) Geometrical
interpretation of $w$. The figure shows the significance of $w$ in a two-
source two-class scenario (as in Fig. 1). Intuitively, $w$ is a measure of
classifier confidence averaged over each classifier. Target samples that are
further into the classifier agreement region exhibit a higher value of $w$.

f) Using curriculum for target samples. We form a curriculum for the target
samples using the average classifier margin $w(\mathbf{x}_{t},f,h)$ as a
weight. Fig. 7c shows the geometrical interpretation of $w$, that measures how
far into the agreement region a target sample falls. Thus, $w$ can be seen as
a measure of the confidence in the prediction. As studied by prior methods
[15, 45, 53], high confidence predictions are often correct. We show this in
Fig. 7b where we plot the precision of target pseudo-labels at various
confidence percentiles (in descending order of $w$). The accuracy shows a
decreasing trend with $w$, validating our hypothesis that $w$ yields an easy-
to-hard curriculum. Although our framework supports confidence thresholding to
further minimize the pseudo-label noise, we do not employ thresholds for the
main results (Table 1) as it introduces sensitive hyperparameters. See
Supplementary for an empirical analysis with confidence thresholding.

##  5 Conclusion

In this paper, we demonstrated Self-supervised Implicit Alignment (SImpAl),
that serves as a simple method to perform Multi-Source Domain Adaptation
(MSDA). We observed that deep models exhibit the potential to implicitly align
features under label supervision, even in the presence of domain-shift. We
demonstrated the use of classifier agreement in SImpAl - to obtain pseudo-
labeled target samples, to perform latent space alignment and to determine the
training convergence. Extensive empirical analysis demonstrates the efficacy
of SImpAl for MSDA.

Our work can facilitate the study of simple and effective algorithms for
unsupervised domain adaptation. The insights obtained from our study can be
used to explain the efficacy of a number of related self-supervised
approaches. A potential direction of research is to develop efficient
adaptation algorithms that are devoid of sensitive hyperparameters. Exploring
SImpAl for scenarios such as Universal Domain Adaptation [63] would also be of
future interest.

## Broader Impact

This work presents a simple and effective solution for Multi-Source Domain
Adaptation, that has a two-fold positive impact. First, the method is aimed at
improving the performance of prediction models by mitigating the bias caused
by domain-shift between the training dataset and the test data encountered
when deployed in a real-world environment. This is of growing interest in the
machine learning community. Secondly, the insights presented in this work
facilitate the study of efficient methods to perform domain adaptation,
motivating the innovation of, for instance, energy-efficient methods to
generalize deep models. While the method shows promising results under domain-
shift, one should be cautious of the use of the pseudo-labeling procedure in
the presence of adversarial samples, where the pseudo-labels may be less
reliable and may result in performance degradation.

## Acknowledgments and Disclosure of Funding

This work was supported by a project grant from MeitY (No.4(16)/2019-ITEA),
Govt. of India and a WIRIN project. We would also like to thank the anonymous
reviewers for their valuable suggestions.

See pages 1-1 of MSDA_suppl_compressed.pdf See pages 2-2 of
MSDA_suppl_compressed.pdf See pages 3-3 of MSDA_suppl_compressed.pdf See pages
4-4 of MSDA_suppl_compressed.pdf See pages 5-5 of MSDA_suppl_compressed.pdf
See pages 6-6 of MSDA_suppl_compressed.pdf See pages 7-7 of
MSDA_suppl_compressed.pdf See pages 8-8 of MSDA_suppl_compressed.pdf

## References

  * [1] Mahsa Baktashmotlagh, Mehrtash Harandi and Mathieu Salzmann “Distribution-Matching Embedding for Visual Domain Adaptation” In _JMLR_ 17.108, 2016, pp. 1–30
  * [2] Shai Ben-David et al. “A theory of learning from different domains” In _Machine learning_ 79.1-2 Springer, 2010, pp. 151–175
  * [3] Shai Ben-David, John Blitzer, Koby Crammer and Fernando Pereira “Analysis of representations for domain adaptation” In _NeurIPS_ , 2007
  * [4] Yoshua Bengio, Jérôme Louradour, Ronan Collobert and Jason Weston “Curriculum learning” In _ICML_ , 2009
  * [5] Zhangjie Cao, Mingsheng Long, Jianmin Wang and Michael I Jordan “Partial transfer learning with selective adversarial networks” In _CVPR_ , 2018
  * [6] Woong-Gi Chang et al. “Domain-Specific Batch Normalization for Unsupervised Domain Adaptation” In _CVPR_ , 2019
  * [7] Minmin Chen, Kilian Q Weinberger and John Blitzer “Co-training for domain adaptation” In _NeurIPS_ , 2011
  * [8] Yun-Chun Chen, Yen-Yu Lin, Ming-Hsuan Yang and Jia-Bin Huang “CrDoCo: Pixel-level domain transfer with cross-domain consistency” In _CVPR_ , 2019
  * [9] M. Everingham et al. “The Pascal Visual Object Classes (VOC) Challenge” In _IJCV_ 88.2, 2010, pp. 303–338
  * [10] Yaroslav Ganin and Victor Lempitsky “Unsupervised Domain Adaptation by Backpropagation” In _ICML_ , 2015
  * [11] Yaroslav Ganin et al. “Domain-adversarial training of neural networks” In _JMLR_ 17.1 JMLR. org, 2016, pp. 2096–2030
  * [12] Boqing Gong, Yuan Shi, Fei Sha and Kristen Grauman “Geodesic flow kernel for unsupervised domain adaptation” In _CVPR_ , 2012
  * [13] Ian Goodfellow et al. “Generative adversarial nets” In _NeurIPS_ , 2014
  * [14] Gregory Griffin, Alex Holub and Pietro Perona “Caltech-256 object category dataset” In _Tech. Rep. California Institute of Technology_ , 2007 URL: https://authors.library.caltech.edu/7694/
  * [15] Jiang Guo, Darsh J Shah and Regina Barzilay “Multi-Source Domain Adaptation with Mixture of Experts” In _EMNLP_ , 2018
  * [16] Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun “Deep residual learning for image recognition” In _CVPR_ , 2016
  * [17] Judy Hoffman, Mehryar Mohri and Ningshan Zhang “Algorithms and theory for multiple-source adaptation” In _NeurIPS_ , 2018
  * [18] Judy Hoffman et al. “Cycada: Cycle-consistent adversarial domain adaptation” In _ICML_ , 2018
  * [19] Diederik P Kingma and Jimmy Ba “Adam: A method for stochastic optimization” In _arXiv preprint arXiv:1412.6980_ , 2014
  * [20] Jogendra Nath Kundu, Phani Krishna Uppala, Anuj Pahuja and R. Venkatesh Babu “AdaDepth: Unsupervised Content Congruent Adaptation for Depth Estimation” In _CVPR_ , 2018
  * [21] Jogendra Nath Kundu, Nishank Lakkakula and R Venkatesh Babu “UM-Adapt: Unsupervised Multi-Task Adaptation Using Adversarial Cross-Task Distillation” In _ICCV_ , 2019
  * [22] Jogendra Nath Kundu et al. “Class-Incremental Domain Adaptation” In _ECCV_ , 2020
  * [23] Jogendra Nath Kundu, Naveen Venkat, Rahul M V and R. Babu “Universal Source-Free Domain Adaptation” In _CVPR_ , 2020
  * [24] Jogendra Nath Kundu et al. “Towards Inheritable Models for Open-Set Domain Adaptation” In _CVPR_ , 2020
  * [25] Dong-Hyun Lee “Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks” In _ICML-W_ , 2013
  * [26] Yanghao Li et al. “Adaptive batch normalization for practical domain adaptation” In _Pattern Recognition_ 80 Elsevier, 2018, pp. 109–117
  * [27] Yunsheng Li, Lu Yuan and Nuno Vasconcelos “Bidirectional learning for domain adaptation of semantic segmentation” In _CVPR_ , 2019
  * [28] Hong Liu et al. “Separate to Adapt: Open Set Domain Adaptation via Progressive Separation” In _CVPR_ , 2019
  * [29] Mingsheng Long, Yue Cao, Jianmin Wang and Michael Jordan “Learning Transferable Features with Deep Adaptation Networks” In _ICML_ , 2015
  * [30] Mingsheng Long, Zhangjie Cao, Jianmin Wang and Michael I Jordan “Conditional adversarial domain adaptation” In _NeurIPS_ , 2018
  * [31] Mingsheng Long, Han Zhu, Jianmin Wang and Michael I Jordan “Deep transfer learning with joint adaptation networks” In _ICML_ , 2017
  * [32] Mingsheng Long, Han Zhu, Jianmin Wang and Michael I Jordan “Unsupervised domain adaptation with residual transfer networks” In _NeurIPS_ , 2016, pp. 136–144
  * [33] Laurens van der Maaten and Geoffrey Hinton “Visualizing data using t-SNE” In _JMLR_ 9.Nov, 2008, pp. 2579–2605
  * [34] Yishay Mansour, Mehryar Mohri and Afshin Rostamizadeh “Domain adaptation with multiple sources” In _NeurIPS_ , 2009
  * [35] Yishay Mansour, Mehryar Mohri and Afshin Rostamizadeh “Multiple source adaptation and the Rényi divergence” In _UAI_ , 2012
  * [36] Zak Murez et al. “Image to image translation for domain adaptation” In _CVPR_ , 2018, pp. 4500–4509
  * [37] Duc Tam Nguyen et al. “Self: Learning to filter noisy labels with self-ensembling” In _ICLR_ , 2020
  * [38] Maxime Oquab, Leon Bottou, Ivan Laptev and Josef Sivic “Learning and transferring mid-level image representations using convolutional neural networks” In _CVPR_ , 2014
  * [39] Sinno Jialin Pan, Ivor W Tsang, James T Kwok and Qiang Yang “Domain adaptation via transfer component analysis” In _IEEE Trans. Neural Netw._ 22.2 IEEE, 2010, pp. 199–210
  * [40] Yingwei Pan et al. “Transferrable prototypical networks for unsupervised domain adaptation” In _CVPR_ , 2019
  * [41] Adam Paszke et al. “PyTorch: An Imperative Style, High-Performance Deep Learning Library” In _NeurIPS_ , 2019
  * [42] Zhongyi Pei, Zhangjie Cao, Mingsheng Long and Jianmin Wang “Multi-adversarial domain adaptation” In _AAAI_ , 2018
  * [43] Xingchao Peng et al. “Moment matching for multi-source domain adaptation” In _ICCV_ , 2019
  * [44] Subhankar Roy et al. “Unsupervised domain adaptation using feature-whitening and consensus loss” In _CVPR_ , 2019
  * [45] Sebastian Ruder, Parsa Ghaffari and John G Breslin “Knowledge adaptation: Teaching to adapt” In _arXiv:1702.02052_ , 2017
  * [46] Olga Russakovsky et al. “Imagenet large scale visual recognition challenge” In _IJCV_ 115.3 Springer, 2015, pp. 211–252
  * [47] Kate Saenko, Brian Kulis, Mario Fritz and Trevor Darrell “Adapting visual category models to new domains” In _ECCV_ , 2010
  * [48] Kuniaki Saito, Donghyun Kim, Stan Sclaroff and Kate Saenko “Universal domain adaptation through self supervision” In _arXiv preprint arXiv:2002.07953_ , 2020
  * [49] Kuniaki Saito, Yoshitaka Ushiku and Tatsuya Harada “Asymmetric Tri-training for Unsupervised Domain Adaptation” In _ICML_ , 2017, pp. 2988–2997
  * [50] Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku and Tatsuya Harada “Maximum classifier discrepancy for unsupervised domain adaptation” In _CVPR_ , 2018
  * [51] Kuniaki Saito, Shohei Yamamoto, Yoshitaka Ushiku and Tatsuya Harada “Open set domain adaptation by backpropagation” In _ECCV_ , 2018
  * [52] Swami Sankaranarayanan, Yogesh Balaji, Carlos D Castillo and Rama Chellappa “Generate to adapt: Aligning domains using generative adversarial networks” In _CVPR_ , 2018
  * [53] Kihyuk Sohn et al. “Fixmatch: Simplifying semi-supervised learning with consistency and confidence” In _arxiv preprint arXiv:2001.07685_ , 2020
  * [54] Baochen Sun and Kate Saenko “Deep coral: Correlation alignment for deep domain adaptation” In _ECCV_ , 2016
  * [55] Antonio Torralba and Alexei A Efros “Unbiased look at dataset bias” In _CVPR_ , 2011
  * [56] Eric Tzeng, Judy Hoffman, Trevor Darrell and Kate Saenko “Simultaneous deep transfer across domains and tasks” In _ICCV_ , 2015, pp. 4068–4076
  * [57] Eric Tzeng, Judy Hoffman, Kate Saenko and Trevor Darrell “Adversarial discriminative domain adaptation” In _CVPR_ , 2017, pp. 7167–7176
  * [58] Eric Tzeng et al. “Deep domain confusion: Maximizing for domain invariance” In _arxiv preprint arXiv:1412.3474_ , 2014
  * [59] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty and Sethuraman Panchanathan “Deep hashing network for unsupervised domain adaptation” In _CVPR_ , 2017
  * [60] Jindong Wang et al. “Visual domain adaptation with manifold embedded distribution alignment” In _ACMMM_ , 2018
  * [61] Ruijia Xu et al. “Deep cocktail network: Multi-source unsupervised domain adaptation with category shift” In _CVPR_ , 2018
  * [62] Jason Yosinski, Jeff Clune, Yoshua Bengio and Hod Lipson “How transferable are features in deep neural networks?” In _NeurIPS_ , 2014
  * [63] Kaichao You et al. “Universal Domain Adaptation” In _CVPR_ , 2019
  * [64] Xingrui Yu et al. “How does Disagreement Help Generalization against Label Corruption?” In _ICML_ , 2019
  * [65] Qiming Zhang, Jing Zhang, Wei Liu and Dacheng Tao “Category anchor-guided unsupervised domain adaptation for semantic segmentation” In _NeurIPS_ , 2019
  * [66] Han Zhao et al. “Adversarial multiple source domain adaptation” In _NeurIPS_ , 2018
  * [67] Sicheng Zhao, Bo Li, Pengfei Xu and Kurt Keutzer “Multi-source Domain Adaptation in the Deep Learning Era: A Systematic Survey” In _arXiv preprint arXiv:2002.12169_ , 2020
  * [68] Yongchun Zhu, Fuzhen Zhuang and Deqing Wang “Aligning domain-specific distribution and classifier for cross-domain classification from multiple sources” In _AAAI_ , 2019
  * [69] Yang Zou et al. “Confidence regularized self-training” In _ICCV_ , 2019
  * [70] Yang Zou, Zhiding Yu, BVK Vijaya Kumar and Jinsong Wang “Unsupervised domain adaptation for semantic segmentation via class-balanced self-training” In _ECCV_ , 2018


# Reverse resonant nonspecular reflection controlled by intensity of light

Sergiy Prosvirnin1,2, Vyacheslav Khardikov1,2, Vladimir Yachin1, Vadym
Plakhtii2, and Nataliia Sydorchuk1 1 Institute of Radio Astronomy, Kharkiv
61002, Ukraine  2 Karazin Kharkiv National University, Kharkiv 61022, Ukraine
<EMAIL_ADDRESS>
###### Abstract

Nonlinear optical features of a plane structure consisted of silicon disks
placed periodically on a silver substrate have been studied in the Littrow
reflection scenario. The structure manifests a bistable resonant reflective
ability. Values of both specular reflectance and reverse one can be tuned by
varying an intensity of incident light.

##  1 Introduction

A flat metallic mirror is a simplest optical device being used since antiquity
[1]. A remarkable feature of a mirror is exhibited by extremely useful
capability of changing sharply the direction of light rays and returning light
back at normal incidence over a wide spectral bandwidth. Today photonics
requires mirrors with specific reflection selectivity both in electromagnetic
field intensity and phase depending on a light wavelength. Spatial variation
of the field phase along the reflection surface is specially adapted in order
to manipulate, redirect and concentrate light intensity.

Metasurfaces open prospects to create specific reflectors with extraordinary
properties [2]. The wavelength-selective properties of mirrors are necessary
to design precise sensors [3, 4, 5], in particular, biological material
sensors [6]. Another aspect of the application of resonant mirrors involves
placement of a certain quantum system which can have different energy states
in an intense field near the reflecting surface. In this case, the challenge
is to obtain the maximum radiation intensity or extra absorption. In the light
of these problems, along with classical metasurfaces, resonant arrays radiated
only one or a few diffracted orders besides main one look attractive for
practical applications due to their promising features [7, 8].

Modern optics technologies give the opportunity to produce resonant non-
specular and selectively reflecting metasurfaces. A reflecting metasurface is
usually a planar double periodic subwavelength patterned metal-dielectric or
all-dielectric layer placed on a metal substrate [9]. The thickness of a
metasurface is usually very small in comparison with an electromagnetic
wavelength in free space. However, the periodic structured surface provides
conditions for the excitation of different types of resonances. A spectacular
metasurface response manifests in resonance reflection and absorption [10], in
electromagnetic radiation enhancement by using a laser medium [11], in exotic
electromagnetic field boundary values, which may be the same as on a surface
of an artificial magnetic wall [12], and in the confinement of the intensive
electromagnetic field inside the structure [13].

Among numbers of important electromagnetic properties, the ability of flat
periodic structures to reflect an incident wave in a preferable non-specular
direction is extremely attractive for applications in photonics [9, 2]. The
first researches of a subject have long history and initially were related to
single-periodic reflective gratings for suppressing unwanted reflection of
microwave radiation [14, 15, 16]. In laser technology of optical electronics,
inclined echelette grating mirrors were used to reflect light in the direction
strictly opposite to the direction of arrival of the incident wave. The
corresponding scenario of diffraction by a structure periodic in one direction
is known as the Littrow scheme [17, 18] or an autocollimation regime. Another
practically important diffraction scenario is when the obliquely incident wave
is reflected in the direction normal to the metasurface. In particular, it is
of interest for the design of quasi-optical power pulse compression devices
[19].

As it is known, in a particular diffraction scenario, the propagation
directions of diffraction orders depend only on a geometric shape and sizes of
a unit cell of the periodic metasurface in comparison with the wavelength and
on the direction of the incident electromagnetic wave. On the other hand,
power distribution between diffraction orders is determined only by resonance
properties of meta-atoms that make up the periodic structure and their mutual
coupling. Therefore, a change in the resonance properties of the meta-atoms
opens up the possibility of switching the structure reflectivity between
states of specular and non-specular reflection.

For changing the properties of meta-atoms, one of opportunities is to use
materials in which polarizability depends on intensity of the applied
electromagnetic field [20]. In this case, switching between the states of
specular and non-specular reflection of light is achieved by changing the
intensity of the incident wave. In particular, the optical Kerr effect is
defined as an intensity dependent refractive index $n=n_{0}+n_{2}I$ where $I$
is the intensity of the optical field, $n_{0}$ is the linear refractive index
in the low light intensity regime and $n_{2}$ is the nonlinear refractive
index.

The optical Kerr effect is a third-order effect, which can manifest optical
bistability of the structure response. This phenomenon can be applied to
design devices as a toggle switch. Third order nonlinear optical properties is
inherent to many different media (both centrosymmetric and
noncentrosymmetric), in particular silicon that is most using medium of
photonics [21].

In this paper, for the first time, we claim a theoretical demonstration of the
controllable features of the uncommon resonant regime of the reverse
reflection realised with the patterned silicon-on-metal structure. The regime
is associated with excitation of one additional diffraction order besides the
main partial wave. Ratio of the non-specular to specular reflectance values
can be controlled by a variation of the incident wave intensity. To provide
the non-specular reflection for any incident wave direction we have proposed
an array of silicon disks with the excitation of the Mie-resonance that has a
specific symmetry of the field distribution [22] used in antennas for mobil
communication.

In our research we have assumed electromagnetic fields of combination
frequencies negligibly small and considered the field as monochromatic. A
basis for such approach is the use of resonant elements on the metasurface.
The research was fulfilled with a finite element method and full-wave
numerical simulation.

##  2 Problem statement

Let us study reflection of the plane electromagnetic wave

${\bf{E}}^{i}={\bf e}^{i}E\exp(-i{\bf k}^{i}{\bf r})$ (1)

by a doubly-periodic reflecting structure placed in free space parallel to the
$xy$-plane (see Fig. 1). In expression (1), the vector ${\bf e}^{i}$ is an
unit polarization vector, $E$ is a field strength of the incident wave and

${\bf k}^{i}={\bf e}_{x}k\sin\theta_{i}\cos\varphi_{i}+{\bf
e}_{y}k\sin\theta_{i}\sin\varphi_{i}-{\bf e}_{z}k\cos\theta_{i}$

is a wave vector, where $\varphi_{i}$ and $\theta_{i}$ are the azimuth and
polar angles of the incidence direction respectively,
$k=\omega/c=2\pi/\lambda$, vectors ${\bf e}_{x}$, ${\bf e}_{y}$ and ${\bf
e}_{z}$ are unit basis vectors along the axes $x$, $y$ and $z$.

Figure 1: Schematic of the doubly-periodic planar reflective structure under
study. The incident ray (red arrow) and also specular and reversely reflected
ones (blue and green arrows respectively) are placed in the plane of incidence
which is shown by a dashed parallelogram.

In the region $z>0$, the field is a superposition of the incident wave field
and fields of partial diffracted waves propagating away from the structure and
along its surface:

${\bf{E}}={\bf{E}}^{i}+\sum_{m,n=-\infty}^{\infty}{\bf{a}}_{mn}\exp(-i{\bf{k}}_{mn}{\bf
r}),\qquad z>0,$ (2)

where ${\bf{a}}_{mn}$ and ${\bf{k}}_{mn}={\bf{g}}_{mn}+{\bf e}_{z}\gamma_{mn}$
are vector amplitudes and wave vectors of partial waves of the reflected field
respectively,

${\bf g}_{mn}={\bf e}_{x}(k_{x}^{i}+\frac{2\pi m}{d})+{\bf
e}_{y}(k_{y}^{i}+\frac{2\pi n}{d})$ (3)

is projection of the wave vector of Floquet harmonic on the $xOy$ plane,

$\gamma_{mn}=\sqrt{k^{2}-g_{mn}^{2}},\quad\mbox{Re}\gamma_{mn}\geq
0,\quad\mbox{Im}\gamma_{mn}\leq 0$

is the component of the wave vector along the $Oz$-axis.

Each spatial harmonic having nonzero real component $\gamma_{mn}$ of the wave
vector corresponds to one of the plane waves, which transport energy from the
array plane to the free space. The spatial harmonic with subscripts $m=0$ and
$n=0$ is a plane wave propagated along a direction of specular reflection of
the incident wave. Thus, propagation directions of spatial harmonics or
diffraction orders of the reflected field are defined by pitches of the array,
a wavelength and a direction of the incident wave. However, an intensity
distribution of spatial harmonic spectrum depends on the scattering properties
of the structure meta-atoms.

For the sake of simplicity and shortness we have further considered the case
$\varphi_{i}=0$ The incident wave is supposed to have TE-polarization i.e.
${\bf e}^{i}={\bf e}_{y}$.

A normalized frequency defined as $\kappa=d/\lambda$ is convenient for using.
An equation $\gamma_{mn}(\kappa)=0$ is the equation on cutoff frequencies
$\kappa_{mn}$ for the diffraction orders. If a normalized frequency is lower
then $\kappa_{-10}=(1+\sin\theta_{i})^{-1}$ the only specular ray propagates
away from the metasurface. A cutoff frequency of the diffraction order with
$m=-2$ and $n=0$ is $\kappa_{-20}=2(1+\sin\theta_{i})^{-1}$. In the frequency
band $(\kappa_{-10},\kappa_{-20})$ a far-zone reflected field consists of two
rays which are (0,0) and (-1,0) diffraction orders.

At the normalized frequency $\kappa=(2\sin\theta_{i})^{-1}$, the diffraction
order (-1,0) propagates strictly opposite to the incident wave propagation
direction as can be easily seen [18].

When choosing resonant elements for the metasurface, our preference has been
made for the silicon disks. Silicon was proposed as it is a most used
dielectric material of photonics. The resonant features of a single dielectric
disk located in free space and on a dielectric substrate are well studied
[23]. The plane arrays of disks were also studied in details in the case when
their unit cell is smaller than a wavelength [13]. Next, the attractive
opportunities of using the electromagnetic features of dielectric disks placed
on a metal substrate as an antenna elements of the mobile communication
devices were discussed in [24].

##  3 Results of simulation and analysis

The silicon disks of the array are assumed to be placed periodically on a
silver plane thick substrate (see Fig. 1). The array pitch $d$ is chosen equal
to 750 nm. The disk radius is $a=130$ nm. The disk thickness is $h=130$ nm.
The linear and nonlinear refractive indexes of silicon are approximated by
data presented in [25, 26, 27]. At frequencies close to the frequency of
reverse reflection, they are $n_{0}=3.48$ and $n_{2}=10^{-18}$ m2/W
respectively. The substrate thickness is assumed so large that the light field
does not penetrate via the substrate. The refractive index of silver are
referenced from [28]. The angle of incidence is assumed $\theta_{i}=35$
degrees.

Such choice of the structure parameters gives us an opportunity to research a
resonant diffraction efficiency of the specular reflected wave and (-1,0)
diffraction order propagating opposite to the incident wave.

In linear regime the designed reflect-array manifests the strong resonant
response at the frequency close to the frequency of auto-collimation
diffraction scenario by excitation of intensive (-1,0) diffraction order. In
the Fig. 2, a frequency dependance of reflectance that is a measure of how
much optical power is reflected into the designated direction compared to the
power incident onto the reflective item has been presented.

The designed structure manifests sharp resonant excitation of
(-1,0)-diffraction order due to its adjustment to Mi-resonance of the silicon
disk elements. The resonant field type is close to TE 01$\delta$ resonance of
the dielectric cylinder of a finite length in free space.

A value of resonant transformation of the specular ray intensity to intensity
of the reverse reflection ray depends on dissipative properties of the
reflect-array. In the base of a non-dissipative theoretical model of the
reflecting structure, we have shown a complete transformation of the incident
wave intensity into the intensity of the reverse wave (see [29]). A power
dissipated by the array is defined by absorption value $A=(1-R_{total})$ (see
Fig. 2).

Figure 2: Reflectances versus the normalized frequency in linear regime. The
approximate normalized frequency value of 0.872 corresponding to the auto-
collimation diffraction scenario has been marked by a dashed vertical line.

Simulation results for the frequency dependences of the array reflectance are
presented in figures 3 \- 4 versus the power of the incident wave per unit
cell of the array. We have observed a bistable reflection properties of the
array if the power exceeds some value. Our estimation of the power threshold
of the reflection bistability is approximately 0.7 W.

Under bistable conditions the value of reflectance at each power level depends
on whether the bistable state is reached by transition from low or from high
frequencies. With an increase in the intensity of the incident wave, the
frequency zone of bistability expands and shifts to lower frequencies.

Frequency dependences of reflectance are the hystereses loops. In the
bistability frequency band, the value of reflectance depends on the kind of
way to reach bistable state.

Figure 3: Specular reflectance R00 versus the normalized frequency and the
incident wave power per unit cell of the reflect-array. Figure 4: Reverse
reflectance R-10 versus the normalized frequency and the incident wave power
per unit cell of the reflect-array.

##  4 Conclusion

In summary, we have shown by a full wave simulation that a structure of
identical silicon disks placed periodically on a silver substrate can be used
as an effectively excited and adjusted resonant nonlinear dielectric-metal
hybrid reflect-array in the Littrow non-specular diffraction scenario. It is
found that the nonlinear response of the silicon disks can be employed for
realising the bistable reverse reflection which is controllable by varying the
intensity of the incident wave. The nonlinear tunability of silicon-on-silver
structure is promising to extend the working area of classical subwavelength
metamaterials providing new opportunities in applications with light-matter
interactions.

Authors are grateful to the National Research Foundation of Ukraine
(2020.02/0218) and the National Academy of Sciences of Ukraine for a support
of this work.

## References

## References

  * [1] Jay M. Enoch.  History of mirrors dating back 8000 years.  Optometry and Vision Science, 83:775–781, October 2006.
  * [2] Stanislav B. Glybovski, Sergei A. Tretyakov, Pavel A. Belov, Yuri S. Kivshar, and Constantin R. Simovski.  Metasurfaces: From microwaves to visible.  Physics Reports, 634:1–72, 2016.
  * [3] Ben-Xin Wang, Xiang Zhai, Gui-Zhen Wang, Wei-Qing Huang, and Ling-Ling Wang.  A novel dual-band terahertz metamaterial absorber for a sensor application.  Journal of Applied Physics, 117(1):014504, 2015.
  * [4] Riad Yahiaoui, Siyu Tan, Longqing Cong, Ranjan Singh, Fengping Yan, and Weili Zhang.  Multispectral terahertz sensing with highly flexible ultrathin metamaterial absorber.  Journal of Applied Physics, 118(8):083103, 2015.
  * [5] N Sydorchuk and S Prosvirnin.  Analysis of terahertz wave reflection by an array of double dielectric elements placed on a reflective substrate.  In XXIInd Intern. Seminar/Workshop on Direct and Inverse Problems of Electromagnetic and Acoustic Wave Theory (DIPED), pages 58–63, Dnipro, Ukraine, September 2017.
  * [6] Y. Lee, S. J. Kim, H. Park, and B. Lee.  Metamaterials and metasurfaces for sensor applications.  Sensors, 17:1726, Jul 2017.
  * [7] Stéphane Collin.  Nanostructure arrays in free-space: optical properties and applications.  Reports on Progress in Physics, 77(12):126402, nov 2014.
  * [8] Li Zhu, Jonas Kapraun, James Ferrara, and Connie J. Chang-Hasnain.  Flexible photonic metastructures for tunable coloration.  Optica, 2(3):255–258, Mar 2015.
  * [9] Majid Esfandyarpour, Erik C. Garnett, Yi Cui, Michael D. McGehee, and Mark L. Brongersma.  Metamaterial mirrors in optoelectronic devices.  Nature Nanotechnology, 9(7):542–547, 2014.
  * [10] Jingshi Meng, Trevon Badloe, Jungho Mun, and Junsuk Rho.  Metasurfaces-based absorption and reflection control: Perfect absorbers and reflectors.  Journal of Nanomaterials, 2017:2361042, 2017.
  * [11] Michael S Eggleston, Kevin Messer, Liming Zhang, Eli Yablonovitch, and Ming C Wu.  Optical antenna enhanced spontaneous emission.  Proceedings of the National Academy of Sciences of the United States of America, 112(6):1704–1709, 02 2015.
  * [12] D. C. Li, F. Boone, M. Bozzi, L. Perregrini, and K. Wu.  Concept of virtual electric/magnetic walls and its realization with artificial magnetic conductor technique.  IEEE Microwave and Wireless Components Letters, 18(11):743–745, 2008.
  * [13] Saman Jahani and Zubin Jacob.  All-dielectric metamaterials.  Nature Nanotechnology, 11:23–36, 1 2016.
  * [14] V. P. Shestopalov, L. M. Lytvynenko, S. A. Masalov, and V. G. Sologub.  Diffraction of waves by gratings.  Kharkov State University, Kharkov, USSR, in russian edition, 1973.
  * [15] E. Jull and G. Ebbeson.  The reduction of interference from large reflecting surfaces.  IEEE Transactions on Antennas and Propagation, 25(4):565–570, 1977\.
  * [16] S. A. Masalov and Iu. K. Sirenko.  Excitation of reflecting lattices by a plane wave in the autocollimation mode.  Radiophysics and Quantum Electronics, 23(4):332–338, Oct 1980.
  * [17] T. M. Hard.  Laser wavelength selection and output coupling by a grating.  Appl. Opt., 9:1825–1830, Aug 1970.
  * [18] Haim Lotem.  Littrow-mounted diffraction grating cavity.  Applied Optics, 33(6):930–934, Feb 1994.
  * [19] Aleksander V Gribovsky and Oleg A Yeliseyev.  Nonspecular reflection of gaussian wave beams on a two-dimensional periodic array with shorted waveguides of rectangular cross-section.  Journal of Optics, 16(3):035701, Feb 2014.
  * [20] Natalia M. Litchinitser and Jingbo Sun.  Optical meta-atoms: Going nonlinear.  Science, 350(6264):1033–1034, 2015.
  * [21] Robert W. Boyd.  Nonlinear optics.  Academic Press, 3 edition, 2008.
  * [22] Sergey L Prosvirnin, Vyacheslav V Khardikov, Kateryna L Domina, Olexandr A Maslovskiy, Ludmila A Kochetova, and Vladimir V Yachin.  Non-specular reflection by a planar resonant metasurface.  http://arxiv.org/abs/2103.01010, 2021.
  * [23] J. van de Groep and A. Polman.  Designing dielectric resonators on substrates: Combining magnetic and electric resonances.  Opt. Express, 21(22):26285–26302, Nov 2013.
  * [24] M. Ene-Dobre, Marian Banciu, Liviu Nedelcu, G. Stoica, Cristina Busuioc, and Horia Alexandru.  Microwave antennas based on $\mathrm{Ba_{1-x}Pb_{x}Nd_{2}Ti_{5}O_{14}}$.  Journal of Optoelectronics and Advanced Materials, 13:1298–1304, 10 2011.
  * [25] M. Dinu, F. Quochi, and H. Garcia.  Third-order nonlinearities in silicon at telecom wavelengths.  Applied Physics Letters, 82(18):2954–2956, 2003.
  * [26] Faezeh Gholami, Sanja Zlatanovic, Aleksandar Simic, Lan Liu, David Borlaug, Nikola Alic, Maziar P. Nezhad, Yeshaiahu Fainman, and Stojan Radic.  Third-order nonlinearity in silicon beyond 2350 nm.  Applied Physics Letters, 99(8):081102, 2011.
  * [27] Ting Wang, Nalla Venkatram, Jacek Gosciniak, Yuanjing Cui, Guodong Qian, Wei Ji, and Dawn T. H. Tan.  Multi-photon absorption and third-order nonlinearity in silicon at mid-infrared wavelengths.  Opt. Express, 21(26):32192–32198, Dec 2013.
  * [28] Wolfgang S. M. Werner, Kathrin Glantschnig, and Claudia Ambrosch-Draxl.  Optical constants and inelastic electron-scattering data for 17 elemental metals.  Journal of Physical and Chemical Reference Data, 38(4):1013–1092, 2009.
  * [29] Sergiy Prosvirnin, Kateryna Domina, Vyacheslav Khardikov, and Vladimir Yachin.  Non-specular reflection of light controlled by light.  In 2021 IEEE 26th International Seminar/Workshop on Direct and Inverse Problems of Electromagnetic and Acoustic Wave Theory (DIPED), pages 82–85, 2021.