# Open Set Domain Adaptation By Novel Class Discovery

###### Abstract

In Open Set Domain Adaptation (OSDA), large amounts of target samples are
drawn from the implicit categories that never appear in the source domain. Due
to the lack of their specific belonging, existing methods indiscriminately
regard them as a single class “ _unknown_ ”. We challenge this broadly-adopted
practice that may arouse unexpected detrimental effects because the decision
boundaries between the implicit categories have been fully ignored. Instead,
we propose Self-supervised Class-Discovering Adapter (SCDA) that attempts to
achieve OSDA by gradually discovering those implicit classes, then
incorporating them to restructure the classifier and update the domain-
adaptive features iteratively. SCDA performs two alternate steps to achieve
implicit class discovery and self-supervised OSDA, respectively. By jointly
optimizing for two tasks, SCDA achieves the state-of-the-art in OSDA and shows
a competitive performance to unearth the implicit target classes.

Index Terms—  Domain Adaptation, Open Set Recognition, Class Discovery,
Unsupervised Learning

##  1 Introduction

Unsupervised Domain Adaptation (UDA) methods aim to transfer the knowledge
from a labeled source domain to classify unlabeled samples in a target domain
by minimizing the cross-domain distribution discrepancy. Despite their
impressive progress, UDA methods usually operate under the _close-set_
assumption, _i.e._ , the source categories and the target categories should be
exactly identical. However, this assumption is probably violated in the real
world since target samples are collected from diverse classes even beyond
source categories. Therefore, _Open Set Domain Adaptation_ (OSDA) [1, 2] has
attracted increasing attention, where the target domain may contain _implicit
classes_ that never appear in the source domain.

Fig. 1: Existing OSDA methods unitedly treat implicit classes as a single
negative class “ _unknown_ ”. Our work challenges this broadly-accepted
practice, since ignoring decision boundaries between implicit classes results
in all implicit-class features converging together due to the cluster
assumption, which further causes a detrimental effect. For instance, suppose
_husky_ , _lion_ denote the implicit classes and _wolf_ , _deer_ denote the
known classes. Given a known class similar with an implicit class, _e.g._ ,
_wolf_ v.s. _husky_ , the _wolf_ target features could be attracted to the
feature field of _lion_ since _husky_ and _wolf_ features are hard to
distinguish, yet _husky_ and _lion_ tend to converge to the identical center
in the “ _unknown_ ”.

Due to the absence of both the corresponding categories for target samples and
the number for implicit classes, existing OSDA methods regard all target
samples of implicit classes as a single class “ _unknown_ ”. This practice is
straightforward but probably problematic. Specifically, existing OSDA methods
[3, 4, 5] aim to minimize the cross-entropy losses of known classes and the “
_unknown_ ” class. Under the cluster assumption [6], the features of unknown
target samples are optimized to converge to an identical center due to the
shared labels. Whereas, since their intrinsic structure and diversity have
been ignored, the convergence is hard to achieve in practice. Especially when
unknown target samples contain more categories, the features of unknown
samples will be more probably mixed up with the known-class features around
the decision boundaries between known classes and the “ _unknown_ ” class
(Figure.1.b), hence, breeding the potential performance drop.

In this paper, we focus on a new methodology to achieve OSDA from another
point of view. Instead of fabricating the “ _unknown_ ”, we aim to transfer
the source knowledge along with discovering implicit classes [7] in unknown
target samples. The process automatically estimates the number of implicit
classes and how they are distributed, then leverages this self-supervised
information to progressively update the OSDA model to improve the performance
of OSDA.

Fig. 2: An overview of SCDA. SCDA starts from a pre-training to obtain unknown
target samples. Then SCDA designs an algorithm to discover the implicit
classes in the unknown target samples; Afterwards, SCDA restructures the
classifier and updates it to recognize them.

To this end, we propose Self-supervised Class-Discovering Adapter (SCDA).
After an adversarial pre-training, SCDA separates unknown target samples
roughly. Unlike previous OSDA methods that regard all unknown target samples
as the “ _unknown_ ” and neglect their inter-class structure, SCDA designs an
unsupervised algorithm, _i.e._ , implicit class discovery, to estimate the
number of the implicit classes and group the unknown samples belonging to each
implicit class. Based on the results of implicit class discovery, SCDA
restructures and updates the model to push features of different classes away
from each other to reduce the risk of confusing them. Our contributions are
summarized in two aspects:

  * •

We consider OSDA from a new point of view: Instead of regarding target samples
beyond categories in the source domains as the “ _unknown_ ” class, we attempt
to discover their structure along with the domain adaption.

  * •

We propose Self-supervised Class-Discovering Adapter (SCDA) which combines
discovering the implicit classes and learning domain-invariant features in a
framework.

  * •

Extensive experiments on three OSDA benchmarks are conducted to evaluate SCDA.
The results evidence the superiority of SCDA on both OSDA and implicit class
discovery.

##  2 Related Work

Close-set domain adaptation (UDA). Assuming labeled source and the unlabeled
target domains share their label sets, closed-set UDA methods aim at reducing
the domain shift across the source and target domains. Most existing
algorithms are based on either domain discrepancy matching [8, 9] or
adversarial learning [10]. The former employs various measurements to minimize
the distribution discrepancy; The latter adversarially learns domain invariant
features.

Open-set domain adaptation (OSDA). OSDA learners categorize target samples
into the known classes or _unknown_. OSBP [2] learns representations to
separate unknown target samples through an adversarial method. STA [11] trains
a binary classifier to perform fine separation on all target samples and
weight each target sample to alleviate negative-transfer caused by unknown
target samples. Besides, there are a few works, _e.g._ , TIM [4], SHOT [3],
JPOT [5] and PGL [12]. However, all these approaches simply assign target
samples beyond the label-set of source classes to the unknown, while our
method further classifies each novel class inside them.

Novel-class discovery. Provided the labeled data from related but different
classes, novel-class discovery aims to find novel classes in unlabeled data.
However, existing methods, _e.g._ , DTC [13] and [7], assume that the labeled
and unlabeled data are drawn from identical distribution and non-overlapping
classes. They can not be directly adopted for OSDA, since the domain shift
between the labeled and unobserved data would incur a severe performance
degeneration.

##  3 Methodology

In this section, we first define the problem setting of OSDA; then we
introduce the architecture of our SCDA; finally, we describe the training
process in detail.

###  3.1 Problem Setting

Suppose $n_{s}$ labeled images
$\mathcal{S}={(\boldsymbol{x}_{i}^{s},\boldsymbol{y}_{i}^{s})}_{i=1}^{n_{s}}$
are drawn from a source density
$P_{\mathcal{S},\mathcal{C}_{\mathcal{S}}}(\boldsymbol{x},\boldsymbol{y})$ and
$n_{t}$ unlabeled images
$\mathcal{T}={(\boldsymbol{x}_{i}^{t})}_{i=1}^{n_{t}}$ are drawn from a target
density $P_{\mathcal{T},\mathcal{C}_{\mathcal{S}}}(\boldsymbol{x})=\int
P_{\mathcal{T},\mathcal{C}_{\mathcal{S}}}(\boldsymbol{x},\boldsymbol{y})d\boldsymbol{y}$.
$\mathcal{C}_{\mathcal{S}}$ denotes the set of the source classes,
$\mathcal{C}_{\mathcal{T}}$ denotes that of the target, and
$\mathcal{C}_{\mathcal{T}}/\mathcal{C}_{\mathcal{S}}$ denotes the implicit
classes in $\mathcal{T}$. In OSDA, due to
$\mathcal{C}_{\mathcal{T}}/\mathcal{C}_{\mathcal{S}}\neq\emptyset$, we are
required to classify target samples of $|\mathcal{C}_{\mathcal{S}}|$ known
classes correctly ($|A|$ indicates the number of members in $A$) and
simultaneously reject the unknown target samples belonging to
$\mathcal{C}_{\mathcal{T}}/\mathcal{C}_{\mathcal{S}}$.

###  3.2 Overall Architecture

SCDA can be flexibly deployed to existing neural network architectures. As
shown in Figure.2, the architecture of SCDA consists of two modules: a
_feature extractor_ $F$ proposed to learn class-aware domain-invariant
features; a _dynamically restructuring classifier_ $C$ proposed to classify
unlabeled target samples into the known classes $\mathcal{C}_{\mathcal{S}}$
and the implicit classes $\mathcal{C}_{\mathcal{T}}$. The output dimension of
$C$ is initialized as $|\mathcal{C}_{\boldsymbol{S}}|+1$, where the
$(|\mathcal{C}_{\boldsymbol{S}}|+1)^{\prime}$-th class indicates the unknown
target samples. It is worth noting that, according to the results of implicit
class discovery, the output dimension of $C$ alters to
$|\mathcal{C}_{\mathcal{S}}|+k_{t}^{\ast}$, where $k*$ refers to the newly
discovered classes.

###  3.3 Self-supervised Class-Discovering Adapter

The pipeline of SCDA mainly consists of two alternate steps, _i.e._ , implicit
class discovery (Sec.3.3.2) and self-supervised OSDA (Sec.3.3.3). Briefly,
SCDA employs a pre-training to roughly separate the unknown target samples.
Then, it alternately performs two steps to discover the implicit classes in
$\mathcal{C}_{\mathcal{T}}/\mathcal{C}_{\mathcal{S}}$ and further improve the
performance of OSDA based on the results of discovery. First, SCDA discovers
the implicit classes in $\mathcal{C}_{\mathcal{T}}/\mathcal{C}_{\mathcal{S}}$
by estimating their number and constructing the pseudo implicit classes
through a clustering assignment. Then based on the pseudo implicit classes,
SCDA restructures the $C$ to recognize newly discovered classes; SCDA trains
$C$ along with $F$ to diminish the domain gap so that the model can be
generalized to classify the target samples in implicit classes. SCDA
repeatedly executes two alternate steps until the maximal epoch is reached. We
elaborate the pre-training and the two steps in the following three
subsections. The pipeline of SCDA is found in Supplementary Algorithm.2.

####  3.3.1 Pre-training

Due to the absence of target samples’ labels, SCDA utilizes adversarial
learning in this pre-training step to preliminarily separate the unknown
target samples. In brief, $C$ is trained to confuse the known and unknown
target samples while $F$ is trained oppositely to distinguish them. We utilize
the correlation confusion derived from [14] to implement the adversarial
training. Specifically, given a mini-batch of $m$ target samples, each element
in a class correlation matrix $\mathbf{R}$ presents as:

$\mathbf{R}_{i,j}=\mathbf{\hat{y}}^{\top}_{i,\cdot}\frac{m\Big{(}1+\exp\big{(}-H(\boldsymbol{x}_{i}^{(t)};F,C)\big{)}\Big{)}}{\sum^{m}_{i^{\prime}=1}\Big{(}1+\exp\big{(}-H(\boldsymbol{x}_{i^{\prime}}^{(t)};F,C)\big{)}\Big{)}}\mathbf{\hat{y}}_{j,\cdot}$
(1)

where $\mathbf{\hat{y}}_{j,\cdot}$ represents the softmax output for the
class-$j$ prediction over $m$ target examples in the mini-batch and $H$ (Eq.7)
is a measure to increase the weight of the reliable examples.

The $j^{\prime}$-th column in $\mathbf{R}$ measures the correlation between
the $j^{\prime}$-th class and other classes when $C$ classifies a mini-batch
of samples. The higher $\mathbf{R}_{j,{j}^{\prime}}$ implies that $C$ will
more probably classify the samples drawn from the $j$-th class to the
$j^{\prime}$-th class. So we can adjust the value of $\mathbf{R}$ between
known and unknown classes to cause confusion. To be specific, after
normalizing $\mathbf{R}$ by the sum of each row, we obtain $\mathbf{\hat{R}}$
where the summation over each row is 1. Then we optimize the value of
$\mathbf{\hat{R}}_{j,|\mathcal{C}_{\boldsymbol{S}}|+1}$ to 0.5 which means the
probability that $C$ classifies samples into the unknown class or
${j}^{\prime}$-th known class is equal, _i.e._ , $C$ can not distinguish the
known and unknown samples. While, we train $F$ in the opposite direction by
inserting a reversed gradient layer [10] between $C$ and $F$. The adversarial
learning loss is defined as:

$L_{adv}=\mathbb{E}_{(\boldsymbol{x})\sim\mathcal{T}}L_{bce}(\frac{1}{|\mathcal{C}_{\boldsymbol{S}}|}\sum_{j=1}^{|\mathcal{C}_{\boldsymbol{S}}|}\mathbf{\hat{R}}_{j,|\mathcal{C}_{\boldsymbol{S}}|+1},\frac{1}{2})$
(2)

Simultaneously, provided with source labeled data $\mathcal{S}$, we have a
standard cross-entropy loss $L_{\rm s}$ to correctly categorize the known
classes in $\mathcal{C}_{\mathcal{S}}$:

$L_{\rm s}=-\ \mathbb{E}_{(\boldsymbol{x},\boldsymbol{y})\sim\mathcal{S}}\
\boldsymbol{y}^{T}\log\ C\big{(}F(\boldsymbol{x})\big{)}$ (3)

Besides, to alleviate the cross-known-class confusion caused by domain shift,
we optimize $L_{\rm kcc}$ (Eq.4). It is worth noting that, $L_{\rm kcc}$ does
not punish the confusion to
$\mathcal{C}_{\mathcal{T}}/\mathcal{C}_{\mathcal{S}}$, leading to the cross-
domain features only aligned on $\mathcal{C}_{\mathcal{S}}$.

$L_{kcc}=\mathbb{E}_{(\boldsymbol{x})\sim\mathcal{T}}\
\frac{1}{|\mathcal{C}_{\boldsymbol{S}}|}\sum_{j=1}^{|\mathcal{C}_{\boldsymbol{S}}|}\sum_{{j}^{\prime}\neq
j}^{|\mathcal{C}_{\boldsymbol{S}}|}\mathbf{\hat{R}}_{j,{j}^{\prime}}$ (4)

Combing the above items, the overall pre-train objective is formulated as:

$\mathop{}_{F}^{\min}L_{\rm s}-L_{\rm adv}+L_{\rm kcc}$ (5)
$\mathop{}_{C}^{\min}L_{\rm s}+L_{\rm adv}+L_{\rm kcc}$ (6)

After preparation, SCDA alternately runs two steps to achieve implicit class
discovery and self-supervised OSDA.

Algorithm 1 Implicit Class Discovery in Sec.3.3.2

Input: Target dataset $\mathcal{T}$; pre-trained feature extractor $F$ and
classifier ${C}$; max implicit classes number $k_{max}$.
Output: The estimation number $k^{\ast}$ of implicit classes; pseudo-labeled
known target data $\hat{\mathcal{T}}_{\rm kn}$; pseudo-labeled newly
discovered target data $\\{\hat{\mathcal{T}}_{i}\\}^{k^{\ast}}_{i=1}$.

1: Compute the entropy for $\boldsymbol{x}^{t}\sim\mathcal{T}$ by Eq. 7. Sort
$\boldsymbol{x}^{t}$ based on their entropies. Select samples according to the
entropy to build $\hat{\mathcal{T}}_{\rm kn}$ with pseudo labels
$\hat{\boldsymbol{y}_{t}}$ and $\hat{\mathcal{T}}_{\rm im}$.

2: Extract feature of $\hat{\mathcal{T}}_{\rm kn}$ and $\hat{\mathcal{T}}_{\rm
im}$ using $F$.

3: For $0\leq k\leq k_{max}$ do

4: Run k-means++ on the extracted feature with $k$ clusters.

5: Compute CA for $\hat{\mathcal{T}}_{\rm kn}$ and SSE for
$\hat{\mathcal{T}}_{\rm kn}\cup\hat{\mathcal{T}}_{\rm im}$.

6: End for

7: Let $\hat{k}$=$(k^{\ast}_{\rm CA}+k^{\ast}_{\rm elbow})/2$. $k^{\ast}_{\rm
CA}$ is the value of $k$ maximizes CA. $k^{\ast}_{\rm elbow}$ is generated by
the elbow method.

8: Let $k^{\ast}$=$\hat{k}-|\mathcal{C}_{\mathcal{S}}|$. Run k-means++ on the
features of $\hat{\mathcal{T}}_{\rm im}$ to obtain $k^{\ast}$ clusters
$\\{\hat{\mathcal{T}}_{i}\\}^{k^{\ast}}_{i=1}$. Categorize
$\hat{\mathcal{T}}_{\rm im}$ into
$\\{\hat{\mathcal{T}}_{i}\\}^{k^{\ast}}_{i=1}$ with pseudo labels.

9: Return $k^{\ast}$=$|\mathcal{C}_{\mathcal{O}}|$; $\hat{\mathcal{T}}_{\rm
kn}$; $\\{\hat{\mathcal{T}}_{i}\\}^{k^{\ast}}_{i=1}$

####  3.3.2 Implicit class discovery

In this step, SCDA attempts to determine the number of implicit classes in
$\mathcal{T}$ with the help of the labeled data. However, if we directly use
the labeled source data, the domain shift between source and target domain
would affect the accuracy of the estimation. Hence, SCDA first constructs two
_high-confident target candidates_ sets $\hat{\mathcal{T}}_{\rm kn}$ and
$\hat{\mathcal{T}}_{\rm im}$ with pseudo labels, indicating target samples in
known classes and implicit classes, respectively. Then SCDA estimates
$|\mathcal{C}_{\mathcal{T}}/\mathcal{C}_{\mathcal{S}}|$ by evaluating the
_clustering consistency_ between $\hat{\mathcal{T}}_{\rm kn}$ and
$\hat{\mathcal{T}}_{\rm im}$, and assigns pseudo labels to the newly
discovered classes. This step has been summarized in Algorithm.1.

High-confident target candidates. Instead of analyzing whole $\mathcal{T}$, we
select target candidates with higher cross-domain classification consistency,
because the target samples with higher consistency are more reliable. The
cross-domain classification consistency can be measured by Eq.7. The lower
entropy to classify target samples with a source classifier implies the higher
consistency.

$H(\boldsymbol{x}^{t};F,C)=-\sum_{i=1}^{|\mathcal{C}_{\mathcal{S}}|+k^{\ast}}C_{i}(F(\boldsymbol{x}^{(t)}))\log
C_{i}(F(\boldsymbol{x}^{(t)}))$ (7)

where $C_{i}(F(\boldsymbol{x}^{(t)}))$ denotes the softmax value of
$\boldsymbol{x}^{t}$ with respect to the $i^{\prime}$-th class;
$k^{\ast}$=$\mathcal{C}_{\mathcal{D}}$ denotes the optimal estimation of
$|\mathcal{C}_{\mathcal{T}}/\mathcal{C}_{\mathcal{S}}|$ in the previous epoch
($k^{\ast}$=$1$ in the initialization). $\mathcal{C}_{\mathcal{D}}$ indicates
the newly discovered target classes and the goal of SCDA is to iteratively
update $\mathcal{C}_{\mathcal{D}}$ to approximate
$\mathcal{C}_{\mathcal{T}}/\mathcal{C}_{\mathcal{S}}$.

Specifically, for each class in $\mathcal{C}_{\mathcal{S}}$, SCDA picks out
the samples in with $\mathcal{T}$ the corresponding pseudo label and selects
the first half of them with low entropies to construct the target candidate
subset of known classes $\hat{\mathcal{T}}_{\rm kn}$. Similarly, as for those
in $\mathcal{C}_{\mathcal{D}}$, SCDA also selects the half of them to
construct the target candidate subset of implicit classes
$\hat{\mathcal{T}}_{\rm im}$. Obviously, the domain gap between
$\hat{\mathcal{T}}_{\rm kn}$ and $\hat{\mathcal{T}}_{\rm im}$ has been erased,
and thus, SCDA executes a dynamical cluster algorithm which splits the
features of $\hat{\mathcal{T}}_{\rm kn}\cup\hat{\mathcal{T}}_{\rm im}$ by
varying the clustering number $k$ then compares their clustering consistency
value to obtain an optimal $k$ as the estimation of
$|\mathcal{C}_{\mathcal{T}}/\mathcal{C}_{\mathcal{S}}|$.

Criteria for the clustering consistency. We employ two criteria to evaluate
the clustering consistency value. The first criterion refers to the elbow
method [15] widely adopted in clustering analysis. It plots the sum of squared
error (SSE) as a function of $k$ to search the elbow point. Specifically, we
take the kneedle algorithm [16] to locate the point with the ideal cluster
number $k^{\ast}_{\rm elbow}$. The elbow method balances the diversity and the
granularity of the clusters, but it can not reflect the prior knowledge of
$\mathcal{C}_{\mathcal{S}}$.

Hence, as a supplement, we compute the clustering accuracy (Eq.8) on
$\hat{\mathcal{T}}_{\rm kn}$ to measure the clustering quality. CA measures
the clustering accuracy between the clustering assignment and the pseudo label
over $\hat{\mathcal{T}}_{\rm kn}$. Higher CA indicates the clustering results
are more consistent with $\mathcal{C}_{\mathcal{S}}$ in the target domain. We
select the $k^{\ast}_{\rm CA}$ with the highest CA.

$k_{\rm
CA}=\underset{k\in\\{1+|\mathcal{C}_{\mathcal{S}}|,\cdots,k_{\max}+|\mathcal{C}_{\mathcal{S}}|\\}}{\arg\max}\frac{1}{n}\sum_{i=1}^{n}\mathbf{1}_{\hat{\boldsymbol{y}}_{i}=M(\boldsymbol{c}_{i})}$
(8)

where $\mathbf{1}$ denotes the indicator function and $M(\boldsymbol{c}_{i})$
is permutation mapping that maps each cluster label $\boldsymbol{c}_{i}$ to
the pseudo label $\hat{\boldsymbol{y}}_{i}$ over total $n$
$\boldsymbol{x}_{i}\in\hat{\mathcal{T}}_{\rm kn}$.

SCDA takes $\hat{k}=(k^{\ast}_{\rm CA}+k^{\ast}_{\rm elbow})/2$ as the optimal
clustering number. Excluding $|\mathcal{C}_{\mathcal{S}}|$ source classes, we
consider $k^{\ast}=\hat{k}-|\mathcal{C}_{\mathcal{S}}|$ as the optimally
estimated number of $\mathcal{C}_{\mathcal{T}}$/$\mathcal{C}_{\mathcal{S}}$,
and use the corresponding clustering assignment to divide
$\mathcal{\hat{T}}_{\rm im}$ into $k^{\ast}$ pseudo classes
$\\{\hat{\mathcal{T}}_{i}\\}^{k^{\ast}}_{i=1}$. It refers to the update of
$\mathcal{C}_{\mathcal{D}}$.

####  3.3.3 Self-supervised open-set adaptation

Step 1 has provided an estimation result for the open-set class discovery,
while it is not able to improve OSDA since the result has not been fed back to
the domain-invariant feature learning yet. Provided with this, we develop the
self-supervised OSDA which enables the classifier $C$ to recognize more target
samples that belong to the classes in
$\mathcal{C}_{\mathcal{T}}/\mathcal{C}_{\mathcal{S}}$, further improving the
OSDA performance.

Restructuring $C$. Since $\mathcal{C}_{\mathcal{D}}$ dynamically changes to
approximate $\mathcal{C}_{\mathcal{T}}/\mathcal{C}_{\mathcal{S}}$, the softmax
classifier $C$ is also dynamically restructured in order to classify
$\mathcal{C}_{\mathcal{D}}$: Its output dimension alters from
$|\mathcal{C}_{\mathcal{S}}|+k^{\ast}_{t-1}$ to
$|\mathcal{C}_{\mathcal{S}}|+k_{t}^{\ast}$, in which the first
$|\mathcal{C}_{\mathcal{S}}|$ corresponds the number of classes in
$\mathcal{C}_{\mathcal{S}}$ and the latter refers to $k*$ classes in
$\mathcal{C}_{\mathcal{D}}$. The parameters are reset and trained in terms of
$\mathcal{C}_{\mathcal{S}}$ and the current $\mathcal{\hat{C}_{\mathcal{D}}}$.

Dynamic class correlation matrix. The crucial problem is to improve the
generalization ability of $F$ and $C$ in terms of the newly discovered
classes. To this end, we reconsider the class correlation matrix proposed in
Eq.1. Indeed, this class decorrelating technique augments various close-set
UDAs to reap the transfer gain. However, it is rarely applied in OSDA since
its confusion mechanism naturally repels the “ _unknown_ ”, thus, reducing
their confusion would eliminate the intrinsic diversity of
$\mathcal{C}_{\mathcal{T}}/\mathcal{C}_{\mathcal{S}}$. This concern found an
echo of our motivation, inspiring us to generalize the class correlation
matrix to suit OSDA. Specifically, we reconfigure Eq.1 by using our
restructuring softmax classifier output as $\mathbf{\hat{y}}_{i,\cdot}$.
Hence, the dimension of $\mathbf{R}$ changes from
$(|\mathcal{C}_{\mathcal{S}}|+1)\times(|\mathcal{C}_{\mathcal{S}}|+1)$ to
$(|\mathcal{C}_{\mathcal{S}}|+k^{\ast})\times(|\mathcal{C}_{\mathcal{S}}|+k^{\ast})$,
extending the confusion measurement from the known classes in
$\mathcal{C}_{\mathcal{S}}$ and the “ _unknown_ ” class, to the newly
discovered classes in $\mathcal{C}_{\mathcal{D}}$ and the correlation between
$\mathcal{C}_{\mathcal{S}}$ and $\mathcal{C}_{\mathcal{D}}$. OSDA with the
dynamic class correlation matrix aims to minimize:

$\underset{C,F}{\min}\ L_{\rm
tcc}=\mathbb{E}_{\\{\boldsymbol{x}^{(t)}_{i}\\}^{m}_{i=1}\sim\mathcal{\mathcal{T}}}\
\frac{1}{|\mathcal{C}_{\mathcal{S}}|+{k}^{\ast}}\sum_{j=1}^{|\mathcal{C}_{\mathcal{S}}|+{k}^{\ast}}\sum_{{j}^{\prime}\neq
j}^{|\mathcal{C}_{\mathcal{S}}|+{k}^{\ast}}{\hat{\mathbf{R}}}_{j,{j}^{\prime}}$
(9)

Compared with Eq.4, Eq.9 iteratively changes its dimension to measure the
confusion of $\mathcal{C}_{\mathcal{S}}\cup\mathcal{\hat{C}}_{\mathcal{D}}$.
It disambiguates the pseudo class assignment produced by Algorithm.1 and helps
$F$ learn more discriminative features for self-supervised open-set
adaptation.

To preserve the knowledge from the known classes, we also keep training C with
the source samples (Eq.3). In order to further approach an ideal performance,
we simultaneously incorporate the pseudo-labeled target samples drawn from
$\hat{{\mathcal{T}}}_{\rm kn}\cup\hat{\mathcal{T}}_{\rm im}$ to learn
transferable features:

$L_{t}=-\
\mathbb{E}_{(\boldsymbol{x},\boldsymbol{\hat{y}})\sim\hat{{\mathcal{T}}}_{\rm
kn}\cup\hat{\mathcal{T}}_{\rm im}}\ \boldsymbol{\hat{y}}^{T}\log\
C(F(\boldsymbol{x}))$ (10)

where $\boldsymbol{\hat{y}}$ denotes the corresponding pseudo label.

$\mathop{}_{F,C}^{\min}\ L_{\rm s}+L_{\rm t}+L_{\rm tcc}$ (11)

In summary, the final objective of this step is formulated as Eq.11. Note that
we do not use any hyperparameter to balance each term in all objectives (Eq.6,
Eq.5 and Eq.11).

Table 1: Results on Office-31 for OSDA. ∘ indicates our re-implementation with the officially released code. Method |  A$\rightarrow$W |  A$\rightarrow$D |  D$\rightarrow$W |  W$\rightarrow$D |  D$\rightarrow$A |  W$\rightarrow$A | Avg
---|---|---|---|---|---|---|---
OS | OS* | OS | OS* | OS | OS* | OS | OS* | OS | OS* | OS | OS* | OS | OS*
OSBP |  86.5$\pm$2.0 |  87.6$\pm$2.1 |  88.6$\pm$1.4 |  89.2$\pm$1.3 |  97.0$\pm$1.0 |  96.5$\pm$0.4 |  97.9$\pm$0.9 |  98.7$\pm$0.6 |  88.9$\pm$2.5 |  90.6$\pm$2.3 |  85.8$\pm$2.5 |  84.9$\pm$1.3 | 90.8 | 91.3
STA |  89.5$\pm$0.6 |  92.1$\pm$0.5 |  93.7$\pm$1.5 |  96.1$\pm$0.4 |  97.5$\pm$0.2 |  96.5$\pm$0.5 |  99.5$\pm$0.2 |  99.6$\pm$0.1 |  89.1$\pm$0.5 |  93.5$\pm$0.8 |  87.9$\pm$0.9 |  87.4$\pm$0.6 | 92.9 | 94.1
TIM |  91.3$\pm$0.7 |  93.2$\pm$1.2 |  94.2$\pm$1.1 | 97.1$\pm$0.8 |  96.5$\pm$0.5 |  97.4$\pm$0.7 |  99.5$\pm$0.2 |  99.4$\pm$0.3 |  90.1$\pm$0.2 |  91.5$\pm$0.2 |  88.7$\pm$1.3 |  88.1$\pm$0.9 | 93.4 | 94.5
JPOT |  92.8$\pm$0.6 |  92.2$\pm$0.4 |  95.2$\pm$0.9 |  96.0$\pm$0.6 |  98.1$\pm$0.3 |  96.2$\pm$0.4 |  99.5$\pm$0.1 |  98.6$\pm$0.2 | 93.0$\pm$0.7 | 94.1$\pm$0.4 |  88.9$\pm$1.0 |  88.4$\pm$0.4 | 94.6 | 94.3
SHOT∘ |  88.8$\pm$0.7 |  91.4$\pm$0.4 |  90.3$\pm$0.5 |  92.6$\pm$0.3 |  96.2$\pm$0.4 |  97.0$\pm$0.4 |  97.4 $\pm$0.3 |  97.9$\pm$0.5 |  91.6$\pm$0.5 |  93.4$\pm$0.7 |  91.2$\pm$0.5 |  93.5$\pm$0.4 | 92.6 | 94.3
PGL∘ |  89.2$\pm$0.5 |  90.1$\pm$0.7 |  89.6$\pm$0.8 |  91.6$\pm$0.5 |  95.3$\pm$0.4 |  95.1$\pm$0.5 |  96.7$\pm$0.5 |  97.6$\pm$0.6 |  71.0$\pm$0.9 |  72.0$\pm$0.5 |  73.0$\pm$0.4 |  77.6$\pm$0.6 | 85.8 | 87.4
SCDA | 95.7$\pm$0.1 | 97.5$\pm$0.2 | 95.9$\pm$0.4 |  96.5$\pm$0.4 | 99.2$\pm$0.1 | 99.7$\pm$0.1 | 99.8$\pm$0.1 | 100$\pm$0.0 |  92.1$\pm$0.1 |  93.7$\pm$0.3 | 92.2$\pm$0.1 | 93.6$\pm$0.1 | 95.8 | 96.8
Table 2: Results on Office-Home for OSDA. △ indicates the method does not report the variance of their results. Method |  Ar$\rightarrow$Cl |  Pr$\rightarrow$Cl |  Rw$\rightarrow$Cl |  Ar$\rightarrow$Pr |  Cl$\rightarrow$Pr |  Rw$\rightarrow$Pr |  Cl$\rightarrow$Ar |  Pr$\rightarrow$Ar |  Rw$\rightarrow$Ar |  Ar$\rightarrow$Rw |  Cl$\rightarrow$Rw |  Pr$\rightarrow$Rw | Avg
---|---|---|---|---|---|---|---|---|---|---|---|---|---
OSBP |  56.7$\pm$1.9 |  51.5$\pm$2.1 |  49.2$\pm$2.4 |  67.5$\pm$1.5 |  65.5$\pm$1.5 |  74.0$\pm$1.5 |  62.5$\pm$2.0 |  64.8$\pm$1.1 |  69.3$\pm$1.1 |  80.6$\pm$0.9 |  74.7$\pm$2.2 |  71.5$\pm$1.9 | 65.7
STA |  58.1$\pm$0.6 |  53.1$\pm$0.9 |  54.4$\pm$1.0 |  71.6$\pm$1.2 |  69.3$\pm$1.0 |  81.9$\pm$0.5 |  63.4$\pm$0.5 |  65.2$\pm$0.8 |  74.9$\pm$1.0 |  85.0$\pm$0.2 |  75.8$\pm$0.4 |  80.8$\pm$0.3 | 69.5
TIM |  60.1$\pm$0.7 |  54.2$\pm$1.0 |  56.2$\pm$1.7 |  70.9$\pm$1.4 |  70.0$\pm$1.7 |  78.6$\pm$0.6 |  64.0$\pm$0.6 |  66.1$\pm$1.3 |  74.9$\pm$0.9 |  83.2$\pm$0.9 |  75.7$\pm$1.3 |  81.3$\pm$1.4 | 69.6
JPOT |  59.6$\pm$0.5 |  54.2$\pm$0.7 |  54.6$\pm$0.9 |  72.3$\pm$1.1 |  70.1$\pm$0.6 |  82.1$\pm$0.9 |  62.9$\pm$0.7 |  68.3$\pm$0.8 |  75.1$\pm$1.1 |  84.8$\pm$0.4 |  77.4$\pm$0.5 |  81.2$\pm$0.4 | 70.2
SHOT△ |  64.5$\pm$0.0 | 59.3$\pm$0.0 |  64.6$\pm$0.0 | 80.4$\pm$0.0 | 75.4$\pm$0.0 |  82.3$\pm$0.0 |  63.1$\pm$0.0 |  65.3$\pm$0.0 |  69.6$\pm$0.0 |  84.7$\pm$0.0 |  81.2$\pm$0.0 |  83.3$\pm$0.0 | 72.8
PGL△ | 61.6$\pm$0.0 |  58.4$\pm$0.0 | 65.0$\pm$0.0 |  77.1$\pm$0.0 |  72.0$\pm$0.0 |  83.0$\pm$0.0 |  68.8$\pm$0.0 |  72.2$\pm$0.0 | 78.6$\pm$0.0 | 85.9$\pm$0.0 |  82.8$\pm$0.0 |  82.6$\pm$0.0 | 74.0
SCDA |  59.9$\pm$0.3 |  59.0$\pm$0.3 |  62.8$\pm$0.5 |  79.6$\pm$0.4 |  73.8$\pm$1.0 | 83.7$\pm$0.8 | 70.9$\pm$0.5 | 72.3$\pm$0.6 |  75.5$\pm$0.4 |  85.3$\pm$0.6 | 82.9$\pm$0.3 | 85.7$\pm$0.9 | 74.3

##  4 Experiment

In this section, we evaluate SCDA on three benchmarks to demonstrate its
superior performance on both OSDA and discovering implicit classes.

Benchmarks. We use two famous datasets: Office-31 and Office-Home, and choose
the same label sets of classes to build $\mathcal{C}_{\mathcal{S}}$ and
$\mathcal{C}_{\mathcal{T}}$ following [11]. Besides, we introduce the
challenging DomainNet. To simulate a real-world adaptation scenario, we
combine the _Real_ and _Clipart_ domains in DomainNet with the Rw and Cl in
Office-Home, respectively, to build two target domains Rw⋆ and Cl⋆. After
merging the same categories, the combined target domains have 362 categories,
279 classes of which are DomainNet-specific. Then we randomly select $1/4$
classes from them to induce the scarcity: we select 10 samples for each of the
classes and abandon the rest. It breeds a benchmark DomainNet⋆ with extremely
imbalanced target domains Rw⋆ and Cl⋆.

Baselines. We compare SCDA with a variety of state-of-the-art OSDA approaches,
including OSBP, STA, TIM, JOPT, SHOT, and PGL. We are also interested in the
performance of discovering implicit classes. To this, we compare SCDA’s class-
discovering ability with some state-of-the-art baselines, _i.e._ , Silhouette
coefficient (SC) and DTC.

Evaluation Criteria. For a fair comparison, we employ two evaluation metrics
in line with [2], _i.e._ , OS: averaging the class-wise target accuracy for
all the classes including the unknown as one class; OS*: averaging the class-
wise target accuracy only on known classes. Besides, in terms of class
discovery, we compare SCDA with SC and DTC to estimate the number of unknown
implicit classes $k^{\ast}$.

Implementation. Following [2], we evaluate all methods on all datasets with
ResNet-50 pre-trained on ImageNet as the backbone. We implement SCDA in
PyTorch and use momentum SGD with a learning rate of $10^{-3}$. More
implementation details can be found in supplementary material 7.

###  4.1 Results for OSDA

Office-31 and Office-Home. In Table 1, SCDA outperforms other baselines on
most transfer tasks in Office-31 with significant margins. In the hard tasks,
_e.g._ , A$\rightarrow$W, SCDA outperforms the second with a larger gap
(2.9%). As illustrated in Table 2, our SCDA still achieves the best
performance in Office-Home dataset. Notably, the second best model in
Office-31 (TIM) and Office-Home (PGL) both perform poorly in the other
dataset. It is probably due to the changing setup of _unknown_. In a
comparison, SCDA is designed to analyze the inter-class structure of
$\mathcal{C}_{\mathcal{T}}$/$\mathcal{C}_{\mathcal{S}}$. Thus, our method
presents the more robust generalization ability in OSDA.

Table 3: OSDA from OfficeHome to DomainNet⋆ Method |  Ar$\rightarrow$Rw⋆ |  Ar$\rightarrow$Cl⋆ |  Pr$\rightarrow$Rw⋆ |  Pr$\rightarrow$Cl⋆ | Avg
---|---|---|---|---|---
OS | OS* | OS | OS* | OS | OS* | OS | OS* | OS | OS*
OSBP | 58.1 | 57.8 | 33.0 | 32.3 | 59.5 | 59.3 | 30.5 | 29.8 | 45.4 | 44.8
STA | 60.5 | 60.4 | 40.1 | 39.6 | 59.1 | 59.0 | 32.3 | 33.9 | 48.0 | 48.2
SHOT | 64.6 | 65.1 | 45.2 | 45.7 | 65.4 | 65.9 | 40.3 | 40.4 | 53.9 | 54.3
Ours | 67.8 | 68.0 | 44.2 | 44.3 | 68.2 | 69.1 | 40.9 | 41.0 | 55.3 | 55.6
Table 4: Unknown categories number estimation results Dataset | GT | SC | DTC | Ours
---|---|---|---|---
$\hat{k}$ | Error | $\hat{k}$ | Error | $\hat{k}$ | Error
Office-31 | 11 |  8$\sim$33 | 6.8 |  4$\sim$9 | 4.2 |  9$\sim$11 | 0.9
Office-Home | 40 |  0$\sim$7 | 38.5 |  9$\sim$23 | 21.4 |  32$\sim$37 | 6.1
DomainNet⋆ | 297 |  5$\sim$8 | 290.7 |  46$\sim$71 | 238.2 |  227$\sim$255 | 58.5

Real-world Scenarios. To further investigate the baselines in more complicated
real world applications, we set up the transfer tasks from Pr and Ar in
Office-Home to the challenging blending-target domains Rw⋆ and Cl⋆ in
DomainNet⋆. As shown in Table 3, although Rw⋆ and Cl⋆ are noisy, and extremely
imbalanced, SCDA still achieves the state of the art in $3$ from $4$ transfer
combination. Besides, SCDA also presents a faster convergence rate and a
higher upper-bound performance in the complicated scenarios (see Fig.5 in SM).

###  4.2 Results for Implicit Class Discovery

In Table 4, we report the results for unknown class number estimation by SC,
DTC, and SCDA. SC performs the worst across all the benchmarks. DTC is poor in
Office-Home and DomainNet⋆ with numerous implicit classes. By contrast, SCDA
shows surprisingly accurate results to estimate the class number in Office-31,
where the average error is less than 1. Despite the large implicit class
number in Office-Home and DomainNet⋆, SCDA produces a low average error. The
results validate the reliability of SCDA to estimate the implicit class
number. More experiments for class discovery can be found in supplementary
material 8.1.

Fig. 3:  Average results of SCDA

###  4.3 Ablation study

The motivation of SCDA rises from the conjecture that the unobserved class
discovery may help OSDA. Our ablation is designed to justify the conjecture.
In particular, we compare SCDA with the following modifications. (1) Pre-
training: we use the network only pre-trained by subsection 3.3.1. (2)
_unknown_ ($k$=1): we utilize the pseudo labels to update the model. However,
without class discovery, we regard them as a single negative class _unknown_.
(3) $k^{*}$ w/o iters: SCDA trains the model with pseudo discovered classes
but without further iteration. (4) $k^{(GT)}$ w/o iters: we provide with the
true number of $\mathcal{C}_{\mathcal{T}}$ but without further iteration. (5)
$k^{(GT)}$ w iters: the algorithm is provided with $k^{(GT)}$, then we
alternatively train the model.

As illustrated in Figure 3, we report the average OS and OS* across all six
transfer tasks in Office-31. By comparing (2) with (3), discovering the
unobserved classes has a better performance than regarding them as the “
_unknown_ ”. It further verifies our motivation: discovering the structure of
the unobserved classes can improve the performance of OSDA. The results of
(3-5) draw an interesting conclusion. Without further iterations, training
with $k^{(GT)}$ outperforms training with the estimated class number. However,
their results are almost the same when we train model iteratively. It suggests
that the precise prediction of the unobserved classes number and the iterative
optimization are both important and their combination play a key role in
addressing OSDA. Besides, more analysis of the ratio of unobserved classes and
visualization are illustrated in Supplementary Material 8.2.

##  5 Conclusion

In this paper, we pay attention to a nontrivial challenge in OSDA: discovering
all implicit classes in the unknown target samples. The mixed unknown chunk
conceives category mismatching risk. To tackle the problem, we propose Self-
supervised Class-Discovering Adapter (SCDA). SCDA utilizes adversarial
learning to preliminarily separate unknown target samples. Then, SCDA employs
an alternate approach to discover novel target categories and update our model
with the discovery results. Through extensive empirical evaluations, we
demonstrate the superiority of our SCDA by the state-of-the-art OSDA
performance and the remarkable ability to discover unknown implicit classes.

## References

  * [1] Pau Panareda Busto and Juergen Gall,  “Open set domain adaptation,”  in ICCV, 2017, pp. 754–763.
  * [2] Kuniaki Saito, Shohei Yamamoto, et al.,  “Open set domain adaptation by backpropagation,”  in ECCV, 2018, pp. 156–171.
  * [3] Jian Liang et al.,  “Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation,”  in ICML, 2020, pp. 6028–6039.
  * [4] Jogendra Nath Kundu et al.,  “Towards inheritable models for open-set domain adaptation,”  in CVPR, 2020.
  * [5] Renjun Xu et al.,  “Joint partial optimal transport for open set domain adaptation.,”  in IJCAI, 2020, pp. 2540–2546.
  * [6] Rui Shu et al.,  “A dirt-t approach to unsupervised domain adaptation,”  arXiv preprint arXiv:1802.08735, 2018.
  * [7] Kai Han et al.,  “Automatically discovering and learning new visual categories with ranking statistics,”  in ICLR, 2020.
  * [8] Mingsheng Long et al.,  “Learning transferable features with deep adaptation networks,”  in ICML, 2015, pp. 97–105.
  * [9] Mingsheng Long et al.,  “Deep transfer learning with joint adaptation networks,”  in ICML, 2017, pp. 2208–2217.
  * [10] Yaroslav Ganin, Evgeniya Ustinova, et al.,  “Domain-adversarial training of neural networks,”  Domain Adaptation in Computer Vision Applications, p. 189, 2017\.
  * [11] Hong Liu et al.,  “Separate to adapt: Open set domain adaptation via progressive separation,”  in CVPR, 2019.
  * [12] Yadan Luo, Zijian Wang, Zi Huang, and Mahsa Baktashmotlagh,  “Progressive graph learning for open-set domain adaptation,”  in ICML, 2020, pp. 6468–6478.
  * [13] Kai Han, Andrea Vedaldi, and Andrew Zisserman,  “Learning to discover novel visual categories via deep transfer clustering,”  in ICCV, 2019, pp. 8401–8409.
  * [14] Ying Jin et al.,  “Minimum class confusion for versatile domain adaptation,”  in ECCV, 2020.
  * [15] Robert L Thorndike,  “Who belongs in the family?,”  Psychometrika, vol. 18, no. 4, pp. 267–276, 1953.
  * [16] V. Satopaa et al.,  “Finding a ”kneedle” in a haystack: Detecting knee points in system behavior,”  in ICDCS Workshops.
  * [17] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell,  “Adapting visual category models to new domains,”  in Proceedings of the European Conference on Computer Vision (ECCV). 2010, pp. 213–226, Springer.
  * [18] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan,  “Deep hashing network for unsupervised domain adaptation,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 5018–5027.
  * [19] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang,  “Moment matching for multi-source domain adaptation,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 1406–1415.
  * [20] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.,  “Imagenet large scale visual recognition challenge,”  International journal of computer vision (IJCV), vol. 115, no. 3, pp. 211–252, 2015.
  * [21] Sergey Ioffe and Christian Szegedy,  “Batch normalization: Accelerating deep network training by reducing internal covariate shift,”  arXiv preprint arXiv:1502.03167, 2015.
  * [22] David Arthur and Sergei Vassilvitskii,  “k-means++: The advantages of careful seeding,”  in SODA, 2007, pp. 1027–1035.
  * [23] Leonard Kaufman and Peter J Rousseeuw,  Finding groups in data: an introduciton to cluster analysis, vol. 344,  John Wiley & Sons, 2009.
  * [24] Junyuan Xie, Ross Girshick, and Ali Farhadi,  “Unsupervised deep embedding for clustering analysis,”  in International Conference on Machine Learning (ICML), 2016, pp. 478–487.

Supplementary Material of Open Set Domain Adaptation
By Novel Class Discovery

##  6 The pipeline of SCDA

The pipeline of SCDA is found in Algorithm.2

Algorithm 2 Self-supervised Class-Discovering Adapter

Input: Labeled source dataset $\mathcal{S}$; Unlabeled target dataset
$\mathcal{T}$; initiated feature extractor $F$ and classifier $C$; max
iteration epoch $E$.
Output: Well-trained $F^{*}$ and $C^{*}$; The predicted number of unknown
classes $k^{\ast}$.

1: Pre-train $F$ by Eq.5 and $C$ by Eq.6

2: for $0:E$ do

3: Step 1: Implicit Class Discovery

4: Get $k^{\ast}$, $\hat{\mathcal{T}}_{\rm sh}$, $\hat{\mathcal{T}}_{\rm im}$
and $\\{\mathcal{\hat{T}}_{i}\\}^{k^{\ast}}_{i=1}$ by Algorithm.1.

5: Step 2: Self-supervised OSDA

6: Reset $C$ with the dimension of $|\mathcal{C}_{\mathcal{S}}|+{k}^{\ast}$.

7: Retrain $F$ and $C$ by Eq.11.

8: End for

9: Return $\\{\mathcal{\hat{T}}_{i}\\}^{k^{\ast}}_{i=1}$; ${k}^{\ast}$;
$F^{*}=F$; $C^{*}=C$.

##  7 Experimental Implementation

In this section, we present more experimental details. The code will be
released later.

###  7.1 Architectures

For fair comparisons with existing methods [2, 11, 4], we evaluate SCDA and
all baselines on Office-31 dataset [17], Office-Home dataset [18], and
DomainNet* [19] with ResNet-50 as the backbone. ResNet-50 is pre-trained on
ImageNet [20], then replaced the last fc layer of ResNet-50 with four fully
connected layers with batch normalization [21]. The detail architectures of
$F$ and $C$ are illustrated in Figure 4.

Fig. 4: The feature extractor $F$ and dynamically restructuring classifier $C$
we used in the experiments about visual recognition. (Best viewed in color)

###  7.2 Network Training

We implement the framework in PyTorch and use momentum SGD with learning rate
of $10^{-3}$, the momentum is set as 0.9 and the weight decay as $5\times
10^{-4}$. During the Self-supervised OSDA pre-training step (_i.e._ , Eq.6,7
in the paper), we do not update the parameters of the backbone network. When
turning to the alternative steps, the parameters of the feature extractor
($F$) would be iteratively updated while for the restructuring classifier
($C$), its parameter would be reset and trained from scratch for each
iteration in order to suit its dynamical structure. The hyper-parameters are
shown in Table 7.

###  7.3 Acceleration.

It is worth noticing that SCDA is an iterative algorithm where each iteration
relies on Algorithm.1, which estimates the implicit class number by
iteratively performing $k$-means. Given a linear computation complexity of a
$k$-means implementation in Pytorch, the complexity of Algorithm.1 refers to

$O(Td_{F}k^{2}_{max}n_{\rm ct})$

where $T$ denotes the max iteration number of the $k$-means implementation;
$d_{F}$ denotes the feature dimension for clustering; $n_{\rm ct}$ denotes the
total number of the confidential target subsets $\hat{\mathcal{T}}_{\rm
ex}\cup\hat{\mathcal{T}}_{\rm im}$; $k_{max}$ indicates the maximum estimation
to the number of all implicit classes. As the target domain $\mathcal{T}$
become diverse, $k_{max}$ will increase and slow the algorithm. To this end,
PCA is applied to reduce the output feature dimension $d_{F}$, and we also
employ GPU-driven $k$-means++ [22] to seed the cluster centroids and run
$k$-means, which implies a trivial $T$ to achieve the convergence in practice.
Beyond these operations, to accelerate the whole SCDA algorithm, we also
encourage the practitioners to reduce the max number of outer loop iteration
$E$ in terms of a benchmark with diverse implicit classes, _e.g._ , DomainNet.

As shown in Table.7, SCDA allows a large $k_{max}$ setup to search the
implicit classes as many as possible. Specifically, the main learning
procedure of SCDA consumes 3 hours for OfficeHome and 12 hours for DomainNet∗
by using a single Geforce GTX TITAN-X.

Table 5: Implicit categories number estimation results of DomainNet* and Office-31 Dataset | DomainNet* | Office-31
---|---|---
Task | Ar$\rightarrow$Rw* | Ar$\rightarrow$Cl* | Pr$\rightarrow$Rw* | Pr$\rightarrow$Cl* | Error | A$\rightarrow$W | A$\rightarrow$D | D$\rightarrow$W | W$\rightarrow$D | D$\rightarrow$A | W$\rightarrow$A | Avg | Error
SC | 8.3 | 5.0 | 6.7 | 5.0 | 290.7 | 6.3 | 21.3 | 36.7 | 18 | 13.3 | 11.0 | 17.8 | 6.8
DTC | 71.0 | 55.7 | 62.3 | 46.0 | 238.2 | 8.0 | 6.3 | 5.3 | 4.7 | 8.3 | 8.3 | 6.8 | 4.2
SCDA | 227.3 | 237.3 | 255.3 | 234.0 | 58.5 | 10.7 | 11.0 | 9.7 | 10.7 | 9.3 | 9.3 | 10.1 | 0.9
GT | 6 | 6 | 6 | 6 | - | 11 | 11 | 11 | 11 | 11 | 11 | 11 | -
Table 6: Implicit categories number estimation results of Office-Home Task | A$\rightarrow$C | P$\rightarrow$C | R$\rightarrow$C | A$\rightarrow$P | C$\rightarrow$P | R$\rightarrow$P | C$\rightarrow$A | P$\rightarrow$A | R$\rightarrow$A | A$\rightarrow$R | C$\rightarrow$R | P$\rightarrow$R | Avg | Error
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
SC | 1.3 | 0.3 | 0.3 | 0.3 | 6.3 | 5.3 | 1.3 | 0.0 | 0.0 | 0.0 | 2.3 | 0.7 | 1.5 | 38.5
DTC | 20.0 | 13.3 | 22.3 | 19.3 | 22.3 | 17.0 | 18.7 | 18.0 | 16.7 | 17.7 | 21.0 | 17.0 | 18.6 | 21.4
SCDA | 33.3 | 34.3 | 35.3 | 34.7 | 32.7 | 36.3 | 32.3 | 31.3 | 31.7 | 33.3 | 36.3 | 35.3 | 33.9 | 6.1
GT | 40 | 40 | 40 | 40 | 40 | 40 | 40 | 40 | 40 | 40 | 40 | 40 | 40 | -
Table 7: The hyper-parameters setting in our experiment. Dataset | Office-31 | Office-Home | DomainNet*
---|---|---|---
backbone | ResNet-50 | ResNet-50 | ResNet-50
batch size | 32 | 32 | 64
lr | 0.001 | 0.001 | 0.001
image size | 227$\times$227 | 227$\times$227 | 227$\times$227
$E$ | 400 | 400 | 40
$k_{max}$ | 40 | 120 | 650

###  7.4 Evaluation Criteria.

For a fair comparison, we employ two OSDA evaluation metrics in line with [2,
11], i.e., OS: averaging the class-wise target accuracy for all the classes
including the unknown as one class; OS*: averaging the class-wise target
accuracy only on shared classes.

In terms of implicit class discovery, we evaluate our algorithm according to
two criterion:

  1. 1.

Implicit class number ($k^{\ast}$) estimation: How many the implicit classes
the algorithm has estimated?

  2. 2.

Implicit class correspondence: How many the real implicit classes the
algorithm-discovered classes have corresponded to?

Towards the first criteria, we follow [13] and run the evaluated algorithms
three times for each transfer task, then report their mean of the estimated
implicit class number, _i.e._ , $k^{\ast}$. To this, Avg indicates the average
result of all the transfer tasks in each dataset. We compare it with the
ground truth number of the implicit classes (GT) and their absolute difference
between Avg and GT (smaller indicates better estimation performance, see
Table.5 ,6).

Towards the second criteria, we compute the probabilities of the unknown
samples and categorized them into the discovered implicit classes in terms of
the specific algorithm. For the samples chosen in each discovered class, we
sort them by the prediction confidence then choose the top $n$ samples with
the highest probabilities. So provided overall $nk^{\ast}$ samples, we check
how many real implicit classes they have referred to. Obviously, the value is
less than the ground truth (GT) number of real implicit classes. The value
closer to the ground truth indicates better (see Table.8).

Table 8: Matching real implicit classes in terms of top-$n$ evaluation Dataset | DTC | SCDA | GT
---|---|---|---
$n$=1 |  $n$=3 |  $n$=5 |  $n$=1 |  $n$=3 |  $n$=5 | GT
Office-31 | 4 | 7.6 | 8.9 | 8.8 | 9.3 | 10 | 11
Office-Home | 10.4 | 20.2 | 25 | 20.6 | 26.7 | 30 | 40
DomainNet* | 42.4 | 91.4 | 121.5 | 117 | 141 | 153.5 | 297

##  8 Results

In this section, we have provided more evaluation results skipped in the
paper. The faster convergence and a higher upper-bound performance in the
complicated scenarios are shown in Figure.5.

Fig. 5: The OS in task Pr$\rightarrow$Rw* and Ar$\rightarrow$Cl* during
training

###  8.1 Results for implicit class discovery.

Here we detailed the evaluation results of implicit class discovery in three
OSDA benchmarks Office-31, Office-Home and DomainNet*.

Implict class number ($k^{\ast}$) estimation. SC [23] and DTC [13] are
introduced as the baselines besides of our algorithm. SC (Silhouette
Coefficients) is a method of validation of consistency within clusters of data
and Kaufman et al. [23] introduced it to estimate the clusters $k$. Deep
Transfer Clustering (DTC) [13] clusters unlabeled data by DEC [24] with a
pretrained model and transfers knowledge from the set of source-known classes
to implicit target samples.

In Table 5 and Table 6, we report the results of the estimated $k^{*}$ of all
transfer tasks in three datasets. As shown in Table 5 and Table 6, SC
performed the worst across all benchmarks and totally failed in Office-Home.
DTC poorly performed in Office-Home and DomainNet*. With the number of
implicit classes increasing, the results of SC and DTC both became worse.
Compared with DTC and SC, our SCDA show the minimum error and the most stable
performance. It validates the effectiveness of our approach.

Implicit classes corrspondence. We further investigate how many implicit
classes SCDA may discover. To be specific, after training, SCDA gets a well-
trained classifier $C^{*}$ with output dimension
$|\mathcal{C}_{s}|+k^{*}$(_e.g._ , $m+k^{*}$), where $k^{*}$ indicates $k^{*}$
novel classes. For each novel class, we consider the probabilities of the
unknown samples belonging to this class according to the softmax output of the
restructuring classifier. We take them to feed the implicit class
correspondence in the second criteria. As for DTC, we choose the top $n$
samples with the highest probability of assigning data points to a class-aware
cluster, and check how many implicit classes they have corresponded. We report
the average results of all transfer tasks in each dataset.

As shown in Table 8, with the identical $n$, our method has corresponded to
more implicit classes. Especially when $n$=1, for all datasets, our SCDA is
twice the number of the discovered classes than DTC.

Fig. 6: Classification and class discovery performance of SCDA in
A$\rightarrow$W and A$\rightarrow$D when the ratio of unknown classes varies.

Fig. 7: OS* and unknown samples accuracy in A$\rightarrow$W when the ratio of
unknown classes changes. Fig. 8: (Left).T-SNE visualizations of the features
learned by PGL and SCDA on task A$\rightarrow$D in OSDA setup. Green points
indicate source features, blue points indicate target known features, and red
points indicate target unknown features. (Right).We magnify the region that
contains the features of unobserved classes and use different colors separate
their categories.

###  8.2 Analysis

The ratio of unobserved classes. The traditional OSDA benchmarks usually fix
the ratio of unobserved classes into 0.5. But in real-world tasks, we are not
aware of the ratio which can vary drastically. To validate the robustness to
the changing ratio, we conduct the experiments on Office-31 with variant
ratios. As illustrated in Fig 7, no matter how the ratio changes, SCDA
performs stably and precisely estimates the unobserved class number.
Furthermore, we compare SCDA with STA and OSBP in task A$\rightarrow$W on
Office-31 by varying the ratio. As shown in Fig 7, all baselines preserve
consistent OS* regardless of the changing ratio. But the accuracies of OSBP
and STA descend sharply as the unobserved class number increases. In a
comparison, SCDA achieves the more stable performance in OS* along with the
persistent accuracy of unknown samples. It reveals that SCDA is insensitive to
the change of unobserved class number.

Visualization. For the task A$\rightarrow$D in Office-31, we visualize the
last-layer features from PGL, and SCDA by t-SNE. In Figures 8, PGL tends to
push unknown target features together. However, the groups of unknown features
are mixed with the known-classes samples. It suggests that unifying all
unobserved classes into “ _unknown_ ” causes confusing decision boundaries
that decrease the OSDA performance. While the features generated from SCDA
present significant classification margins across domain-shared and unobserved
classes. Further, we magnify the region of the unknown features and use
different colors to indicate different unknown categories. As shown in the
dotted circles, SCDA features present more significant margins between
unobservable categories whereas STA simply mixes them together. The
illustration verifies the hypothesis in our introduction and demonstrates
SCDA’s capability to discover unseen classes in target samples.