# Active Source Free Domain Adaptation

Fan Wang
Shandong University
<EMAIL_ADDRESS>
&Zhongyi Han
Shandong University
<EMAIL_ADDRESS>
Zhiyan Zhang
Shandong University
<EMAIL_ADDRESS>
&Yilong Yin
Shandong University
<EMAIL_ADDRESS>

###### Abstract

Source free domain adaptation (SFDA) aims to transfer a trained source model
to the unlabeled target domain without accessing the source data. However, the
SFDA setting faces an effect bottleneck due to the absence of source data and
target supervised information, as evidenced by the limited performance gains
of newest SFDA methods. In this paper, for the first time, we introduce a more
practical scenario called active source free domain adaptation (ASFDA) that
permits actively selecting a few target data to be labeled by experts. To
achieve that, we first find that those satisfying the properties of neighbor-
chaotic, individual-different, and target-like are the best points to select,
and we define them as the minimum happy (MH) points. We then propose minimum
happy points learning (MHPL) to actively explore and exploit MH points. We
design three unique strategies: neighbor ambient uncertainty, neighbor
diversity relaxation, and one-shot querying, to explore the MH points.
Further, to fully exploit MH points in the learning process, we design a
neighbor focal loss that assigns the weighted neighbor purity to the cross-
entropy loss of MH points to make the model focus more on them. Extensive
experiments verify that MHPL remarkably exceeds the various types of baselines
and achieves significant performance gains at a small cost of labeling.

##  1 Introduction

Transferring a trained source model instead of the source data to the
unlabeled target domain, source-free domain adaptation (SFDA) has drawn much
attention recently. Since it prevents the external leakage of source data,
SFDA meets privacy persevering [1, 2], data security [3, 4], and data silos
[5]. Moreover, it has important potential in many applications, _e.g.,_ object
detection [6], object recognition [7, 8], and semantic segmentation [9, 10].
However, the SFDA setting faces an _effect bottleneck_ due to the absence of
source data and target supervised information. In literature, the state-of-
the-art A2Net [11] is a very powerful method, but it only improved the mean
accuracy of the pioneering work (SHOT [7]) from 71.8% to 72.8% on the
challenging Office-Home dataset [12].

In this paper, we introduce a new setting, _active source free domain
adaptation (ASFDA)_ , to break through the effect bottleneck. Table 1 reports
the main differences between active learning (AL), domain adaptation (DA),
Active DA, SFDA, and ASFDA. It is balanced that ASFDA permits actively
selecting a few samples in the target domain to be labeled by experts without
accessing the source data and with distribution shifts across domains. ASFDA
simultaneously obeys data security maximization, domain discrepancy impact
minimization, labeling cost minimization, and performance benefit
maximization. It is also more practical than SFDA because seeking domain
experts to annotate a few samples is very common in real-world applications,
such as COVID-19 screening [13, 14], autonomous driving [15, 16], and speech
recognition [17, 18].

Table 1: Comparison of ASFDA and other existing settings (‘$\checkmark$’
indicates that the condition is met, ‘$\times$’ indicates the opposite, and
‘-’ indicates that the condition does not exist).

Conditions | AL | DA | Active DA | SFDA | ASFDA
---|---|---|---|---|---
source data | - | $\checkmark$ | $\checkmark$ | $\times$ | $\times$
a few actively labeled target data (training data) | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$
domain shift | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$

Exploring and exploiting the most informative querying samples play key roles
in ASFDA, while previous AL and Active DA strategies cannot reach this. In
this paper, we find the best informative samples are minimum happy (MH) points
that satisfy the properties of neighbor-chaotic, individual-different, and
target-like. 1) The property of neighbor-chaotic measures the sample
uncertainty through their label-chaotic neighbors identified in high
dimensional inherent feature space. Under domain shift, existing AL [19, 20,
21] and Active DA [22, 23, 24] strategies are difficult to measure the sample
uncertainty due to model miscalibration [25] and unavailable source data in
ASFDA. 2) The property of individual-different guarantees the sample diversity
and further reduces the redundant querying. Previous methods [26, 27] ensure
diversity by selecting the samples closest to the centroids or searching a
core-set [28]. However, the selected samples would be easily-adaptive target
samples that lie in the source distribution and are unrepresentative of the
target domain; 3) The informative samples should be target-like because the
representative target samples can improve the generalization of the target
domain. Existing Active DA methods [22, 23] select target-like samples by a
domain discriminator that requires the source data but which is unavailable in
ASFDA. Concerning the exploitation of actively selected samples, most methods
of AL [19, 28, 29, 30, 31] and Active DA [22, 23, 26, 32] deem these samples
as ordinary labeled samples and use standard supervised losses, but which are
difficult to fully exploit the selected samples, resulting in limited
performance gains.

We propose the Minimum Happy Points Learning (MHPL) to explore and exploit the
informative MH points. The comprehensive framework of MHPL is shown in Fig. 1.
We propose a novel neighbor ambient uncertainty based on neighbor purity and
neighbor affinity to measure the neighbor-chaotic sample uncertainty. Then we
propose the neighbor diversity relaxation to guarantee the individual
difference. Further, we conduct the one-shot querying, selecting target
samples at once based on the source model, as the source model without fine-
tuning can better describe the distribution discrepancy across domains and the
target-like samples are more likely to be explored. Moreover, the selected
samples are fully exploited by a new-designed neighbor focal loss, which
assigns the weighted neighbor purity to the cross-entropy loss w.r.t. MH
points to make the model focus more on them.

Our contributions can be summarized as follows: (i) To the best of our
knowledge, it is the first time that the new setting of active source free
domain adaptation is introduced, which breaks through the effect bottleneck of
SFDA; (ii) We find the minimum happy points which enhance the generalization
performance on the target domain and achieve successful ASFDA; (iii) We
propose a novel MHPL framework to explore and exploit the minimum happy points
with the neighbor ambient uncertainty, neighbor diversity relaxation, one-shot
querying, and neigh focal loss; (iv) We lay a promising baseline for future
works of active source free domain adaptation.

Figure 1: The framework of Minimum Happy Points Learning (MHPL).

##  2 Minimum Happy Points Learning

Notation. In active source free domain adaptation (ASFDA), we can access a
source model $h_{s}:{\mathcal{X}_{s}}\rightarrow{\mathcal{Y}_{s}}$ well
trained on the source data $D_{s}$ and an unlabeled target domain $D_{t}$ =
$\\{x^{i}\\}_{i=1}^{n_{t}}$ from different distributions. In the multi-
classification task, ${\mathcal{Y}}$ $\in$ $\\{1,...,K\\}$, and $K$ represents
the number of classes. $m$ target samples of the target domain are selected
for querying, $m$ $\ll$ $n_{t}$. Further, we denote the small selected labeled
target data by $D_{t}^{L}$ and the unlabeled target data by $D_{t}^{U}$. Our
method is based on a two-part network: a feature extractor $f$, and a
classifier $g$. The feature learned by the feature extractor is denoted as
$z_{i}$ = $f(x_{i})$. The goal of ASFDA is to train a target model $h_{t}$ =
$g_{t}(f_{t}(x))$ with satisfactory performance while querying as few as
labeled data $D_{t}^{L}$ from $D_{t}$ as possible.

###  2.1 Minimum Happy Points

We find the most informative samples for ASFDA are minimum happy points (MH
points) that have the characteristics of neighbor-chaotic, individual-
different, and target-like at the same time.

(a) initial pseudo-labels

(b) BVSB [21]

(c) MHPL

Figure 2:  Feature visualization for the source model with 5% actively labeled
target data on the Cl$\rightarrow$Pr task. Different colors in (a) represent
different classes of pseudo-labels by clustering, where blue blocks include
easily-adaptive source-like samples with label-clean neighbors, and red blocks
include the hardly-adaptive target-like samples with label-chaotic neighbors.
In (b) and (c), the dark green indicates that the pseudo-label is consistent
with the true label, and light blue indicates the opposite. The red stars
indicate the selected samples based on BVSB and our MHPL, respectively.

(1) Neighbor-chaotic samples among label-chaotic neighbors are uncertain,
meanwhile, their neighbors are heavily uncertain. Annotation of these samples
can not only achieve self-correction but also guide the learning of their
confusing neighbors, bringing in great performance gains. Further, the sample
with chaotic neighbors lies potentially in the boundary of multiple
categories, so learning the sample would promote accurate classification
boundaries. As shown in Fig. 2, MHPL could explore more samples (red blocks)
with label-chaotic neighbors than BVSB [21] because the strategies based
solely on model uncertainty can only measure the self-uncertainty of samples.

(2) Individual-different samples are dissimilar to each other, representing
sample diversity. By ensuring the individual difference, we not only avoid
similar instances with high redundancy but also obtain the samples that are
representative of the entire target distribution and ensure class balance. It
is clear to see in Fig. 2 that both BVSB and MHPL can ensure sample diversity,
and MHPL can also avoid selecting outliers, but existing strategies cannot.

(3) Target-like samples are biased to the target distribution. Following the
covariate shift assumption [33], the target data $D_{t}$ can be divided into
easily-adaptive source-like instances $D_{t}^{s}$ and hardly-adaptive target-
like instances $D_{t}^{t}$. Selecting samples from $D_{t}^{s}$ would lead to
limited generalization gain on the target domain since $D_{t}^{s}$ represents
the source distribution. It is clear to see in Fig. 2 that the samples
selected by BVSB have more chances to fall in the blue boxes, which represent
the source-like data, while the samples selected by our MHPL are mostly
target-like.

###  2.2 Minimum Happy Points Exploration

We propose the neighbor ambient uncertainty, neighbor diversity relaxation,
and one-shot querying to explore the neighbor-chaotic, individual-different,
and target-like samples.

Neighbor Ambient Uncertainty. Instead of relying heavily on the self-
uncertainty of target samples, the neighbor ambient uncertainty evaluates
target sample uncertainty by measuring the neighbor environments they are in.
Given a target sample $x$, its neighbor ambient uncertainty $\text{NAU}(x)$ is
defined by multiplying the neighbor purity $\text{NP}(x)$ and the neighbor
affinity $\text{NA}(x)$:

$\displaystyle\text{NAU}(x)=\text{NP}(x)*\text{NA}(x)\,.$ (1)

Note that $\text{NP}(x)\geq 0$ and $0\leq\text{NA}(x)\leq 1$. A sample with a
high value of NAU would have noisy and close neighbors, satisfying the
neighbor-chaotic characteristic of MH points. As shown in Fig. 1, the sample
$a$ has more chances to be selected as an MH point. Next, we first define the
concept of neighbor and then introduce the neighbor purity and the neighbor
affinity, orderly.

Based on the feature space $\mathcal{F}=\\{z_{1},z_{2},..,z_{n_{t}}\\}$ of
target data and the cosine similarity measurement [7], we define the samples
that are close to $x$ in the feature space as the neighbors of $x$:

$\displaystyle S_{N}(x)=\\{N_{1},N_{2},...,N_{q}\\},\text{and}\
S_{N}^{Y}(x)=\\{Y_{N_{1}},Y_{N_{2}},...,Y_{N_{q}}\\}\,,$ (2)

where $q$ denotes the number of neighbors, $S_{N}$ represents the neighbor
sample space of $x$, and $S_{N}^{Y}$ represents the neighbor pseudo-label
space of $x$. The pseudo-labels of target samples are obtained by the standard
deep clustering [7]. $Y_{N_{q}}$ represents the pseudo-label of the neighbor
sample $N_{q}$.

Intuitively, if a sample has the same pseudo-label as its most neighbors, its
uncertainty is low. In contrast, the sample in label-chaotic neighbor
environments has large uncertainty. Based on this intuitive insight, we design
the neighbor purity as follows.

Neighbor purity describes the chaotic degree of neighbor labels around a
sample. For calculating the neighbor purity $\text{NP}(x)$, we first establish
the neighbor class probability distribution space $S_{N}^{p}(x)$:

$\displaystyle S_{N}^{p}(x)=\\{p_{1},p_{2},p_{3},...,p_{k}\\}\,,$ (3)

where $p_{k}$ represents the proportion of samples labeled as class $k$ in $q$
neighbors. Take samples $a$ and $b$ in the Fig. 1 as examples,
$S_{N}^{p}(a)=[0.3,0.4,0.3]$, $S_{N}^{p}(b)=[0.2,0.1,0.7]$.

Then the neighbor purity of sample $x$ is measured by the neighbor entropy:

$\displaystyle{\text{NP}}(x)=-{\sum}_{k=1}^{K}p_{k}\text{log}p_{k},\
\text{\text{s.t.}}\ p_{k}\in S_{N}^{p}(x)\,,$ (4)

The sample with a high value of NP has label-noisy neighbors, and such a
sample is easy to be selected as an uncertain one. However, only using
neighbor purity would select outliers, resulting in biased estimation, so we
propose the neighbor affinity as a strong complement to neighbor purity.

Neighbor affinity describes how close a sample is to its neighbors. To measure
the close degree, we first define the neighbor similarity space $S_{N}^{s}$
for each sample by

$\displaystyle S_{N}^{s}(x)=\\{S_{N_{1}},S_{N_{2}},...,S_{N_{p}}\\}\,,$ (5)

where $S_{N_{1}}$ represents the cosine similarity between $x$ and its
neighbor $N_{1}$. Further, the neighbor affinity is measured by the average
similarity between $x$ and its neighbors:

$\displaystyle\text{NA}(x)=\frac{S_{N_{1}}+S_{N_{2}}+...+S_{N_{q}}}{q},\
\text{s.t.}\ S_{N_{q}}\in S_{N}^{s}(x)\,.$ (6)

The more low is the neighbor affinity, the more far is the sample to its
neighbors, and such that it is more likely to be outliers because outliers do
not have compact neighbors.

The advantages of NAU are two-fold. Firstly, previous SFDA works [34, 35] have
proved that the target samples’ deep features extracted by the source model
can still form clean clusters under domain shift to a certain extent. Fig. 2
shows the visualization of target features and further verifies this
phenomenon. NAU measures the sample uncertainty from its neighbors that would
be better than the strategies that rely solely on the miscalibrated
uncertainty of model predictions [19, 20, 21]. Secondly, NAU has the
advantages of ensemble learning [36] and lets numerous neighbor crowds
calibrate the individual uncertainty and improve the fault tolerance of target
sample selection.

Algorithm 1 Neighbor Diversity Relaxation.

Require: target data $D_{t}$, labeled target set $D_{t}^{L}=\emptyset$, $m$;

1: Sort the samples in $D_{t}$ in reverse order by the value of NAU, and Let
$i$ = 0;

2: while $\text{Length}(D_{t}^{L})$ $\leq$ $m$ do

3: Select the candidate sample $x_{i}$ and obtain its nearest neighbor
$N_{x_{i}}^{o}$,

4: if $N_{x_{i}}^{o}$ $\notin$ $D_{t}^{L}$ then $D_{t}^{L}$ $\leftarrow$
$x_{i}$,

5: else Skip the selection of sample $x_{i}$,

6: end while

Neighbor Diversity Relaxation. We propose the neighbor diversity relaxation
(NDR) to guarantee individual differences in candidate samples. In existing
methods [26, 27], sample diversity is ensured by choosing samples closest to
cluster centers, which ignores domain shift and selects uninformative and
source-like samples. Instead, NDR guarantees sample diversity by performing
nearest neighbor relaxation on candidate samples with high neighbor ambient
uncertainty. The main procedure of NDR is shown in algorithm 1. For a
candidate sample $x$ with a high $\text{NAU}(x)$, we first evaluate its
nearest neighbor sample $N_{x_{i}}^{o}$ before putting it in $D_{t}^{L}$. If
$N_{x_{i}}^{o}$ has been put in the $D_{t}^{L}$, then $x_{i}$ has no chance to
be selected for expert annotation. In this way, those redundant samples with
similar features would have few chances to be selected, ensuring the
individual difference and sample diversity.

One-shot Querying. In order to explore target-like samples, we perform one-
shot querying, selecting samples at once according to the raw source model.
The reasons are two-fold. For one thing, without the source data, we can
measure the domain discrepancy by observing how the target data behaves on the
untrained source model. The samples that can be easily classified by the
source model are commonly source-like instances, which mainly occur in label-
clean clusters [34, 37]. Otherwise, the samples with label-chaotic neighbors
are severely misclassified and target-like instances which can be effectively
explored by our proposed criterion. For another, as the training model is
biased toward the target domain gradually, it is difficult to determine if the
samples are source-like or target-like.

###  2.3 Minimum Happy Points Exploitation

Neighbor Focal Loss is designed to make the model focus more on chaotic and
representative MH points to ensure the generalization of the target domain. NF
loss is inspired by, but not the same as, focal loss [38]. For one side, focal
loss aims to alleviate the overfitting problem of cross-entropy loss for
imbalanced object detection datasets, while NF loss is proposed to fully
exploit the MH points. For another, the focal loss assigns the weights from
the miscalibrated model’s unreliable predictions to the hard samples, while NF
loss assigns the neighbor purity and a larger weight $\alpha$ to the MH
points:

$\displaystyle
L_{NF}(f_{t};{\mathcal{D}_{t}^{L}})=-{\mathbb{E}}_{x\in{\mathcal{D}}_{t}^{L}}{\sum}_{k=1}^{K}\alpha\text{NP}(x)l_{k}\text{log}({\delta}_{k}(f_{t}(x)))\,,$
(7)

where $\delta$ is the softmax function, and $l_{k}$ is ‘1’ for the ground-
truth class. Focusing on informative MH points can increase the importance of
active labeled samples and further avoid the model to overfit on the samples
with wrong pseudo-labels. And assigning neighbor purity could penalty the
mistakes on neighbor-chaotic samples that are more important than ordinary
samples for generalization.

Meanwhile, the NF loss assigns a smaller weight $\beta$ to the rest target
samples $D_{t}^{U}$. NF loss uses their pseudo-labels from clustering and
makes the model do not tilt toward them during training:

$\displaystyle
L_{NF}(f_{t};{\mathcal{D}_{t}^{U}})=-{\mathbb{E}}_{x\in{\mathcal{D}}_{t}^{U}}{\sum}_{k=1}^{K}\beta
l_{k}\text{log}({\delta}_{k}(f_{t}(x)))\,.$ (8)

In summary, NF loss pays the more attention to MH points and the low attention
to non-MH points:

$\displaystyle
L_{NF}(f_{t};{\mathcal{D}_{t}})=L_{NF}(f_{t};{\mathcal{D}_{t}^{L}})+L_{NF}(f_{t};{\mathcal{D}_{t}^{U}})\,.$
(9)

Entropy loss and KL divergence are introduced to guarantee the unambiguous and
balanced classes [39, 40], which has been widely used in clustering [41, 42],
and several DA works [7, 37, 43, 44]:

$\displaystyle
L_{ent}(f_{t};{\mathcal{D}_{t}})=-{\mathbb{E}}_{x\in{\mathcal{D}}_{t}}{\sum}_{k=1}^{K}{\delta}_{k}(f_{t}(x))\text{log}({\delta}_{k}(f_{t}(x)))\,,$
(10) $\displaystyle
L_{div}(f_{t};{\mathcal{D}_{t}})=-{\mathbb{E}}_{x\in{\mathcal{D}}_{t}}{\sum}_{k=1}^{K}\text{KL}(\hat{p}_{k}||q_{k}),\text{and}\
q_{\\{k=1,...,K\\}}=\frac{1}{K}\,,$

where $\hat{p}_{k}=\frac{1}{n_{t}}\sum\delta(f_{t}(x))^{(k)}$ is the mean
prediction of the $k$-th target data.

Algorithm 2 Minimum Happy Points Learning.

Require: source model $h_{s}$, target data $D_{t}$, labeled target set
$D_{t}^{L}=\emptyset$, maximum number of epochs $E$, trade-off parameters
$\alpha$, $\beta$, $m$;

1: Initialize the target model $h_{t}$ with $h_{s}$ and obtain pseudo-labels
of target data based clustering;

2: $\forall x\in D_{t}$, compute $\text{NP}(x)$ and $\text{NA}(x)$ to serve as
$\text{NAU}(x)$ with Eq. (1 \- 6);

3: $D_{t}^{L}$ $\leftarrow$ select $m$ samples with NAU and algorithm 1;

4: Let epoch $=1$, $iter\\_num=0$;

5: while epoch $\leq$ $E$ do

6: Obtain pseudo-labels of $D_{t}^{U}$ from deep clustering;

7: while $iter\\_num<n_{b}$ do

8: Calculate neighbor focal loss, entropy loss and KL divergence with Eq. (7\-
10);

9: Update model $f_{t}$ via Eq. (11).

10: end while

11: end while

To conclude, the workflow of MHPL is illustrated in Algorithm 2. The final
objective can be stated as

$\displaystyle L=L_{NF}+L_{ent}+L_{div}\,.$ (11)

##  3 Experiments

Benchmarks. We adopt three benchmark datasets. Office-31 [45] is a small-scale
DA dataset with 31 classes and 3 domains: Amazon, Dslr, and Webcom. Office-
Home [12] is a more challenging DA dataset with 65 classes and 4 domains:
Artistic images, Clip Art images, Product images, and Real-world images.
VisDA-2017 [46] is a large simulation-to-real dataset with 12 classes.

Baselines. We construct three types of methods as baselines. (i) Source free
domain adaptation: SFDA [47], SHOT [7], MA [8], HCL [44], CPGA [48], NRC [34],
and A2Net [11]. (ii) Active learning: (1) Random: random samples. (2) Entropy
[20]: samples with highest entropy. (3) BVSB [21]: samples with the smallest
difference between top-2 class probabilities. (4) LC (least confidence) [19]:
samples with smallest probability. (5) CoreSet [28]: samples selected by a
set-cover problem. (6) CTC: samples that are closest to clustering centers.
(7) BADGE [31]: construct diverse batches by running KMeans++ [49]. (iii)
Active DA: AADA [22], TQS [23], CLUE [26], and EADA [50].

Implementation. We report the main results upon the backbone of ResNet-50 [51]
for Office-Home and Office-31, as well as ResNet-101 for VisDA-2017. We adopt
the same network architecture as SHOT [7]. We conduct SGD with momentum 0.9
and batch size of 64 for all datasets. The learning rate is set to 1e-2 for
Office-31 and Office-Home, and 1e-3 for VisDA-2017. For the number of
neighbors $q$, we set nine for Office-31, 20 for Office-Home, and five for
VisDA-2017. Further, we set $\alpha$ = 3, $\beta$ = 0.3 for all datasets. The
more implementation details are in Appendix C. The full results of other
backbones and the sensitivity analysis of the hyperparameters are in Appendix
D.

###  3.1 Main Results

Table 2: Accuracy (%) on Office-Home (ResNet50) under different settings with
5% labeled target samples ("SF" in tables denotes source data free, _i.e.,_
adaptation without source data).

Categories | Method | SF | Ar$\rightarrow$Cl | Ar$\rightarrow$Pr | Ar$\rightarrow$Re | Cl$\rightarrow$Ar | Cl$\rightarrow$Pr | Cl$\rightarrow$Re | Pr$\rightarrow$Ar | Pr$\rightarrow$Cl | Pr$\rightarrow$Re | Re$\rightarrow$Ar | Re$\rightarrow$Cl | Re$\rightarrow$Pr | Avg
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
None | Source-only | $\checkmark$ | 45.5 | 68.4 | 75.2 | 53.4 | 63.7 | 65.6 | 52.4 | 41.0 | 73.6 | 65.9 | 46.3 | 78.2 | 60.8
SFDA | SFDA [47] | $\checkmark$ | 48.5 | 71.3 | 75.6 | 63.9 | 69.0 | 72.1 | 62.4 | 43.5 | 76.0 | 70.4 | 50.1 | 76.1 | 64.9
CPGA [48] | $\checkmark$ | 59.3 | 78.1 | 79.8 | 65.4 | 75.5 | 76.4 | 65.7 | 58.0 | 81.0 | 72.0 | 64.4 | 83.3 | 71.6
SHOT [7] | $\checkmark$ | 57.1 | 78.1 | 81.5 | 68.0 | 78.2 | 78.1 | 67.4 | 54.9 | 82.2 | 73.3 | 58.8 | 84.3 | 71.8
NRC [34] | $\checkmark$ | 57.7 | 80.3 | 82.0 | 68.1 | 79.8 | 78.6 | 65.3 | 56.4 | 83.0 | 71.0 | 58.6 | 85.6 | 72.2
| A2Net [11] | $\checkmark$ | 58.4 | 79.0 | 82.4 | 67.5 | 79.3 | 78.9 | 68.0 | 56.2 | 82.9 | 74.1 | 60.5 | 85.0 | 72.8
Active DA | AADA [22] | $\times$ | 56.6 | 78.1 | 79.0 | 58.5 | 72.7 | 71.0 | 60.1 | 53.1 | 77.0 | 70.6 | 57.0 | 84.5 | 68.3
TQS [23] | $\times$ | 58.6 | 81.1 | 81.5 | 61.1 | 76.1 | 73.3 | 61.2 | 54.7 | 79.7 | 73.4 | 58.9 | 86.3 | 72.5
Clue [26] | $\times$ | 58.0 | 79.3 | 80.9 | 68.8 | 77.5 | 76.7 | 66.3 | 57.9 | 81.4 | 75.6 | 60.8 | 86.3 | 72.5
EADA [50] | $\times$ | 63.6 | 84.4 | 83.5 | 70.7 | 83.7 | 80.5 | 73.0 | 63.5 | 85.2 | 79.4 | 65.4 | 88.6 | 76.7
ASFDA | Base | $\checkmark$ | 57.2 | 78.5 | 81.5 | 68.5 | 79.1 | 78.6 | 67.5 | 56.3 | 82.2 | 73.7 | 58.5 | 83.6 | 72.1
Random | $\checkmark$ | 63.8 | 81.4 | 83.9 | 71.3 | 82.2 | 81.4 | 68.8 | 62.4 | 83.3 | 76.1 | 63.8 | 85.8 | 75.2
CTC | $\checkmark$ | 60.8 | 78.7 | 82.2 | 69.3 | 79.2 | 79.8 | 68.6 | 59.4 | 82.2 | 74.6 | 61.7 | 84.4 | 73.4
CoreSet [28] | $\checkmark$ | 61.8 | 81.8 | 83.3 | 71.1 | 82.9 | 81.6 | 70.7 | 60.5 | 84.7 | 76.1 | 61.7 | 86.1 | 75.2
BADGE [31] | $\checkmark$ | 62.4 | 82.7 | 83.9 | 71.5 | 83.0 | 81.8 | 71.2 | 62.7 | 84.6 | 76.2 | 62.9 | 87.8 | 75.9
Entropy [20] | $\checkmark$ | 65.0 | 84.0 | 85.9 | 71.8 | 83.8 | 82.6 | 70.7 | 63.8 | 85.1 | 77.8 | 64.1 | 88.1 | 76.9
BVSB [21] | $\checkmark$ | 64.8 | 84.4 | 85.5 | 72.0 | 83.2 | 83.4 | 70.4 | 63.9 | 85.0 | 77.5 | 65.0 | 88.1 | 76.9
LC [19] | $\checkmark$ | 65.0 | 84.0 | 85.4 | 72.1 | 83.0 | 82.8 | 71.0 | 64.9 | 85.1 | 78.0 | 64.8 | 87.9 | 77.0
| MHPL | $\checkmark$ | 67.8 | 85.9 | 86.1 | 72.8 | 87.1 | 84.5 | 72.5 | 67.6 | 86.2 | 79.7 | 68.6 | 89.9 | 79.1

Table 3: Accuracy (%) on VisDA-2017 (ResNet-101) and Office-31 (ResNet-50)
with 5% labeled target samples ("SF" in tables denotes source data free,
_i.e.,_ adaptation without source data).

Categories | Method | SF | VisDA-2017 | A$\rightarrow$D | A$\rightarrow$W | D$\rightarrow$A | D$\rightarrow$W | W$\rightarrow$A | W$\rightarrow$D | Avg
---|---|---|---|---|---|---|---|---|---|---
None | Source-only | $\checkmark$ | 50.0 | 79.3 | 76.7 | 61.8 | 96.7 | 63.0 | 98.8 | 79.4
SFDA | SHOT [7] | $\checkmark$ | 82.4 | 94.0 | 90.1 | 74.7 | 98.4 | 74.3 | 99.9 | 88.6
CPGA [48] | $\checkmark$ | 86.0 | 94.4 | 94.1 | 76.0 | 98.4 | 76.6 | 99.8 | 89.9
HCL [44] | $\checkmark$ | 83.5 | 90.8 | 91.3 | 72.7 | 98.2 | 72.7 | 100.0 | 87.6
NRC [34] | $\checkmark$ | 85.9 | 96.0 | 90.8 | 75.3 | 99.0 | 75.0 | 100.0 | 89.4
| A2Net [11] | $\checkmark$ | 84.3 | 94.5 | 94.0 | 76.7 | 99.2 | 76.1 | 100.0 | 90.1
Active DA | AADA [22] | $\times$ | - | 89.2 | 87.3 | 78.2 | 99.5 | 78.7 | 100.0 | 88.8
TQS [23] | $\times$ | - | 92.8 | 92.2 | 80.6 | 100.0 | 80.4 | 100.0 | 91.1
Clue [26] | $\times$ | - | 92.0 | 87.3 | 79.0 | 99.2 | 79.6 | 99.8 | 89.5
EADA [50] | $\times$ | - | 97.7 | 96.6 | 82.1 | 100.0 | 82.8 | 100.0 | 93.2
ASFDA | Base | $\checkmark$ | 83.3 | 93.6 | 90.6 | 75.5 | 98.9 | 75.4 | 99.8 | 89.0
Random | $\checkmark$ | 85.1 | 94.2 | 94.8 | 78.2 | 98.9 | 77.4 | 99.6 | 90.5
CTC | $\checkmark$ | 84.0 | 94.0 | 90.1 | 77.1 | 98.6 | 76.1 | 99.8 | 89.3
CoreSet [28] | $\checkmark$ | 85.9 | 93.6 | 91.8 | 78.6 | 99.1 | 78.0 | 99.8 | 90.2
BADGE [31] | $\checkmark$ | 86.0 | 95.2 | 91.5 | 78.2 | 99.1 | 78.2 | 99.8 | 90.3
Entropy [20] | $\checkmark$ | 86.7 | 96.0 | 94.0 | 80.5 | 99.0 | 79.3 | 100.0 | 91.5
BVSB[21] | $\checkmark$ | 86.5 | 95.4 | 94.0 | 80.3 | 99.1 | 80.1 | 100.0 | 91.5
LC [19] | $\checkmark$ | 86.7 | 96.2 | 95.2 | 80.1 | 99.1 | 79.7 | 100.0 | 91.7
| MHPL | $\checkmark$ | 91.2 | 98.6 | 96.7 | 82.3 | 99.3 | 83.9 | 100.0 | 93.5

Table 4: Accuracy (%) on VisDA-2017 (ResNet-50) with 5% labeled target
samples.

Method | AADA [22] | TQS [23] | Clue[26] | EADA [50] | MHPL
---|---|---|---|---|---
Acc ($\%$) | 80.8 $\pm$ 0.4 | 83.1 $\pm$ 0.4 | 85.2 $\pm$ 0.4 | 88.3 $\pm$ 0.1 | 89.5 $\pm$ 0.1

Results on object recognition. Our method MHPL significantly outperforms
existing SFDA methods, successfully breaking through the effect bottleneck of
SFDA with limited annotations. Firstly, MHPL achieves the state-of-the-art on
the ASFDA setting. As shown in Table 2, MHPL surpasses on average by 6.3% over
the state-of-the-art SFDA method A2Net on Office-Home with only 5% of labeled
target data. Especially in challenging tasks, MHPL achieves significant
improvements, _e.g.,_ the accuracy of MHPL is 11.4% and 9.5% higher than that
of A2Net on tasks Ar$\rightarrow$Cl and Pr$\rightarrow$Cl, respectively. As
shown in Table 3, the accuracy of MHPL also remarkably outperforms all SFDA
baselines on VisDA-2017 and Office-31. Secondly, MHPL can better explore and
exploit MH points and maximize the performance gains compared to existing
active learning baselines. As shown in Tables 2, 3, MHPL outperforms all
existing active learning strategies on all tasks, especially for challenging
tasks. Finally, as shown in Tables 2, 3, 4, it is very interesting that MHPL
without accessing the source data can outperform all active domain adaptation
methods, showing the practical significance of ASFDA.

###  3.2 Analysis

Ablation on neighbor ambient uncertainty and neighbor diversity relaxation
(NDR). To investigate the efficacy of key components of our criterion for
selecting minimum happy points, we firstly conduct an ablation study with the
following variants on Ar$\rightarrow$Cl in various sample selection ratios
from 1% to 10%: (i) MHPL w/o neighbor purity; (ii) MHPL w/o neighbor affinity;
(iii) MHPL w/o NDR; (iv) MHPL. As shown in Fig. 3 (a), the full method
outperforms other variants in various selection ratios, indicating the
necessity of each key component. In addition, as shown in Fig. 3 (b) and (c),
the samples selected with NDR have more diversity among the high overlapping
regions.

(a) Ar$\rightarrow$Cl

(b) Ar$\rightarrow$Cl without NDR

(c) Ar$\rightarrow$Cl

(d) Cl$\rightarrow$Pr

(e) Pr$\rightarrow$Cl

(f) Accuracy on neighbors

Figure 3: Ablation studies on the selection criterion. (a) represents the
accuracy of key components from NAU and NDR at different selection ratios. (b)
and (c) represent the feature visualization with and without NDR,
respectively. They demonstrate the consistent advantages of NDR on
representation learning as Fig. 2. (d) and (e) show the model benefits brought
by samples selected at different epochs. (f) shows the accuracy of initial
neighbors before and after training.

Ablation on one-shot querying. In order to verify the effectiveness of one-
shot querying, we compare the effect with samples selected at different epochs
of model training. As shown in Fig. 3 (d) and (e), the abscissa indicates the
epoch of model training for sample selection, where epoch zero represents the
source model. It is observed that the samples selected by the source model
obtain larger model performance gains on Cl$\rightarrow$Pr and
Pr$\rightarrow$Cl. Additionally, when sample selection is done on larger
epochs, the model performance is significantly degraded, as the trained model
cannot properly reflect the distribution discrepancy, resulting in the
difficulty to explore the target-like samples.

Effect on correcting neighbors. We further analyze the role of selected
samples in correcting the pseudo-labels of their neighbors and compare MHPL
with the state-of-the-art LC [19]. Fig. 3 (f) shows the accuracy of the
selected sample’s initial neighbors before (dark color) and after training
(light color). The lighter the color, the better the performance. It is
obvious that MHPL is better at correcting neighbors than LC in all tasks,
especially in challenging tasks Ar$\rightarrow$Cl, Pr$\rightarrow$Cl, and
Re$\rightarrow$Cl. After training, the accuracy of initial neighbors improves
more significantly than it did before training, demonstrating that selected
samples have better guidance on their confusing neighbors.

Ablation on loss functions. To demonstrate the effect of loss functions in Eq.
11, we perform ablation studies on Ar$\rightarrow$Cl at various sample
selection ratios. As shown in Fig. 4 (a), the effect of NF loss increases as
more active samples are chosen. Meanwhile, entropy loss and KL divergence also
promote model learning. After removing NF loss (yellow), the accuracy of
different selection ratios remains the same because the selected samples do
not participate in training and the model is always trained using entropy loss
and KL divergence. To verify the versatility of NF loss, we integrate it into
other sample selection strategies. Specially, we assign the weight,
$\alpha\text{NP}$, to the samples selected by entropy, and the value of NP is
calculated by entropy [20]. As shown in Fig. 4 (b), the accuracy with
$\alpha\text{NP}$ (green) is always better than entropy with standard cross-
entropy loss (blue) on Ar$\rightarrow$Cl, while their results are still lower
than our MHPL (red). Moreover, after removing the weight $\alpha\text{NP}$ of
NF Loss on Ar$\rightarrow$Cl and Re$\rightarrow$Cl, the accuracy becomes
lower, as shown in Fig. 4 (c) and (d).

(a) Ar$\rightarrow$Cl

(b) Ar$\rightarrow$Cl

(c) Ar$\rightarrow$Cl

(d) Re$\rightarrow$Cl

Figure 4:  Ablation studies on loss functions.

##  4 Related Work

Domain Adaptation (DA) aims to transfer the knowledge from amounts of labeled
source data to the unlabeled target domain. Most DA works attempt to align the
feature distributions across domains with moment matching [52, 53, 54] or
adversarial learning [55, 56, 57]. Recently, semi-supervised DA [58, 59, 60]
and few-shot DA [61] have verified that utilizing a few labeled target data is
helpful to performance benefits. However, the success of the above methods
depends on amounts of annotated source data, which is unrealistic and
unpractical in privacy-persevering scenarios.

Source Free Domain Adaptation (SFDA) is considered in the literature recently
for relaxing the dependence on the source data. Existing SFDA works have made
a great effort based on two categories of methods: _i.e.,_ source model
exploration and data generation, while they face the _effect bottleneck_ with
limited performance gains. The works based on source model exploration, SHOT
[7], HCL [44], NRC [34], and A2Net [11], aim to utilize the potential
distribution information stored in the given model. However, these methods
cannot guarantee the positive transfer with limited information, without more
than 1% improvement over the initial SHOT [7] on Office-Home [12]. Another
line of work based on data generation, MA [8] and CPGA [48], intends to
generate the source or target data to obtain supervised information, costing
amounts of time and resources. But these works also perform worse in practice
as the real samples or correct feature prototypes are difficult to generate.
Hypothesis Transfer Learning (HTL) [62, 63] aims to transfer the knowledge
using learned source hypotheses and limited labeled target data without the
source data. However, naively selecting labeled target data would be
inefficient. In this paper, we introduce the active source free domain
adaptation, aiming to break through the _effect bottleneck_ with a few
actively labeled target data.

Active Learning (AL) aims to obtain a satisfactory model at the cost of a
limited annotation budget. Existing AL methods are mainly based on two
categories: (1) Uncertainty, _e.g.,_ entropy [20] and BVSB [21]; (2)
Diversity, _e.g.,_ CoreSet [28]. However, neither method is suitable for
domain adaptation under domain shift. Recently, Active DA introduces active
learning to domain adaptation. AADA [22] leverages the domain discriminator to
qualify the target-like samples. TQS [23] utilizes ensemble learning to select
informative samples by transferable committee, uncertainty, and domainness.
Clue [26] combines uncertainty and diversity to select samples. EADA [32]
determines the key samples with free energy [64]. Due to domain shift and
unavailable source data, it is impossible to directly apply the existing
criteria of AL and Active DA to ASFDA.

##  5 Conclusion

In this paper, we propose a new setting, active source free domain adaptation
(ASFDA), for the first time. ASFDA maximizes the model benefits with the
minimum cost of data annotation under domain shift in the privacy-preserving
scenarios. We firstly find the essential minimum happy (MH) points that
satisfy the properties of neighbor-chaotic, individual-different, and target-
like. We then design the minimum happy points learning to explore and exploit
the MH points well. Extensive experiments verify that we lay a promising
baseline of ASFDA for further works.

## References

  * [1] Jian Wang, Yongcheng Luo, Yan Zhao, and Jiajin Le.  A survey on privacy preserving data mining.  In 2009 First International Workshop on Database Technology and Applications, pages 111–114. IEEE, 2009.
  * [2] Inda Kreso, Amra Kapo, and Lejla Turulja.  Data mining privacy preserving: Research agenda.  Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 11(1):e1392, 2021.
  * [3] Chandra Thapa and Seyit Camtepe.  Precision health data: Requirements, challenges and existing techniques for data security and privacy.  Computers in biology and medicine, 129:104130, 2021.
  * [4] Dongpo Zhang.  Big data security and privacy protection.  In 8th International Conference on Management and Computer Science (ICMCS 2018), volume 77, pages 275–278. Atlantis Press, 2018.
  * [5] Serban Stan and Mohammad Rostami.  Privacy preserving domain adaptation for semantic segmentation of medical images.  arXiv preprint arXiv:2101.00522, 2021.
  * [6] Xianfeng Li, Weijie Chen, Di Xie, Shicai Yang, Peng Yuan, Shiliang Pu, and Yueting Zhuang.  A free lunch for unsupervised domain adaptive object detection without source data.  In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 8474–8481. AAAI Press, 2021.
  * [7] Jian Liang, Dapeng Hu, and Jiashi Feng.  Do we really need to access the source data ? source hypothesis transfer for unsupervised domain adaptation.  In International Conference on Machine Learning, pages 6028–6039. PMLR, 2020.
  * [8] Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Si Wu.  Model adaptation: Unsupervised domain adaptation without source data.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9641–9650, 2020.
  * [9] Marvin Klingner, Jan-Aike Termöhlen, Jacob Ritterbach, and Tim Fingscheidt.  Unsupervised batchnorm adaptation (UBNA): A domain adaptation method for semantic segmentation without using source domain representations.  In IEEE/CVF Winter Conference on Applications of Computer Vision Workshops, WACV - Workshops, Waikoloa, HI, USA, January 4-8, 2022, pages 210–220. IEEE, 2022.
  * [10] Yuang Liu, Wei Zhang, and Jun Wang.  Source-free domain adaptation for semantic segmentation.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 1215–1224. Computer Vision Foundation / IEEE, 2021.
  * [11] Haifeng Xia, Handong Zhao, and Zhengming Ding.  Adaptive adversarial network for source-free domain adaptation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9010–9019, 2021.
  * [12] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan.  Deep hashing network for unsupervised domain adaptation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5018–5027, 2017.
  * [13] Zhongyi Han, Benzheng Wei, Yanfei Hong, Tianyang Li, Jinyu Cong, Xue Zhu, Haifeng Wei, and Wei Zhang.  Accurate screening of COVID-19 using attention-based deep 3d multiple instance learning.  IEEE Trans. Medical Imaging, 39(8):2584–2594, 2020.
  * [14] Zhongyi Han, Rundong He, Tianyang Li, Benzheng Wei, Jian Wang, and Yilong Yin.  Semi-supervised screening of COVID-19 from positive and unlabeled data with constraint non-negative risk estimator.  In Aasa Feragen, Stefan Sommer, Julia A. Schnabel, and Mads Nielsen, editors, Information Processing in Medical Imaging - 27th International Conference, IPMI 2021, Virtual Event, June 28-June 30, 2021, Proceedings, volume 12729 of Lecture Notes in Computer Science, pages 611–623. Springer, 2021.
  * [15] Navjot Singh, Håkon Hukkelås, and Frank Lindseth.  Deep active learning for autonomous perception.  In NIKT: Norsk IKT-konferanse for forskning og utdanning 2020. Bibsys Open Journal Systems, 2020.
  * [16] Mahesh M Dhananjaya, Varun Ravi Kumar, and Senthil Yogamani.  Weather and light level classification for autonomous driving: Dataset, baseline and active learning.  In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pages 2816–2821. IEEE, 2021.
  * [17] G. Riccardi and D. Hakkani-Tur.  Active learning: theory and applications to automatic speech recognition.  IEEE Transactions on Speech and Audio Processing, 13(4):504–511, 2005.
  * [18] Yang Yuan, Soo-Whan Chung, and Hong-Goo Kang.  Gradient-based active learning query strategy for end-to-end speech recognition.  In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2832–2836. IEEE, 2019.
  * [19] Tao He, Xiaoming Jin, Guiguang Ding, Lan Yi, and Chenggang Yan.  Towards better uncertainty sampling: Active learning with multiple views for deep convolutional neural network.  In 2019 IEEE International Conference on Multimedia and Expo (ICME), pages 1360–1365. IEEE, 2019.
  * [20] Dan Wang and Yi Shang.  A new active labeling method for deep learning.  In 2014 International Joint Conference on Neural Networks, IJCNN 2014, Beijing, China, July 6-11, 2014, pages 112–119. IEEE, 2014.
  * [21] Ajay J. Joshi, Fatih Porikli, and Nikolaos Papanikolopoulos.  Multi-class active learning for image classification.  In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pages 2372–2379. IEEE Computer Society, 2009.
  * [22] Jong-Chyi Su, Yi-Hsuan Tsai, Kihyuk Sohn, Buyu Liu, Subhransu Maji, and Manmohan Chandraker.  Active adversarial domain adaptation.  In IEEE Winter Conference on Applications of Computer Vision, WACV 2020, Snowmass Village, CO, USA, March 1-5, 2020, pages 728–737. IEEE, 2020.
  * [23] Bo Fu, Zhangjie Cao, Jianmin Wang, and Mingsheng Long.  Transferable query selection for active domain adaptation.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 7272–7281. Computer Vision Foundation / IEEE, 2021.
  * [24] François Deheeger, Mathilde MOUGEOT, Nicolas Vayatis, et al.  Discrepancy-based active learning for domain adaptation.  In International Conference on Learning Representations, 2021.
  * [25] Jasper Snoek, Yaniv Ovadia, Emily Fertig, Balaji Lakshminarayanan, Sebastian Nowozin, D. Sculley, Joshua V. Dillon, Jie Ren, and Zachary Nado.  Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift.  In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 13969–13980, 2019.
  * [26] Viraj Prabhu, Arjun Chandrasekaran, Kate Saenko, and Judy Hoffman.  Active domain adaptation via clustering uncertainty-weighted embeddings.  In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 8485–8494. IEEE, 2021.
  * [27] Anurag Singh, Naren Doraiswamy, Sawa Takamuku, Megh Bhalerao, Titir Dutta, Soma Biswas, Aditya Chepuri, Balasubramanian Vengatesan, and Naotake Natori.  Improving semi-supervised domain adaptation using effective target selection and semantics.  In IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2021, virtual, June 19-25, 2021, pages 2709–2718. Computer Vision Foundation / IEEE, 2021.
  * [28] Ozan Sener and Silvio Savarese.  Active learning for convolutional neural networks: A core-set approach.  In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
  * [29] Hiranmayi Ranganathan, Hemanth Venkateswara, Shayok Chakraborty, and Sethuraman Panchanathan.  Deep active learning for image classification.  In 2017 IEEE International Conference on Image Processing (ICIP), pages 3934–3938. IEEE, 2017.
  * [30] Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal.  Deep batch active learning by diverse, uncertain gradient lower bounds.  arXiv preprint arXiv:1906.03671, 2019.
  * [31] Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal.  Deep batch active learning by diverse, uncertain gradient lower bounds.  In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
  * [32] Binhui Xie, Longhui Yuan, Shuang Li, Chi Harold Liu, Xinjing Cheng, and Guoren Wang.  Active learning for domain adaptation: An energy-based approach.  arXiv preprint arXiv:2112.01406, 2021.
  * [33] Shai Ben David, Tyler Lu, Teresa Luu, and Dávid Pál.  Impossibility theorems for domain adaptation.  In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 129–136. JMLR Workshop and Conference Proceedings, 2010.
  * [34] Shiqi Yang, Joost van de Weijer, Luis Herranz, Shangling Jui, et al.  Exploiting the intrinsic neighborhood structure for source-free domain adaptation.  Advances in Neural Information Processing Systems, 34, 2021.
  * [35] Shiqi Yang, Yaxing Wang, Joost van de Weijer, Luis Herranz, and Shangling Jui.  Generalized source-free domain adaptation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8978–8987, 2021.
  * [36] Zhi-Hua Zhou.  Ensemble methods: foundations and algorithms.  CRC press, 2012.
  * [37] Hui Tang, Ke Chen, and Kui Jia.  Unsupervised domain adaptation via structurally regularized deep clustering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [38] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017.
  * [39] John Bridle, Anthony Heading, and David MacKay.  Unsupervised classifiers, mutual information and’phantom targets.  Advances in neural information processing systems, 4, 1991.
  * [40] Weihua Hu, Takeru Miyato, Seiya Tokui, Eiichi Matsumoto, and Masashi Sugiyama.  Learning discrete representations via information maximizing self-augmented training.  In International conference on machine learning, pages 1558–1567. PMLR, 2017.
  * [41] Kamran Ghasedi Dizaji, Amirhossein Herandi, Cheng Deng, Weidong Cai, and Heng Huang.  Deep clustering via joint convolutional autoencoder embedding and relative entropy minimization.  In Proceedings of the IEEE international conference on computer vision, pages 5736–5745, 2017.
  * [42] Andreas Krause, Pietro Perona, and Ryan Gomes.  Discriminative clustering by regularized information maximization.  Advances in neural information processing systems, 23, 2010.
  * [43] Yuan Shi and Fei Sha.  Information-theoretical learning of discriminative clusters for unsupervised domain adaptation.  In ICML, 2012.
  * [44] Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu.  Model adaptation: Historical contrastive learning for unsupervised domain adaptation without source data.  Advances in Neural Information Processing Systems, 34, 2021.
  * [45] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell.  Adapting visual category models to new domains.  In European conference on computer vision, pages 213–226. Springer, 2010.
  * [46] Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko.  Visda: The visual domain adaptation challenge.  arXiv preprint arXiv:1710.06924, 2017.
  * [47] Youngeun Kim, Donghyeon Cho, and Sungeun Hong.  Towards privacy-preserving domain adaptation.  IEEE Signal Processing Letters, 27:1675–1679, 2020.
  * [48] Zhen Qiu, Yifan Zhang, Hongbin Lin, Shuaicheng Niu, Yanxia Liu, Qing Du, and Mingkui Tan.  Source-free domain adaptation via avatar prototype generation and adaptation.  arXiv preprint arXiv:2106.15326, 2021.
  * [49] David Arthur and Sergei Vassilvitskii.  k-means++: The advantages of careful seeding.  Technical report, Stanford, 2006.
  * [50] Binhui Xie, Longhui Yuan, Shuang Li, Chi Harold Liu, Xinjing Cheng, and Guoren Wang.  Active learning for domain adaptation: An energy-based approach.  CoRR, abs/2112.01406, 2021.
  * [51] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [52] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan.  Learning transferable features with deep adaptation networks.  In International conference on machine learning, pages 97–105. PMLR, 2015.
  * [53] Jiayuan Huang, Arthur Gretton, Karsten Borgwardt, Bernhard Schölkopf, and Alex Smola.  Correcting sample selection bias by unlabeled data.  Advances in neural information processing systems, 19:601–608, 2006\.
  * [54] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan.  Deep transfer learning with joint adaptation networks.  In International conference on machine learning, pages 2208–2217. PMLR, 2017.
  * [55] Yaroslav Ganin and Victor Lempitsky.  Unsupervised domain adaptation by backpropagation.  In International conference on machine learning, pages 1180–1189. PMLR, 2015.
  * [56] Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada.  Maximum classifier discrepancy for unsupervised domain adaptation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3723–3732, 2018.
  * [57] Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan.  Conditional adversarial domain adaptation.  In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 1647–1657, 2018.
  * [58] Kuniaki Saito, Donghyun Kim, Stan Sclaroff, Trevor Darrell, and Kate Saenko.  Semi-supervised domain adaptation via minimax entropy.  In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pages 8049–8057. IEEE, 2019.
  * [59] Taekyung Kim and Changick Kim.  Attract, perturb, and explore: Learning a feature alignment network for semi-supervised domain adaptation.  In European conference on computer vision, pages 591–607. Springer, 2020.
  * [60] Pin Jiang, Aming Wu, Yahong Han, Yunfeng Shao, Meiyu Qi, and Bingshuai Li.  Bidirectional adversarial training for semi-supervised domain adaptation.  In IJCAI, pages 934–940, 2020.
  * [61] Takeshi Teshima, Issei Sato, and Masashi Sugiyama.  Few-shot domain adaptation by causal mechanism transfer.  In International Conference on Machine Learning, pages 9458–9469. PMLR, 2020.
  * [62] Ilja Kuzborskij and Francesco Orabona.  Stability and hypothesis transfer learning.  In International Conference on Machine Learning, pages 942–950. PMLR, 2013.
  * [63] Sk Miraj Ahmed, Aske R Lejbolle, Rameswar Panda, and Amit K Roy-Chowdhury.  Camera on-boarding for person re-identification using hypothesis transfer learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12144–12153, 2020.
  * [64] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li.  Energy-based out-of-distribution detection.  Advances in Neural Information Processing Systems, 33:21464–21475, 2020.

## Checklist

  1. 1.

For all authors…

    1. (a)

Do the main claims made in the abstract and introduction accurately reflect
the paper’s contributions and scope? [Yes]

    2. (b)

Did you describe the limitations of your work? [Yes] The limitations is
attached in Appendix A.

    3. (c)

Did you discuss any potential negative societal impacts of your work? [Yes]
The potential negative social impacts is attached in Appendix B.

    4. (d)

Have you read the ethics review guidelines and ensured that your paper
conforms to them? [Yes]

  2. 2.

If you are including theoretical results…

    1. (a)

Did you state the full set of assumptions of all theoretical results? [N/A]

    2. (b)

Did you include complete proofs of all theoretical results? [N/A]

  3. 3.

If you ran experiments…

    1. (a)

Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)? [Yes]
We attach the code in the supplemental material.

    2. (b)

Did you specify all the training details (e.g., data splits, hyperparameters,
how they were chosen)? [Yes] Detail implementation details is attached in
Appendix C.

    3. (c)

Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)? [Yes] All main results are average over three
running times in all experiments

    4. (d)

Did you include the total amount of compute and the type of resources used
(e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Detail
information is attached in Appendix C.

  4. 4.

If you are using existing assets (e.g., code, data, models) or
curating/releasing new assets…

    1. (a)

If your work uses existing assets, did you cite the creators? [Yes]

    2. (b)

Did you mention the license of the assets? [No]

    3. (c)

Did you include any new assets either in the supplemental material or as a
URL? [No]

    4. (d)

Did you discuss whether and how consent was obtained from people whose data
you’re using/curating? [No]

    5. (e)

Did you discuss whether the data you are using/curating contains personally
identifiable information or offensive content? [No]

  5. 5.

If you used crowdsourcing or conducted research with human subjects…

    1. (a)

Did you include the full text of instructions given to participants and
screenshots, if applicable? [N/A]

    2. (b)

Did you describe any potential participant risks, with links to Institutional
Review Board (IRB) approvals, if applicable? [N/A]

    3. (c)

Did you include the estimated hourly wage paid to participants and the total
amount spent on participant compensation? [N/A]