# TOHAN: A One-step Approach towards
Few-shot Hypothesis Adaptation

Haoang Chi Equal contributions. Institute for Quantum Information $\&$ State
Key Laboratory of High Performance Computing, National University of Defense
Technology  Feng Liu∗ DeSI Lab, AAII, University of Technology Sydney  Wenjing
Yang  Institute for Quantum Information $\&$ State Key Laboratory of High
Performance Computing, National University of Defense Technology  Long Lan
Institute for Quantum Information $\&$ State Key Laboratory of High
Performance Computing, National University of Defense Technology  Tongliang
Liu  TML Lab, University of Sydney
Bo Han  Department of Computer Science, Hong Kong Baptist University  William
K. Cheung  Department of Computer Science, Hong Kong Baptist University  James
T. Kwok  Department of Computer Science and Engineering, Hong Kong University
of Science and Technology

###### Abstract

In _few-shot domain adaptation_ (FDA), classifiers for the target domain are
trained with _accessible_ labeled data in the _source domain_ (SD) and few
labeled data in the _target domain_ (TD). However, data usually contain
private information in the current era, e.g., data distributed on personal
phones. Thus, the private information will be leaked if we directly access
data in SD to train a target-domain classifier (required by FDA methods). In
this paper, to thoroughly prevent the privacy leakage in SD, we consider a
very challenging problem setting, where the classifier for the TD has to be
trained using few labeled target data and a well-trained SD classifier, named
_few-shot hypothesis adaptation_ (FHA). In FHA, we cannot access data in SD,
as a result, the private information in SD will be protected well. To this
end, we propose a _target orientated hypothesis adaptation network_ (TOHAN) to
solve the FHA problem, where we generate highly-compatible unlabeled data
(i.e., an intermediate domain) to help train a target-domain classifier. TOHAN
maintains two deep networks simultaneously, where one focuses on learning an
intermediate domain and the other takes care of the intermediate-to-target
distributional adaptation and the target-risk minimization. Experimental
results show that TOHAN outperforms competitive baselines significantly.

##  1 Introduction

In _Domain Adaptation_ (DA) [45, 36, 37, 16, 6, 52], we aim to train a target-
domain classifier with data in source and target domains. Based on the
availability of data in the target domain (e.g., fully-labeled data,
partially-labeled data and unlabeled data), DA is divided into three
categories: _supervised DA_ (SDA) [38], semi-supervised DA [15] and
_unsupervised DA_ (UDA) [34, 55, 7, 23]. Since SDA methods outperform UDA
methods for the same quantity of target data [28], it becomes attractive if we
can train a good target-domain classifier using labeled-source data and few
labeled-target data [40].

Figure 1: The _few-shot hypothesis adaptation_ (FHA) and existing domain
adaptation problem settings. In FHA, we aim to train a classifier for the
target domain only using few labeled target data and a well-trained source-
domain classifier. Namely, we do not access any source data when training the
target-domain classifier. This thoroughly prevents the information leakage of
the source domain. The lock means we cannot access data in the source domain.

Hence, _few-shot domain adaptation_ (FDA) methods [28] are proposed to train a
target-domain classifier with _accessible_ labeled data from the source domain
and few labeled data from the target domain. Compared to SDA and UDA methods,
FDA methods only require few data in the target domain, which is suitable to
solve many problems, e.g., medical image processing [42]. Existing FDA methods
involve many approaches and applications. Structural casual model [40] has
been proposed to overcome the problem caused by apparent distribution
discrapancy. Since deep neural networks tend to overfit the few-labeled data
in the training process, a meta-learning method becomes an effective solution
to the FDA problem [39]. Besides, FDA methods perform well in face of
generation [46] and virtual-to-real scene parsing [50].

However, it is risky to directly access source data for training a target-
domain classifier (required by FDA methods) due to the private information
contained in the source domain. In the current era, labeled data are
distributed over different physical devices and usually contain private
information, e.g., data on personal phones or from surveillance cameras [21].
Since FDA methods [40] require abundant labeled source data to train a target-
domain classifier, they may leak private information in the training process,
maybe resulting in massive loss [14].

In this paper, to thoroughly prevent the private-information leakage of the
source domain in existing FDA methods, we propose a novel and very challenging
setting, where the classifier for the target domain has to be trained using
few labeled target data and a well-trained source-domain classifier, named
_few-shot hypothesis adaptation_ (FHA, see Figure 1). In the literature,
researchers have adapted source-domain hypothesis to be a target-domain
classifier when abundant unlabeled target data are available [21]. However,
since these methods require abundant target data, they cannot address the FHA
problem well, which has been empirically verified in Table 1 and Table 2.

The key benefit of FHA is that we do not need to access any source data, which
wisely avoids the private-information leakage of the source domain. Moreover,
the scales of images of most domains tend to be larger in the real world.
Thus, existing FDA methods will take a long time to train a target-domain
classifier. However, in FHA, we can train a target-domain classifier only
using a well-trained source-domain classifier and few labeled target data.

To address FHA, we first revisit the theory related to learning from few
labeled data and try to find out if FHA can be addressed in principle.
Fortunately, we find that, in _semi-supervised learning_ (SSL) where only few
labeled data available, researchers have already shown that, a good classifier
can be learned if we have abundant unlabeled data that are compatible with the
labeled data. Thus, motivated by the SSL, we aim to address FHA via gradually
generating highly compatible data for the target domain. To this end, we
propose a _target orientated hypothesis adaptation network_ (TOHAN) to solve
the FHA problem. TOHAN maintains two deep networks simultaneously, where one
focuses on learning an intermediate domain (i.e., learning compatible data)
and the other takes care of the intermediate-to-target distributional
adaptation (Figure 2).

Specifically, due to the scarcity of target data, we cannot directly generate
compatible data for the target domain. Thus, we first generate an intermediate
domain where data are compatible with the given source classifier and the few
labeled target data. Then, we conduct the intermediate-to-target
distributional adaptation to make the generated intermediate domain close to
the target domain. Eventually, we embed the above procedures into our one-step
solution, TOHAN, to make us be able to gradually generate an intermediate
domain that contains highly compatible data for the target domain. According
to learnability of SSL, with the generated “target-like” intermediate domain,
TOHAN can learn a good target-domain classifier.

We conduct experiments on $8$ FHA tasks on $5$ datasets (_MNIST_ , _SVHN_ ,
_USPS_ , _CIFAR_ -$10$ and _STL_ -$10$). We compare TOHAN with $5$ competitive
baselines. Experiments show that TOHAN effectively transfers knowledge of the
source hypothesis to train a target-domain classifier when we only have few
labeled target data. In a word, our paper opens a new door to domain
adaptation field, which solves privacy leakage and data shortage
simultaneously.

##  2 Few-shot Hypothesis Adaptation

In this section, we formalize a novel and challenging problem setting, called
_few-shot hypothesis adaptation_ (FHA). Let $\mathcal{X}\subset\mathbb{R}^{d}$
be a feature (input) space and $\mathcal{Y}:=\\{1,\dots,N\\}$ be a label
(output) space, and $N$ is the number of classes. A domain for the FHA problem
is defined as follows.

Figure 2: Overview of _target orientated hypothesis adaptation network_
(TOHAN). It consists of generator _G_ , encoder _$g_{s}$_ , _$g_{t}$_
(initialize _$g_{t}$_ =_$g_{s}$_), classifier _$h_{s}$_ , _$h_{t}$_
(initialize _$h_{t}$_ =_$h_{s}$_) and group discriminator _D_. (a) Firstly, we
train a generator _G_ using the source classifier _$g_{s}$_ , _$h_{s}$_ and
target data _$D_{t}$_ , then we generate intermediate data between two
domains. (b) We freeze _$g_{t}$_ and _$h_{t}$_ and update group discriminator
_D_. (c) We freeze _D_ and update _$g_{t}$_ and _$h_{t}$_. In subfigures (b)
and (c), they show a data pair from $\mathcal{G}_{2}$ which two data come from
the same class but different domain.

######  Definition 1 (Domains for FHA).

Given random variables $X_{s},X_{t}\in\mathcal{X}$,
$Y_{s},Y_{t}\in\mathcal{Y}$, the source and target domains are joint
distributions $P\left(X_{s},Y_{s}\right)$ and $P\left(X_{t},Y_{t}\right)$,
where the joint distributions
$P\left(X_{s},Y_{s}\right)\,\neq\,P\left(X_{t},Y_{t}\right)$ and $\mathcal{X}$
is compact.

Then the FHA problem is defined as follows.

######  Problem 1 (FHA).

Given a model (consisting of an encoder $g_{s}$ and a classifier $h_{s}$)
trained on the source domain $P\left(X_{s},Y_{s}\right)$ and independent and
identically distributed (i.i.d.) labeled data
$D_{t}=\left\\{\left(x_{t}^{i},y_{t}^{i}\right)\right\\}_{i=1}^{n_{t}}$
($n_{t}\leq 7$, following [32]) drawn from the target domain
$P\left(X_{t},Y_{t}\right)$, the aim of FHA is to train a classifier
$\emph{$h_{t}$}:\mathcal{X}\to\mathcal{Y}$ with $g_{s}$, $h_{s}$ and $D_{t}$
such that $h_{t}$ can accurately classify target data drawn from
$P\left(X_{t},Y_{t}\right)$.

Comparison with Few-shot Learning. The main difference between FHA and FSL is
the prior knowledge [44, 24, 47, 25]. The prior knowledge of FSL mainly
includes various types of numerical information and comes from the same
distribution with their tasks [44]. For example, [43] uses the data itself as
prior knowledge, and [35] uses the pairwise similarity, which is relatively
weaker than the former. In addition, _model-agnostic meta learning_ (MAML)
requires data to optimize a meta-learner as prior knowledge [8]. However, the
prior knowledge of FHA is just a well-trained classifier and training data of
this classifier come from different distribution with $P(X_{t},Y_{t})$.

Comparison with UDA. The main differences between FHA and UDA [53, 51, 31, 48]
focus on amount and label of data on two domains. For source domain, UDA
requires a large amount of labeled data, while FHA only requires a well-
trained model. For target domain, UDA requires a large amount of unlabeled
data, while FHA requires _few_ labeled data.

Comparison with Few-shot Domain Adaptation. With the development of FSL,
researchers also apply ideas of FSL into domain adaptation, called _few-shot
domain adaptation_ (FDA). FADA [28] is a representative FDA method, which
pairs data from source domain and data from target domain and then follows the
adversarial domain adaptation method. Casual mechanism transfer [40] is
another novel FDA method dealing with a meta-distributional scenario, in which
the data generating mechanism is invariant among domains. Nevertheless, FDA
methods still need to access many labeled source data for training, which may
cause the private-information leakage of the source domain.

Comparison with Hypothesis Transfer Learning. In the _hypothesis transfer
learning_ (HTL), we can only access a well-trained source-domain classifier
and small labeled or abundant unlabeled target data. [19] requires small
labeled target data and uses the Leave-One-Out error find the optimal transfer
parameters. Later, SHOT [21] is proposed to solve the HTL with many unlabeled
target data by freezing the source-domain classifier and learning a target-
specific feature extraction module. As for the universal setting, a two-stage
learning process [18] has been proposed to address the HTL problem. Compared
with FHA, HTL still requires at least small target data (e.g., at least $12$
samples in binary classification problem [19], or at least two of labeling
percentage [1]). In FHA, we focus on a more challenging situation: only few
data (e.g., one sample per class) are available. Besides, previous solutions
to HTL mainly focus on mortifying existing hypotheses or loss functions used
for fine-tuning. However, our solution stems from the learnability of semi-
supervised learning (Section 3) and try to generate more compatible data,
which is quite different from previous works. In this paper, we modify the
newest HTL method, SHOT [21], as one of our baselines. The modified SHOT can
leverage labeled target data to train a good target-domain classifier.

##  3 How to Learn from Few-shot Data in Principle

From the view of statistical learning theory [41], it is unrealistic to
directly learn an accurate target-domain classifier only with few labeled
data. However, the amount of labeled data in _semi-supervised learning_ (SSL)
[56] is also few (e.g., one sample per class), but SSL methods still achieves
good performance across various learning tasks, which motivates us to consider
solving FHA in the view of SSL. First, we will show theoretical analysis
regarding learnability of SSL.

### Learnability of SSL.

For simplicity, we consider the $0$-$1$ semi-supervised classification
problem. Let $c^{*}:\mathcal{X}\to\\{0,1\\}$ be the optimal target classifier
and $\mathcal{H}=\\{h:\mathcal{X}\to\\{0,1\\}\\}$ is a hypothesis space. Let
$err(h)=\mathbb{E}_{x\sim P}[h(x)\neq c^{*}(x)]$ be the true error rate of a
hypothesis $h$ over a distribution $P$. In SSL, its learnability mainly
dependents on the compatibility
$\chi:\mathcal{H}\times\mathcal{X}\mapsto[0,1]$ that measures how “compatible”
$h$ is to an unlabeled data $x$. Let $\chi(h,P)=\mathbb{E}_{x\sim
P}[\chi(h,x)]$ be the expectation of compatibility of data from $P$ on a
classifier $h$. If the unlabeled data and $c^{*}$ are highly compatible (i.e.,
$\chi(c^{*},P)$ closes to $1$), then, in theory, we can learn a good
classifier with few labeled data and sufficient unlabeled data. Specifically,
we have the following theorem (see proof in Appendix B).

######  Theorem 1.

Let $\hat{\chi}(h,S)=\frac{1}{|S|}\sum_{x\in S}\chi(h,x)$ be the empirical
compatibility over unlabeled dataset $S$. Let
$\mathcal{H}_{0}=\\{h\in\mathcal{H}:\widehat{err}(h)=0\\}$. If
$c^{*}\in\mathcal{H}$ and $\chi(c^{*},P)=1-t$, then $m_{u}$ unlabeled data and
$m_{l}$ labeled data are sufficient to learn to error $\epsilon$ with
probability $1-\delta$, for

$m_{u}=\mathcal{O}\left(\frac{VCdim(\chi(\mathcal{H}))}{\epsilon^{2}}\log\frac{1}{\epsilon}+\frac{1}{\epsilon^{2}}\log\frac{2}{\delta}\right)$
(1)

and

$m_{l}=\frac{2}{\epsilon}\left[\ln(2\mathcal{H}_{P,\chi}(t+2\epsilon)[2m_{l},P])+\ln\frac{4}{\delta}\right],$
(2)

where $\chi(\mathcal{H})=\\{\chi_{h}:h\in\mathcal{H}\\}$,
$\chi_{h}(\cdot)=\chi(h,\cdot)$, and
$\mathcal{H}_{P,\chi}(t+2\epsilon)[2m_{l},P]$ is the expected number of splits
of $2m_{l}$ data drawn from $P$ using hypotheses in $\mathcal{H}$ of
compatibility more than $1-t-2\epsilon$. In particular, with probability at
least $1-\delta$, we have $err(\hat{h})\leq\epsilon$, where

$\hat{h}=\mathop{\arg\max}_{h\in\mathcal{H}_{0}}\hat{\chi}(h,S).$ (3)

######  Remark 1.

If unlabeled data are highly compatible to $c^{*}$, $t$ is small, which
results in a smaller $m_{l}$. Namely, with the smaller $m_{l}$, we can still
achieve a low error rate. In view of Theorem 1, it is clear that SSL will be
learnable if many compatible unlabeled data are available. Motivated by SSL,
we wonder if we can generate compatible data to help our learning task. The
answer is _affirmative_.

### Solving FHA in Principle.

Motivated by Theorem 1, finding many highly compatible unlabeled data is a
breakthrough point for FHA. Hence, generating unlabeled target data is a
straightforward solution. However, due to the shortage of existing target
data, directly generating them is unrealistic. To solve this problem, we can
ask for help from the source classifier. In our paper, we first try to
generate intermediate domain $P_{m}$ containing knowledge of source and target
domains, which are compatible with both source classifier and target
classifier, i.e.,

$\displaystyle
P_{m}=\mathop{\arg\max}\limits_{P}[\chi(h_{s},P)+\chi(h_{t},P)],$ (4)

where $\chi(h_{s},P)$ (resp. $\chi(h_{t},P)$) measures how compatible $h_{s}$
(resp. $h_{t}$) is with unlabeled data distribution $P$. Then, we will adapt
intermediate domain $P_{m}$ to the target domain via distributional adaptation
with the training procedure going on. Finally, we can obtain many unlabeled
data that is compatible with $h_{s}$ and $h_{t}$ (more compatible with
$h_{t}$), meaning that, based on Theorem 1, we can address FHA in principle.
According to Eq. (4), it can be seen that we can have two straightforward
solutions: maximizing $\chi(h_{s},P)$ or $\chi(h_{t},P)$, corresponding to
S+FADA and T+FADA in benchmark solutions. The results in Table 1 and Table 2
indicate that these two straightforward solutions cannot address FHA well,
which motivate us to maximize them simultaneously, which is realized below.

##  4 Target Orientated Hypothesis Adaptation Network for FHA Problem

In this section, we propose a powerful one-step approach: _target orientated
hypothesis adaptation network_ (TOHAN, see Figure 2). TOHAN can generate data
that are highly compatible with both source classifier and target classifier
and adapt the knowledge of these data to target domain gradually.

### Intermediate domain generation.

The first step of TOHAN is to generate intermediate domain data (see Figure
2a). We input Gaussian random noise _z_ to a generator $G_{n}$ (taking the
$n^{th}$ class for an example), then the generator outputs the generated data.
We aim to generate data satisfying two conditions: (1) the generated data
$G_{n}(z)$ can be correctly classified by the given source classifier
_$f_{s}=h_{s}\circ g_{s}$_ , and (2) $G_{n}(z)$ becomes closer to target
domain with training procedure going on. Thus, there are two loss functions
regarding to the intermediate domain generation. The first one is introduced
below.

Without loss of the generality, we assume $G_{n}(z)$ generates $B$ images,
where $B$ is the batchsize in the training process of TOHAN. When $G_{n}(z)$
is inputted to the source-domain classifier _$f_{s}$_ , we will obtain an
$B\times N$ matrix $\mathbf{G}_{n}^{M}$, where the $i^{th}$ row in
$\mathbf{G}_{n}^{M}$ represents probability of the $i^{th}$ generated image
belonging to each class. Thus, the $n^{th}$ column in $\mathbf{G}_{n}^{M}$
represents the probability that the $B$ generated images belongs to the
$n^{th}$ class, and we denote the $n^{th}$ column in $\mathbf{G}_{n}^{M}$ as
$l_{n}$. Since $G_{n}(z)$ aims to generate data belonging to the $n^{th}$
class, we should update parameters of $\mathbf{G}_{n}^{M}$ to make each
element in $l_{n}$ close to $1$. Namely, the first loss function to train the
$G_{n}$ can be defined as

$\mathcal{L}_{G_{n}}^{s}=\frac{1}{B}\left\|l_{n}-\mathbbm{1}\right\|_{2}^{2},$
(5)

where $\mathbbm{1}$ is a $B$-by-$1$ vector whose elements are $1$.

As discussed before, we also want to reduce the distance between the generated
data $G_{n}(z)$ and the target data whose labels are $n$. In this way, we can
make the generated data close to the target domain and attain an intermediate
domain $\mathcal{D}_{m}$. Following [22], we adopt an augmented $L_{1}$
distance $\|X-Y\|_{1}=\sum_{i}\omega_{i}\left|X_{i}-Y_{i}\right|$, where
$\omega_{i}=\left|X_{i}-Y_{i}\right|^{2}/\|X-Y\|_{2}$. Compared to ordinary
$\ell_{1}$ norm, the augmented $L_{1}$ distance encourages larger gradients
for feature dimensions with higher residual error [22]. Compared to the
$\ell_{2}$ norm, since $L_{1}$ distance is more robust to outliers [30], it is
better to measure the distance between generated images and target images.
Thus, the second loss to train $G_{n}$ is defined as follows,

$\mathcal{L}_{G_{n}}^{t}=\frac{1}{MBK}\sum_{i=1}^{B}\sum_{k=1}^{K}\left\|x_{m}^{i}-x_{t}^{k}\right\|_{1},$
(6)

where $M=\max\limits_{x_{1},x_{2}\in\mathcal{X}}\|x_{1}-x_{2}\|_{1}$
($\mathcal{X}$ is compact and $\|\cdot\|_{1}$ is continuous) and
$G_{n}(z):=\\{x_{m}^{i}\\}_{i=1}^{B}$. Combining Eq. (5) and Eq. (6), we
obtain the total loss to train the generator $G_{n}$:

$\displaystyle\mathcal{L}_{G_{n}}$
$\displaystyle=\mathcal{L}_{G_{n}}^{s}+\lambda\mathcal{L}_{G_{n}}^{t}=\frac{1}{B}\left\|l_{n}-\mathbbm{1}\right\|_{2}^{2}+\frac{\lambda}{MBK}\sum_{i=1}^{B}\sum_{k=1}^{K}\left\|x_{m}^{i}-x_{t}^{k}\right\|_{1},$
(7)

where $\lambda$ is a hyper-parameter between two losses to tradeoff the weight
of knowledge of source-domain and target-domain. To ensure that generated data
are high-quality images, we train the generator $G_{n}$ ($n=1,\dots,N$) for
some steps all alone. And we claim that optimizing Eq. (7) is corresponding to
Eq. (4). More specifically, Eq (5) (resp. Eq. (6)) is corresponding to
$\chi(h_{s},P_{m})$ (resp. $\chi(h_{t},P_{m})$). Then we conduct intermediate-
to-target distributional adaptation (see the next paragraph) and generation
simultaneously.

### Intermediate-to-target distributional adaptation.

Now, we focus on how to construct _domain-invariant representations_ (DIP)
between the intermediate domain and the target domain. Through DIP, classifier
for the intermediate domain can be used to classify target data well.

Since we only have few target data per class, so we aim to “augment” them.
Following [28], we can overcome the shortage of target data by pairing them
with corresponding intermediate data. Specifically, we create $4$ groups of
data pairs: $\mathcal{G}_{1}$ consists of data pairs from the intermediate
domain with the same label, $\mathcal{G}_{2}$ consists of pairs from different
domains (one from the intermediate and one from the target domain) but with
the same label, $\mathcal{G}_{3}$ consists of pairs from the same domain with
different labels, and $\mathcal{G}_{4}$ consists of pairs from different
domains (one from the intermediate and one from the target domain) and with
different labels.

Based on the above four groups, we construct a four-class group discriminator
_D_ to decide which of the four groups a given data pair belongs to, which
differs from classical adversarial domain adaptation [9, 15]. The group
discriminator _D_ aims to classify the data pair groups. As a classification
problem, we train _D_ with the standard categorical cross-entropy loss:

$\mathcal{L}_{D}=-\hat{\mathbb{E}}\left[\sum_{i=1}^{4}y_{\mathcal{G}_{i}}\log\left(D\left(\phi\left(\mathcal{G}_{i}\right)\right)\right)\right],$
(8)

where $\hat{\mathbb{E}}[\cdot]$ represents the empirical mean value,
$y_{\mathcal{G}_{i}}$ is the label of group $\mathcal{G}_{i}$, and
$\phi(\mathcal{G}_{i}):=\left[g_{t}(x_{1}),g_{t}(x_{2})\right]$,
$(x_{1},x_{2})\in\mathcal{G}_{i}$, and $g_{t}$ is the encoder on target
domain. Note that we will freeze _$g_{t}$_ when minimizing the above loss
function (see Figure 2b).

Next, we turn to train _$g_{t}$_ and _$h_{t}$_ with the group discriminator
_D_ fixed, which confuses _D_ unable to distinguish between $\mathcal{G}_{1}$
and $\mathcal{G}_{2}$ (also $\mathcal{G}_{3}$ and $\mathcal{G}_{4}$). However,
we need _D_ to correctly discriminate positive pairs ($\mathcal{G}_{1}$,
$\mathcal{G}_{2}$) from negative pairs ($\mathcal{G}_{3}$, $\mathcal{G}_{4}$).
This means that domain confusion and classification are realized at the same
time. We firstly initial _$g_{t}$_ and _$h_{t}$_ with the same weight as
_$g_{s}$_ and _$h_{s}$_ , respectively. Motivated by non-saturating game [10],
we minimize the following loss to update _$g_{t}$_ and _$h_{t}$_ (see Figure
2c):

$\mathcal{L}_{h\circ
g}=-\beta\hat{\mathbb{E}}\left[y_{\mathcal{G}_{1}}\log\left(D\left(\phi\left(\mathcal{G}_{2}\right)\right)\right)-y_{\mathcal{G}_{3}}\log\left(D\left(\phi\left(\mathcal{G}_{4}\right)\right)\right)\right]+\hat{\mathbb{E}}\left[\ell\left(f_{t}\left(X_{t}\right),f_{t}^{*}(X_{t})\right)\right],$
(9)

where $\beta$ is a hyper-parameter to tradeoff confusion and classification
and $\ell$ is the cross-entropy loss. $f_{t}:=g_{t}\circ h_{t}$ is the target
model and $f_{t}^{*}$ is the optimal target model. Corresponding to Theorem 1,
optimizing the first term in Eq. (9) increases compatibility of target
classifier with intermediate data, and optimizing the second term in Eq. (9)
reduces $\hat{err}(h_{t})$, resulting in a smaller $err(h_{t})$. Compared to
[28], Eq. (9) means that we train target-domain classifier by confusing $D$
and improving classification accuracy simultaneously.

Input: encoder _$g_{s}$_ , classifier _$h_{s}$_ ,
$D_{t}=\left\\{x_{t}^{i},y_{t}^{i}\right\\}_{i=1}^{n_{t}}$, learning rate
$\gamma_{1}$, $\gamma_{2}$, $\gamma_{3}$ and $\gamma_{4}$, total epoch
$T_{max}$, pretraining _D_ epoch $T_{d}$, adaptation epoch $T_{f}$, network
parameter $\left\\{\theta_{G_{n}}\right\\}_{n=1}^{N}$, $\theta_{h\circ g}$,
$\theta_{D}$.

1: Initialize $\left\\{\theta_{G_{n}}\right\\}_{n=1}^{N}$ and $\theta_{D}$;

for  _$t=1,2,.....,T_{max}$_ do

2: Initialize $\mathcal{D}_{m}=\varnothing$

for  _$n=0,1,\dots,N-1$_ do

3: Generate random noise $z$;

4: Generate data $G_{n}(z)$ then add them to $\mathcal{D}_{m}$

5: Update
$\theta_{G_{n}}\leftarrow\theta_{G_{n}}-\gamma_{1}\nabla\mathcal{L}_{G_{n}}\left(z,D_{t}\right)$
using Eq. (7);

end for

if  _$t=T_{max}-T_{f}$_ then

for  _$i=1,2,\dots,T_{d}$_ do

6: Sample $\mathcal{G}_{1}$, $\mathcal{G}_{3}$ from
$\mathcal{D}_{m}\times\mathcal{D}_{m}$;

7: Sample $\mathcal{G}_{2}$, $\mathcal{G}_{4}$ from
$\mathcal{D}_{m}\times\mathcal{D}_{t}$;

8: Update
$\theta_{D}\leftarrow\theta_{D}-\gamma_{2}\nabla\mathcal{L}_{D}\left(\\{\mathcal{G}_{i}\\}_{i=1}^{4}\right)$
using Eq. (8);

end for

end if

if  _$t\geq T_{max}-T_{f}$_ then

9: Sample $\mathcal{G}_{1}$, $\mathcal{G}_{3}$ from
$\mathcal{D}_{m}\times\mathcal{D}_{m}$;

10: Sample $\mathcal{G}_{2}$, $\mathcal{G}_{4}$ from
$\mathcal{D}_{m}\times\mathcal{D}_{t}$;

11: Update $\theta_{h\circ g}\leftarrow\theta_{h\circ
g}-\gamma_{3}\mathcal{L}_{h\circ
g}(\\{\mathcal{G}_{i}\\}_{i=1}^{4},x_{m},x_{t})$ using Eq. (9);

12: Update
$\theta_{D}\leftarrow\theta_{D}-\gamma_{4}\nabla\mathcal{L}_{D}\left(\\{\mathcal{G}_{i}\\}_{i=1}^{4}\right)$
using Eq. (8);

end if

end for

Output: the neural network $h_{t}\circ g_{t}$.

Algorithm 1 Target orientated hypothesis adaptation network (TOHAN)

### TOHAN: A one-step solution to FHA.

Although we can sequentially combine the above two steps to solve the FHA
problem (i.e., a two-step solution), the fixed intermediate domain (generated
by the first step) may have large distributional discrepancy with target
domain. As a result, such two-step solution may not obtain a good target-
domain classifier. To address this issue, we introduce a one-step solution
TOHAN. The ablation study verifies that TOHAN outperforms such two-step
solution (see ST+F and TOHAN in Table 3).

The entire training procedures of TOHAN are shown in Algorithm 1. Since the
convergence speed of generator _G_ is relatively slow, the quality of
generated data is poor at the beginning of the training process of _G_. Thus,
we will train the generator _G_ for certain epochs before doing intermediate-
to-target distributional adaptation (lines $2$ to $5$). When the generator can
generate high-quality images, we will train the generator and conduct
adaptation altogether.

We train every generator $G_{n}$ ($n=1,2,\dots,N$) separately, and we generate
intermediate domain data using the latest generators. Then, we pair
intermediate data with target data and pre-train the group discriminator _D_
(lines $6$ to $8$). Next, we pair the intermediate data with target data and
conduct the adaptation (lines $9$ to $12$). After conducting intermediate-to-
target distributional adaptation, we will obtain better _$g_{t}$_ and
_$h_{t}$_ , i.e. classifying the intermediate domain data more accurately.
With the better target-domain classifier, we can make the generated
intermediate data get closer to the target domain, in turn, these generated
intermediate data furthermore promote the adaptation performance.

### Why does TOHAN prevent the leakage of private information effectively?

As mentioned above, TOHAN generates intermediate domain data containing
knowledge of source domain and target domain. The knowledge of source domain
is mainly the underlying features, dominating which class an intermediate data
belongs to. However, the high-level, visual and useful features of source
domain are rare in the generated intermediate data (Figure 4). Thus, it is
clear that the high-level features of intermediate data mostly come from
target domain. Therefore, the useful knowledge of the source domain is
completely unaccessible. Therefore, the privacy information in the source
domain is protected strictly.

Table 1: Classification accuracy$\pm$standard deviation ($\%$) on $6$ digits FHA tasks. Bold value represents the highest accuracy on each column. Tasks | WA | FHA | Number of Target Data per Class
---|---|---|---
Methods | 1 | 2 | 3 | 4 | 5 | 6 | 7
_M_ $\rightarrow$_S_ | 24.1 | FT |  26.7$\pm$1.0 |  26.8$\pm$2.1 |  26.8$\pm$1.6 |  27.0$\pm$0.7 |  27.3$\pm$1.2 |  27.5$\pm$0.8 |  28.3$\pm$1.5
SHOT |  25.7$\pm$2.2 |  26.9$\pm$1.2 |  27.9$\pm$2.6 |  29.1$\pm$0.4 |  29.1$\pm$1.4 |  29.6$\pm$1.7 |  29.8$\pm$1.5
S+F |  25.6$\pm$1.3 |  27.7$\pm$0.5 |  27.8$\pm$0.7 |  28.2$\pm$1.3 |  28.4$\pm$1.4 |  29.0$\pm$1.0 |  29.6$\pm$1.9
T+F |  25.3$\pm$1.0 |  26.3$\pm$0.8 |  28.9$\pm$1.0 |  29.1$\pm$1.3 |  29.2$\pm$1.3 |  31.9$\pm$0.4 |  32.4$\pm$1.8
TOHAN | 26.7$\pm$0.1 | 28.6$\pm$1.1 | 29.5$\pm$1.4 | 29.6$\pm$0.4 | 30.5$\pm$1.2 | 32.1$\pm$0.2 | 33.2$\pm$0.8
_S_ $\rightarrow$_M_ | 70.2 | FT |  70.2$\pm$0.0 |  70.6$\pm$0.3 |  70.7$\pm$0.1 |  70.8$\pm$0.3 |  70.9$\pm$0.2 |  71.1$\pm$0.3 |  71.1$\pm$0.4
SHOT |  72.6$\pm$1.9 |  73.6$\pm$2.0 |  74.1$\pm$0.6 |  74.6$\pm$1.2 |  74.9$\pm$0.7 |  75.4$\pm$0.3 |  76.1$\pm$1.5
S+F |  74.4$\pm$1.5 |  83.1$\pm$0.7 |  83.3$\pm$1.1 |  85.9$\pm$0.5 |  86.0$\pm$1.2 |  87.6$\pm$2.6 |  89.1$\pm$1.0
T+F |  74.2$\pm$1.8 |  81.6$\pm$4.0 |  83.4$\pm$0.8 |  82.0$\pm$2.3 |  86.2$\pm$0.7 |  87.2$\pm$0.8 |  88.2$\pm$0.6
TOHAN | 76.0$\pm$1.9 | 83.3$\pm$0.3 | 84.2$\pm$0.4 | 86.5$\pm$1.1 | 87.1$\pm$1.3 | 88.0$\pm$0.5 | 89.7$\pm$0.5
_M_ $\rightarrow$_U_ | 69.7 | FT |  74.4$\pm$0.7 |  76.7$\pm$1.9 |  76.9$\pm$2.2 |  77.3$\pm$1.1 |  77.6$\pm$1.4 |  78.3$\pm$2.1 |  78.3$\pm$1.6
SHOT |  87.2$\pm$0.2 |  87.9$\pm$0.3 |  87.8$\pm$0.4 |  88.0$\pm$0.4 |  87.9$\pm$0.5 |  88.0$\pm$0.3 |  88.4$\pm$0.3
S+F |  83.7$\pm$0.9 |  86.0$\pm$0.4 |  86.1$\pm$1.1 |  86.5$\pm$0.8 |  86.8$\pm$1.4 |  87.0$\pm$0.6 |  87.2$\pm$0.8
T+F |  84.2$\pm$0.1 |  84.2$\pm$0.3 |  85.2$\pm$0.9 |  85.2$\pm$0.6 |  86.0$\pm$1.5 |  86.8$\pm$1.5 |  87.2$\pm$0.5
TOHAN | 87.7$\pm$0.7 | 88.3$\pm$0.5 | 88.5$\pm$1.2 | 89.3$\pm$0.9 | 89.4$\pm$0.8 | 90.0$\pm$1.0 | 90.4$\pm$1.2
_U_ $\rightarrow$_M_ | 82.9 | FT |  83.5$\pm$0.4 |  84.3$\pm$2.4 |  84.5$\pm$0.7 |  85.5$\pm$1.3 |  86.6$\pm$1.0 |  87.2$\pm$0.7 |  88.1$\pm$2.7
SHOT |  83.1$\pm$0.5 | 85.5$\pm$0.3 | 85.8$\pm$0.6 |  86.0$\pm$0.2 |  86.6$\pm$0.2 |  86.7$\pm$0.2 |  87.0$\pm$0.1
S+F |  83.2$\pm$0.2 |  84.0$\pm$0.3 |  85.0$\pm$1.2 |  85.6$\pm$0.5 |  85.7$\pm$0.6 |  86.2$\pm$0.6 |  87.2$\pm$1.1
T+F |  82.9$\pm$0.7 |  83.9$\pm$0.2 |  84.7$\pm$0.8 |  85.4$\pm$0.6 |  85.6$\pm$0.7 |  86.3$\pm$0.9 |  86.6$\pm$0.7
TOHAN | 84.0$\pm$0.5 |  85.2$\pm$0.3 |  85.6$\pm$0.7 | 86.5$\pm$0.5 | 87.3$\pm$0.6 | 88.2$\pm$0.7 | 89.2$\pm$0.5
_S_ $\rightarrow$_U_ | 64.3 | FT |  64.9$\pm$1.1 |  66.5$\pm$1.5 |  66.7$\pm$1.7 |  67.3$\pm$1.1 |  68.1$\pm$2.3 |  68.3$\pm$0.5 |  69.7$\pm$1.4
SHOT |  74.7$\pm$0.3 |  75.5$\pm$1.4 |  75.6$\pm$1.0 |  75.8$\pm$0.7 |  77.1$\pm$2.1 |  77.8$\pm$1.6 |  79.6$\pm$0.6
S+F |  72.2$\pm$1.4 |  73.6$\pm$1.4 |  74.7$\pm$1.4 |  76.2$\pm$1.3 |  77.2$\pm$1.7 |  77.8$\pm$3.0 |  79.7$\pm$1.9
T+F |  71.7$\pm$0.6 |  74.3$\pm$1.9 |  74.5$\pm$0.8 |  75.9$\pm$2.1 |  77.7$\pm$1.5 |  76.8$\pm$1.8 |  79.7$\pm$1.9
TOHAN | 75.8$\pm$0.9 | 76.8$\pm$1.2 | 79.4$\pm$0.9 | 80.2$\pm$0.6 | 80.5$\pm$1.4 | 81.1$\pm$1.1 | 82.6$\pm$1.9
_U_ $\rightarrow$_S_ | 17.3 | FT |  23.4$\pm$1.8 |  23.6$\pm$2.7 |  23.8$\pm$1.6 |  24.6$\pm$1.4 |  24.6$\pm$1.2 |  24.8$\pm$0.7 |  25.5$\pm$1.8
SHOT | 30.3$\pm$1.2 | 31.6$\pm$0.4 |  29.8$\pm$0.5 |  29.4$\pm$0.3 |  29.7$\pm$0.5 |  29.8$\pm$0.8 |  30.1$\pm$0.9
S+F |  28.1$\pm$1.2 |  28.7$\pm$1.3 |  29.0$\pm$1.2 |  30.1$\pm$1.1 |  30.3$\pm$1.3 |  30.7$\pm$1.0 |  30.9$\pm$1.5
T+F |  27.5$\pm$1.4 |  27.9$\pm$0.9 |  28.4$\pm$1.3 |  29.4$\pm$1.8 |  29.5$\pm$0.7 |  30.2$\pm$1.0 |  30.4$\pm$1.7
TOHAN |  29.9$\pm$1.2 |  30.5$\pm$1.2 | 31.4$\pm$1.1 | 32.8$\pm$0.9 | 33.1$\pm$1.0 | 34.0$\pm$1.0 | 35.1$\pm$1.8

##  5 Experiments

In this section, we compare TOHAN with possible benchmark solutions on five
standard supervised DA datasets: _MNIST_ $\left(\emph{M}\right)$, _SYHN_
$\left(\emph{S}\right)$, _USPS_ $\left(\emph{U}\right)$, _CIFAR_
-$10\left(\emph{CF}\right)$, _STL_ -$10\left(\emph{SL}\right)$. We follow the
standard domain-adaptation protocols [34] and compare average accuracy of $5$
independent repeated experiments. For digital datasets (i.e., _M_ , _S_ , and
_U_), we choose the number of target data from $1$ to $7$ following [28]. For
objective datasets (i.e., _CF_ and _SL_), we choose the number of target data
as $10$. Details regarding these datasets can be found in Appendix C.

(a) $CF$ $\to$ $SL$ (b) $SL$ $\to$ $CF$

Figure 3: TOHAN vs DAPN.

### Benchmark solutions for FHA.

Although the FHA is a new problem setting, we still design $5$ benchmark
solutions to this new problem. (1) Without adaptation (WA): to classify target
domain with the source classifier (encoder _$g_{s}$_ and classifier
_$h_{s}$_). (2) Fine tuning (FT): to train the _classifier_ _$g_{s}$_ with few
owned target data. (3) SHOT: a novel HTL method, where we modify it to use
labeled target data instead of only using unlabeled target data. [21]. (4)
S+FADA (S+F): to generate fake source data with the source classifier then
apply them into DANN [9]. (5) T+FADA (T+F): to generate fake target data with
few real target data then apply them into DANN. We demonstrate details of $5$
benchmark solutions in Appendix D. Experimental details can be found in
Appendix E. Moreover, we conduct additional experiments about existing HTL
methods in Appendix F.

### Results on digits FHA tasks.

We conduct experiments on $6$ digits FHA tasks: _M_ $\rightarrow$_S_ , _S_
$\rightarrow$_M_ , _M_ $\rightarrow$_U_ , _U_ $\rightarrow$_M_ , _S_
$\rightarrow$_U_ and _U_ $\rightarrow$_S_. Table 1 reports target-domain
classification accuracy of $6$ methods on $6$ digits FHA tasks. It is clear
that TOHAN performs the best on almost every tasks. On _M_ $\rightarrow$_S_ ,
_S_ $\rightarrow$_M_ , _M_ $\rightarrow$_U_ and _S_ $\rightarrow$_U_ , TOHAN
outperforms all benchmark solutions obviously. However, on the tasks of _U_
$\rightarrow$_M_ and _U_ $\rightarrow$_S_ , the accuracy of TOHAN is slightly
lower than SHOT when the amount of target data is too small ($n=1,2$). This
abnormal phenomenon shows that TOHAN cannot generate intermediate domain data
effectively with very little target data, especially when the resolution of
source data is much smaller than that of target data. In this case, the data
we generate is close to source domain, so TOHAN degrades to S+FADA.

In Appendix G, we use t-SNE to visualize the feature extracted by TOHAN and
$5$ benchmark solutions on _M_ $\rightarrow$_U_ task (see Figure 7 in Appendix
G). When we use WA and FT methods, nearly all classes mix together. Although
the classification accuracy of SHOT, S+F and T+F are relatively high, there
are still a little mixtures among classes. For TOHAN, it can be seen that all
classes are separated well, which demonstrates that TOHAN works well for
solving FHA problem.

### Results on objects FHA tasks.

Following [34], we also evaluate TOHAN and benchmark solutions on $2$ objects
FHA tasks: $SL$ $\rightarrow$ $CF$ and $CF$ $\rightarrow$ $SL$. Considering
the complexity of datasets and the difficulty of our problem setting, we do
not have amazing results like digits tasks. In $SL$ $\rightarrow$ $CF$, we
achieve of $4.8\%$ improvement over WA and a performance accuracy of $56.9\%$.
Note that because the number of pixels per image of $CF$ and $SL$ are quite
different, the images from $SL$ will lose a lot of information when inputted
to the pre-trained model of $CF$, thus making the effects of TOHAN and
benchmark solutions are not obvious for $CF$ $\rightarrow$ $SL$.

### Comparing TOHAN with FSL methods.

As mentioned above, FHA is a difficult case of FSL where the prior knowledge
is a pre-trained model of another domain. To test the effectiveness of FSL
methods in FHA, we compare TOHAN with a novel FSL method called _domain-
adaptive few-shot learning_ (DAPN) [54].

Figure 4: Visualization of S+FADA and TOHAN.

Note that we use the same pre-trained model in both TOHAN and DAPN. Taking
$CF$ $\leftrightarrow$ $SL$ with five target data (per class) as an example,
we solve FHA with TOHAN and DAPN and show the results in Figure 3. It is clear
that TOHAN outperforms DAPN when the training epoch ($t$) is relatively large.

### Ablation Study.

Finally, we study the advantages of one-step method over other two-step
methods. We consider the following baselines: S+F, T+F and _ST+FADA_ (ST+F).
We have explained S+F and T+F previously. ST+F denotes the two-step version of
TOHAN, i.e., to conduct intermediate domain generation and intermediate-to-
target distributional adaptation separately. We make ablation study on three
digital datasets mentioned before as an example.

As shown in Table 3, it is clear that TOHAN works better than other baselines.
The generator of S+F uses the loss $\mathcal{L}_{G_{n}}^{s}$, which merely
contains knowledge from source domain. The generator of T+F uses the loss
$\mathcal{L}_{G_{n}}^{t}$ and ignores the knowledge contained in the source-
domain classifier. Compared to them, TOHAN uses both $\mathcal{L}_{G_{n}}^{s}$
and $\mathcal{L}_{G_{n}}^{t}$. As a result, TOHAN achieves higher accuracy
than S+F and T+F. Besides, generators and classifiers in TOHAN will promote
each other in the training process, which results in that TOHAN performs
better than the ST+F. In Figure 4, we visualize the data generated by S+FADA
and TOHAN. It is clear that data generated by S+FADA are chaotic that contain
little useful information. However, data generated by TOHAN contain many
target domain high-level and visual features, and they can be classified by
source classifier accurately, resulting in a better performance in FHA. The
detailed analysis of ablation study can be found in Appendix G.

Table 2: Classification accuracy$\pm$standard deviation ($\%$) on 2 objects FHA tasks: CIFAR-$10\to$ STL-$10$ (CF$\to$SL) and STL-$10\to$ CIFAR-$10$ (SL$\to$CF). Bold value represents the highest accuracy ($\%$) among TOHAN and benchmark solutions. Methods | WA | FT | SHOT | S+F | T+F | TOHAN
---|---|---|---|---|---|---
CF$\to$SL | 70.6 |  71.5$\pm$1.0 |  71.9$\pm$0.4 |  72.1$\pm$0.4 |  71.3$\pm$0.5 | 72.8$\pm$0.1
SL$\to$CF | 51.8 |  54.3$\pm$0.5 |  53.9$\pm$0.2 | 56.9$\pm$0.5 |  55.8$\pm$0.8 |  56.6$\pm$0.3
Table 3: Ablation study. We show the average accuracy of $6$ tasks on digits datasets in this table. Bold value represents the highest accuracy ($\%$) on each column. See full results in Appendix G. FHA | Number of Target Data per Class
---|---
Methods | 1 | 2 | 3 | 4 | 5 | 6 | 7
S+F | 61.2 | 63.0 | 64.3 | 65.4 | 65.7 | 66.4 | 67.2
T+F | 61.0 | 63.0 | 64.2 | 64.5 | 65.7 | 66.5 | 67.4
ST+F | 61.8 | 64.5 | 64.9 | 65.8 | 66.5 | 67.3 | 68.4
TOHAN | 63.3 | 65.4 | 66.4 | 67.5 | 68.0 | 68.9 | 70.0

##  6 Conclusion

This paper presents a very challenging problem setting called _few-shot
hypothesis adaptation_ (FHA), which trains a target-domain classifier with
only few labeled target data and a well-trained source-domain classifier.
Since we can only access a well-trained source-domain classifier in FHA, the
private information in the source domain will be protected well. To this end,
we propose a novel one-step FHA method, called _target orientated hypothesis
adaptation network_ (TOHAN). Experiments conducted on $8$ FHA tasks confirm
that TOHAN effectively adapts the source-domain classifier to the target
domain and outperforms competitive benchmark solutions to the FHA problem.

## References

  * [1] Sk Miraj Ahmed, Aske R. Lejbølle, Rameswar Panda, and Amit K. Roy-Chowdhury.  Camera on-boarding for person re-identification using hypothesis transfer learning.  In CVPR, pages 12141–12150, 2020.
  * [2] Sercan Ömer Arik, Jitong Chen, Kainan Peng, Wei Ping, and Yanqi Zhou.  Neural voice cloning with a few samples.  In NeurIPS, 2018.
  * [3] Adam Coates, Andrew Ng, and Honglak Lee.  An Analysis of Single Layer Networks in Unsupervised Feature Learning.  In AISTATS, 2011.
  * [4] Luc Devroye, László Györfi, and Gábor Lugosi.  A Probablistic Theory of Pattern Recognition.  Springer, 1996.
  * [5] Harrison Edwards and Amos J. Storkey.  Towards a neural statistician.  In ICLR, 2017.
  * [6] Tongtong Fang, Nan Lu, Gang Niu, and Masashi Sugiyama.  Rethinking importance weighting for deep learning under distribution shift.  In NeurIPS, 2020.
  * [7] Zhen Fang, Jie Lu, Feng Liu, Junyu Xuan, and Guangquan Zhang.  Open set domain adaptation: Theoretical bound and algorithm.  IEEE Transactions on Neural Networks and Learning Systems, 2020\.
  * [8] Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In ICML, 2017.
  * [9] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky.  Domain-adversarial training of neural networks.  Journal of Machine Learning Research, 17(59):1–35, 2016.
  * [10] Ian Goodfellow.  Neurips 2016 tutorial: Generative adversarial networks.  arXiv preprint arXiv:1701.00160, 2016.
  * [11] Yunzhong Hou and Liang Zheng.  Source free domain adaptation with image translation.  arXiv preprint arXiv:2008.07514, 2020.
  * [12] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In CVPR, 2017.
  * [13] Jonathan J. Hull.  A database for handwritten text recognition research.  IEEE Trans. Pattern Anal. Mach. Intell., 16(5):550–554, 1994\.
  * [14] Bargav Jayaraman and David Evans.  Evaluating differentially private machine learning in practice.  In USENIX, 2019.
  * [15] Pin Jiang, Aming Wu, Yahong Han, Yunfeng Shao, Meiyu Qi, and Bingshuai Li.  Bidirectional adversarial training for semi-supervised domain adaptation.  In IJCAI, 2020.
  * [16] Yongcheng Jing, Xiao Liu, Yukang Ding, Xinchao Wang, Errui Ding, Mingli Song, and Shilei Wen.  Dynamic instance normalization for arbitrary style transfer.  In AAAI, 2020.
  * [17] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  Technical Report, 2009.
  * [18] Jogendra Nath Kundu, Naveen Venkat, R Venkatesh Babu, et al.  Universal source-free domain adaptation.  In CVPR, 2020.
  * [19] Ilja Kuzborskij and Francesco Orabona.  Stability and hypothesis transfer learning.  In ICML, 2013.
  * [20] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324, 1998.
  * [21] Jian Liang, Dapeng Hu, and Jiashi Feng.  Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation.  In ICML, 2020.
  * [22] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In ICCV, 2017.
  * [23] F. Liu, J. Lu, B. Han, G. Niu, G. Zhang, and M. Sugiyama.  Butterfly: A panacea for all difficulties in wildly unsupervised domain adaptation.  In NeurIPS LTS Workshop, 2019.
  * [24] Lu Liu, William Hamilton, Guodong Long, Jing Jiang, and Hugo Larochelle.  A universal representation transformer layer for few-shot image classification.  In ICLR, 2021.
  * [25] Lu Liu, Tianyi Zhou, Guodong Long, Jing Jiang, Lina Yao, and Chengqi Zhang.  Prototype propagation networks (PPN) for weakly-supervised few-shot learning on category graph.  In IJCAI, 2019.
  * [26] M. Long, Z. Cao, J. Wang, and M.I Jordan.  Conditional adversarial domain adaptation.  In NeurIPS, 2018.
  * [27] Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel.  A simple neural attentive meta-learner.  In ICLR, 2018.
  * [28] Saeid Motiian, Quinn Jones, Seyed Iranmanesh, and Gianfranco Doretto.  Few-shot adversarial domain adaptation.  In NeurIPS, 2017.
  * [29] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng.  Reading digits in natural images with unsupervised feature learning.  In NeurIPS, 2011.
  * [30] Feiping Nie, Heng Huang, Xiao Cai, and Chris H. Q. Ding.  Efficient and robust feature selection via joint $\mathscr{l}2,1$-norms minimization.  In NeurIPS, 2010.
  * [31] Luca Oneto, Michele Donini, Giulia Luise, Carlo Ciliberto, Andreas Maurer, and Massimiliano Pontil.  Exploiting MMD and Sinkhorn divergences for fair and transferable representation learning.  In NeurIPS, 2020.
  * [32] Seonwook Park, Shalini De Mello, Pavlo Molchanov, Umar Iqbal, Otmar Hilliges, and Jan Kautz.  Few-shot adaptive gaze estimation.  In ICCV, 2019.
  * [33] Alec Radford, Luke Metz, and Soumith Chintala.  Unsupervised representation learning with deep convolutional generative adversarial networks.  In ICLR, 2015.
  * [34] Rui Shu, Hung H. Bui, Hirokazu Narui, and Stefano Ermon.  A DIRT-T approach to unsupervised domain adaptation.  In ICLR, 2018.
  * [35] Jake Snell, Kevin Swersky, and Richard S. Zemel.  Prototypical networks for few-shot learning.  In NeurIPS, 2017.
  * [36] Jie Song, Yixin Chen, Xinchao Wang, Chengchao Shen, and Mingli Song.  Deep model transferability from attribution maps.  In NeurIPS, pages 6179–6189, 2019.
  * [37] Jie Song, Yixin Chen, Jingwen Ye, Xinchao Wang, Chengchao Shen, Feng Mao, and Mingli Song.  DEPARA: deep attribution graph for deep knowledge transferability.  In CVPR, 2020.
  * [38] S. Sukhija, N.C. Krishnan, and G. Singh.  Supervised heterogeneous domain adaptation via random forests.  In IJCAI, 2016.
  * [39] Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele.  Meta-transfer learning for few-shot learning.  In CVPR, 2019.
  * [40] Takeshi Teshima, Issei Sato, and Masashi Sugiyama.  Few-shot domain adaptation by causal mechanism transfer.  In ICML, 2020.
  * [41] Vladimir N. Vapnik.  Statistical Learning Theory.  Wiley, 1998.
  * [42] Dong Wang, Yuan Zhang, Kexin Zhang, and Liwei Wang.  Focalmix: Semi-supervised learning for 3d medical image detection.  In CVPR, 2020.
  * [43] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool.  Temporal segment networks for action recognition in videos.  IEEE Trans. Pattern Anal. Mach. Intell., 41(11):2740–2755, 2019\.
  * [44] Yaqing Wang, Quanming Yao, James T. Kwok, and Lionel M. Ni.  Generalizing from a few examples: A survey on few-shot learning.  ACM Comput. Surv., 53(3):63:1–63:34, 2020.
  * [45] Ying Wei, Yu Zhang, Junzhou Huang, and Qiang Yang.  Transfer learning via learning to transfer.  In ICML, 2018.
  * [46] Chao Yang and Ser-Nam Lim.  One-shot domain adaptation for face generation.  In CVPR, 2020.
  * [47] Shuo Yang, Lu Liu, and Min Xu.  Free lunch for few-shot learning: Distribution calibration.  In ICLR, 2021.
  * [48] Xiyu Yu, Tongliang Liu, Mingming Gong, Kun Zhang, Kayhan Batmanghelich, and Dacheng Tao.  Label-noise robust domain adaptation.  In ICML, 2020.
  * [49] Yiqin Yu, Xu Min, Shiwan Zhao, Jing Mei, Fei Wang, Dongsheng Li, Kenney Ng, and Shaochun Li.  Dynamic knowledge distillation for black-box hypothesis transfer learning.  arXiv:2007.12355, 2020.
  * [50] Junyi Zhang, Ziliang Chen, Junying Huang, Liang Lin, and Dongyu Zhang.  Few-shot structured domain adaptation for virtual-to-real scene parsing.  In ICCV, 2019.
  * [51] Kun Zhang, Mingming Gong, Petar Stojanov, Biwei Huang, QINGSONG LIU, and Clark Glymour.  Domain adaptation as a problem of inference on graphical models.  In NeurIPS, 2020.
  * [52] Tianyi Zhang, Ikko Yamane, Nan Lu, and Masashi Sugiyama.  A one-step approach to covariate shift adaptation.  In ACML, 2020.
  * [53] Yiyang Zhang, Feng Liu, Zhen Fang, Bo Yuan, Guangquan Zhang, and Jie Lu.  Learning from a complementary-label source domain: Theory and algorithms.  IEEE Transactions on Neural Networks and Learning Systems, 2021\.
  * [54] An Zhao, Mingyu Ding, Zhiwu Lu, Tao Xiang, Yulei Niu, Jiechao Guan, Ji-Rong Wen, and Ping Luo.  Domain-adaptive few-shot learning.  arXiv preprint arXiv:2003.08626, 2020.
  * [55] Li Zhong, Zhen Fang, Feng Liu, Jie Lu, Bo Yuan, and Guangquan Zhang.  How does the combined risk affect the performance of unsupervised domain adaptation approaches?  AAAI, 2021.
  * [56] Xiaojin Zhu.  Semi-supervised learning.  Encyclopedia of Machine Learning, pages 892–897, 2010.

##  Appendix A Related Work

In this section, we briefly review _few-shot learning_ (FSL) and two domain
adaptation settings related to the FHA problem, which include FDA, and
_source-data-free UDA_ (SFUDA).

Few-shot Learning. Existing FSL methods can be divided into three categories:
(1) Augmenting training data set by prior knowledge. Data augmentation via
hand-crafted rules serves as pre-processing in FSL methods. For instance, we
can use reflection [5]; and (2) Constraining hypothesis space by prior
knowledge [27]; and (3) Altering search strategy in hypothesis space by prior
knowledge. For instance, we can use early-stopping [2]. Note that our method
belongs to category (1). However, the prior knowledge we have is more
difficult to leverage than the prior knowledge that FSL methods have.

Few-shot Domain Adaptation. With the development of FSL, researchers also
apply ideas of FSL into domain adaptation, called _few-shot domain adaptation_
(FDA). FADA [28] is a representative FDA method, which pairs data from source
domain and data from target domain and then follows the adversarial domain
adaptation method. Casual mechanism transfer [40] is another novel FDA method
dealing with a meta-distributional scenario, in which the data generating
mechanism is invariant among domains. Nevertheless, FDA methods still need to
access many labeled source data for training, which may cause the private-
information leakage of the source domain.

Hypothesis Transfer Learning. In the _hypothesis transfer learning_ (HTL), we
can only access a well-trained source-domain classifier and small labeled or
abundant unlabeled target data. [19] requires small labeled target data and
uses the Leave-One-Out error find the optimal transfer parameters. Later, SHOT
[21] is proposed to solve the HTL with many unlabeled target data by freezing
the source-domain classifier and learning a target-specific feature extraction
module. [11] proposes an image translation method that transfers the style of
target images to that of unseen source images. As for the universal setting, a
two-stage learning process [18] has been proposed to address the HTL problem.
Compared with FHA, HTL still requires at least small target data (e.g., at
least $12$ samples in binary classification problem [19], or at least two of
labeling percentage [1]). In FHA, we focus on a more challenging situation:
only few data (e.g., one sample per class) are available.

##  Appendix B Proof of Theorem 1

We state here two known generalization bounds [4] used in our proof.

######  Lemma 1.

Suppose that $\mathcal{H}$ is a set of functions from $\mathcal{X}$ to
$\\{0,1\\}$ with finite $VC$-dimension $V\geq 1$. For any distribution $P$
over $\mathcal{X}$, any target function, and any $\epsilon$, $\delta>0$, if we
draw a set of data from $P$ of size

$m(\epsilon,\delta,V)=\frac{64}{\epsilon^{2}}\left(2V\ln\left(\frac{12}{\epsilon}\right)+\ln\left(\frac{4}{\delta}\right)\right),$

then with probability at least $1-\delta$, we have
$|err(h)-\widehat{err}(h)|\leq\epsilon$ for all $h\in\mathcal{H}$.

######  Lemma 2.

Suppose that $\mathcal{H}$ is a set of functions from $\mathcal{X}$ to
$\\{0,1\\}$ with finite $VC$-dimension $V\geq 1$. For any probability
distribution $P$ over $\mathcal{X}$, any target function $c^{*}$, we have

$\bm{Pr}\left[\sup\limits_{h\in\mathcal{H},\widehat{err}(h)=0}|err(h)-\widehat{err}(h)\geq\epsilon|\right]\leq
2\mathcal{H}[2m,P]e^{-m\epsilon/2}.$

So, for any $\epsilon$, $\delta>0$, if we draw a set of data from $P$ of size

$m\geq\frac{2}{\epsilon}\left(2\ln(\mathcal{H}[2m,P])+\ln\left(\frac{2}{\delta}\right)\right),$

then with probability at least $1-\delta$, we have that all functions with
$\widehat{err}(h)=0$ satisfy

$err(h)\leq\epsilon.$

Now we begin the proof of Theorem 1.

###### Proof.

Let $S$ be the set of $m_{u}$ unlabeled data. By standard VC-dimension bounds
(e.g., Lemma 1), the number of unlabeled data given is sufficient to ensure
that with probability at least $1-\frac{\delta}{2}$ we have

$|\bm{Pr}_{x\sim\widebar{S}}[\chi_{h}(x)=1]-\bm{Pr}_{x\sim
P}[\chi_{h}(x)=1]|\leq\epsilon\quad\text{for all
}\chi_{h}\in\chi(\mathcal{H}).$

Since $\chi_{h}(x)=\chi(h,x)$, this implies that we have

$|\chi(h,D)-\hat{\chi}(h,S)|\leq\epsilon\quad\text{for all }h\in\mathcal{H}.$

Therefore, the set of hypotheses with $\hat{\chi}(h,S)\geq 1-t-\epsilon$ is
contained in $\mathcal{H}_{P,\chi}(t+2\epsilon)$.

The bound on the number of labeled data now follows directly from known
concentration results using the expected number of partitions instead of the
maximum in the standard VC-dimension bounds (e.g., Lemma 2). This bound
ensures that with probability $1-\frac{\delta}{2}$, none of the functions
$h\in\mathcal{H}_{P,\chi}(t+2\epsilon)$ with $err(h)\geq\epsilon$ have
$\widehat{err}(h)=0$.

The above two arguments together imply that with probability $1-\delta$, all
$h\in\mathcal{H}$ with $\widehat{err}(h)=0$ and $\hat{\chi}(h,S)\geq
1-t-\epsilon$ have $err(h)\geq\epsilon$, and furthermore $c^{*}$ has
$\hat{\chi}(c^{*},S)\geq 1-t-\epsilon$. This in turn implies that with
probability at least $1-\delta$, we have $err(\hat{h})\leq\epsilon$, where

$\hat{h}=\mathop{\arg\max}_{h\in\mathcal{H}_{0}}\hat{\chi}(h,S).$

∎

##  Appendix C Datasets

### Digits.

Following the evaluation protocol of [28], we conduct experiments on $6$
adaptation scenarios: _M_ $\rightarrow$_S_ , _S_ $\rightarrow$_M_ , _M_
$\rightarrow$_U_ , _U_ $\rightarrow$_M_ , _S_ $\rightarrow$_U_ and _U_
$\rightarrow$_S_. MNIST [20] images have been size-normalized and centered in
a fixed-size ($28\times 28$) image. USPS [13] images are $16\times 16$
grayscale pixels. SVHN [29] images are $32\times 32$ pixels with $3$ channels.

### Objects.

We also evaluate TOHAN and benchmark solutions on CIFAR-10 [17] and STL-10
[3], following [34]. The CIFAR-10 dataset contains $60,000$ $32\times 32$
color images in 10 categories. The STL-10 dataset is inspired by the CIFAR-10
dataset but with some modifications. However, these two datasets only contain
nine overlapping classes. We removed the non-overlapping classes (“frog” and
“monkey”) [34].

(a) Fine-tuning

(b) SHOT

(c) S+FADA

(d) T+FADA

Figure 5: Overview of benchmark solutions to the FHA problem. (a) We freeze
the source encoder _$g_{s}$_ and train the source classifier _$h_{s}$_ with
the target data $D_{t}$. (b) We first train source encoder _$g_{s}$_ and
classifier _$h_{s}$_ , then we transfer them to the target domain. We generate
pseudo labels for target data, then we use them to train the target model with
classifier _$h_{t}$_ freezed. (c) We generate some source-like data under the
guidance of the source classifier, then we combine them with FADA. (d) We
generate some data close to target domain, i.e. decreasing the distance
between data and target domain, then we combine them with FADA.

##  Appendix D Benchmark Solutions for FHA

To solve the FHA problem, this section presents $5$ benchmark solutions that
directly combine existing techniques used in the deep learning and domain
adaptation fields.

### Without adaptation.

Since we have a source-domain classifier, we can directly use it to classify
the target data, which is a frustrating solution to the FHA problem.

### Fine-tuning.

See Figure 5(a). Fine-tuning is a basic solution to the FHA problem. We freeze
the source encoder _$g_{s}$_ and train the source classifier _$h_{s}$_ with
the target data $D_{t}$. In this way, knowledge about target domain is filled
into source hypothesis.

### SHOT.

See Figure 5(b). SHOT is a novel method for source hypothesis transfer [21].
It learns the optimal target-specific feature learning module to fit the
source hypothesis with only the source classifier. We first train source
encoder _$g_{s}$_ and classifier _$h_{s}$_ , and then we transfer them to the
target domain. SHOT is an UDA method. Thus, we generate pseudo labels for
target data, and then we use them to train the target model with classifier
_$h_{t}$_ freezed. Although SHOT is suitable for our FHA problem, it requires
a lot of target data, which is an obstacle for FHA.

### S+FADA.

See Figure 5(c). As mentioned in Figure 6, a straightforward solution to the
FHA problem is a two-step approach. We can train a source-data generator _G_
under the guidance of source hypothesis, and then we use it to generate source
data. First, we input Gaussian random noise _z_ to _G_ , then _G_ outputs
various disordered data. Second, these data is inputted into _$g_{s}$_
$\circ$_$h_{s}$_ , and then _$h_{s}$_ outputs the probability of $G(z)$
belonging to each class. Third, if we would like to generate data belonging to
$n^{th}$ class, we should optimize $G$ to push the probability of $G(z)$
belonging to $n$ near to $1$. Finally, we can apply the restored source data
into an adversarial DA method to train a target domain classifier _$h_{t}$_.

Figure 6: A straightforward solution for FHA is a two-step approach. Namely,
we can first generate source data and then train a target-domain classifier
using the generated source data and an FDA method (e.g., _few-shot adversarial
domain adaptation_ (FADA)). A visualized comparison between S+FADA and TOHAN
(take MNIST$\rightarrow$USPS as an example) is displayed in subfigures (a) and
(b). On the left side, Subfigure (a) illustrates source-domain data generated
by a two-step method: S+FADA. It is clear that the generated data are just
noise and do not contain useful information about the source domain. In
subfigure (b), we illustrate the intermediate-domain data generated by our
method (i.e., TOHAN). It is clear that the generated intermediate-domain data
contain useful information about two domains. In subfigure (c), the histogram
shows classification accuracy of the two methods, and TOHAN outperforms S+FADA
clearly.

### T+FADA.

See Figure 5(d). Different from S+FADA, here we train a generator with the
help of target data instead, and then we will generate data close to target
domain. We input Gaussian random noise _z_ to generator _G_ and minimize the
distance between $G(z)$ and target data simultaneously. Finally, we
sequentially combine these generated data with adversarial DA method to train
a target-domain classifier.

(a) WA.

(b) FT.

(c) SHOT.

(d) S+F.

(e) T+F.

(f) TOHAN.

Figure 7: The t-SNE visualization for a 10-way classification task (taking
_MNIST_ $\rightarrow$_USPS_ as an example). When we use WA and FT methods,
nearly all classes mix together. Although the classification accuracy of SHOT,
S+F and T+F are relatively high, there are still a little mixtures among
classes. For TOHAN, it can be seen that all classes are separated well.
Namely, TOHAN works well for solving FHA problem.

##  Appendix E Implementation Details

We implement all methods by PyTorch 1.7.1 and Python 3.7.6, and conduct all
the experiments on two NVIDIA RTX 2080Ti GPUs.

### Network architecture.

We select architecture of generators _$G_{n}$_ ($n=1,\dots,N$) from DCGAN
[33]. For digits tasks, the encoder _g_ , classifier _h_ and group
discriminator _D_ share the same architecture in all $6$ tasks, following FADA
[28]. As for encoder _g_ , we employ the backbone network of LeNet-$5$ with
batch normalization and dropout. For classifier _h_ , we adopt one fully
connected layer with softmax fuction. For group discriminator _D_ , we adopt
$3$ connected layers with softmax function. For objects tasks, we employ
Densenet-$169$ [12] as encoder _g_.

### Network hyper-parameters.

We set fixed hyper-parameters in every method which is irrelevant to dataset,
based on the common protocol of domain adaptation [34]. The batch size of
generator is set to $32$, and the batch size of group discriminator, encoder,
classifier is all $64$. We pre-train the group discriminator for $100$ epochs.
Meanwhile, the numbers of training steps of generator, group discriminator,
encoder, classifier are set to $500$, $50$, $50$, $50$, respectively. Adam
optimizer is with the same learning rate of $1\times 10^{-3}$ in generators,
encoder, classifier and group discriminator. The tradeoff parameter $\beta$ in
Eq. (9) is set to $\frac{2}{1+\text{exp}(-10\dot{q})}-1$, same as [26]. And
the tradeoff parameter $\lambda$ in Eq. (7) is set to $0.2$ fixed. For the
fair comparisons, we only resize and normalize the image and do not use any
addition data augment or transformation. Note than, for each experiment, we
report the result of the model trained in the last epoch.

Table 4: Comparison between dkdHTL and TOHAN. We report the classification accuracy$\pm$standard deviation ($\%$) on $6$ digits FHA tasks. Bold value represents the highest accuracy on each column. Tasks | FHA | Number of Target Data per Class
---|---|---
Methods | 1 | 2 | 3 | 4 | 5 | 6 | 7
_M_ $\rightarrow$_S_ | dkdHTL |  24.1$\pm$0.7 |  24.1$\pm$0.3 |  24.5$\pm$0.6 |  24.4$\pm$1.1 |  25.4$\pm$0.8 |  25.7$\pm$0.5 |  26.1$\pm$1.1
TOHAN | 26.7$\pm$0.1 | 28.6$\pm$1.1 | 29.5$\pm$1.4 | 29.6$\pm$0.4 | 30.5$\pm$1.2 | 32.1$\pm$0.2 | 33.2$\pm$0.8
_S_ $\rightarrow$_M_ | dkdHTL |  71.2$\pm$1.2 |  83.4$\pm$0.4 | 88.5$\pm$0.6 | 88.2$\pm$0.7 | 89.5$\pm$0.7 | 89.6$\pm$0.4 | 90.3$\pm$0.2
TOHAN | 76.0$\pm$1.9 |  83.3$\pm$0.3 |  84.2$\pm$0.4 |  86.5$\pm$1.1 |  87.1$\pm$1.3 |  88.0$\pm$0.5 |  89.7$\pm$0.5
_M_ $\rightarrow$_U_ | dkdHTL |  65.2$\pm$0.6 |  70.5$\pm$1.3 |  74.4$\pm$0.6 |  77.8$\pm$0.6 |  78.6$\pm$0.9 |  78.8$\pm$1.1 |  79.0$\pm$1.3
TOHAN | 87.7$\pm$0.7 | 88.3$\pm$0.5 | 88.5$\pm$1.2 | 89.3$\pm$0.9 | 89.4$\pm$0.8 | 90.0$\pm$1.0 | 90.4$\pm$1.2
_U_ $\rightarrow$_M_ | dkdHTL |  83.2$\pm$0.2 | 85.5$\pm$0.5 | 85.9$\pm$0.4 |  85.7$\pm$0.8 |  86.2$\pm$0.2 |  86.2$\pm$0.4 |  86.8$\pm$0.3
TOHAN | 84.0$\pm$0.5 |  85.2$\pm$0.3 |  85.6$\pm$0.7 | 86.5$\pm$0.5 | 87.3$\pm$0.6 | 88.2$\pm$0.7 | 89.2$\pm$0.5
_S_ $\rightarrow$_U_ | dkdHTL | 76.3$\pm$0.3 | 77.6$\pm$0.5 |  78.9$\pm$0.4 |  79.5$\pm$0.4 |  80.2$\pm$0.5 |  80.7$\pm$0.4 |  82.1$\pm$0.4
TOHAN |  75.8$\pm$0.9 |  76.8$\pm$1.2 | 79.4$\pm$0.9 | 80.2$\pm$0.6 | 80.5$\pm$1.4 | 81.1$\pm$1.1 | 82.6$\pm$1.9
_U_ $\rightarrow$_S_ | dkdHTL |  20.5$\pm$0.8 |  20.9$\pm$0.4 |  21.7$\pm$0.3 |  23.8$\pm$0.2 |  24.5$\pm$0.7 |  25.5$\pm$0.6 |  25.7$\pm$0.4
TOHAN | 29.9$\pm$1.2 | 30.5$\pm$1.2 | 31.4$\pm$1.1 | 32.8$\pm$0.9 | 33.1$\pm$1.0 | 34.0$\pm$1.0 | 35.1$\pm$1.8

##  Appendix F Additional Experiments about HTL

In this section, we compare TOHAN with another novel HTL method, i.e.,
_dynamic knowledge distillation for HTL_ (dkdHTL) [49]. It is worth noting
that dkdHTL is a black-box HTL method. That is, we cannot access the
parameters of source model. Therefore, in distillation loss, we cannot get the
logits, which is used to compute the soften probabilities with a high
temperature $T>1$. To address this problem, they tried to solve the logits
through soften probabilities approximately. For the sake of fairness, we
convert dkdHTL to white-box version. Specifically, we use the standard softmax
function in distillation loss, instead of the approximate version. Moreover,
we initial the parameters of target model by source model. Then, we show the
results of dkdHTL in Table 4 and Table 5.

We find that TOHAN outperforms dkdHTL in most tasks significantly. However, in
_S_ $\rightarrow$_M_ , _U_ $\rightarrow$_M_ , and _S_ $\rightarrow$_U_ , there
exists few tasks that dkdHTL outperforms TOHAN. In these three tasks, the
complexity of source domain is high, while the complexity of target domain is
low. TOHAN cannot generate qualified intermediate data effectively when the
number of target data is very few and the source domain is highly complex
simultaneously. dkdHTL is only suitable for tasks with uncomplicated target
domain. The main reason is that the training data of dkdHTL are only the few
target data. If the target domain is complex, it is very easy to overfit.
Therefore, as for tasks with complex target domains, TOHAN has the upper hand.

Table 5: Comparison between dkdHTL and TOHAN. We report the classification accuracy$\pm$standard deviation ($\%$) on 2 objects FHA tasks: CIFAR-$10\to$ STL-$10$ and STL-$10\to$ CIFAR-$10$. Bold value represents the highest accuracy ($\%$) among TOHAN and benchmark solutions. Tasks | dkdHTL | TOHAN
---|---|---
CIFAR-$10\to$ STL-$10$ |  70.8$\pm$0.7 | 72.8$\pm$0.1
STL-$10\to$ CIFAR-$10$ |  52.4$\pm$0.5 | 56.6$\pm$0.3

##  Appendix G Additional Analysis

### Visualization of Results.

We use t-SNE to visualize the feature (the penultimate layer of the
classifier) extracted by TOHAN and $5$ benchmark solutions on _M_
$\rightarrow$_U_ task (see Figure 7). When we use WA and FT methods, nearly
all classes mix together. Although the classification accuracy of SHOT, S+F
and T+F are relatively high, there are still a little mixtures among classes.
For TOHAN, all classes are separated well, which demonstrates that TOHAN works
well for solving FHA problem.

### Detailed Analysis of Ablation Study.

Table 6 shows the full results of ablation study. It is clear that TOHAN
performs better than the corresponding two-step approach ST+FADA. However,
when the number of target data is too small, ST+FADA may outperform TOHAN with
a small probability. The reason for this abnormal phenomenon may be the
limitation of target data. Although we use the technique of paring data,
overfitting still occurs when data are scarce.

Table 6: Ablation Study. Bold value represents the highest accuracy ($\%$) on each column. Data behind ’$\pm$’ is the standard derivation. Tasks | FHA | Number of target data
---|---|---
Methods | 1 | 2 | 3 | 4 | 5 | 6 | 7
_M_ $\rightarrow$_S_ | S+FADA |  25.6$\pm$1.3 |  27.7$\pm$0.5 |  27.8$\pm$0.7 |  28.2$\pm$1.3 |  28.4$\pm$1.4 |  29.0$\pm$1.0 |  29.6$\pm$1.9
T+FADA |  25.3$\pm$1.0 |  26.3$\pm$0.8 |  28.9$\pm$1.0 |  29.1$\pm$1.3 |  29.2$\pm$1.3 |  31.9$\pm$0.4 |  32.4$\pm$1.8
ST+FADA |  25.7$\pm$0.7 |  28.1$\pm$0.9 |  28.5$\pm$1.2 |  29.2$\pm$1.0 |  29.2$\pm$0.8 |  31.3$\pm$1.7 |  32.0$\pm$0.8
TOHAN | 26.7$\pm$0.1 | 28.6$\pm$1.1 | 29.5$\pm$1.4 | 29.6$\pm$0.4 | 30.5$\pm$1.2 | 32.1$\pm$0.23 | 33.2$\pm$0.8
_S_ $\rightarrow$_M_ | S+FADA |  74.4$\pm$1.5 |  83.1$\pm$0.7 |  83.3$\pm$1.1 |  85.9$\pm$0.5 |  86.0$\pm$1.2 |  87.6$\pm$2.6 |  89.1$\pm$1.0
T+FADA |  74.2$\pm$1.8 |  81.6$\pm$4.0 |  83.4$\pm$0.8 |  82.0$\pm$2.3 |  86.2$\pm$0.7 |  87.2$\pm$0.8 |  88.2$\pm$0.6
ST+FADA |  74.3$\pm$1.2 | 83.7$\pm$1.0 |  83.8$\pm$0.8 |  85.8$\pm$0.6 |  86.0$\pm$0.9 |  87.7$\pm$0.8 |  89.0$\pm$0.6
TOHAN | 76.0$\pm$1.9 |  83.3$\pm$0.3 | 84.2$\pm$0.4 | 86.5$\pm$1.1 | 87.1$\pm$1.3 | 88.0$\pm$0.5 | 89.7$\pm$0.5
_M_ $\rightarrow$_U_ | S+FADA |  83.7$\pm$0.9 |  86.0$\pm$0.4 |  86.1$\pm$1.1 |  86.5$\pm$0.8 |  86.8$\pm$1.4 |  87.0$\pm$0.6 |  87.2$\pm$0.8
T+FADA |  84.2$\pm$0.1 |  84.2$\pm$0.3 |  85.2$\pm$0.9 |  85.2$\pm$0.6 |  86.0$\pm$1.5 |  86.8$\pm$1.5 |  87.2$\pm$0.5
ST+FADA |  86.1$\pm$1.5 |  87.1$\pm$1.6 |  86.9$\pm$0.7 |  87.9$\pm$1.1 |  88.0$\pm$1.2 |  88.3$\pm$0.7 |  88.5$\pm$1.3
TOHAN | 87.7$\pm$0.7 | 88.3$\pm$0.5 | 88.5$\pm$1.2 | 89.3$\pm$0.9 | 89.4$\pm$0.8 | 90.0$\pm$1.0 | 90.4$\pm$1.2
_U_ $\rightarrow$_M_ | S+FADA |  83.2$\pm$0.2 |  83.9$\pm$0.3 |  84.9$\pm$1.2 |  85.6$\pm$0.5 |  85.7$\pm$0.6 |  86.2$\pm$0.6 |  87.2$\pm$1.1
T+FADA |  82.9$\pm$0.7 |  83.9$\pm$0.2 |  84.7$\pm$0.8 |  85.4$\pm$0.6 |  85.6$\pm$0.7 |  86.3$\pm$0.9 |  86.6$\pm$0.7
ST+FADA | 84.0$\pm$0.7 |  84.2$\pm$0.5 |  85.3$\pm$1.0 |  85.6$\pm$1.2 |  86.7$\pm$1.0 |  86.5$\pm$0.5 |  88.0$\pm$1.0
TOHAN |  84.0$\pm$0.5 | 85.2$\pm$0.3 | 85.6$\pm$0.7 | 86.5$\pm$0.5 | 87.3$\pm$0.6 | 88.2$\pm$0.7 | 89.2$\pm$0.5
_S_ $\rightarrow$_U_ | S+FADA |  72.2$\pm$1.4 |  73.6$\pm$1.4 |  74.7$\pm$1.4 |  76.2$\pm$1.3 |  77.2$\pm$1.7 |  77.8$\pm$3.0 |  79.7$\pm$1.9
T+FADA |  71.7$\pm$0.6 |  74.3$\pm$1.9 |  74.5$\pm$0.8 |  75.9$\pm$2.1 |  77.7$\pm$1.5 |  76.8$\pm$1.8 |  79.7$\pm$1.9
ST+FADA |  73.1$\pm$0.9 |  75.2$\pm$1.3 |  75.9$\pm$0.8 |  76.3$\pm$1.5 |  78.3$\pm$1.6 |  79.1$\pm$1.7 |  79.7$\pm$1.6
TOHAN | 75.8$\pm$0.9 | 76.8$\pm$1.2 | 79.4$\pm$0.9 | 80.2$\pm$0.6 | 80.5$\pm$1.4 | 81.1$\pm$1.1 | 82.6$\pm$1.9
_U_ $\rightarrow$_S_ | S+FADA |  28.1$\pm$1.2 |  28.7$\pm$1.3 |  29.0$\pm$1.2 |  30.1$\pm$1.1 |  30.3$\pm$1.3 |  30.7$\pm$1.0 |  30.9$\pm$1.5
T+FADA |  27.5$\pm$1.4 |  27.9$\pm$0.9 |  28.4$\pm$1.3 |  29.4$\pm$1.8 |  29.5$\pm$0.7 |  30.2$\pm$1.0 |  30.4$\pm$1.7
ST+FADA |  28.1$\pm$1.3 |  28.9$\pm$0.7 |  29.2$\pm$1.5 |  29.8$\pm$1.2 |  31.0$\pm$0.9 |  31.2$\pm$0.9 |  33.2$\pm$1.7
TOHAN | 29.9$\pm$1.2 | 30.5$\pm$1.2 | 31.4$\pm$1.1 | 32.8$\pm$0.9 | 33.1$\pm$1.0 | 34.0$\pm$1.0 | 35.1$\pm$1.8