# Cross-Domain Latent Modulation for Variational Transfer Learning

Jinyong Hou, Jeremiah D. Deng, Stephen Cranefield, Xuejie Ding
Department of Information Science, University of Otago
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We propose a cross-domain latent modulation mechanism within a variational
autoencoders (VAE) framework to enable improved transfer learning. Our key
idea is to procure deep representations from one data domain and use it as
perturbation to the reparameterization of the latent variable in another
domain. Specifically, deep representations of the source and target domains
are first extracted by a unified inference model and aligned by employing
gradient reversal. Second, the learned deep representations are cross-
modulated to the latent encoding of the alternate domain. The consistency
between the reconstruction from the modulated latent encoding and the
generation using deep representation samples is then enforced in order to
produce inter-class alignment in the latent space. We apply the proposed model
to a number of transfer learning tasks including unsupervised domain
adaptation and image-to-image translation. Experimental results show that our
model gives competitive performance.

##  1 Introduction

In machine learning, one can rarely directly apply a pre-trained model to a
new dataset or a new task, as the performance of a learned model often plunges
significantly for the new data which may have significant sampling bias or
even belong to different distributions. Transfer learning can help us utilize
the learned knowledge from a previous domain (the ‘source’) to improve
performance on a related domain or task (the ‘target’) [31, 35, 41].

From the perspective of probabilistic modeling [23, 39, 34], the key challenge
in achieving cross-domain transfer is to learn a joint distribution of data
from different domains. Once the joint distribution is learned, it can be used
to generate the marginal distribution of the individual domains [14, 23].
Under the variational inference scenario, an inferred joint distribution is
often applied to the latent space. Due to the coupling theory, inferring the
joint distribution from the marginal distributions of different domains is a
highly ill-posed problem [21]. To address this problem, UNIT [23] makes an
assumption that there is a shared latent space for the two domains. Usually,
this can be achieved by applying the adversarial strategy to the domains’
latent spaces. Another line of research focuses on the use of a complex prior
to improve the representation performance for the input data [28, 36, 12].
However, the previous works neglect the role of the generation process for the
latent space which could be helpful for cross-domain transfer scenarios.

In this paper, we propose a novel latent space reparameterization method, and
employ a generative process to cater for the cross-domain transferability.
Specifically, we incorporate a cross-domain component into the
reparameterization transformation, which builds the connection between the
variational representations and domain features in a cross-domain manner. The
generated transfer latent space is further tuned by domain-level adversarial
alignment and domain consistency between images obtained through
reconstruction and generations. We apply our model to the homogeneous transfer
scenarios, such as unsupervised domain adaptation and image-to-image
translation. The experimental results show the efficiency of our model.

The rest of the paper is organized as follows. In Section 2, some related work
is briefly reviewed. In Section 3, we outline the overall structure of our
proposed model and develop the learning metrics with defined losses. The
experiments are presented and discussed in Section 4. We conclude our work in
Section 5, indicating our plan of future work.

##  2 Related Work

Latent space manipulation: As discussed above, for a joint distribution,
manipulation of the latent space is common [17, 23, 22] for the cross-domain
adaptation situations. One approach focuses on a shared latent space, where
the latent encodings are regarded as common representations for inputs across
domains. Some adversarial strategy is usually used to pool them together so
that the representations are less domain-dependent. For the variational
approach, works in [28, 48, 15] adopt complex priors for multi-modal latent
representations, while other works [23, 34, 22] still assume a standard
Gaussian prior. Another aproach is to use disentangled latent representations
where the latent encoding is divided into some defined parts (e.g. style and
content parts), then the model learns separated representations and swaps them
for the transfer [9, 46, 6, 20]. Our method is different from these
approaches. In our model, learned auxiliary deep representation is used to
generate perturbations to the latent space through a modified
reparameterization using variational information from the counterpart domain.
It helps generate cross-domain image translation. The transfer is carried out
by a reparameterization transformation, using statistical moments retaining
specific information for one domain, and deep representation providing
information from another domain.

Varied Homogeneous transfer tasks: The manipulation on the latent space is
often interwoven with the homogeneous image transfer together, such as
unsupervised domain adaptation and image translation [29, 30, 5]. In the
domain separation networks [4], separate encoding modules are employed to
extract the invariant representation and domain-specific representations from
the domains respectively, with the domain-invariant representations being used
for the domain adaptation. References [3, 33, 11] transfer the target images
into source-like images for domain adaptation. References [23, 47, 13] map the
inputs of different domains to a single shared latent space, but cycle
consistency is required for the completeness of the latent space. The UFDN
[22] utilizes a unified encoder to extract the multi-domain images to a shared
latent space, and the latent domain-invariant coding is manipulated for image
translation between different domains.

In contrast, we adopt the pixel-level adaptation between domains from the
cross-domain generation, but the proposed model can also be used at the
feature-level due to the latent space alignment. Our model also has a unified
inference model, but the consistency is imposed in a straightforward way, with
reduced computational complexity.

##  3 Proposed Model

###  3.1 Problem setting

Let $\mathcal{X}\subset\mathbb{R}^{d}$ be a $d$-dimensional data space, and
$\bm{X}=\\{\bm{x}_{1},\bm{x}_{2},\ldots,\bm{x}_{n}\\}\in\mathcal{X}$ the
sample set with marginal distribution $p(\bm{X})$. The source domain is
denoted by a tuple $(\mathcal{X}_{s},p(\bm{X}_{s}))$, and the target domain by
$(\mathcal{X}_{t},p(\bm{X}_{t}))$. In our paper, we consider the homogeneous
transfer with domain shift, i.e. $\mathcal{X}_{s}\approx\mathcal{X}_{t}$, but
$p(\bm{X}_{s})\neq p(\bm{X}_{t})$. For the unsupervised pixel-level domain
adaptation scenario, the label set is $\bm{Y}=\\{y_{1},y_{2},\ldots
y_{n}\\}\in\mathcal{Y}$ ($\mathcal{Y}$ is the label space), and a task
$\mathcal{T}=p(\bm{Y}|\bm{X})$ is considered too. However, only the source
domain’s label set $\bm{Y}_{s}$ is available during transfer learning.

###  3.2 Transfer Latent Space

As any given marginal distribution can be yielded by an infinite number of
joint distributions, we need to build an inference framework with some
constraints. Under the variational autoencoder (VAE) framework, the latent
space is one of the manipulation targets. We propose the transfer latent space
as follows.

######  Definition 1.

Transfer Latent Space $\ddot{\bm{Z}}$. Let $\bm{x}_{s}\in\bm{X}_{s}$,
$\bm{x}_{t}\in\bm{X}_{t}$ be the domain samples. Let us have a map $f$ that
extracts domain information $\bm{\Omega}$ and a feature representation
$\bm{h}$ given an input $\bm{x}$:

$f:\bm{x}\longrightarrow(\bm{\Omega},\bm{h}),\bm{x}\in\bm{X}_{s}\cup\bm{X}_{t}.$

Suppose we construct a transfer map $\mathcal{G}$ that generates a latent
variable $\bm{\ddot{z}}$ from $\bm{\Omega}$ and $\bm{h}$ with domain
crossovers:

$\bm{\ddot{z}}_{st}=\mathcal{G}(\bm{\Omega}_{s},\bm{h}_{t}),$
$\bm{\ddot{z}}_{ts}=\mathcal{G}(\bm{\Omega}_{t},\bm{h}_{s}).$

The joint space formed by $\bm{\ddot{z}}_{st}$ and $\bm{\ddot{z}}_{ts}$
samples is defined as a transfer latent space, denoted by $\ddot{\bm{Z}}$.

The transfer latent space is intended to become a “mixer” for the two domains,
as the resulted latent variables are under cross-domain influences. Hence the
transfer latent space can be regarded as a generalization of the latent space.

###  3.3 Framework

Our framework is shown in Fig. 1. In our framework, we build the cross-domain
generation by a unified inference model $\mathbf{E}_{\phi}(\cdot)$ (as an
implementation of the map $f$) and a generative model for the desired domain
$\mathbf{D}_{\theta}(\cdot)$, e.g., the source domain in our model. A
discriminator $\Xi$ is utilized for the adversarial training. We use the terms
“inference model” and “encoder” for $\mathbf{E}_{\phi}(\cdot)$, and
“generative model” and “decoder” for $\mathbf{D}_{\theta}(\cdot)$
interchangeably.

Figure 1: Architectural view of the proposed model. It encourages an image
from target domain (blue hexagon) to be transformed to a corresponding image
in the source domain (black hexagon). The transfer latent distributions
$p(\bm{\ddot{z}}_{ts}|\bm{x}_{t},\bm{h}_{s})$ and
$p(\bm{\ddot{z}}_{st}|\bm{x}_{s},\bm{h}_{t})$ are learned which are used to
generate corresponding images by the desired decoder. The deep representations
are integrated into the reparameterization transformation with standard
Gaussian auxiliary noise. Blue lines are for the target domain and black ones
are for the source domain.

As discussed in section 3.2, under the variational framework, the domain
information $\bm{\Omega}$ (here we remove the domain subscript for simplicity)
is usually the pair $(\bm{\mu},\bm{\sigma})$. Let $\bm{h}^{\prime}$ be the
flattened activations of the last convolution layer in $\mathbf{E}_{u}$. Then,
following the treatment in [16], $\bm{\mu}$ and $\bm{\sigma}$ can be obtained
by $\bm{\mu}=\mathbf{W}_{\bm{\mu}}\bm{h}^{\prime}+\bm{b}_{\bm{\mu}}$ and
$\bm{\sigma}=\mathbf{W}_{\bm{\sigma}}\bm{h}^{\prime}+\bm{b}_{\bm{\sigma}}$,
where
$\mathbf{W}_{\bm{\mu}},\mathbf{W}_{\bm{\sigma}},\bm{b}_{\bm{\mu}},\bm{b}_{\bm{\sigma}}$
are the weights and biases for $\bm{\mu}$ and $\bm{\sigma}$.

From our observations, both shallow (e.g. PCA features) and deep
representations can be used to obtain domain information $\bm{h}$, in our end-
to-end model we use the latter. We choose the high-level activation of the
last convolutional layer, i.e.,
$\bm{h}=\text{sigmoid}(\mathbf{W}_{\bm{h}}\bm{h}^{\prime}+\bm{b}_{\bm{h}})$ as
the deep representation [44], where $\mathbf{W}_{\bm{h}},\bm{b}_{\bm{h}}$ are
the weights and biases for the deep abstractions.

Having obtained the domain information $\bm{\Omega}$ and deep representation
$\bm{h}$, a natural choice for the transfer map $\mathcal{G}$ is through
reparameterization. Here we propose a modified reparameterization trick to
give the sampling from the transfer latent space as follows:

$\bm{\ddot{z}}_{st}=\mathcal{G}((\bm{\mu}_{s},\bm{\sigma}_{s}),\bm{h}_{t})=\bm{\mu}_{s}+\bm{\sigma}_{s}\odot(\gamma_{1}\bm{h}_{t}+\gamma_{2}\bm{\epsilon}),$
(1)

and

$\bm{\ddot{z}}_{ts}=\mathcal{G}((\bm{\mu}_{t},\bm{\sigma}_{t}),\bm{h}_{s})=\bm{\mu}_{t}+\bm{\sigma}_{t}\odot(\gamma_{1}\bm{h}_{s}+\gamma_{2}\bm{\epsilon}),$
(2)

where $\bm{h}_{s}$ ($\bm{h}_{t}$) is the sample of the deep representation
space $\mathcal{H}_{s}$ ($\mathcal{H}_{t}$); $\bm{\mu}_{s}$ and
$\bm{\sigma}_{s}$ ($\bm{\mu}_{t}$ and $\bm{\sigma}_{t}$) are the mean and
standard deviation of the approximate posterior for the source (target)
domain; $\gamma_{1},\gamma_{2}>0$ are trade-off hyperparameter to balance the
deep feature modulation and the standard Gaussian noise $\bm{\epsilon}$; and
$\odot$ stands for the element-wise product of vectors. Therefore, the
auxiliary noise in VAE resampling is now a weighted sum of a deep
representation from the other domain and Gaussian noise, different from the
standard VAE framework. Because the modified reparameterization allows a
domain’s deep representation to get modulated into another domain’s latent
representation, we call our model “Cross-Domain Latent Modulation”, or CDLM
for short.

Now we have obtained the transfer encodings by a unified encoder. Following
the probabilistic encoder analysis [16], a shared inference model confines the
latent variables into the same latent space. But this cannot guarantee them to
be aligned. To pull the domains close, an adversarial strategy [7, 37] should
be used for the alignment. The gradient reversal layer [7] is used in our
model, by which adversarial learning is introduced to learn transferable
features that are robust to domain shift. The adversarial alignment between
$\mathcal{H}_{s}$ and $\mathcal{H}_{t}$ is for domain-level.

Furthermore, for better interpretation of the modulated reparameterization,
let $\bm{h}_{t}\sim
p(\mathcal{H}_{t})=\mathcal{N}(\bm{\mu}_{\bm{h}_{t}},\bm{\sigma}_{\bm{h}_{t}})$.
Then for $\bm{\ddot{z}}_{st}$, we have
$\bm{\ddot{z}}_{st}\sim\mathcal{N}(\bm{\ddot{z}}_{st};\bm{\mu}_{st},\bm{\sigma}_{st}^{2}\mathbf{I})$.
For the $i$-th element of the distribution moments are given as follows:

$\begin{split}\mu_{st}^{i}&=\mathbb{E}\\{\mu^{i}_{s}+\sigma_{s}^{i}(\gamma_{1}h_{t}^{i}+\gamma_{2}\epsilon^{i})\\}\\\
&=\mu_{s}^{i}+\gamma_{1}\sigma_{s}^{i}\mu_{h_{t}}^{i}.\end{split}$ (3)
$\begin{split}\mathbf{Var}(\ddot{z}_{st}^{i})&=\mathbb{E}\\{(\sigma_{s}^{i}(\gamma_{1}h_{t}^{i}+\gamma_{2}\epsilon^{i})-\gamma_{1}\sigma_{s}^{i}\mu_{h_{t}}^{i})^{2}\\}\\\
&=(\sigma_{s}^{i})^{2}[\gamma_{1}^{2}(\sigma_{h_{t}}^{i})^{2}+\gamma_{2}^{2}].\end{split}$
(4)

Therefore, the $\mu_{st}^{i}$ and $\sigma_{st}^{i}$ are

$\begin{gathered}\mu_{st}^{i}=\mu_{s}^{i}+\gamma_{1}\sigma_{s}^{i}\mu_{\bm{h}_{t}}^{i}\\\
\sigma_{st}^{i}=\sigma_{s}^{i}\sqrt{\gamma_{1}^{2}(\sigma_{\bm{h}_{t}}^{i})^{2}+\gamma_{2}^{2}}.\end{gathered}$
(5)

Here, it is reasonable to assume $\bm{\mu}_{\bm{h}_{t}}\approx\bm{\mu}_{t}$
and $\bm{\sigma}_{\bm{h}_{t}}\approx\bm{\sigma}_{t}$ when the training is
finished. With a practical setting of $\gamma_{1}\gg\gamma_{2}$, and in effect
$\sigma_{s}^{i}=1$, Eq. (5) can be further simplified to
$\mu_{st}^{i}=\mu_{s}^{i}+\gamma_{1}\mu_{t}^{i}$, and
$\sigma_{st}^{i}=\gamma_{1}\sigma_{s}^{i}\sigma_{t}^{i}$. Then we can see that
$\bm{\mu}_{st}$ can be regarded as a location shift of $\bm{\mu}_{s}$ under
the influence of $\bm{\mu}_{t}$, which helps reduce the domain gap;
$\bm{\sigma}_{st}$ can be taken as a recoloring of $\bm{\sigma}_{s}$ under the
influence from the target. The formulation of $\bm{\ddot{z}}_{ts}$ can be
similarly interpreted. These modulated encodings are hence constructed in a
cross-domain manner.

Next, we apply the consistency constraint to the transfer latent space with
modulation for further inter-class alignment. It has been found that
consistency constraints preserve class-discriminative information [34, 13, 8].
For our model, the consistency is applied to the reconstructions from
modulated encodings and the corresponding generations from deep
representations. Let $\mathbf{D}_{\theta}(\cdot)$ be the generative model for
domain image generation from the transfer latent space. The consistency
requirements are

$\begin{gathered}\mathbf{D}_{\theta}(\bm{\ddot{z}}_{st})=\mathbf{D}_{\theta}(\gamma_{1}\bm{h}_{s}+\gamma_{2}\bm{\epsilon})\\\
\mathbf{D}_{\theta}(\bm{\ddot{z}}_{ts})=\mathbf{D}_{\theta}(\gamma_{1}\bm{h}_{t}+\gamma_{2}\bm{\epsilon}),\end{gathered}$
(6)

where $\mathbf{D}_{\theta}(\bm{\ddot{z}}_{st})$ is the reconstruction of the
source ($\widehat{\bm{x}}_{st}$), $\mathbf{D}_{\theta}(\bm{\ddot{z}}_{ts})$ is
for the target ($\widehat{\bm{x}}_{ts}$). $\mathbf{D}_{\theta}(\cdot)$ can
also function as a generative model, generating
$\widetilde{\bm{x}}_{s}=\mathbf{D}_{\theta}(\gamma_{1}\bm{h}_{s}+\gamma_{2}\bm{\epsilon})$
and
$\widetilde{\bm{x}}_{t}=\mathbf{D}_{\theta}(\gamma_{1}\bm{h}_{t}+\gamma_{2}\bm{\epsilon})$
for the source and target domain respectively. Also, the consistencies can
guide the encoder to learn the representations from both domains.

Finally, a desired marginalized decoder, e.g. the source decoder, is trained
to map the target images to be source-like. We render the target’s structured
generation $\widetilde{\bm{x}}_{ts}$ for the test mode. For this end, we do
not need the source to be taken into account for the test. That means a test
image from the target domain $\bm{x}_{t}^{i}$ first passes through the
inference model and obtains its deep feature $\bm{h}_{t}^{i}$. Then it is fed
into the generation model to generate an image with source style but keep its
own class. That is to make the marginal distribution
$p(\widetilde{\bm{x}}_{ts}^{i})\approx p(\bm{x}_{s}^{j})$, but keep its class
$y_{t}^{i}$.

###  3.4 Learning

Our goal is to update the variational parameters to learn a joint distribution
and the generation parameters for the desired marginal distribution. Since the
latent variables are generated with inputs from both domains, we have a
modified formulation adapted from the plain VAE:

$\begin{aligned} \log
p(\bm{x}_{s},\bm{x}_{t})-\mathrm{KL}(q_{\phi}(\bm{\ddot{z}}|\bm{x}_{s},\bm{x}_{t})\|p(\bm{\ddot{z}}|\bm{x}_{s},\bm{x}_{t}))&=\mathbb{E}[\log
p(\bm{x}_{s},\bm{x}_{t}|\bm{\ddot{z}})]\\\
&-\mathrm{KL}(q_{\phi}(\bm{\ddot{z}}|\bm{x}_{s},\bm{x}_{t})\|p(\bm{\ddot{z}})),\end{aligned}$

(7)

where $\mathrm{KL}(\cdot)$ is the Kullback-Leibler divergence, and the
transfer latent variable $\bm{\ddot{z}}$ can be either $\bm{\ddot{z}}_{st}$ or
$\bm{\ddot{z}}_{ts}$. Minimizing
$\mathrm{KL}(q_{\phi}(\bm{\ddot{z}}|\bm{x}_{s},\bm{x}_{t})\|p(\bm{\ddot{z}}|\bm{x}_{s},\bm{x}_{t}))$
is equivalent to maximizing the variational evidence lower bound (ELBO)
$\mathcal{L}(\bm{x}_{s},\bm{x}_{t},\bm{\theta},\bm{\phi})$:

$\mathcal{L}(\bm{x}_{s},\bm{x}_{t},\bm{\theta},\bm{\phi})=\mathbb{E}_{q_{\phi}}[\log
p_{\theta}(\bm{x}_{s},\bm{x}_{t}|\bm{\ddot{z}})]-\mathrm{KL}(q_{\phi}(\bm{\ddot{z}}|\bm{x}_{s},\bm{x}_{t})\|p(\bm{\ddot{z}})),$

(8)

where the first term corresponds to the reconstruction cost
($\mathcal{L}_{\mathit{Rec}}$), and the second term is the K-L divergence
between the learned latent probability and the prior (specified as
$\mathcal{N}(0,\bm{I})$) ($\mathcal{L}_{\mathit{KL}}$). Considering the
reconstruction of $\bm{x}_{s}$, and the K-L divergence for both
$\bm{\ddot{z}}_{st}$ and $\bm{\ddot{z}}_{ts}$, we have

$\begin{split}\mathcal{L}(\bm{x}_{s},\bm{x}_{t},\bm{\theta},\bm{\phi})&=\mathbb{E}_{\bm{\ddot{z}}\sim
q(\bm{\ddot{z}}_{st}|\bm{x}_{s},\bm{x}_{t})}[\log
p_{\bm{\theta}}(\bm{x}_{s}|\bm{\ddot{z}}_{st})]\\\ &-\mathrm{KL}(\log
q_{\bm{\phi}}({\bm{\ddot{z}}_{st}|\bm{x}_{s},\bm{x}_{t}})\|p(\bm{z}))\\\
&-\mathrm{KL}(\log
q_{\bm{\phi}}({\bm{\ddot{z}}_{ts}|\bm{x}_{t},\bm{x}_{s}})\|p(\bm{z})).\\\
\end{split}$ (9)

To align the deep representations of the source and target domains, an
adversarial strategy is employed to regularize the model. The loss function is
given by

$\mathcal{L}_{adv}=\mathbb{E}_{\bm{h}_{s}\sim
p(\bm{h}_{s}|\bm{x}_{s})}[\log\Xi(\bm{h}_{s})]+\mathbb{E}_{\bm{h}_{t}\sim
p(\bm{h}_{t}|\bm{x}_{t})}[\log(1-\Xi(\bm{h}_{t}))],$

(10)

where $\Xi(\cdot)$ is the discriminator to predict from which domain the deep
representation feature is.

From the analysis in Section 3.2, we can introduce a pairwise consistency
between the reconstruction and the generation for the source and the target in
an unsupervised manner respectively. The consistencies regularization improve
the inter-class alignment. For the consistency loss $\mathcal{L}_{c}$, both
the $l^{1}$ and $l^{2}$-norm penalty can be used to regularize the decoder.
Here we simply use MSE. Let $\mathcal{L}_{c}^{s}$ and $\mathcal{L}_{c}^{t}$ be
the consistency for the domains respectively. $\mathcal{L}_{c}$ is given as a
combination of these two components, weighted by two coefficients $\beta_{1}$
and $\beta_{2}$, respectively:

$\begin{aligned}
\mathcal{L}_{c}&=\beta_{1}\mathcal{L}_{c}^{s}+\beta_{2}\mathcal{L}_{c}^{t}\\\
&=\beta_{1}(\mathbb{E}_{\bm{\ddot{z}}\sim
q(\bm{\ddot{z}}|\bm{x}_{s},\bm{x}_{t})}[\log
p(\widehat{\bm{x}}_{s}|\bm{\ddot{z}}_{st})]-\mathbb{E}_{\bm{h}_{s}\sim
p(\bm{h}_{s}|\bm{x}_{s})}[\log p(\widetilde{\bm{x}}_{s}|\bm{h}_{s})])^{2}\\\
&+\beta_{2}(\mathbb{E}_{\bm{\ddot{z}}\sim
q(\bm{\ddot{z}}|\bm{x}_{s},\bm{x}_{t})}[\log
p(\widehat{\bm{x}}_{ts}|\bm{\ddot{z}}_{ts})]-\mathbb{E}_{\bm{h}_{t}\sim
p(\bm{h}_{t}|\bm{x}_{t})}[\log
p(\widetilde{\bm{x}}_{ts}|\bm{h}_{t})])^{2}.\end{aligned}$

(11)

Then, the variational parameters $\bm{\phi}$ and generation parameters
$\bm{\theta}$ are updated by the following rules:

$\begin{gathered}\bm{\phi}\leftarrow\bm{\phi}-\eta_{1}\nabla(\mathcal{L}_{\mathit{adv}}+\lambda_{1}\mathcal{L}_{\mathit{KL}}+\lambda_{2}\mathcal{L}_{\mathit{Rec}})\\\
\bm{\theta}\leftarrow\bm{\theta}-\eta_{2}\nabla(\mathcal{L}_{\mathit{Rec}}+\mathcal{L}_{c}),\end{gathered}$
(12)

where $\eta_{1},\eta_{2}$ are the learning rates. Note, that only data from
the desired domain (the source) are used to train the reconstruction loss. The
$\mathrm{KL}$ items approximate the transfer latent space to their prior.
Hyperparameters $\lambda_{1}$, $\lambda_{2}$ are used to balance the
discriminator loss and reconstruction loss.

##  4 Experiments

Table 1: Mean classification accuracy comparison. The “source only" row is the
accuracy for target without domain adaptation training only on the source. The
“target only" is the accuracy of the full adaptation training on the target.
For each source-target task the best performance is in bold

Source | MNIST | USPS | MNIST | MNISTM | Fashion | Fashion-M | Linemod 3D
---|---|---|---|---|---|---|---
Target | USPS | MNIST | MNISTM | MNIST | Fashion-M | Fashion | Linemod Real
Source Only | 0.634 | 0.625 | 0.561 | 0.633 | 0.527 | 0.612 | 0.632
DANN [7] | 0.774 | 0.833 | 0.766 | 0.851 | 0.765 | 0.822 | 0.832
CyCADA [11] | 0.956 | 0.965 | 0.921 | 0.943 | 0.874 | 0.915 | 0.960
GtA [33] | 0.953 | 0.908 | 0.917 | 0.932 | 0.855 | 0.893 | 0.930
CDAN [27] | 0.956 | 0.980 | 0.862 | 0.902 | 0.875 | 0.891 | 0.936
PixelDA [3] | 0.959 | 0.942 | 0.982 | 0.922 | 0.805 | 0.762 | 0.998
UNIT [23] | 0.960 | 0.951 | 0.920 | 0.932 | 0.796 | 0.805 | 0.964
CDLM ($\widetilde{\bm{x}}_{ts}$) | 0.961 | 0.983 | 0.987 | 0.962 | 0.913 | 0.922 | 0.984
Target Only | 0.980 | 0.985 | 0.983 | 0.985 | 0.920 | 0.942 | 0.998

We conducted extensive evaluations of CDLM in two homogeneous transfer
scenarios including unsupervised domain adaptation and image-to-image
translation. During the experiments, our model was implemented using
TensorFlow [1]. The structures of the encoder and the decoder adopt those of
UNIT [23] which perform well for image translation tasks. A two-layer fully
connected MLP was used for the discriminator. SGD with momentum was used for
updating the variational parameters, and Adam for updating generation
parameters. The batch size was set to 64. During the experiments, we set
$\gamma_{1}=1.0,\gamma_{2}=0.1$, $\lambda_{1}=\lambda_{2}=0.0001$,
$\beta_{1}=0.1$ and $\beta_{2}=0.01$. For the datasets, we considered a few
popular benchmarks, including MNIST [19], MNSITM [7], USPS [18], Fashion-MNIST
[43], Linemod [10, 42], Zap50K-shoes [45] and CelebA [25, 24].

###  4.1 Datasets

We have evaluated our model on a variety of benchmark datasets. They are
described as follows.

MNIST: MNIST handwritten dataset [19] is a very popular machine learning
dataset. It has a training set of 60,000 binary images, and a test set of
10,000. There are 10 classes in the dataset. In our experiments, we use the
standard split of the dataset. MNISTM [7] is a modified version for the MNIST,
with random RGB background cropped from the Berkeley Segmentation
Dataset111URL
https://www2.eecs.berkeley.edu/Research/Projects/CS/vision/bsds/.

USPS: USPS is a handwritten zip digits datasets [18]. It contains 9298 binary
images ($16\times 16$), 7291 of which are used as the training set, while the
remaining 2007 are used as the test set. The USPS samples are resized to
$28\times 28$, the same as MNIST.

Fashion: Fashion [43] contains 60,000 images for training, and 10,000 for
testing. All the images are grayscale, $28\times 28$ in size space. In
addition, following the protocol in [7], we add random noise to the Fashion
images to generate the FashionM dataset, with random RGB background cropped
from the Berkeley Segmentation Dataset.

Linemod 3D images Following the protocol of [3], we render the LineMod [10,
42] for the adaptation between synthetic 3D images (source) and real images
(target). The objects with different poses are located at the center of the
images. The synthetic 3D images render a black background and a variety of
complex indoor environments for real images. We use the RGB images only, not
the depth images.

CelebA: CelebA [25] is a large celebrities face image dataset. It contains
more than 200k images annotated with 40 facial attributes. We select 50K
images randomly, then transform them to sketch images followed the protocol of
[24]. The original and sketched images are used for translation.

UT-Zap50K-shoes: This dataset [45] contains 50K shoes images with 4 different
classes. During the translation, we get the edges produced by canny detector.

###  4.2 Unsupervised Domain Adaptation

We applied our model to unsupervised domain adaptation, adapting a classifier
trained using labelled samples in the source domain to classify samples in the
target domain. For this scenario, only the labels of the source images were
available during training. We chose DANN [7] as the baseline, but also
compared our model with the state-of-the-art domain adaptation methods:
Conditional Domain Adaptation Network (CDAN) [27], Pixel-level Domain
Adaptation (PixelDA) [3], Unsupervised Image-to-Image translation (UNIT) [23],
Cycle-Consistent Adversarial Domain Adaption (CyCADA) [11], and Generate to
Adapt (GtA) [33]. We also used source- and target-only training as the lower
and upper bound respectively, following the practice in [3, 7].

####  4.2.1 Quantitative Results

The performance of domain adaptation for the different tasks is shown in Table
1. There are 4 scenarios and 7 tasks. Each scenario has bidirectional tasks
for adaptation except LineMod. For LineMod, it is adapted from synthetic 3D
image to real objects. For the same adaptation task, we cite the accuracy from
the corresponding references, otherwise the accuracies for some tasks are
obtained by training the open-source code provided by authors with suggested
optimal parameters, for fair comparison.

From Table 1 we can see that the our method has a higher advantage compared
with the baseline and the source-only accuracy, a little lower than the
target-only accuracy from both adaptation directions. In comparison with other
models, our model has a better performance for most of tasks. The CDLM has a
higher adaptation accuracy for the scenarios with seemingly larger domain gap,
such as MNIST$\rightarrow$MNISTM and Fashion$\rightarrow$FashionM. For the 3D
scenario, the performance of our model is a little lower than PixelDA [3], but
outperforms all the other compared methods. In PixelDA, the input is not only
source image but also depth image pairs. It might be helpful for the
generation. Besides, we visualize the t-SNE [38] for latent encodings
($\bm{\ddot{z}}_{st},\bm{\ddot{z}}_{ts}$) w.r.t the source and the target,
respectively. Fig. 4 is the visualization for task MNISTM$\rightarrow$MNIST
and MNIST$\rightarrow$USPS, and it shows that both are aligned well.

####  4.2.2 Qualitative Results

Our model can give the visualization of the adaptation. Fig. 2 is the
visualization for the digits and Fashion adaptation respectively. For the
scenario of MNIST and USPS, the generation for the task USPS$\rightarrow$MNIST
is shown in Fig. 2(a). The target MNIST is transferred to the source USPS
style well, meanwhile it keeps the correspondent content (label). For example,
the digit ‘1’ in MNIST become more leaned and ‘9’ more flatten. Also in Fig.
2(b), the target USPS becomes the MNIST style. For the scenario of
MNIST$\rightarrow$MNISTM, our proposed model can remove and add the noise
background well for adaptation.

(a) USPS$\rightarrow$MNIST

(b) MNIST$\rightarrow$USPS

(c) MNIST$\rightarrow$MNISTM

(d) MNISTM$\rightarrow$MNIST

(e) Fashion$\rightarrow$Fashionm

(f) Fashionm$\rightarrow$Fashion

Figure 2: Visualization for the adaptations. 6 different tasks are
illustrated. For each task, the first row shows target images and the second
row shows the adapted images with source-like style. Figure 3: Linemod 3D
Synthetic$\rightarrow$Real. For a query image (on the left), different
adaptation images (to the right) with various poses can be generated.

(a) Task MNISTM$\rightarrow$MNIST

(b) Task MNIST$\rightarrow$USPS.

Figure 4: t-SNE visualization of cross-domain latent encodings
$\bm{\ddot{z}}_{st},\bm{\ddot{z}}_{ts}$. The $\bm{\ddot{z}}_{st}$ are in blue,
and the $\bm{\ddot{z}}_{ts}$ in red.

For the scenario of Fashion, the fashion items have more complicated texture
and content. In addition, the noisy backgrounds pollute the items randomly,
for example, different parts of a cloth are filled with various colors. For
visualizations, specifically, Fig. 2(e) is for the task Fashion $\rightarrow$
FashionM. The proposed model can remove the noisy background and maintain the
content. On the other hand, Fig. 2(f) shows that the original Fashion images
are added with similar noisy background as the source. This is promising for a
better adaptation performance.

For Linemod3D, the real objects images with different backgrounds are
transferred to the synthetic 3D images with black background. Due to the 3D
style, the generation of the target gives different poses. For example in
Fig.3, different poses of the iron object are obtained for different trials.

(a) “edge” to “shoes”

(b) “sketch” to “face”

(c) “shoes” to “edge”

(d) “face” to “sketch”

Figure 5: Visualization of cross-domain image mapping.

###  4.3 Cross-Domain Image Mapping

The proposed model also can be used for the cross-domain image mapping.

Fig. 5 gives a demonstration of the image style translation. Specifically, for
“shoes” and “edges” in Fig. 5(a) and 5(c), we can see that the proposed model
can translate “edges” to its counterpart quite well. The translation is
stochastic – an “edge” pattern can be used to generate “shoes” in different
colors with different trials. For the more challenging “face” and “sketch”
translations, the proposed model also performs well. The generations have some
variations compared with the original images. In general, our method can
generate realistic translated images. However, we find that compared with the
translation from sketches to real images, the reverse task seems harder. For
example, when a face image is given, the generated sketch loses some details.
The reason may be the low-level feature is neglected when the deep feature
acts as the condition.

For further evaluation, quantitative performance is evaluated for image
mapping. SSIM [40], MSE, and PSNR are used for the evaluation. The results are
shown in Table 2. We can see that our model outperforms E-CDRD [24], which
learns a disentangled latent encoding for the source and the target for domain
adaptation. Meanwhile, it matches the performance of StarGAN [5], which is
designed for multi-domain image translation. The result shows that our model
can map cross-domain images well compared to these prior works.

Table 2: Performance for image mapping.

Models | “Sketch” to “Face”
---|---
SSIM | MSE | PSNR
E-CDRD [24] | 0.6229 | 0.0207 | 16.86
StarGAN [5] | 0.8026 | 0.0142 | 19.04
CDLM | 0.7961 | 0.0140 | 19.89

In addition, we also conduct the classification to evaluate the translation
performance. We take shoes as an example which are labeled to 4 different
classes. The recognition accuracy of our proposed model for task
shoes$\rightarrow$edge is 0.953, which is higher than the results of PixelDA
(0.921) and UNIT (0.916) respectively.

###  4.4 Model Analysis

The effect of encoder settings – depth and different $\gamma_{1},\gamma_{2}$:
In our model, the deep features are utilized to cross-modulate the transfer
latent encoding. Therefore the deep feature is an important factor in our
framework and is influenced by the depth of the encoder. During the
experiments, we use MNIST $\rightarrow$ USPS and Fashion $\rightarrow$
FashionM as the evaluation tasks. For the first one, they have different
content, but with the same background. The second task is a totally different
scenario, the images have the same content but different background. The
outputs of different encoder layers ($k>3$) are used for the experiments.

Table 3: Adaptation accuracy with different layer depth for Tasks MNIST
$\rightarrow$ USPS and Fashion $\rightarrow$ FashionM.

Tasks/Layers | Conv4 | Conv5 | Convlast
---|---|---|---
MNIST $\rightarrow$ USPS | 0.954 | 0.956 | 0.961
Fashion $\rightarrow$ FashionM | 0.890 | 0.905 | 0.913

Table 4: Adaptation accuracy with different ($\gamma_{1},\gamma_{2}$) for
Tasks MNIST $\rightarrow$ USPS and Fashion $\rightarrow$ FashionM.

Tasks / ($\gamma_{1},\gamma_{2}$) | (0.1,1.0) | (0.5,0.5) | (0.9,0.1) | (1.0,0.1) | (1.0, 0)
---|---|---|---|---|---
MNIST $\rightarrow$ USPS | 0.320 | 0.723 | 0.961 | 0.961 | 0.961
Fashion $\rightarrow$ FashionM | 0.226 | 0.513 | 0.912 | 0.913 | 0.913

Table 5: Evaluation on the effect of unsupervised consistency metrics. The
recognition accuracy is shown for four tasks in the unsupervised domain
adaptation scenario. Our model is on the last row with both the
$\mathcal{L}_{c}^{s}$ and $\mathcal{L}_{c}^{t}$, which achieves the best
performance.

Model/Tasks | MNIST$\rightarrow$USPS | USPS$\rightarrow$MNIST | Fashion$\rightarrow$FashionM | FashionM$\rightarrow$Fashion
---|---|---|---|---
CDLM w/o $\mathcal{L}_{c}$ | 0.635 | 0.683 | 0.646 | 0.672
CDLM+$\mathcal{L}_{c}^{t}$ | 0.689 | 0.695 | 0.682 | 0.691
CDLM+$\mathcal{L}_{c}^{s}$ | 0.951 | 0.980 | 0.912 | 0.915
CDLM+$\mathcal{L}_{c}^{s}$+$\mathcal{L}_{c}^{t}$ | 0.961 | 0.983 | 0.913 | 0.922

As the result (Table 3) shows, a higher accuracy is achieved when more layers
are used to extract the deep representations. The accuracy gain of the task
MNIST $\rightarrow$ USPS is lower than that of Fashion $\rightarrow$ FashionM.
This is expected as features extracted by higher layers would normally
eliminate lower-level variations between domains, such as change of background
and illumination in the images.

For $\gamma_{1},\gamma_{2}$, we fixed the last convolutional layer for the
deep representations and evaluate different values. From Table 4, we can see
that the performance drops down significantly with a smaller $\gamma_{1}$
compared with $\gamma_{2}$, and increased with a larger $\gamma_{1}$. The
performance seems to be stabilized when $\gamma_{1}$ is greater than 0.9 while
$\gamma_{2}$ remains 0.1. Following the standard VAE, we keep the noise
$\bm{\epsilon}$ ($\gamma_{2}\neq 0$) in the evaluations. Meanwhile, our model
works well even when $\gamma_{2}=0$. These results suggest the deep
representation plays a crucial role in the cross-domain modulation.

$\mathcal{A}$-Distance: In a theoretical analysis of the domain discrepancy
[2], Ben-David et al. suggests that $\mathcal{A}$-distance can be used as a
measure for the domain discrepancy. As the exact $\mathcal{A}$-distance is
intractable, a proxy is defined as $\hat{d}_{\mathcal{A}}=2(1-2\epsilon)$,
where $\epsilon$ is the generalization error of a binary classifier (e.g.
kernel SVM) trained to distinguish the input’s domain (source or target).
Following the protocol of [26, 32], we calculate the $\mathcal{A}$-distance on
four adaptation tasks under the scenarios of Raw features, DANN features, and
CDLM features respectively. The results are show in Fig. 6. We observe that
both the DANN and CDLM reduce the domain discrepancy compared with the Raw
images scenario, and the $\mathcal{A}$-distance of CDLM is smaller than the
DANN’s. This demonstrates that it is harder to distinguish the source and the
target by the CDLM generations.

Figure 6: $\mathcal{A}$-distances comparison for four tasks.

Convergence: We also conduct the convergence experiment with training error on
task MNSIT-USPS to evaluate our model. As shown in the Fig. 7, our model has a
better convergence than DANN, thought there are some oscillations at the
beginning of the training. In addition, the error of CDLM is lower that the
DANN, which demonstrate that CDLM has a better adaptation performance. This is
consistent with the adaptation performance in Table 1.

Figure 7: Convergence of CDLM compared with DANN.

The effect of unsupervised consistency metrics: In our model, two unsupervised
consistency metrics are added for generation in good effects. The adaptation
accuracy is used for evaluation. Table 5 is the results for the four different
tasks. The performance w/o $\mathcal{L}_{c}$ is dropped down because the
decoder cannot generate realistic cross-domain images. $\mathcal{L}_{c}^{t}$
connects outputs generated from the $\bm{h}_{t}$ and $\ddot{\bm{z}}_{ts}$ only
for the target, which improves the performance slightly. Meanwhile, we can see
that the $\mathcal{L}_{c}^{s}$ loss boosts the accuracy for adaptation
significantly, which connects the two domains with the generations by the
$\bm{h}$. Finally, the scenario with both $\mathcal{L}_{c}^{s}$ and
$\mathcal{L}_{c}^{t}$ gives the best performance in all four tasks. It bridges
both the $\bm{h}$ and $\ddot{\bm{z}}$ between the two domains.

##  5 Conclusion

In this paper, we have presented a novel variational cross-domain transfer
learning model with cross modulation of deep representations from different
domains. A shared transfer latent space is introduced, and the
reparameterization transformation is modified to enforce the connection
between domains. Evaluations carried out in unsupervised domain adaptation and
image translation tasks demonstrate our model’s competitive performance. Its
effectiveness is also clearly shown in visual assessment of the adapted
images, as well as in the alignment of the latent information as revealed by
visualization using t-SNE. Overall, competitive performance has been achieved
by our model despite its relative simplicity.

For future work, we intend to further improve our variational transfer
learning framework and use it for heterogeneous, multi-domain transfer tasks.

## References

  * [1] Mart Abadi et al.  TensorFlow: Large-scale machine learning on heterogeneous distributed systems.  arXiv:1603.04467 [cs.DC], 2016.
  * [2] Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira.  Analysis of representations for domain adaptation.  In Advances in Neural Information Processing Systems (NeurIPS), pages 137–144, 2007.
  * [3] Konstantinos Bousmalis, Nathan Silberman, David Dohan, Dumitru Erhan, and Dilip Krishnan.  Unsupervised pixel-level domain adaptation with generative adversarial networks.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 95–104, 2017.
  * [4] Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan.  Domain separation networks.  In Advances in Neural Information Processing Systems (NeurIPS), pages 343–351, 2016.
  * [5] Yunjey Choi, Minje Choi, Munyoung Kim, Jung Woo Ha, Sunghun Kim, and Jaegul Choo.  StarGAN: Unified generative adversarial networks for multi-domain image-to-image translation.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 8789–8797, 2018.
  * [6] Zunlei Feng, Anxiang Zeng, Xinchao Wang, Dacheng Tao, Chenglong Ke, and Mingli Song.  Dual swap disentangling.  In Advances in Neural Information Processing Systems (NeurIPS), pages 5894–5904, 2018.
  * [7] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor S Lempitsky.  Domain-adversarial training of neural networks.  Journal of Machine Learning Research (JMLR), 17(59):1–35, 2016\.
  * [8] Muhammad Ghifary, W. Bastiaan Kleijn, Mengjie Zhang, David Balduzzi, and Wen Li.  Deep reconstruction-classification networks for unsupervised domain adaptation.  In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, European Conference on Computer Vision (ECCV), pages 597–613, 2016.
  * [9] Abel Gonzalez-garcia, Joost van de Weijer, and Yoshua Bengio.  Image-to-image translation for cross-domain disentanglement.  In Advances in Neural Information Processing Systems (NeurIPS), pages 1294–1305, 2018.
  * [10] Stefan Hinterstoisser, Cedric Cagniart, Slobodan Ilic, Peter Sturm, Nassir Navab, Pascal Fua, and Vincent Lepetit.  Gradient response maps for real-time detection of texture-less objects.  IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 34(5):876–888, 2012.
  * [11] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Yan Zhu, Phillip Isola, Kate Saenko, Alexei A. Efros, and Trevor Darrell.  CyCADA: Cycle-consistent adversarial domain adaptation.  In Jennifer G Dy and Andreas Krause, editors, International Conference on Machine Learning (ICML), volume 80 of PMLR, pages 3162–3174, 2018.
  * [12] Matthew D Hoffman and Matthew J Johnson.  ELBO surgery: Yet another way to carve up the variational evidence lower bound.  In Advances in Neural Information Processing Systems Workshop (NeurIPSW), 2016.
  * [13] Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim.  Learning to discover cross-domain relations with generative adversarial networks.  In Yee Whye Teh Doina Precup, editor, International Conference on Machine Learning (ICML), volume 70 of PMLR, pages 1857–1865, 2017.
  * [14] Diederik P Kingma.  Fast gradient-based inference with continuous latent variable models in auxiliary form.  arXiv:1306.0733 [cs.LG], 2013.
  * [15] Diederik P. Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling.  Improved variational inference with inverse autoregressive flow.  In Advances in Neural Information Processing Systems (NeurIPS), pages 4743–4751, 2016.
  * [16] Diederik P. Kingma and Max Welling.  Auto-encoding variational Bayes.  In International Conference on Learning Representations (ICLR), 2014\.
  * [17] Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther.  Autoencoding beyond pixels using a learned similarity metric.  In Maria Florina Balcan and Kilian Q. Weinberger, editors, International Conference on Machine Learning (ICML), volume 48 of PMLR, pages 1558–1566, 2016.
  * [18] Yann Le Cun, L D Jackel, B Boser, J S Denker, H P Graf, Isabelle Guyon, Don Henderson, R E Howard, and W Hubbard.  Handwritten digit recognition: Applications of neural network chips and automatic learning.  IEEE Communications Magazine, 27(11):41–46, 1989.
  * [19] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324, 1998.
  * [20] Hsin Ying Lee, Hung Yu Tseng, Jia Bin Huang, Maneesh Singh, and Ming Hsuan Yang.  Diverse image-to-image translation via disentangled representations.  In Europen Conference on Computer Vision (ECCV), pages 36–52, 2018\.
  * [21] Torgny Lindvall.  Lectures on the Coupling Method.  Dover, 2002.
  * [22] Alexander H. Liu, Yen Cheng Liu, Yu Ying Yeh, and Yu Chiang Frank Wang.  A unified feature disentangler for multi-domain image translation and manipulation.  In Advances in Neural Information Processing Systems (NeurIPS), pages 2590–2599, 2018.
  * [23] Ming-Yu Yu Liu, Thomas Breuel, and Jan Kautz.  Unsupervised image-to-image translation networks.  In Advances in Neural Information Processing Systems (NeurIPS), pages 700–708, 2017.
  * [24] Yen Cheng Liu, Yu Ying Yeh, Tzu Chien Fu, Sheng De Wang, Wei Chen Chiu, and Yu Chiang Frank Wang.  Detach and adapt: Learning cross-domain disentangled deep representation.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 8867–8876, 2018.
  * [25] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In IEEE International Conference on Computer Vision (ICCV), pages 3730–3738, 2015.
  * [26] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan.  Learning transferable features with deep adaptation networks.  In Francis Bach and David Blei, editors, International Conference on Machine Learning (ICML), volume 37 of PMLR, pages 97–105, 2015.
  * [27] Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan.  Conditional adversarial domain adaptation.  In Samy Bengio, Hanna M Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems (NeurIPS), pages 1640–1650, 2018.
  * [28] Shweta Mahajan, Iryna Gurevych, and Stefan Roth.  Latent normalizing flows for many-to-many cross-domain mappings.  In International Conference on Learning Representations (ICLR), 2020\.
  * [29] Muzammal Naseer, Salman H. Khan, Harris Khan, Fahad Shahbaz Khan, and Fatih Porikli.  Cross-domain transferability of adversarial perturbations.  In Advances in Neural Information Processing Systems (NeuIPS), 2019\.
  * [30] Atsuhiro Noguchi and Tatsuya Harada.  Image generation from small datasets via batch statistics adaptation.  In IEEE Conference on Computer Vision (ICCV), pages 2750–2758, 2019\.
  * [31] Sinno Jialin Pan and Qiang Yang.  A survey on transfer learning.  IEEE Transactions on Knowledge and Data Engineering (TKDE), 22(10):1345–1359, 2010.
  * [32] Xingchao Peng, Zijun Huang, Yizhe Zhu, and Kate Saenko.  Federated adversarial domain adaptation.  In International Conference on Learning Representations (ICLR), 2020\.
  * [33] Swami Sankaranarayanan, Yogesh Balaji, Carlos D Castillo, and Rama Chellappa.  Generate to adapt: Aligning domains using generative adversarial networks.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 8503–8512, 2018.
  * [34] Edgar Schonfeld, Sayna Ebrahimi, Samarth Sinha, Trevor Darrell, and Zeynep Akata.  Generalized zero-and few-shot learning via aligned variational autoencoders.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 8247–8255, 2019.
  * [35] Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, and Chunfang Liu.  A survey on deep transfer learning.  In Vera Kurková, Yannis Manolopoulos, Barbara Hammer, Lazaros S Iliadis, and Ilias Maglogiannis, editors, International Conference on Artificial Neural Networks (ICANN), pages 270–279. Springer, 2018.
  * [36] Jakub M. Tomczak and Max Welling.  VAE with a vampprior.  In Fernando Perez-Cruz Amos Storkey, editor, International Conference on Artificial Intelligence and Statistics (AISTATS), volume 84 of PMLR, pages 1214–1223, 2018.
  * [37] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell.  Adversarial discriminative domain adaptation.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7167–7176, 2017.
  * [38] Laurens van der Maaten and Geoffrey Hinton.  Visualizing data using t-SNE.  Journal of machine learning research (JMLR), 9:2579–2605, 2008\.
  * [39] Liwei Wang, Alexander G. Schwing, and Svetlana Lazebnik.  Diverse and accurate image description using a variational auto-encoder with an additive gaussian encoding space.  In Advances in Neural Information Processing Systems (NeurIPS), pages 5757–5767, 2017.
  * [40] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: From error visibility to structural similarity.  IEEE Transactions on Image Processing, 13(4):600–612, 2004.
  * [41] Karl R Weiss, Taghi M Khoshgoftaar, and Dingding Wang.  A survey of transfer learning.  Journal of Big Data, 3(9):1–40, 2016.
  * [42] Paul Wohlhart and Vincent Lepetit.  Learning descriptors for object recognition and 3d pose estimation.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3109–3118, 2015.
  * [43] Han Xiao, Kashif Rasul, and Roland Vollgraf.  Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms.  arXiv:1708.07747 [cs.LG], 2017.
  * [44] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson.  How transferable are features in deep neural networks?  In Advances in Neural Information Processing Systems (NeurIPS), pages 3320–3328, 2014.
  * [45] Aron Yu and Kristen Grauman.  Semantic jitter: Dense supervision for visual comparisons via synthetic images.  In IEEE Conference on Computer Vision (ICCV), pages 5570–5579, 2017\.
  * [46] Jianfu Zhang, Yuanyuan Huang, Yaoyi Li, Weijie Zhao, and Liqing Zhang.  Multi-attribute transfer via disentangled representation.  In AAAI Conference on Artificial Intelligence (AAAI), volume 33, pages 9195–9202, 2019.
  * [47] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In IEEE Conference on Computer Vision (ICCV), pages 2242–2251, 2017\.
  * [48] Zachary M. Ziegler and Alexander M. Rush.  Latent normalizing flows for discrete sequences.  In Ruslan Salakhutdinov Kamalika Chaudhuri, editor, International Conference on Machine Learning (ICML), volume 97 of PMLR, pages 7673–7682, 2019.