# LRS-DAG: Low Resource Supervised Domain Adaptation with Generalization
Across Domains

Rheeya Uppaal
College of Information and Computer Sciences
University of Massachusetts, Amherst
Amherst, MA 01002
<EMAIL_ADDRESS>
Author webpage at: http://uppaal.github.io/

###### Abstract

Current state of the art methods in Domain Adaptation follow adversarial
approaches, making training a challenge. Existing non-adversarial methods
learn mappings between the source and target domains, to achieve reasonable
performance. However, even these methods do not focus on a key aspect:
maintaining performance on the source domain, even after optimizing over the
target domain. Additionally, there exist very few methods in low resource
supervised domain adaptation. This work proposes a method, LRS-DAG, that aims
to solve these current issues in the field. By adding a set of "encoder
layers" which map the target domain to the source, and can be removed when
dealing directly with the source data, the model learns to perform optimally
on both domains. LRS-DAG showcases its uniqueness by being a new algorithm for
low resource domain adaptaion which maintains performance over the source
domain, with a new metric for learning mappings between domains being
introduced. We show that, in the case of FCNs, when transferring from MNIST to
SVHN, LRS-DAG performs comparably to fine tuning, with the advantage of
maintaining performance over the source domain. LRS-DAG outperforms fine
tuning when transferring to a synthetic dataset similar to MNIST, which is a
setting more representative of low resource supervised domain adaptation.

##  1 Introduction

Domain adaptation (Huang et al. [2007], Ben-David et al. [2010]) aims to
generalize a model from a source domain, with vast amounts of labelled data,
to a target domain. Data in the target domain is almost always a large pool of
unlabelled or partially labelled data. Domain Adaptation is typically achieved
by learning a mapping between the domains.

A popular way of learning these mappings is using Generative Adversarial
Networks (Goodfellow et al. [2014]), using the cycle consistency constraint
from the CycleGAN (Zhu et al. [2017]). This has shown promising results,
(Hoffman et al. [2017], Liu et al. [2017]); however, adversarial models are
known to be notoriously hard to jointly train. (Arjovsky and Bottou [2017])

There has been a series of non-adversarial approaches to learning domain
mappings. (Hoshen and Wolf [2018], Long et al. [2015], Sun et al. [2016], Sun
and Saenko [2016], Haeusser et al. [2017]). However, all the aforementioned
methods focus on the problem of large amounts of unlabelled data in the target
domain. There exist many problems where collecting data at a large scale is
hard. (Motiian et al. [2017a], Patel et al. [2015]) There is limited work in
this domain, (Motiian et al. [2017a], Motiian et al. [2017b], Hosseini-Asl et
al. [2018]), however, the typical approach is to use low capacity models to
learn from this low resource data.

Additionally, there is almost no focus on maintaining performance on the
source domain, while improving the target domain performance. This may be
crucial in tasks where a unified model on both domains must be used, and thus
a paradigm similar to multi-task learning would be required. (Jiang [2008])
For example, in the task of stellar classification, teaching the model to
detect rare Supernovae should not deteriorate performance on detecting regular
stars.

The method proposed in this work aims to address all of these problems: (1)
Identifying method for Supervised Domain Adaptation with limited labelled
data, and (2) Creating a model that maintains performance on the source domain
even after training on the target domain. In addition, the method also trains
in a non-adversarial manner, which is an added advantage. The proposed method
divides the network into two sets of layers, and a set of ‘Encoder’ layers are
inserted between the two sets of layers of the original network. The ‘Encoder
layers’ are trained to map the target distribution to the source (rather than
mapping both into a domain invariant space, as with other methods), without
changing the weights in the original network. Thus, simply removing the
encoder layers assures the original optimal performance of the model on the
source domain. The encoder layers are trained by minimizing a measure of
distance between the two distributions: essentially, the Kullback–Leibler
divergence, and second order statistics have been considered as objective
functions. The proposed method has been implemented on two sets of source-
target datasets, and two different neural network architectures. While the
results are comparable to fine-tuning, the method maintains generalization
across the domains, and shows promising results for future work.

The main contributions of this work are: (1) Proposing and testing a set of
new metrics for minimizing feature covariances across domains (2) Proposing a
new method in the supervised low-resource domain adaptation setting, which in
being non-adversarial is significantly easier to train (3) Proposing a model
which maintains performance over the source domain when learning from the
target, thus displaying better generalization across domains. It must also be
noted that the proposed method can be made to handle the standard case of
domain adaptation of high resource unlabelled data, with minor tweaks. The
implementation of the unsupervised variant is part of future scope.

##  2 Related Work

Domain Adaptation primarily focuses on reducing a domain shift, in three major
ways. The first approach applies a form of regularization to better fit the
model to the target domain (Aytar and Zisserman [2011], Bergamo and Torresani
[2010], Becker et al. [2013]). The second is to transform both domains into a
domain invariant space, and make further inference for the specific task,
based on the features in this space. A popular approach for this is to use
Generative Adversarial Networks (Goodfellow et al. [2014]), using the cycle
consistency constraint from the CycleGAN (Zhu et al. [2017]). This puts the
constraint on a particular example, that is converted from the source to
target and back to the source, such that the same example is obtained.
(Hoffman et al. [2017], Liu et al. [2017]) Manders et al. [2018] align
predicted class probabilities across domains to achieve state of the art
results, in addition to being robust to overfitting. These class of methods
consistently show state of the art results on standard benchmarks. However,
all these methods train models adversarially with a minimax objective which
makes reaching a optimum hard. (Arjovsky and Bottou [2017]) In fact, recent
work shows that the objective function of GANs have no optimum, and must be
treated as equilibriation problems, showing that the use of traditional
optimization algorithms on GANs is ‘broken’. (Gemp and Mahadevan [2018],
Mescheder et al. [2017] )

The third method is to find some form of a mapping from the source domain to
the target domain. The proposed method roughly falls into this category, with
the slight difference that a mapping from the target to the source domain is
learnt. Sun and Saenko [2016] and Sun et al. [2017] present methods closely
related to the proposed method. They align the second order statistics of the
source and target distributions with a non-linear transformation. The loss
used is the CORAL loss, which is the Frobenious norm of the correlations of
the source and target domains.

$\mathcal{L}_{CORAL}=\frac{1}{4d^{2}}||C_{S}-C_{T}||_{F}^{2}$

Unlike LRS-DAG, the method works on unsupervised domain adaptation.
Additionally, the model does not learn maintain performance on the source
domain while learning the target, and does thus not generalize across domains.
It also uses a strong prior for both domains, by plugging in Alexnet at the
base of the network.

Haeusser et al. [2017] follow a very similar setting, with also using an
unlabelled target domain. They learn statistically domain invariant
embeddings, while minimizing the classification error on the labeled source
domain. This models holds the same weaknesses as Deep CORAL.

##  3 Methodology

Figure 1: The architecture for the proposed methodology, with an arbitrary
neural network trained on a classification task. The notation has been
simplified such that S and T denote single datapoints from $\\{S\\}$ and
$\\{T\\}$.

###  3.1 LRS-DAG

The LRS-DAG method works for an arbitrary model trained on the task of
classification on the source data, $S$, where $s\in S$ and $s\in{\rm
I\\!R}^{d}$. The layers of the network are divided into two groups, $N1$ and
$N2$. The model is trained in a standard manner, with the objective being to
minimize an arbitrary classification loss. At the end of training, $N1$ learns
a function $f:{\rm I\\!R}^{d}\rightarrow{\rm I\\!R}^{k}$, which maps $s$ to
$f(s)$, $\forall s\in S$. Similarly, $N2$ learns the function $g:{\rm
I\\!R}^{k}\rightarrow{\rm I\\!R}^{c}$, which maps $f(s)$ to $g(f(s)),\forall
s$.

The key idea to generalizing across both domains is to keep the mappings
created by $f$ and $g$ unaltered, and instead leverage them in their unaltered
condition to optimize performance over the target data, $T$, where $t\in T$
and $t\in{\rm I\\!R}^{d}$. Thus, the weights of $N1$ and $N2$ are kept frozen
in the next phase of training over the target domain data.

A new set of layers, the ‘Encoder layers’, represented by $E$, are introduced
between $N1$ and $N2$, in this phase (as shown in Figure 1). With the target
domain, $E$ gets as input $f(t)\in{\rm I\\!R}^{k}$, and must map that to
$f(s)$. For this, $E$ is trained to learn a function $h:{\rm
I\\!R}^{k}\rightarrow{\rm I\\!R}^{k}$ such that $h(f(t))=f(s)$. This would
allow the input to $N2$ to be $f(s)$ regardless of the current domain, and $g$
can function in the same manner. The objective function to be minimized for
training $E$ would thus be some measure of the difference between $f(s)$ and
$h(f(t))$. Six measures of the objective function have been proposed:

  * •

CLS: $\mathcal{L}\coloneqq L_{class}$

  * •

CLS+MSE:
$\mathcal{L}\coloneqq\frac{1}{|T|}(\sum_{i=0}^{|T|}f(s_{i})-h(f(t_{i})))^{2}+L_{class}$

  * •

CLS+KL: $\mathcal{L}\coloneqq KL(f(S)||h(f(T)))+L_{class}$

  * •

CLS+Norm:
$\mathcal{L}\coloneqq\frac{1}{|T|}(\mu_{source}-\mu_{target})^{2}+\frac{1}{|T|}(\sigma_{source}-\sigma_{target})^{2}+L_{class}$

  * •

CLS+KL-Rev: $\mathcal{L}$ := $KL(h(f(T))||f(S))+L_{class}$

  * •

CORAL: $\mathcal{L}$ := $L_{CORAL}+L_{class}$

where $L_{class}$ can be defined as any classification loss on $t\in T$, which
in this case has been defined as the cross entropy loss between
$\hat{y_{target}}$ and $t$, $L_{CORAL}$ is the CORAL loss from section 2 and
$\mu_{source}$, $\sigma_{source}$, $\mu_{target}$ and $\sigma_{target}$ are
the means and covariances of the source and target sets respectively. Methods
CLS+KL and CLS+KL-Rev have both been included, since KL divergence is not
symmetric. Method CORAL has been included as a comparison method. Since Deep
CORAL is implemented in a different architecture and data setting than LRS-
DAG, only the CORAL loss can be used as a comparison metric.

During inference, depending on the domain the model is currently being applied
to, the encoder layers could be included or ignored from the forward pass ((a)
and (b) in Figure 1).

The proposed domain adaptation method is aimed to be model agnostic. Hence, it
should perform for any arbitrary network, trained on the task of
classification. For this reason, it has been tested with a basic fully
connected network (for initial experimentation and proof of concept), and then
with a standard CNN used for classification on the selected datasets.

###  3.2 Other Aspects of the Training Regime

A practical issue with the above loss metrics is that the number of examples
in $S$ and $T$ vary, i.e. $|S|\neq|T|$, and the metrics use a one-to-one
correspondence from the target domain. Two solutions were considered. The
first is to parameterize the entire source distribution by finding
$\mu_{source}$ and $\sigma_{source},\forall s\in S$. Then sample $|T|$ points
from a multivariate Gaussian with $(\mu_{source},\sigma_{source})$. However,
apart from the possibly incorrect assumption that the source data is Gaussian,
these are just estimates of the mean and covariance of the true source
distribution, and they may be biased. This might lead to $E$ learning a
spurious function.

The other option is to simply sample $|T|$ points from $S$ (In practical
implementation, however, we sample points for one minibatch at a time).
However, an issue with this could be that a certain degree of information
about the observed distribution would be lost. An extreme case of this would
be where all the points are sampled from the tails of the observed source
distribution, thus mapping $h$ to a distribution different from $f(S)$.

Both methods have been tested in the experimental section, and results are
presented in Section 6. For simplicity, the first method shall be referred to
as ‘indirect sampling’ and the second method shall be referred to as ‘random
sampling’.

##  4 Datasets

#### MNIST

The MNIST dataset contains 28x28 sized grayscale images of handwritten digits
labelled from 0 to 9, and predefined training and testing splits of 60,000 and
10,000 examples apiece. The images were scaled to 32x32 and normalized. This
has been used as the source domain.

#### SVHN

The Street View House Number dataset is a real-world image dataset obtained
from Google Street View images. Like MNIST, it contains images of cropped
digits between 0 and 9, but the images come from a significantly harder
problem. The dataset consists of approximately 73,000 training images (out of
which 10% has been retained for the limited labelled data scenario) and 26,000
test images. The images were converted to greyscale and normalized. MNIST-SVHN
is a standard benchmark for domain adaptation tasks, which is why these
datasets have been used for initial testing.

#### Synthetic-MNIST

To see how LRS-DAG performs with different levels of domain shift, this
dataset was created by applying a series of transformations on MNIST. Random
horizontal flips over samples from the data were applied, and images were
sheared. In addition to this, the brightness, contrast and saturation of
images was randomly changed. Like with SVHN, only 10% of the labelled training
data was used.

Validation sets were made from the training splits for these datasets, and
rolled back into the training sets after performing a grid search over the
hyperparameter space, and judging performance over the validation set.

Figure 2: Left: Samples from the MNIST, SVHN and Syn-MNIST datasets. Right:
Distributions of classes over the test data, for MNIST and SVHN.

##  5 Experiments

The goal of this series of experiments was to test the LRS-DAG method, with
all its variants of loss functions. To show that the proposed training regime
would work as an efficient form of Domain Adaptation, it has been tested over
different models, and different sets of datasets. To test for the correctness
of the hypothesis that the method is model agnostic, all experiments have been
run for two networks:

  * •

Model 1 - FCN: A fully connected network with 4 hidden layers. The output
layer generates softmax predictions for all classes. $N1$, $E$ and $N2$
consist of the bottom two, middle two and top two layers of the network. $E$
is ignored when in the source domain. The network has no non-linearities.

  * •

Model 2 - CNN: A CNN used for learning from domains of similar complexity. The
network consists of 5 convolutional layers and one fully connected layer, each
followed by ReLU non-linearity. Softmax is applied over the output of the last
layer to give a confidence score for every class. As with FCN, $N1$, $E$ and
$N2$ consist of the bottom two, middle two and top two layers of the network.
$E$ takes and returns values of the same shape, in both models.

The model is first trained on the source domain for 100 epochs. Following
this, there are three main sets of experiments: $MNIST\rightarrow SVHN$ with
FCN, $MNIST\rightarrow SVHN$ with CNN, $MNIST\rightarrow Syn-MNIST$ with CNN,
where $\rightarrow$ signified transferring across domains. For each of these
sets of experiments, all the loss functions (with the direct and indirect
sampling methods described in Section 3.2) are tested. Additionally, they are
compared with a series of baselines.

#### Basline Methods:

(1) Source Trained: The most rudimentary baseline method considered was
training a model on the source dataset, and performing inference on the target
with no additional training. This gives a lower bound on performance. (2)
Target Trained: Train the model from scratch on the target domain. The high
capacity model is likely to overfit to the data, thus performing poorly on the
target test set. (3) Finetune N2: Finetune the weights of $N2$ on the target
domain, after training the model on the source domain. This is akin to the
most standard method of transfer learning, when limited labelled data is
available. (4) CORAL Loss: Despite the Deep CORAL method (described in Section
2) being targeted towards the setting of a large pool of unlabelled data, the
method is still most similar to LRS-DAG. For this reason, the CORAL loss has
been fit into the LRS-DAG architecture as a loss function. This is expected to
be the best performing method, as the loss minimizes second order statistics
of the source and target distributions.

Additional points of note are that:

  * •

The method was implemented from scratch, using PyTorch 0.4, Scipy, Numpy and
Scikit-Learn. No other existing implementations or frameworks were used.

  * •

The accuracy of the model on the hidden test set of the target domain was used
as a metric of the performance of a model. Confusion Matrices were also used
to further analyse the methods, but have been excluded from this work for
brevity.

  * •

Hyperparameter tuning was done through a grid search over learning rate,
weight decay, and the kind of optimizer. This performance was measured over
the validation set, which was later rolled back into the training set for all
methods. The validation set splits were stored and, for a particular dataset,
the same data points were used as the validation set for all methods.

  * •

The Adam optimizer was used for all models. On average, all methods for a
particular model-source-target triplet required very similar hyperparameters.

  * •

To account for the stochasticity arising from random weight initializations,
every experiment has been run for three trials, and their averaged results
have been showcased in Table 1.

  * •

All models were trained until satisfying the stopping criteria of the
difference in loss between two epochs being less than a particular threshold
(thresholds varied, based on the type of loss function).

  * •

Since intermediate features in a network are not probability distributions,
and the method relies on the assumption they are distributions, and a softmax
function is applied over the features after extracting them from the network,
to convert them into a valid probability distribution.

##  6 Results

#### Experiment Set 1

: Indirect sampling of the source domain consistently outperforms random
sampling. Hence, the information loss while sampling 10% points from the
source is relatively large. All proposed methods were expected to have similar
outcomes, but the CLS+KL method ($KL(f(S)||(h(f(T))$) slightly outperforms the
other methods. CLS+KL-Rev has a very similar performance since it is still a
very similar notion of distance that is being minimized between both methods.

A point worth noting is that, while the Target Trained baseline outperformed
other methods, LRS-DAG with CLS+KL is almost the same as fine tuning. However,
unlike fine tuning, the proposed method maintains generalization across both
domains.

Another notable point is the weak performance of CORAL based model in all
three experiment sets. This may be because the Deep CORAL method
simultaneously trains on the source and target domains, jointly minimizing
estimators of the true $f(S)$, with roughly equal strength in both domains.
With LRS-DAG, the estimate of $f(S)$ from the source is already very accurate,
which might cause $h(f(S))$ to converge to an alternate value.

Table 1: Accuracies (in percentage, averaged over three trials) of baselines and proposed methods, for FCN, when transferring from MNIST to SVHN. Results of the two variants of the model: Without E (the model without the encoder layers, intended for inference over the source domain) and With E (the model with the encoder layers, intended for inference over the target domain) are shown. Method | Sampling Method | Without $E$ | With $E$
---|---|---|---
Source | Target | Source | Target
Target Trained | - | 25.44 | 13.92 | 6.39 | 34.52
Finetune N2 | - | 15.42 | 14.98 | 11.08 | 30.35
CLS | - | 91.91 | 13.93 | 11.46 | 29.97
CLS+MSE | Indirect | 91.91 | 13.93 | 13.53 | 29.64
CLS+MSE | Random | 91.91 | 13.93 | 11.31 | 29.38
CLS+KL | Indirect | 91.91 | 13.93 | 12.18 | 30.28
CLS+KL | Random | 91.91 | 13.93 | 14.25 | 28.85
CLS+Norm | Indirect | 91.91 | 13.93 | 10.55 | 29.66
CLS+Norm | Random | 91.91 | 13.93 | 10.98 | 28.15
CLS+KL-Rev | Indirect | 91.91 | 13.93 | 11.75 | 29.79
CLS+KL-Rev | Random | 91.91 | 13.93 | 7.73 | 30.15
CORAL | Indirect | 91.91 | 13.93 | 11.35 | 19.59
CORAL | Random | 91.91 | 13.93 | 12.46 | 19.32
Table 2: Accuracies (in percentage, averaged over three trials) of baselines and proposed methods, for CNN, when transferring from MNIST to SVHN. Results of the two variants of the model: Without E (the model without the encoder layers, intended for inference over the source domain) and With E (the model with the encoder layers, intended for inference over the target domain) are shown. Method | Sampling Method | Without $E$ | With $E$
---|---|---|---
Source | Target | Source | Target
Finetune N2 | - | 16.44 | 18.47 | 13.53 | 26.77
CLS | - | 93.88 | 20.19 | 24.34 | 21.88
CLS+MSE | Indirect | 93.88 | 20.19 | 26.62 | 21.73
CLS+KL | Indirect | 93.88 | 20.19 | 28.52 | 21.85
CLS+Norm | Indirect | 93.88 | 20.19 | 60.79 | 21.29
CLS+KL-Rev | Indirect | 93.88 | 20.19 | 29.24 | 21.92
CORAL | Indirect | 93.88 | 20.19 | 11.46 | 20.59
Table 3: Accuracies (in percentage, averaged over three trials) of baselines and proposed methods, for FCN, when transferring from MNIST to Syn-MNIST. Results of the two variants of the model: Without E (the model without the encoder layers, intended for inference over the source domain) and With E (the model with the encoder layers, intended for inference over the target domain) are shown. Method | Sampling Strategy | Without $E$ | With $E$
---|---|---|---
Source | Target | Source | Target
Finetune N2 | - | 89.27 | 65.42 | 86.12 | 77.33
CLS | - | 91.91 | 63.13 | 84.52 | 77.09
CLS+MSE | Indirect | 91.91 | 63.13 | 84.98 | 78.09
CLS+KL | Indirect | 91.91 | 63.13 | 85.19 | 78.14
CLS+Norm | Indirect | 91.91 | 63.13 | 85.26 | 77.95
CLS+KL-Rev | Indirect | 91.91 | 63.13 | 84.88 | 78.11
CORAL | Indirect | 91.91 | 63.13 | 76.07 | 67.94

Figure 3: Top Row: t-SNE Visualizations of raw data from MNIST, SVHN and Syn-
MNIST. Second row: $f(S)$ for source domain. Third row: $f(T)$ before
training. Bottom: $h(f(T))$ after feature alignment. Left column:
MNIST$\rightarrow$ SVHN with FCN. Middle column: MNIST$\rightarrow$ Syn-MNIST
with FCN. Right column: MNIST$\rightarrow$ SVHN with CNN.

#### Experiment Set 2

: When using the CNN for adapting from MNIST to SVHN, it seems possible that
the stopping criterion was not accurately applied. This would explain why the
results on the target set in this experiment set is lower than the previous
experiment set, despite the CNN being more powerful. Once again, the loss
function based on KL divergence outperformed other proposed methods, and the
CORAL loss. Additionally, the CLS+KL and CLS+KL-Rev methods have almost
identical results here as well. The results of all methods are extremely
similar in this set, leading to inconclusive results. However, finetuning
clearly surpasses other methods.

#### Experiment Set 3

: The Syn-MNIST domain has a lesser shift from MNIST than SVHN. Thus, the
results seem more promising. Here, the CLS+KL and CLS+KL-Rev methods
outperform finetuning, which is the most promising result so far. It is worth
noting that this is a significant, as in low resource supervised domain
adaptation, it is common to treat a highly similar domain as the source
domain, and finetune over the target. LRS-DAG provides a clear benefit over
other methods in this case.

##  7 Discussion and Conclusion

The LRS-DAG method seems comparable to fine-tuning, except for the case of the
Syn-MNIST dataset, a closely related domain. However, the proposed method
significantly outperforms CORAL, and most importantly, maintains
generalization across both domains.

The method was inconclusive with CNNs. The performance may have been such due
to a bad stopping criteria, or difficulty in aligning domains across
convolutions. However, looking at the t-SNE plots of the aligned target domain
after training (Figure 3, bottom right), the points of all classes have been
clustered together. Thus, the proposed method of LRS-DAG shows promise. Taking
the observations so far into account, and generating a better experimental
setup, may provide more promising results in the future.

A point worth arguing would we whether it would make more sense to add $E$ to
the top of the network, rather than making it handle intermediate features. A
series of experiments (involving training different parts of the network and
analyzing results) showed that the features across domains differ across lower
levels, indicating the positioning of $E$ lower in the network in more
beneficial (results excluded for brevity).

A possible path to pursue in the future would be to align $c$ different
domains for each class, instead of an overall domain loss. Another field to
explore would be mapping this model to the unsupervised domain adaptation
setting. The method currently requires access to the source domain, when
training on the target. Since access to the entire source domain is not always
possible, another addition to the method could be to use the source domain
embeddings to minimize domain difference, rather than using the data itself.

## References

  * Huang et al. [2007] Jiayuan Huang, Arthur Gretton, Karsten M Borgwardt, Bernhard Schölkopf, and Alex J Smola.  Correcting sample selection bias by unlabeled data.  In _Advances in neural information processing systems_ , pages 601–608, 2007.
  * Ben-David et al. [2010] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan.  A theory of learning from different domains.  _Machine learning_ , 79(1-2):151–175, 2010.
  * Goodfellow et al. [2014] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In _Advances in neural information processing systems_ , pages 2672–2680, 2014.
  * Zhu et al. [2017] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  _arXiv preprint_ , 2017.
  * Hoffman et al. [2017] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei A Efros, and Trevor Darrell.  Cycada: Cycle-consistent adversarial domain adaptation.  _arXiv preprint arXiv:1711.03213_ , 2017.
  * Liu et al. [2017] Ming-Yu Liu, Thomas Breuel, and Jan Kautz.  Unsupervised image-to-image translation networks.  In _Advances in Neural Information Processing Systems_ , pages 700–708, 2017.
  * Arjovsky and Bottou [2017] Martin Arjovsky and Léon Bottou.  Towards principled methods for training generative adversarial networks.  _arXiv preprint arXiv:1701.04862_ , 2017.
  * Hoshen and Wolf [2018] Yedid Hoshen and Lior Wolf.  Nam: Non-adversarial unsupervised domain mapping.  _arXiv preprint arXiv:1806.00804_ , 2018.
  * Long et al. [2015] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I Jordan.  Learning transferable features with deep adaptation networks.  _arXiv preprint arXiv:1502.02791_ , 2015.
  * Sun et al. [2016] Baochen Sun, Jiashi Feng, and Kate Saenko.  Return of frustratingly easy domain adaptation.  In _AAAI_ , volume 6, page 8, 2016.
  * Sun and Saenko [2016] Baochen Sun and Kate Saenko.  Deep coral: Correlation alignment for deep domain adaptation.  In _European Conference on Computer Vision_ , pages 443–450. Springer, 2016.
  * Haeusser et al. [2017] Philip Haeusser, Thomas Frerix, Alexander Mordvintsev, and Daniel Cremers.  Associative domain adaptation.  In _International Conference on Computer Vision (ICCV)_ , volume 2, page 6, 2017.
  * Motiian et al. [2017a] Saeid Motiian, Quinn Jones, Seyed Iranmanesh, and Gianfranco Doretto.  Few-shot adversarial domain adaptation.  In _Advances in Neural Information Processing Systems_ , pages 6670–6680, 2017a.
  * Patel et al. [2015] Vishal M Patel, Raghuraman Gopalan, Ruonan Li, and Rama Chellappa.  Visual domain adaptation: A survey of recent advances.  _IEEE signal processing magazine_ , 32(3):53–69, 2015.
  * Motiian et al. [2017b] Saeid Motiian, Marco Piccirilli, Donald A Adjeroh, and Gianfranco Doretto.  Unified deep supervised domain adaptation and generalization.  In _The IEEE International Conference on Computer Vision (ICCV)_ , volume 2, page 3, 2017b.
  * Hosseini-Asl et al. [2018] Ehsan Hosseini-Asl, Yingbo Zhou, Caiming Xiong, and Richard Socher.  Augmented cyclic adversarial learning for domain adaptation.  _arXiv preprint arXiv:1807.00374_ , 2018.
  * Jiang [2008] Jing Jiang.  A literature survey on domain adaptation of statistical classifiers.  _URL: http://sifaka. cs. uiuc. edu/jiang4/domainadaptation/survey_ , 3:1–12, 2008.
  * Aytar and Zisserman [2011] Yusuf Aytar and Andrew Zisserman.  Tabula rasa: Model transfer for object category detection.  In _Computer Vision (ICCV), 2011 IEEE International Conference on_ , pages 2252–2259. IEEE, 2011.
  * Bergamo and Torresani [2010] Alessandro Bergamo and Lorenzo Torresani.  Exploiting weakly-labeled web images to improve object classification: a domain adaptation approach.  In _Advances in neural information processing systems_ , pages 181–189, 2010.
  * Becker et al. [2013] Carlos J Becker, Christos M Christoudias, and Pascal Fua.  Non-linear domain adaptation with boosting.  In _Advances in Neural Information Processing Systems_ , pages 485–493, 2013.
  * Manders et al. [2018] Jeroen Manders, Elena Marchiori, and Twan van Laarhoven.  Simple domain adaptation with class prediction uncertainty alignment.  _arXiv preprint arXiv:1804.04448_ , 2018.
  * Gemp and Mahadevan [2018] Ian Gemp and Sridhar Mahadevan.  Global convergence to the equilibrium of gans using variational inequalities.  _arXiv preprint arXiv:1808.01531_ , 2018.
  * Mescheder et al. [2017] Lars Mescheder, Sebastian Nowozin, and Andreas Geiger.  The numerics of gans.  In _Advances in Neural Information Processing Systems_ , pages 1825–1835, 2017.
  * Sun et al. [2017] Baochen Sun, Jiashi Feng, and Kate Saenko.  Correlation alignment for unsupervised domain adaptation.  In _Domain Adaptation in Computer Vision Applications_ , pages 153–171. Springer, 2017.