# Deep into The Domain Shift: Transfer Learning through Dependence
Regularization

Shumin Ma, Zhiri Yuan, Qi Wu, Yiyan Huang, Xixu Hu, Cheuk Hang Leung, Dongdong
Wang, Zhixiang Huang Shumin Ma is with Guangdong Provincial Key Laboratory of
Interdisciplinary Research and Application for Data Science, BNU-HKBU United
International College; Division of Science and Technology, BNU-HKBU United
International College, Zhuhai 519087, China.Zhiri Yuan, Yiyan Huang, Xixu Hu,
and Cheuk Huang Leung are with the CityU-JD Digits Joint Laboratory in
Financial Technology and Engineering and the School of Data Science at the
City University of Hong Kong, Hong Kong.Qi Wu is with the School of Data
Science, The CityU-JD Digits Joint Laboratory in Financial Technology and
Engineering, and the Institute of Data Science at the City University of Hong
Kong. He is also with The Laboratory for AI-Powered Financial Technologies
Limited, Hong Kong.Dongdong Wang and Zhixiang Huang are with JD Digits
Technology, Beijing, China.Corresponding author: Qi Wu.

###### Abstract

Classical Domain Adaptation methods acquire transferability by regularizing
the overall distributional discrepancies between features in the source domain
(labeled) and features in the target domain (unlabeled). They often do not
differentiate whether the domain differences come from the marginals or the
dependence structures. In many business and financial applications, the
labeling function usually has different sensitivities to the changes in the
marginals versus changes in the dependence structures. Measuring the overall
distributional differences will not be discriminative enough in acquiring
transferability. Without the needed structural resolution, the learned
transfer is less optimal. This paper proposes a new domain adaptation approach
in which one can measure the differences in the internal dependence structure
separately from those in the marginals. By optimizing the relative weights
among them, the new regularization strategy greatly relaxes the rigidness of
the existing approaches. It allows a learning machine to pay special attention
to places where the differences matter the most. Experiments on three real-
world datasets show that the improvements are quite notable and robust
compared to various benchmark domain adaptation models.

###### Index Terms:

domain adaptation, regularization, domain divergence, copula.

##  I Introduction

Unsupervised domain adaptation emerges when one estimates a prediction
function in a given target domain without any labeled samples by exploiting
the knowledge available from a source domain where labels are known. The
critical step in the transfer is to extract feature representations that are
invariant across domains. A large body of work learns the domain-invariant
feature representations by minimizing various metrics on the feature
distributions between domains: Proxy-$\mathcal{A}$ distance [1], total
variation distance [2, 3, 4], maximum mean discrepancy (MMD, [5, 6, 7]),
Wasserstein distance [8, 9], etc. It is worth noting that in most of the
literature, the domain invariance is measured on the overall feature
distributions between the source and the target domains.

Figure 1: Visualization of three 2-d Gaussian distributions:
($P^{\mathbf{X}}$) $N(\begin{bmatrix}0\\\ 1\end{bmatrix},\begin{bmatrix}1&0\\\
0&1\end{bmatrix})$, ($P^{\mathbf{Y}}$) $N(\begin{bmatrix}0\\\
0\end{bmatrix},\begin{bmatrix}1&0\\\ 0&1\end{bmatrix})$, ($P^{\mathbf{Z}}$)
$N(\begin{bmatrix}0\\\ 0\end{bmatrix},\begin{bmatrix}1&\sqrt{1-e^{-1}}\\\
\sqrt{1-e^{-1}}&1\end{bmatrix})$. $P^{\mathbf{X}}$ and $P^{\mathbf{Y}}$ differ
only in the 2nd marginal distribution, while $P^{\mathbf{Z}}$ and
$P^{\mathbf{Y}}$ differ in the dependence structure. One can check that the KL
divergence between $P^{\mathbf{X}}$ and $P^{\mathbf{Y}}$ is $1/2$ and it is
the same as the KL divergence between $P^{\mathbf{Z}}$ and $P^{\mathbf{Y}}$.
That is, one single divergence value ($1/2$ in this example) can not
distinguish whether the distribution difference comes from the marginals or
the dependence structure.

However, the overall feature distribution difference encodes both the
marginals’ distinctions and the dependence difference into one metric value,
making it hard to identify whether the domain difference comes from the
marginals or the dependence structure. As a motivation example, we use Figure
1 to clarify this point. In Figure 1, there are three random vectors
($\mathbf{X}$, $\mathbf{Y}$ and $\mathbf{Z}$) that follow three different
Gaussian distributions ($P^{\mathbf{X}}$, $P^{\mathbf{Y}}$ and
$P^{\mathbf{Z}}$, with the details shown in the caption) respectively. It can
be observed that $P^{\mathbf{X}}$ and $P^{\mathbf{Y}}$ only differ in the 2nd
marginal distribution where $P^{\mathbf{X}}_{2}$ is $N(1,1)$ and
$P^{\mathbf{Y}}_{2}$ is $N(0,1)$. Namely, the overall distribution difference
is completely caused by the marginal distinction. However, for
$P^{\mathbf{Y}}$ and $P^{\mathbf{Z}}$, it is well observed that the marginals
are the same while the covariance matrix differs. Namely, the overall
distribution difference over the latter two distributions is controlled by the
dependence difference. In general cases, any two distributions can differ in
the marginal distributions and the dependence structure simultaneously. One
can check that the KL divergence between $P^{\mathbf{X}}$ and $P^{\mathbf{Y}}$
is the same with that of $P^{\mathbf{Z}}$ and $P^{\mathbf{Y}}$, which is $1/2$
in the example. That is to say, a single divergence value cannot distinguish
between marginal difference and dependence difference. Furthermore, the
overall feature distributions’ divergence sums the marginals’ and dependence
differences up together in a relatively fixed manner. Such rigidness is
undesirable when the prediction function has different sensitivities to the
changes in the marginals versus changes in the dependence structure.
Especially in many financial and business applications, the prediction
function may depend heavily on the dependence structure of the features. For
example, during the market crash, the stocks show remarkably synchronous co-
movement, implying a stronger dependence than regular times. Such observations
motivate us to relax the binding between the marginals’ and dependence
differences in one metric value and pay attention to the difference term that
matters the most in the transfer.

We propose a new domain adaptation model to regularize the domain differences
in the dependence structure via the copula distance, separately from the
marginal divergence. The idea is inspired by Sklar’s Theorem [10] which states
that any multivariate distribution can be decomposed as the product of
marginal distributions and a copula function, and vice versa. It explicitly
shows that the copula function, together with the marginal distributions, is
sufficient to recover the original multivariate distribution. The efficacy and
versatility of our approach are demonstrated with real-world classification
and regression problems.

The contributions of this paper are summarized as follows. (1) We propose a
novel deep domain adaptation framework that allows more flexibility to combine
the marginal difference and the dependence difference into a regularizer. (2)
We explore the structural properties of the copula distance that guarantee the
algorithm convergence of our approach. (3) Our proposed model proves its
efficiency on two novel datasets (a large-scale retail credit dataset and an
intra-day equity price dataset) and one standard UCI dataset.

##  II Related work

Due to the ability of deep neural nets to learn rich feature representations,
deep domain adaptation models have focused on using these networks to learn
invariant representations, i.e., intermediate features whose distributions are
the same in the source and the target domains, while at the same time
achieving a small prediction error on the source domain. The hope is that the
learned representation, together with the hypothesis learned from the source
domain, can generalize to the target domain. There are many ways to measure
the domain invariance [11]. [12] uses $L_{2}$ norm to directly align features
of different domains. [6] proposes a Deep Adaptation Network (DAN)
architecture that embeds the feature representations in a reproducing kernel
Hilbert space (RKHS) and reduces the domain discrepancy through multi-kernel
MMD. [1] proposes a domain adversarial neural network (DANN) to learn the
domain-invariant features with a min-max formulation. [13] characterizes a
fundamental tradeoff between learning invariant representations and achieving
a small joint error on both domains when the marginal label distributions
differ from the source to the target. Furthermore, [14] tackles the knowledge
transfer problem under the generalized covariate shift condition by Bregman
divergence. [15] proposes a novel neural embedding matching method by
enforcing consistent class-wise cross-domain instance distributions in the
embedding space. [16] proposes a two-stage progressive training strategy to
learn invariant, discriminative, and domain-agnostic subspace. Another line of
work proposes to reduce the domain discrepancy through minimizing the optimal
transport loss between the source and target distributions [8, 9]. For
example, [9] minimizes the empirical Wasserstein distance between the source
and target samples. However, these methods focus on the overall distribution
discrepancy and often do not differentiate whether the domain differences come
from the marginals or the dependence structure.

To explicitly encode the dependence difference in the domain adaptation
framework, [17] proposes a correlation alignment (CORAL) model to measure the
dis-similarity by the Frobenius norm of the covariance matrices from the two
domains. [18] further combines CORAL with deep neural networks and verifies
its effectiveness through extensive experiments on the standard benchmark
datasets. [19] matches distributions by aligning the RKHS covariance matrices
across domains. [20] integrates the MMD and CORAL into a unified framework and
exploits the higher-order statistics for domain alignment. Our approach is
more general in that it separates the marginals’ divergence and the dependence
difference and integrates them into one regularizer. It facilitates us to
detect the changes in the marginals and the dependence structure
simultaneously.

Our work is closely related to copulas. Copulas have been successfully used in
many deep learning methods. [21] performs the missing value imputation by
developing a semi-parametric algorithm to estimate copula parameters from
incomplete mixed data. [22] and [23] propose to summarize and measure the
pairwise correlations between variables, which is shown to well capture the
various dependence patterns. By assuming the underlying features have a
specific structure, [24] adopts non-parametric vine copula for semi-supervised
domain adaptation problems. [25] uses copula to generate dependent data in a
segmented way. [26] incorporates copula into a doubly nonparametric sparse
nonnegative matrix factorization framework. [27] well documents the literature
that takes advantage of copulas to model the correlation in multivariate data
in smart grid. We extend the idea of copulas and construct a copula-based
divergence measure to quantify the dependence difference. Our proposed measure
incorporates more statistical information than the commonly-used covariance
matrices that only capture the second-order statistics.

##  III Copula distance

Suppose that a $d$-dimensional random variable
$\mathbf{X}=[X_{1},\dots,X_{d}]$ is characterized by the cumulative
distribution function (CDF) $P$ and its density function is denoted by $p$.
Sklar’s theorem [10] states that there exists a copula $C$ such that
$P(x_{1},\dots,x_{d})=C(P_{1}(x_{1}),\dots,P_{d}(x_{d}))$, where
$P_{i}(\cdot)$ is the marginal CDF. Furthermore, any continuous density
function $p(x_{1},\dots,x_{d})$ can be written in terms of univariate marginal
density functions $\\{p_{i}(x_{i})\\}_{i=1}^{d}$ and a unique copula density
function $c:[0,1]^{d}\rightarrow\mathbb{R}$ which characterizes the dependence
structure:

$p(x_{1},\dots,x_{d})=c(u_{1},\ldots,u_{d})\times\prod\nolimits_{i=1}\nolimits^{d}p_{i}(x_{i}),$
(1)

where $u_{i}:=P_{i}(x_{i}),\,\forall\,1\leq i\leq d$.

One can use the copula function to extract a clean quantification of the
dependence strength between any components [$X_{i}$, $X_{j}$] in the random
vector $\mathbf{X}$. For example, the mutual information between $X_{i}$ and
$X_{j}$, a well-known dependence measure in information theory, is equivalent
to the negative entropy of the copula function, namely,

$\small\mathcal{H}_{KL}(P_{ij},P_{i}P_{j})=\int_{0}^{1}\int_{0}^{1}c_{ij}(u_{i},u_{j})\log
c_{ij}(u_{i},u_{j})\mathrm{d}u_{i}\mathrm{d}u_{j}.$

Here, $\mathcal{H}_{KL}$ denotes the Kullback-Leibler (KL) divergence,
$P_{ij}$ is the joint distribution of $[X_{i},X_{j}]$, and $P_{i}$, $P_{j}$
represent the marginal distributions. $c_{ij}(u_{i},u_{j})$ is short for the
density function $c(1,\ldots,1,u_{i},1,\ldots,1,u_{j},1,\ldots,1)$ where the
$i$-th and the $j$-th arguments are $u_{i}$ and $u_{j}$ respectively, and the
other arguments are all $1$. It should be noted that mutual information is a
special case of a more general dependence measurement framework [22] that
computes the distance $\mathcal{H}(P_{ij},P_{i}P_{j})$ with any divergence
measure $\mathcal{H}$. We list a few examples below where $\mathcal{H}$ takes
$\chi^{2}$ distance [28], Hellinger distance [28], and $\alpha$-divergence
[29] (the detailed derivation can be found in the Appendix):

  * -

$\mathcal{H}_{\chi^{2}}(P_{ij},P_{i}P_{j})=\int_{0}^{1}\int_{0}^{1}(c_{ij}^{2}(u_{i},u_{j})-1)\mathrm{d}u_{i}\mathrm{d}u_{j}$.

  * -

$\mathcal{H}_{H}(P_{ij},P_{i}P_{j})=\int_{0}^{1}\int_{0}^{1}[\sqrt{c_{ij}(u_{i},u_{j})}-1]^{2}\mathrm{d}u_{i}\mathrm{d}u_{j}$.

  * -

$\mathcal{H}_{\alpha}(P_{ij},P_{i}P_{j})=\frac{1}{1-\alpha^{2}}\int_{0}^{1}\int_{0}^{1}[1-{c_{ij}(u_{i},u_{j})}^{-\frac{\alpha+1}{2}}]c_{ij}(u_{i},$
$u_{j})\mathrm{d}u_{i}\mathrm{d}u_{j}$.

For most of the divergence measures $\mathcal{H}$, it can be proved that
$\mathcal{H}(P_{ij},P_{i}P_{j})$ is a function of the copula densities (see
Appendix). For a given random vector $\mathbf{X}$, if
$\mathcal{H}(P_{ij},P_{i}P_{j})$ provides a clean measure of any of its
component pairs [$X_{i}$, $X_{j}$], one can run through the indexes, give them
different weights $\beta_{ij}$, and sum over all the weighted $\mathcal{H}$s.
The resulting quantity would allow one to look into the complete internal
structure of $\mathbf{X}$ along the direction of any user-defined attention
angle using any measure $\mathcal{H}$. This observation motivates us to use
such a measure to quantify the difference in internal dependencies between two
random vectors $\mathbf{X}$ and $\mathbf{Y}$. Below is its precise definition.

TABLE I: For any two adjacent distributions in Figure 1, the overall distribution divergence, the $1^{st}$-d and the $2^{nd}$-d marginal distinction, together with the copula distance (CD) are recorded. Specifically, we take the KL divergence, Jenson-Shannon (JS) divergence and MMD as the distribution divergence measure for illustration purposes.  |  $P^{\mathbf{X}}$ vs. $P^{\mathbf{Y}}$ |  $P^{\mathbf{Z}}$ vs. $P^{\mathbf{Y}}$
---|---|---
$\mathcal{H}$ | $\mathcal{H}(P^{\mathbf{X}},P^{\mathbf{Y}})$ | $\mathcal{H}(P^{\mathbf{X}}_{1},P^{\mathbf{Y}}_{1})$ | $\mathcal{H}(P^{\mathbf{X}}_{2},P^{\mathbf{Y}}_{2})$ | CD | $\mathcal{H}(P^{\mathbf{Z}},P^{\mathbf{Y}})$ | $\mathcal{H}(P^{\mathbf{Z}}_{1},P^{\mathbf{Y}}_{1})$ | $\mathcal{H}(P^{\mathbf{Z}}_{2},P^{\mathbf{Y}}_{2})$ | CD
KL | 0.5 | 0 | 0.5 | 0 | 0.5 | 0 | 0 | 0.5
JS | 0.5 | 0 | 0.5 | 0 | 0.859 | 0 | 0 | 0.859
MMD | 0.107 | 0 | 0.175 | 0 | 0.042 | 0 | 0 | 0.042

######  Definition 1.

(Copula distance) Let $\mathbf{X}=[X_{1},\ldots,X_{d}]\in\mathbb{R}^{d}$ and
$\mathbf{Y}=[Y_{1},\ldots,Y_{d}]\in\mathbb{R}^{d}$ be two random vectors. Let
$P_{ij}^{\mathbf{X}}$ and $P_{ij}^{\mathbf{Y}}$ be the cumulative joint
distributions of any of their component pairs [$X_{i}$, $X_{j}$] and [$Y_{i}$,
$Y_{j}$], respectively. Let $\mathcal{H}(\cdot,\cdot)$ be a measure of
distribution difference. We define the copula distance between $[X_{i},X_{j}]$
and $[Y_{i},Y_{j}]$ ($\forall 1\leq i<j\leq d$) as:

$\displaystyle
CD_{\mathcal{H}}([X_{i},X_{j}],[Y_{i},Y_{j}])=|\mathcal{H}(P^{\mathbf{X}}_{ij},P^{\mathbf{X}}_{i}P^{\mathbf{X}}_{j})-\mathcal{H}(P^{\mathbf{Y}}_{ij},P^{\mathbf{Y}}_{i}P^{\mathbf{Y}}_{j})|.$

The copula distance between random vectors $\mathbf{X}$ and $\mathbf{Y}$
w.r.t. the positive weights $\bm{\beta}=\\{\beta_{ij}\\}_{1\leq i<j\leq d}$ is
defined as:

$\displaystyle
CD_{\mathcal{H}}(\mathbf{X},\mathbf{Y};\bm{\beta})=\sum\limits_{1\leq i<j\leq
d}\beta_{ij}CD_{\mathcal{H}}([X_{i},X_{j}],[Y_{i},Y_{j}]).$

(2)

As explained before, although the copula distance seems irrelevant to the
copulas at first glance, it is actually a function of the copula densities. In
the following context, we will use a set of $d$ terms
$\\{\mathcal{H}(P^{\mathbf{X}}_{i},P^{\mathbf{Y}}_{i})\\}_{i=1}^{d}$ to
measure the marginals’ distinctions and the copula distance
$CD_{\mathcal{H}}(\mathbf{X},\mathbf{Y};\bm{\beta})$ to measure the dependence
difference of any two random vectors $\mathbf{X}\in\mathbb{R}^{d}$ and
$\mathbf{Y}\in\mathbb{R}^{d}$. To explain the concepts in more details, we use
the distributions in Figure 1. We record the marginals’ distinctions and
dependence difference between each two of the three distributions under KL
divergence (KL), Jenson-Shannon divergence (JS) and MMD in Table I.

When the copula density takes some specific function form, it can be proved
that the copula distance has two nice properties: bounded and monotonic. These
two properties guarantee the convergence of our proposed algorithms in the
later sections. There are multiple parametric copula density functions
(Gaussian copula, $t$ copula, Gumbel copula, etc.) that can be effectively
incorporated into our model framework. In this work, we take the copula
density in the form of Gaussian copula. Gaussian copula has been widely used
in the financial literature because it is convenient to capture the dependence
embedded in the random vector [30, 31, 32]. A random vector
$\mathbf{X}\in\mathbb{R}^{d}$ is said to have Gaussian copula with parameter
$\Sigma\in\mathbb{R}^{d\times d}$ if the copula density function
$c(u_{1},\ldots,u_{d})=|\Sigma|^{-\frac{1}{2}}\exp(-\frac{1}{2}\mathbf{x}^{T}(\Sigma^{-1}-\mathbf{I})\mathbf{x})$,
where $\mathbf{x}:=[\Phi^{-1}(u_{1}),\ldots,\Phi^{-1}(u_{d})]^{T}$ with $\Phi$
being the cumulative density function for the standard normal distribution. We
state the boundedness and monotonicity under the Gaussian copula here and
defer the proof to the Appendix.

######  Proposition 1.

(Boundedness) The copula distance defined in Eq. (2) is bounded when the
divergence metric $\mathcal{H}$ is taken to be MMD distance, Wasserstein-2
distance or some of the $\phi$-divergence (including Jensen-Shannon distance,
Hellinger distance and total variation distance).

######  Proposition 2.

(Monotonicity) Let $\Sigma^{\mathbf{X}}$ and $\Sigma^{\mathbf{Y}}$ be the
Gaussian copula parameters for the random vectors
$\mathbf{X}\in\mathbb{R}^{d}$ and $\mathbf{Y}\in\mathbb{R}^{d}$, respectively.
Given all of the other entries in $\Sigma^{\mathbf{X}}$ and
$\Sigma^{\mathbf{Y}}$ fixed, the copula distance
$CD_{\mathcal{H}}([X_{i},X_{j}],[Y_{i},Y_{j}])$ is monotonically increasing
with $|(\Sigma_{ij}^{\mathbf{X}})^{2}-(\Sigma_{ij}^{\mathbf{Y}})^{2}|$,
$\forall~{}1\leq i<j\leq d$. The monotonicity is satisfied by general
probability distribution divergence measures, including MMD distance,
Wasserstein-2 distance, and most of the commonly-used $\phi$-divergence
(including KL divergence, $\chi^{2}$ distance, Hellinger distance, etc.).

##  IV Unsupervised domain adaptation

Notations. In unsupervised domain adaptation, we are given a source domain
$\mathcal{D}_{s}=\\{(\mathbf{x}_{n}^{s},y_{n}^{s})\\}_{n=1}^{N_{s}}$ with
$N_{s}$ labeled examples, and a target domain
$\mathcal{D}_{t}=\\{\mathbf{x}_{n}^{t}\\}_{n=1}^{N_{t}}$ with $N_{t}$
unlabeled examples. It is assumed that the two domains are characterized by
different probability distributions, while they share the same feature space.
In a classification task, the goal is to learn a transferable classifier to
minimize the classification error on the target domain using all the given
data.

Deep domain adaptation methods begin with a feature extractor that can be
implemented by a neural network. The feature extractor is supposed to learn
the domain-invariant feature representations from both domains. Specifically,
the feature extractor learns a function
$F(\mathbf{x};\theta_{f}):\mathbb{R}^{d}\rightarrow\mathbb{R}^{m}$ that maps
an instance to an $m$-dimensional representation with the network parameters
$\theta_{f}$ (as illustrated in Figure 2). For simplicity, we denote the
feature representation of a source instance $\mathbf{x}_{n}^{s}$ as
$\mathbf{F}_{n}^{s}:=F(\mathbf{x}_{n}^{s};\theta_{f})$, and that of a target
instance $\mathbf{x}_{n}^{t}$ as
$\mathbf{F}_{n}^{t}:=F(\mathbf{x}_{n}^{t};\theta_{f})$. We define the source
feature set $\mathcal{F}^{s}:=\\{\mathbf{F}_{n}^{s}\\}_{n=1}^{N_{s}}$ and the
target feature set $\mathcal{F}^{t}:=\\{\mathbf{F}_{n}^{t}\\}_{n=1}^{N_{t}}$.

Figure 2: The network flow in deep domain adaptation models. All samples, no
matter from the source domain $\\{\mathbf{x}_{n}^{s}\\}_{n=1}^{N_{s}}$ (blue)
or the target domain $\\{\mathbf{x}_{n}^{s}\\}_{n=1}^{N_{t}}$ (green), are fed
into a feature extractor $F(\cdot;\theta_{f})$ to extract features that are
both discriminative and domain-invariant. The discriminative features are
achieved by training a discriminator $D(\cdot;\theta_{d})$ to minimize the
loss function $\mathcal{L}_{cls}(\theta_{f},\theta_{d})$, while the domain
invariance is measured by $\mathcal{H}(\mathcal{F}^{s},\mathcal{F}^{t})$.

When domain adaptation is applied to a classification problem, the feature
extractor is followed by a discriminator trained with samples from the source
domain $\mathcal{D}_{s}$. Given the feature representations $\mathcal{F}^{s}$
computed by the feature extractor on the source domain, together with the
labels $\\{y_{n}^{s}\\}_{n=1}^{N_{s}}$ ($y_{n}^{s}\in\\{1,2,\ldots,l\\}$), we
can train a discriminator
$D(\cdot~{};\theta_{d}):\mathbb{R}^{m}\rightarrow\mathbb{R}^{l}$ that is
characterized by the parameters $\theta_{d}$. The discriminator loss function
is defined as the cross-entropy between the predicted probabilistic
distribution and the one-hot encoding of the class labels:

$\small\mathcal{L}_{cls}(\theta_{f},\theta_{d}):=-\frac{1}{N_{s}}\sum\limits_{n=1}\limits^{N_{s}}\sum\limits_{i=1}\limits^{l}\mathbf{1}(y_{n}^{s}=i)\cdot\log
D\big{(}F(\mathbf{x}_{n}^{s};\theta_{f});\theta_{d}\big{)}_{i},$ (3)

where $\mathbf{1}(\cdot)$ is the indicator function and
$D(\cdot~{};\theta_{d})_{i}$ represents the $i$-th element in the predicted
distribution $D(\cdot~{};\theta_{d})$.

Besides the discriminator that is trained to learn the discriminative
features, the feature extractor is also followed by a discrepancy term that
measures the difference between the source features $\mathcal{F}^{s}$ and the
target features $\mathcal{F}^{t}$ to learn domain-invariant feature
representations. To be specific, for a given discrepancy mesure $\mathcal{H}$,
the empirical discrepancy between the source feature distribution and the
target feature distribution is given by
$\mathcal{H}(\mathcal{F}^{s},\mathcal{F}^{t})$. In literature, there have been
multiple choices of the discrepancy measures $\mathcal{H}$, such as MMD
distance [6], Wasserstein distance [9], JS distance [33], Proxy-$\mathcal{A}$
distance [34], etc. We call the latter few measures as adversarial distance
because they are implemented with a domain classifier to quantify the
invariance between $\mathcal{F}^{s}$ and $\mathcal{F}^{t}$. We illustrate the
commonly-chosen discrepancy measures in domain adaptation models in Figure
3(a)-(b).

Figure 3: There can be various choices of $\mathcal{H}$ to measure the feature
difference in Eq. (4), such as (a) MMD distance and (b) Wasserstein distance,
JS distance, Proxy-$\mathcal{A}$ distance, etc. Our proposed model (c) looks
into the detailed feature structures and regularizes the source error by
dynamically adjusting the marginal divergence and the copula distance.

In summary, the detailed objective function to train a deep domain adaptation
network is:

$\min_{\theta_{f},\theta_{d}}\,\big{\\{}\mathcal{L}_{cls}(\theta_{f},\theta_{d})+\lambda\mathcal{H}(\mathcal{F}^{s},\mathcal{F}^{t})\big{\\}},$
(4)

where the coefficient $\lambda$ controls the tradeoff between discriminative
and transferrable feature learning.

###  IV-A Copula-based domain adaptation networks

In this work, we propose a copula-based domain adaptation network (CDAN) to
allow more flexibility and emphasis on the dependence structure. To be
specific, on the source domain, we split the distribution of the source
features $\mathcal{F}^{s}\subseteq\mathbb{R}^{N_{s}\times m}$ into $m$
marginal distributions and the copula between the marginals. We do the same on
the target domain. This facilitates us to evaluate the feature differences
between the source domain and the target domain by the sum of two terms: (1)
the sum of marginal feature differences
$\\{\mathcal{H}(\mathcal{F}_{i}^{s},\mathcal{F}_{i}^{t})\\}_{i=1}^{m}$, and
(2) the copula distance between the source features and the target features
$CD_{\mathcal{H}}(\mathcal{F}^{s},\mathcal{F}^{t};\bm{\beta})$. Here, for
$*\in\\{s,t\\}$,
$\mathcal{F}_{i}^{*}:=\\{\mathbf{F}_{n,i}^{*}\\}_{n=1}^{N_{*}}\subseteq\mathbb{R}^{N_{*}\times
1}$ with $\mathbf{F}_{n,i}^{*}$ being the $i$-th dimensional value of
$\mathbf{F}_{n}^{*}$. In the later context, we call (1) the marginal
divergence (MD) and call (2) the copula distance (CD). By adding the marginal
divergence (with hyperparameters $\\{\alpha_{i}\\}_{i=1}^{m}$) and the copula
distance (with hyperparameters $\bm{\beta}$) as regularization terms, we
arrive at the objective function to train a CDAN, namely:

$\min_{\theta_{f},\theta_{d}}\,\big{\\{}\mathcal{L}_{cls}(\theta_{f},\theta_{d})+\sum\limits_{i=1}\limits^{m}\alpha_{i}\mathcal{H}_{1}(\mathcal{F}_{i}^{s},\mathcal{F}_{i}^{t})+CD_{\mathcal{H}_{2}}(\mathcal{F}^{s},\mathcal{F}^{t};\bm{\beta})\big{\\}}.$
(5)

The detailed divergence framework is illustrated in Figure 3(c). Notice that
in Eq. (5), we differentiate the divergence metric $\mathcal{H}_{1}$ that is
used to calculate the marginal divergence with $\mathcal{H}_{2}$ that is for
the copula distance calculation. It is worth noting that our model can well
accommodate the commonly-used divergence measures in literature.

Our proposed model has three advantages. On one hand, rather than encoding the
divergence of each marginal feature and the dependence difference in a single
value as in Eq. (4), we split the divergence of the joint feature
distributions. Thus we can identify to what extent the marginal feature
differences and the copula distance contribute to the target risk
respectively. Moreover, using hyperparameters $\\{\alpha_{i}\\}_{i=1}^{m}$ and
$\bm{\beta}$ to separately control the marginal divergence and copula distance
allows us to dynamically adjust the hyperparameters in a data-driven manner.
It implicitly shows that there can be a tradeoff between the marginal
divergence and the copula distance. Finally, different from using one distance
metric to measure both the marginal feature differences and the dependence
difference, our proposed model provides a more convenient and elaborate way to
detect the changes of the marginal distributions and the dependence structure.

It is straightforward to generalize our CDAN model to the regression tasks.
Same as in the classification setting, a domain adaptation model for a
regression task combines the feature extractor $F(\cdot;\theta_{f})$ together
with a regressor network $D(\cdot;\theta_{d})$ to form the basis of a
supervised learning network. We denote the predicted value for a sample
$\mathbf{x}_{n}^{s}$ as
$\widehat{y}_{n}^{s}:=D(F(\mathbf{x}_{n}^{s};\theta_{f});\theta_{d})$. The
regressor loss function is defined as the mean squared error between the
predicted values $\\{\widehat{y}_{n}^{s}\\}_{n=1}^{N_{s}}$ and the ground-
truth values $\\{y_{n}^{s}\\}_{n=1}^{N_{s}}$ on the source domain:
$\mathcal{L}_{rgr}(\theta_{f},\theta_{d}):=\sum_{n=1}^{N_{s}}(\widehat{y}_{n}^{s}-y_{n}^{s})^{2}/N_{s}=\sum_{n=1}^{N_{s}}\big{(}D(F(\mathbf{x}_{n}^{s};\theta_{f});\theta_{d})-y_{n}^{s}\big{)}^{2}/N_{s}$.
Adding the marginal divergence and the copula distance as regularizer, we
obtain the objective function to train a CDAN for a regression task:

$\min_{\theta_{f},\theta_{d}}\,\big{\\{}\mathcal{L}_{rgr}(\theta_{f},\theta_{d})+\sum\limits_{i=1}\limits^{m}\alpha_{i}\mathcal{H}_{1}(\mathcal{F}_{i}^{s},\mathcal{F}_{i}^{t})+CD_{\mathcal{H}_{2}}(\mathcal{F}^{s},\mathcal{F}^{t};\bm{\beta})\big{\\}}.$

###  IV-B Algorithm

The complete process of CDAN algorithm is presented in Algorithm 1. In
particular, we provide a detailed description on how to learn the Gaussian
copula parameter $\Sigma$ and how to update the model parameters $\theta_{f}$
and $\theta_{d}$.

Learning the Gaussian copula parameter $\Sigma$. [35] proposes a moment-
matching approach to learn the Gaussian copula parameter $\Sigma$ through
Kendall’s tau. Denote $\rho_{\tau}(X_{i},X_{j})$ as Kendall’s tau between
random variables $X_{i}$ and $X_{j}$, that is,
$\rho_{\tau}(X_{i},X_{j}):=\mathbb{E}[\text{sign}\big{(}(X_{i}-\widetilde{X}_{i})(X_{j}-\widetilde{X}_{j})\big{)}]$,
where $[\widetilde{X}_{i},\widetilde{X}_{j}]$ is an independent copy of
$[X_{i},X_{j}]$. Then it can be proved that
$\Sigma_{ij}=\sin\frac{\pi}{2}\rho_{\tau}(X_{i},X_{j})$. However, computing
Kendall’s tau by definition incurs a complexity of $O(N^{2})$ when the sample
size is $N$, making it expensive to train deep neural networks. Moreover,
gradient vanishing occurs during training the neural network because of the
sign function.

1

Input : Source data
$\mathcal{D}_{s}=\\{(\mathbf{x}_{n}^{s},y_{n}^{s})\\}_{n=1}^{N_{s}}$, Target
data $\mathcal{D}_{t}=\\{\mathbf{x}_{n}^{t}\\}_{n=1}^{N_{t}}$, Maximum
training epoch $S$, Divergence metric $\mathcal{H}_{1}$, $\mathcal{H}_{2}$,
Parameters $\\{\alpha_{i}\\}_{i=1}^{m}$, $\bm{\beta}$.

Output : Optimal model parameters $\theta_{f},\theta_{d}$.

2

3for  _epoch $=1$ to $S$_ do

4
$\mathcal{F}^{s}\leftarrow\\{F(\mathbf{x}_{n}^{s};\theta_{f})\\}_{n=1}^{N_{s}}$;

5
$\mathcal{F}^{t}\leftarrow\\{F(\mathbf{x}_{n}^{t};\theta_{f})\\}_{n=1}^{N_{t}}$;

6
$MD(\theta_{f})\leftarrow\sum\limits_{i=1}\limits^{m}\alpha_{i}\mathcal{H}_{1}(\mathcal{F}^{s}_{i},\mathcal{F}^{t}_{i})$;

7 $CD(\theta_{f})\leftarrow
CD_{\mathcal{H}_{2}}(\mathcal{F}^{s},\mathcal{F}^{t};\bm{\beta})$ as
calculated in Eq. (2);

8 Get the source error $\mathcal{L}_{cls}(\theta_{f},\theta_{d})$ with Eq.
(3);

9
$Loss\leftarrow\mathcal{L}_{cls}(\theta_{f},\theta_{d})+MD(\theta_{f})+CD(\theta_{f})$;

10 $Loss.backward()$

Algorithm 1 CDAN algorithm

To address the gradient vanishing issue, we propose to replace the $sign$
function with the $\tanh$ function with parameter $a$, namely,
$\rho_{\tau}(X_{i},X_{j})\approx\rho(X_{i},X_{j};a):=\mathbb{E}[\tanh\big{(}a(X_{i}-\widetilde{X}_{i})(X_{j}-\widetilde{X}_{j})\big{)}]$.
We prove that
$\text{lim}_{a\to\infty}\rho(X_{i},X_{j};a)=\rho_{\tau}(X_{i},X_{j})$ (with
the proof attached in Appendix.)

######  Proposition 3.

For two random variables $X_{1}$ and $X_{2}$, it holds that
$\lim\limits_{a\to\infty}\rho(X_{1},X_{2};a)=\rho_{\tau}(X_{1},X_{2})$.

To further reduce the computational complexity, we adopt the unbiased estimate
of Kendall’s tau that can be computed with linear complexity. More
specifically, given
$\\{[x_{n,1},x_{n,2}]\\}_{n=1}^{N}\subseteq\mathbb{R}^{N\times 2}$ as $N$
realizations of $[X_{i},X_{j}]$, an unbiased estimator for
$\rho_{\tau}(X_{i},X_{j})$ is
$\rho_{\tau}(X_{i},X_{j})=\frac{2}{N}\sum_{n=1}^{N/2}\tanh\big{(}a(x_{2n-1,1}-x_{2n,1})(x_{2n-1,2}-x_{2n,2})\big{)}$.
It significantly reduces the computational cost of Kendall’s tau from
$O(N^{2})$ to $O(N)$.

Learning the model parameters $\theta_{f}$ and $\theta_{d}$. For illustration
convenience, we conduct the calculation in the classification setting. It can
be similarly done for the regression problem, thus we omit here. From Eq. (5),
we have:

$\displaystyle\bigtriangledown_{\theta_{d}}$
$\displaystyle=\partial_{\theta_{d}}\mathcal{L}_{cls}(\theta_{f},\theta_{d})$
$\displaystyle=-\sum\limits_{n=1}^{N_{s}}\sum\limits_{i=1}^{l}\mathbf{1}(y_{n}^{s}=i)\cdot\partial_{\theta_{d}}D(\mathbf{F}_{n}^{s};\theta_{d})_{i}/\big{(}N_{s}D(\mathbf{F}_{n}^{s};\theta_{d})_{i}\big{)},$
$\displaystyle\bigtriangledown_{\theta_{f}}=\sum\limits_{i=1}^{m}\alpha_{i}\partial_{\theta_{f}}\mathcal{H}_{1}(\mathcal{F}_{i}^{s},\mathcal{F}_{i}^{t})+\partial_{\theta_{f}}CD_{\mathcal{H}_{2}}+\partial_{\theta_{f}}\mathcal{L}_{cls}(\theta_{f},\theta_{d}),$

where $CD_{\mathcal{H}_{2}}$ is the abbreviation for
$CD_{\mathcal{H}_{2}}(\mathcal{F}^{s},\mathcal{F}^{t};\bm{\beta})$.

By the chain rule,

$\displaystyle\partial_{\theta_{f}}\mathcal{L}_{cls}(\theta_{f},\theta_{d})$
$\displaystyle=$
$\displaystyle-\sum\limits_{n=1}^{N_{s}}\sum\limits_{i=1}^{l}\frac{\mathbf{1}(y_{n}^{s}=i)\cdot\partial_{\mathbf{F}_{n}^{s}}D(\mathbf{F}_{n}^{s};\theta_{d})_{i}\cdot\partial_{\theta_{f}}F(\mathbf{x}_{n}^{s};\theta_{f})}{N_{s}D(\mathbf{F}_{n}^{s};\theta_{d})_{i}}.$

The derivatives
$\partial_{\theta_{f}}\mathcal{H}_{1}(\mathcal{F}_{i}^{s},\mathcal{F}_{i}^{t})$
and
$\partial_{\theta_{f}}CD_{\mathcal{H}_{2}}(\mathcal{F}^{s},\mathcal{F}^{t};\bm{\beta})$
depend on the choice of $\mathcal{H}_{1}$ and $\mathcal{H}_{2}$. In the
experiment, we will take $\mathcal{H}_{1}$ as the MMD distance and
$\mathcal{H}_{2}$ as the KL divergence. Specifically, if $\mathcal{H}_{1}$ is
MMD distance with the characteristic kernel function $k$, the unbiased
estimate of squared MMD distance between $\mathcal{F}_{i}^{s}$ and
$\mathcal{F}_{i}^{t}$ is given as [6]:

$\displaystyle\mathcal{H}_{\text{MMD}}^{2}(\mathcal{F}_{i}^{s},\mathcal{F}_{i}^{t}):=$
$\displaystyle\sum_{n,n^{\prime}=1}^{N_{s}}\frac{k(\mathbf{F}_{n,i}^{s},\mathbf{F}_{n^{\prime},i}^{s})}{N_{s}^{2}}+\sum_{n,n^{\prime}=1}^{N_{t}}\frac{k(\mathbf{F}_{n,i}^{t},\mathbf{F}_{n^{\prime},i}^{t})}{N_{t}^{2}}$
$\displaystyle-\sum_{n=1}^{N_{s}}\sum_{n^{\prime}=1}^{N_{t}}\frac{2k(\mathbf{F}_{n,i}^{s},\mathbf{F}_{n^{\prime},i}^{t})}{N_{s}N_{t}}.$

Thus,

$\displaystyle\partial_{\theta_{f}}\mathcal{H}_{\textup{MMD}}(\mathcal{F}_{i}^{s},\mathcal{F}_{i}^{t})$
$\displaystyle=$
$\displaystyle\Big{[}\sum\limits_{n,n^{\prime}=1}^{N_{s}}\frac{\partial_{\theta_{f}}k(\mathbf{F}_{n,i}^{s},\mathbf{F}_{n^{\prime},i}^{s})}{N_{s}^{2}}+\sum\limits_{n,n^{\prime}=1}^{N_{t}}\frac{\partial_{\theta_{f}}k(\mathbf{F}_{n,i}^{t},\mathbf{F}_{n^{\prime},i}^{t})}{N_{t}^{2}}$
$\displaystyle-\sum\limits_{n=1}^{N_{s}}\sum\limits_{n^{\prime}=1}^{N_{t}}\frac{2\partial_{\theta_{f}}k(\mathbf{F}_{n,i}^{s},\mathbf{F}_{n^{\prime},i}^{t})}{N_{s}N_{t}}\Big{]}/\big{(}2\mathcal{H}_{\textup{MMD}}(\mathcal{F}^{s},\mathcal{F}^{t})\big{)}.$

When $\mathcal{H}_{2}$ is the KL divergence, then

$\displaystyle\partial_{\theta_{f}}CD_{\mathcal{H}_{KL}}=\sum\limits_{i<j}\frac{\beta_{ij}\partial_{\theta_{f}}|\log(1-(\Sigma_{ij}^{s})^{2})/(1-(\Sigma_{ij}^{t})^{2})|}{2},$

where $\Sigma_{ij}^{s}$ ($\Sigma_{ij}^{t}$) is the Gaussian copula parameter
of $\\{[\mathbf{F}^{s}_{n,i},\mathbf{F}^{s}_{n,j}]\\}_{n=1}^{N_{s}}$
($\\{[\mathbf{F}^{t}_{n,i},\mathbf{F}^{t}_{n,j}]\\}_{n=1}^{N_{t}}$).

##  V Experiments

To prove the efficacy of our proposed model, we test it on one toy dataset and
three real-world datasets, with the details of the three real-world datasets
summarized in Table II. All experiments in this section are run on Dell 7920
with Intel(R) Xeon(R) Gold 6250 CPU at 3.90GHz, and a set of NVIDIA Quadro RTX
6000 GPU. The code is available at
https://github.com/yzR1991/Deep_into_The_Domain_Shift. We run our python code
on the anaconda virtual environment with Microsoft Windows Server 2019
Standard as OS. The code can also be run in other OS, device, or environment,
with Pytorch version 1.7 or above.

TABLE II: Information of the real-world datasets used in this paper. Dataset |  $\sharp$Instances |  $\sharp$Features | Task
---|---|---|---
Retail credit data | 1100000 | 69 | Classification
Equity price data | 71242 | 22 | Regression
UCI wine quality | 6197 | 12 | Regression

###  V-A Toy problem: Two inter-twinning moons

The source domain considered here is the classical binary problem with two
inter-twinning moons, each class corresponding to one moon. Specifically, for
the blue moon in Figure 4, the points roughly falls on the upper half circle
of $y=\sqrt{1-x^{2}}$, with points for the red moon falling on lower half
circle of $y=0.5-\sqrt{1-(1-x)^{2}}$. We then consider 4 different target
domains by stretching the circle into ellipses where the length of the major
axis can be 2, 3, 4 and 5 times that of the minor axis. Such stretching from
source domain to target domain strongly affects the relationship between the
vertical ordinates and the horizontal ordinates of each point, thus making
changes to the internal dependence structure of the data. For each domain, we
generate 1024 instances (512 of each class).

For each transfer task, we compare the CDAN model with DAN [6], CORAL[17] and
the no-adaptation baseline (MLP). Each trial of the models is repeated 10
times, and we report the average accuracy in Table III. We remark that the
larger the length of the major axis, the more difficult the problem becomes,
as all of the four models unanimously show a weaker adaptation ability. Our
CDAN provides the best performance in all of the four transfer tasks,
indicating that CDAN actually captures the dependence difference precisely.

Figure 4: Illustration of the four transfer tasks on the synthetic dataset. The two classes of the source samples are blue and red, and points that are more transparent represent the target samples. TABLE III: Accuracy for the inter-twinning moons dataset. | 2 times | 3 times | 4 times | 5 times
---|---|---|---|---
MLP | 88.99 | 80.88 | 78.73 | 77.10
CORAL | 97.45 | 94.40 | 92.32 | 90.26
DAN | 97.04 | 94.17 | 91.18 | 91.04
CDAN | 97.91 | 94.42 | 93.17 | 91.54

###  V-B Retail credit classification

In this section, we apply our methods to improve the classification accuracy
of a credit risk model on a novel real-world anonymous dataset. The dataset is
kindly provided by one of the largest global technology firms that operates in
both the e-commerce business and the lending business. It records the monthly
credit status of roughly one-half million customers, spanning from 2016
January to 2020 June. The credit status shows whether a customer is in
default. In addition to the credit status, customers’ monthly shopping,
purchasing, and loan history are also included in the dataset in detail. We
collect 69 features for each customer in each month from the raw dataset and
normalize them to the range $[0,1]$. A binary classification model is
constructed to distinguish customers who default (labeled as 0) from those who
have paid off all the debts on time (labeled as 1).

Domain adaptation is needed when one forecasts customers’ credit risk with the
classification model trained with the past data because significant
distribution shifts exist between months, especially between the off-season
and the peak season, between before-COVID times and after-COVID times. We
record the distribution shifts between two consecutive months in Figure 5.
Specifically, we collect 2 features that show great importance for the
classification: each customer’s monthly total purchase and his monthly credit
ratio (available credit amount / total credit limit). To illustrate the
distribution shifts, we take the three points on 19Jun in Figure 5 as an
example. The blue (blue-dashed, resp.) point records the MMD distance of
monthly purchase (credit ratio, resp.) distribution between May and June, and
the black point records the copula distance between May and June. From Figure
5, we can identify three peak periods circled by a red box (19Jun, 19Nov,
20Feb) that show substantial marginal differences, verifying the necessity of
transfer learning. What’s more, the copula distance remains high over the
whole year, suggesting that special attention to the dependence difference is
in urgent need.

Figure 5: Distribution shift in the raw data. TABLE IV: Performance of retail credit classification in 8 transfer tasks. |  19May$\rightarrow$Jun |  19Oct$\rightarrow$Nov |  19Dec$\rightarrow$20Jan |  20Jan$\rightarrow$Feb |  20Feb$\rightarrow$Mar |  20Mar$\rightarrow$Apr |  20Apr$\rightarrow$May |  20May$\rightarrow$Jun
---|---|---|---|---|---|---|---|---
MLP |  73.85 $\pm$ 0.05 |  56.11 $\pm$ 0.05 |  90.66 $\pm$ 0.005 |  75.60 $\pm$ 0.03 |  76.55 $\pm$ 0.04 |  90.78 $\pm$ 0.009 |  91.61 $\pm$ 0.005 |  88.06 $\pm$ 0.018
DAN |  79.00 $\pm$ 0.03 |  79.31 $\pm$ 0.05 |  92.09 $\pm$ 0.004 |  87.41 $\pm$ 0.01 |  91.04 $\pm$ 0.01 |  91.88 $\pm$ 0.008 |  92.78 $\pm$ 0.002 |  90.88 $\pm$ 0.009
CORAL |  77.86 $\pm$ 0.04 |  76.23 $\pm$ 0.05 | 92.12 $\pm$ 0.003 |  87.13 $\pm$ 0.01 |  89.15 $\pm$ 0.01 |  91.94 $\pm$ 0.008 | 92.79 $\pm$ 0.003 |  90.94 $\pm$ 0.009
AFN |  77.53 $\pm$ 0.02 |  80.01 $\pm$ 0.02 |  83.73 $\pm$ 0.02 |  82.72 $\pm$ 0.02 |  83.32 $\pm$ 0.01 |  84.52 $\pm$ 0.01 |  86.24 $\pm$ 0.01 |  84.71 $\pm$ 0.02
MCD |  75.30 $\pm$ 0.10 |  71.08 $\pm$ 0.11 |  82.60 $\pm$ 0.08 |  78.40 $\pm$ 0.09 |  76.94 $\pm$ 0.08 |  86.48 $\pm$ 0.04 |  86.00 $\pm$ 0.04 |  84.85 $\pm$ 0.05
CDAN | 80.44 $\pm$ 0.04 | 80.79 $\pm$ 0.06 |  91.93 $\pm$ 0.005 | 88.20 $\pm$ 0.02 | 91.51 $\pm$ 0.01 | 92.46 $\pm$ 0.010 |  92.74 $\pm$ 0.003 | 91.39 $\pm$ 0.010

We first evaluate our methods on the transfer between the off-seasons and the
peak seasons (specifically, the sales seasons June and November), and build
two transfer tasks: 19May$\rightarrow$Jun, 19Oct$\rightarrow$Nov. We further
investigate the COVID impact on transferring the classification models and
include the evaluation on 6 more transfer tasks: 19Dec$\rightarrow$20Jan,
20Jan$\rightarrow$Feb, 20Feb$\rightarrow$Mar, 20Mar$\rightarrow$Apr,
20Apr$\rightarrow$May, 20May$\rightarrow$Jun. In each transfer task, the
source domain consists of samples from the former month and the target domain
samples are from the latter month. The sample size for each domain is in the
magnitude of $10^{5}$ customers.

We mainly follow standard evaluation protocol [36] for unsupervised domain
adaptation and use all source samples with binary labels and all target
samples without labels [6]. We compare our CDAN model to 4 classical domain
adaptation models: DAN [6], CORAL [17], AFN [37], MCD [38] as well as the no-
adaptation baseline (MLP), which is a fully connected neural network with
multiple hidden layers. For our model CDAN, we set $\mathcal{H}_{1}$ to be the
MMD distance [6], and set $\mathcal{H}_{2}$ to be the KL divergence [39]. We
set $\alpha_{i}=\alpha\,(\forall\,i)$ and $\beta_{ij}=\beta\,(\forall\,i,j)$,
and select the hyperparameter pair $(\alpha,\beta)$ by grid search. The
detailed implementation procedures are summarized in the Appendix.

In Table IV, we record the averages and standard errors of AUC over 100
randomized trials for each model in each task. CDAN outperforms the other
models in 6 transfer tasks. It deserves attention that the outperformance is
significant in that the increase in the AUC score is far larger than the
standard deviation. As an illustration, we plot the density of the learned
features’ MD and CD in Figure 6 for each model. The density is estimated from
the 100 trials for each model in a specific transfer task
(20Jan$\rightarrow$Feb). We find that CDAN again shows superiority in terms of
contracting both the marginal and the dependence differences. Such
observation, together with the optimized hyperparameters, explains why CDAN
outperforms other domain adaptation models.

Figure 6: Distribution of the learned features’ marginal divergence (left) and
copula distance (right) over 100 trials for each model.

###  V-C Intra-day equity price regression

We collect the intraday 5-minute asset prices of 22 stocks selected from HKEX
according to the market cap and daily turnover. The 22 stocks cover 8
industries (according to the Hang Seng Industry Classification System) that
include Information Technology, Financials, Consumer Discretionary, etc. The
data spans from Dec 1st, 2014 to Dec 31st, 2020, and consists of 71242
observations with information of the first half-hour in each trading day
excluded. We divide the observations into two domains according to the Hang
Seng Index (HSI) daily returns. Specifically, the target domain includes
observations of the days when the daily return is less than the 0.1-quantile
of the whole 6-year return series, and the source domain includes the
remaining observations. The goal is to forecast the next 5-minute price of the
22 stocks with their last-hour price as the input.

TABLE V: The mean $\alpha$-quantiles $Q^{s}_{\alpha}$ ($Q^{t}_{\alpha}$) over all 22 stocks and the mean quantile dependence $\tau^{s}_{\alpha}$ ($\tau^{t}_{\alpha}$) over all stock pairs in the source (target) domain. Values in the brackets record the corresponding minimum and maximum. | $\alpha=0.03$ | $\alpha=0.10$ | $\alpha=0.25$
---|---|---|---
$Q^{s}_{\alpha}$ |  | 78.5
---
(37.7, 114.8)
| 87.4
---
(39.8, 147.6)
| 103.9
---
(44.3, 198.3)
$Q^{t}_{\alpha}$ |  | 76.6
---
(36.7, 131.3)
| 84.2
---
(38.7, 149.0)
| 97.7
---
(43.5, 158.7)
$\tau^{s}_{\alpha}$ |  | 28.9
---
(4.3e-2, 88.0)
| 36.8
---
(1.6e-1, 89.6)
| 48.5
---
(8.4e-1, 98.6)
$\tau^{t}_{\alpha}$ |  | 42.6
---
(1.7e-1, 100.0)
| 47.0
---
(2.2, 93.8)
| 58.4
---
(2.4, 99.7)
TABLE VI: Efficiency of equity price forecast for the 6 models. Quantities are averaged over 100 runs. | RMSE | RE(%) | Q1 RE(%) | Q2 RE(%) | Q3 RE(%) | LRE(%) | SRE(%)
---|---|---|---|---|---|---|---
RNN |  0.0507 $\pm$ 3.4e-3 |  3.33 $\pm$ 0.22 | 3.23 | 3.35 | 3.47 |  9.15 $\pm$ 0.90 |  1.02 $\pm$ 0.12
LSTM |  0.0446 $\pm$ 1.5e-3 |  3.04 $\pm$ 0.12 | 2.96 | 3.02 | 3.12 |  8.62 $\pm$ 0.70 |  0.94 $\pm$ 0.10
DANN |  0.0475 $\pm$ 9.6e-3 |  3.00 $\pm$ 0.56 | 2.59 | 2.90 | 3.32 |  8.44 $\pm$ 1.60 |  0.98 $\pm$ 0.22
CORAL |  0.0449 $\pm$ 1.5e-3 |  3.06 $\pm$ 0.14 | 2.97 | 3.06 | 3.15 |  8.68 $\pm$ 0.87 |  0.96 $\pm$ 0.09
DAN |  0.0255 $\pm$ 2.5e-3 |  1.89 $\pm$ 0.21 | 1.75 | 1.89 | 2.05 |  6.44 $\pm$ 1.55 |  0.45 $\pm$ 0.10
CDAN | 0.0235 $\pm$ 2.3e-3 | 1.77 $\pm$ 0.21 | 1.63 | 1.75 | 1.89 | 6.06 $\pm$ 1.53 | 0.43 $\pm$ 0.09

To illustrate the distribution shift of the two domains, we record the
quantiles and the quantile dependence of each domain in Table V. We denote
$X_{i}^{s}$ ($X_{i}^{t}$) as the price series for stock $i$ in the source
(target) domain and define its whole price series as $X_{i}:=X_{i}^{s}\bigcup
X_{i}^{t}$. Each $X_{i}$ is normalized by its first price. For a given
quantile level $\alpha$ and $*\in\\{s,t\\}$ indicating the domain, the average
$\alpha$-quantile of domain $*$ is defined by
$Q_{\alpha}^{*}:=\sum_{i=1}^{22}Q_{\alpha,i}^{*}/22$, where $Q_{\alpha,i}^{*}$
is the $\alpha$-quantile of the price series$X_{i}^{*}$. The quantile
dependence is given by $\tau_{\alpha}^{*}:=\sum_{1\leq i<j\leq
22}\tau_{\alpha,[i,j]}^{*}/231$, where $\tau_{\alpha,[i,j]}^{*}:=100\times
Pr(X_{j}^{*}\leq Q_{\alpha,j}|X_{i}^{*}\leq Q_{\alpha,i})$ and $Q_{\alpha,i}$
is the $\alpha$-quantile of $X_{i}$. We see that the marginal quantiles of
either domain do not differ that much, but as $\alpha$ increases, the
marginals’ differences between the two domains get larger. Furthermore, the
quantile dependence significantly differs irrespective of the $\alpha$ value.

We compare to the following 5 models: RNN, LSTM, DANN [1], CORAL [18] and DAN
[6]. Specifically, RNN and LSTM serve as the no-adaptation benchmarks and only
utilize the source samples to do the training. CORAL and DAN are the same as
in Section V-B, except that they use an LSTM as a feature extractor in this
section. Though AFN and MCD are SOTA models, they are not originally designed
for sequential data and perform not well in this case, so we do not list the
corresponding results. For more implementation details, see the Appendix.

(a) Comparison of test RMSE (left) and RE distribution (right).

(b) DAN model (left) v.s. CDAN model (right).

Figure 7: The performance of forecasting the equity price in a particular
(typical) trial. All values are in the base 10 logarithms.

In Table VI, we summarize the experimental results of the 6 models. There are
7 performance metric columns in Table VI. The 1st column represents the mean
and the standard deviation of the RMSE over 100 trials. The 2nd-5th columns
record the detailed information of the relative errors (RE) over 100 trials.
Specifically, the 2nd column records the mean and the standard deviation. The
3rd-5th columns record the 0.25-quantile, 0.5-quantile and 0.75-quantile of
the RE over 100 trials. In addition to the RE over 100 trials, we also record
the maximal (in the 6th column LRE) and the minimal RE (in the 7th column SRE)
among the 22 stocks. We find that CDAN achieves the best performance.
Moreover, its Q2 RE is close to its mean RE, and the standard deviation of RE
is quite small, showing that the CDAN model is pretty stable in terms of the
model performance. We plot additional visualizing pictures in Figure 7 to have
a better understanding of the results. In Figure 7(a), CDAN converges fast in
terms of the test RMSE, and it’s efficient in controlling the relative errors.
Diving deeper into the training details, from Fig 7(b) we observe that the CD
of CDAN decreases more significantly (to around $10^{-2}$) than that of DAN
(to around $10^{-1}$). It shows that CDAN does capture the dependence
difference which explains and contributes to the outperformance of CDAN.

###  V-D Wine quality regression

The UCI wine quality dataset [40] contains records of red and white vinho
verde wine samples from the north of Portugal, with sample size 1599 and 4598
respectively. Each record has 12 features, such as pH, alcohol and quality.
The red wine and white wine samples differ in the feature distributions. Our
goal is to predict the wine quality with two transfer tasks, from white wine
to red wine (W$\rightarrow$R) and from red wine to white wine
(R$\rightarrow$W).

We compare CDAN to 6 neural network baselines, namely MLP, AFN [37], MCD [38],
DANN [1], CORAL [18], and DAN [6]. Each neural network model has 2 hidden
layers and each hidden layer has 8 units. For each model, we run 100 trials
and record the RMSE, R2 scores, and relative errors.

TABLE VII: The RMSE, R2 score (R2) and relative error (RE) for wine quality prediction. |  W$\rightarrow$R |  R$\rightarrow$W
---|---|---
| RMSE | R2 | RE | RMSE | R2 | RE
MLP | 0.125 | 0.131 | 0.110 | 0.143 | 0.067 | 0.115
AFN | 0.129 | 0.087 | 0.119 | 0.145 | 0.032 | 0.118
MCD | 0.125 | 0.137 | 0.109 | 0.144 | 0.042 | 0.116
DANN | 0.127 | 0.104 | 0.115 | 0.147 | 0.006 | 0.119
DAN | 0.122 | 0.175 | 0.109 | 0.138 | 0.123 | 0.112
CORAL | 0.125 | 0.136 | 0.109 | 0.144 | 0.054 | 0.115
CDAN | 0.120 | 0.201 | 0.108 | 0.133 | 0.177 | 0.109

Figure 8: R2 score distributions over 100 runs for the transfer task W$\rightarrow$R (left) and R$\rightarrow$W (right). TABLE VIII: Parameter sensitivity for the CDAN model in 8 transfer tasks on the retail credit dataset. $(\alpha,\beta)$ |  19May $\to$ Jun |  19Oct $\to$ Nov |  19Dec $\to$ Jan |  20Jan $\to$ Feb |  20Feb $\to$ Mar |  20Mar $\to$ Apr |  20Apr $\to$ May |  20May $\to$ Jun
---|---|---|---|---|---|---|---|---
(0.01,0.01) |  80.02$\pm$0.04 |  79.33$\pm$0.06 | 92.23$\pm$0.005 |  88.04$\pm$0.02 |  91.14$\pm$0.01 |  92.28$\pm$0.01 |  92.75$\pm$0.003 |  91.12$\pm$0.01
(0.01,0.1) |  79.97$\pm$0.04 |  80.95$\pm$0.04 |  91.89$\pm$0.005 |  87.14$\pm$0.02 |  90.71$\pm$0.01 |  91.65$\pm$0.01 |  92.67$\pm$0.004 |  90.71$\pm$0.01
(0.01, 1) |  79.84$\pm$0.03 | 81.02$\pm$0.03 |  91.84$\pm$0.005 |  86.82$\pm$0.02 |  90.36$\pm$0.02 |  91.63$\pm$0.01 |  92.54$\pm$0.003 |  91.15$\pm$0.01
(0.1,0.01) |  78.98$\pm$0.04 |  78.90$\pm$0.06 |  92.18$\pm$0.004 |  87.80$\pm$0.02 |  91.40$\pm$0.01 |  92.44$\pm$0.01 | 92.78$\pm$0.003 |  91.20$\pm$0.01
(0.1,0.1) |  79.79$\pm$0.03 |  79.95$\pm$0.05 |  91.85$\pm$0.006 |  86.90$\pm$0.02 |  90.94$\pm$0.01 |  91.85$\pm$0.01 |  92.68$\pm$0.003 |  90.91$\pm$0.01
(0.1,1) |  79.95$\pm$0.03 |  80.87$\pm$0.04 |  91.86$\pm$0.005 |  87.17$\pm$0.02 |  90.54$\pm$0.01 |  91.50$\pm$0.02 |  92.52$\pm$0.004 |  91.02$\pm$0.01
(1,0.01) | 80.44$\pm$0.04 |  80.79$\pm$0.06 |  91.93$\pm$0.005 | 88.20$\pm$0.02 | 91.51$\pm$0.01 | 92.46$\pm$0.01 |  92.74$\pm$0.003 | 91.39$\pm$0.01
(1,0.1) |  80.26$\pm$0.04 |  79.66$\pm$0.06 |  91.90$\pm$0.006 |  87.64$\pm$0.02 |  90.91$\pm$0.01 |  91.97$\pm$0.01 |  92.76$\pm$0.003 |  91.07$\pm$0.01
TABLE IX: Parameter sensitivity for the CDAN model on the equity price dataset. $(\alpha,\beta)$ | RMSE | RE(%) | Q1 RE(%) | Q2 RE(%) | Q3 RE(%) | LRE(%) | SRE(%)
---|---|---|---|---|---|---|---
(0.01,0.01) | 0.0235 $\pm$ 2.3e-3 | 1.77 $\pm$ 0.21 | 1.63 | 1.75 | 1.89 | 6.06 $\pm$ 1.53 |  0.43 $\pm$ 0.09
(0.01, 0.1) |  0.0237 $\pm$ 1.9e-3 |  1.80 $\pm$ 0.19 | 1.66 | 1.79 | 1.92 |  6.14 $\pm$ 1.58 | 0.43 $\pm$ 0.08
(0.1,0.01) |  0.0243 $\pm$ 2.6e-3 |  1.83 $\pm$ 0.26 | 1.66 | 1.79 | 1.98 |  6.52 $\pm$ 2.00 |  0.44 $\pm$ 0.10
(0.1, 0.1) |  0.0243 $\pm$ 2.6e-3 |  1.84 $\pm$ 0.25 | 1.66 | 1.80 | 2.00 |  6.14 $\pm$ 1.51 |  0.44 $\pm$ 0.09
(0.1, 1) |  0.0249 $\pm$ 2.5e-3 |  1.87 $\pm$ 0.26 | 1.70 | 1.86 | 2.03 |  6.40 $\pm$ 1.80 |  0.46 $\pm$ 0.09
(1, 0.1) |  0.0247 $\pm$ 2.3e-3 |  1.87 $\pm$ 0.23 | 1.72 | 1.87 | 1.98 |  6.64 $\pm$ 1.86 |  0.44 $\pm$ 0.10
(1, 1) |  0.0259 $\pm$ 2.4e-3 |  1.94 $\pm$ 0.23 | 1.80 | 1.95 | 2.06 |  6.68 $\pm$ 1.67 |  0.48 $\pm$ 0.09
(1, 10) |  0.0306 $\pm$ 4.7e-3 |  2.18 $\pm$ 0.35 | 1.94 | 2.09 | 2.31 |  6.88 $\pm$ 2.01 |  0.59 $\pm$ 0.13
(10, 1) |  0.0276 $\pm$ 3.2e-3 |  2.04 $\pm$ 0.27 | 1.83 | 2.01 | 2.20 |  6.87 $\pm$ 1.85 |  0.50 $\pm$ 0.11

The results are summarized in Table VII. We conclude that the CDAN model
outperforms the other benchmarks by achieving the highest R2 score, the
smallest relative error and the smallest RMSE. Furthermore, we plot the R2
score distribution over the 100 runs for the two transfer tasks in Figure 8.
We see that the R2 score for CDAN model is more concentrated than its
competitors, showing that the outperformance is quite stable.

###  V-E Parameter sensitivity and ablation study

To further look into the sensitivity of the parameters $\alpha$ and $\beta$,
we compare the model performance on the retail credit dataset and the equity
price dataset. For the retail dataset, we list the model performance under
various combinations of $\alpha$ and $\beta$ in the 8 transfer tasks in Table
VIII. Among the best performances in each transfer task, we find that the
coefficient of MD is in most cases larger than that of CD, indicating that the
marginal differences and the dependence difference weigh differently in
measuring the overall domain divergence. For the equity price dataset, we test
9 candidate pairs of hyperparameters $(\alpha,\beta)$ and record corresponding
model performances in Table IX. It shows that as the coefficients increase,
the model performance tends to get worse. And the change in the CD parameter
$\beta$ can significantly affect the model performance. It thus confirms the
motivation of learning deep features by jointly adapting marginal divergence
and copula distance, since a good trade-off between them could enhance feature
transferability.

To evaluate the efficiency of taking MD and CD separately into the
regularizer, we run experiments on the wine quality dataset. We compare the
results of either $\alpha=0$ or $\beta=0$ and summarize them in Table X. The
ablation study shows that MD and CD are both essential in terms of a good
model performance.

TABLE X: Ablation study on wine quality prediction. $(\alpha,\beta)$ |  W$\rightarrow$R R2 |  R$\rightarrow$W R2
---|---|---
$(0,0)$ | 0.131 | 0.067
$(0,0.1)$ | 0.145 | 0.012
$(0,1)$ | 0.130 | 0.069
$(0,10)$ | 0.121 | 0.070
$(0.1,0)$ | 0.151 | 0.137
$(1,0)$ | 0.181 | 0.158
$(10,0)$ | 0.093 | 0.153
$(1,1)$ | 0.201 | 0.177
TABLE XI: Comparison of different divergence measures for wine quality prediction. |  W$\rightarrow$R |  R$\rightarrow$W
---|---|---
$\mathcal{H}_{1}$ \+ $\mathcal{H}_{2}$ | RMSE | R2 | RE | RMSE | R2 | RE
MMD + KL | 0.120 | 0.201 | 0.108 | 0.133 | 0.177 | 0.109
MMD + W1 | 0.135 | 0.140 | 0.124 | 0.146 | 0.178 | 0.115
MMD + $\chi^{2}$ | 0.135 | 0.143 | 0.124 | 0.147 | 0.176 | 0.114
KL + KL | 0.128 | 0.175 | 0.112 | 0.150 | 0.036 | 0.118
KL + W1 | 0.126 | 0.210 | 0.112 | 0.149 | 0.150 | 0.117
KL + $\chi^{2}$ | 0.133 | 0.140 | 0.115 | 0.147 | 0.122 | 0.118
W1 + KL | 0.124 | 0.191 | 0.110 | 0.145 | 0.112 | 0.115
W1 + W1 | 0.126 | 0.208 | 0.113 | 0.144 | 0.179 | 0.114
W1 + $\chi^{2}$ | 0.125 | 0.200 | 0.110 | 0.144 | 0.151 | 0.114

###  V-F Comparison of different divergence measures

As we have mentioned, there can be multiple choices over the divergence
measures $\mathcal{H}_{1}$ and $\mathcal{H}_{2}$. In this section, we
investigate the performance difference caused by the various divergence
measures. Specifically, the candidate divergence measures for
$\mathcal{H}_{1}$ include KL divergence, W1 (abbreviated for Wasserstein-1)
distance and MMD. And the candidate divergence measures for $\mathcal{H}_{2}$
include KL divergence, $\chi^{2}$ (abbreviated for Pearson $\chi^{2}$)
divergence and W1 distance. In Table XI, we record the model performance of
CDAN with the various combinations of $\mathcal{H}_{1}$ and $\mathcal{H}_{2}$
on the UCI wine quality dataset. From the table, we see that, the CDAN model
with $\mathcal{H}_{1}$ taking MMD distance and $\mathcal{H}_{2}$ taking KL
divergence performs the best in terms of RMSE and RE. Also, it should be noted
that the performance difference caused by the divergence measures can be as
large as that brought by different models. That reminds us to be prudent in
choosing the suitable divergence measures.

##  VI Conclusion

This work proposes a new domain adaptation framework that facilitates a user
to detect whether the domain difference in a transfer task comes from the
marginals’ differences or the dependence difference. Specifically, we quantify
the dependence difference with copula distance, a difference measure endowed
with boundedness and monotonicity to guarantee the algorithm convergence. By
optimizing the relative weights between the marginal divergence and the copula
distance, we can acquire transferability across domains in a more flexible
way. Experiments on the real-world datasets demonstrate the efficacy and
robustness of our approach compared to a variety of existing domain adaptation
models.

## Acknowledgments

Shumin Ma acknowledges the support from: Guangdong Provincial Key Laboratory
of Interdisciplinary Research and Application for Data Science, BNU-HKBU
United International College (2022B1212010006), Guangdong Higher Education
Upgrading Plan (2021-2025) (UIC R0400001-22) and UIC (UICR0700019-22). Qi Wu
acknowledges the support from the Hong Kong Research Grants Council [General
Research Fund 14206117, 11219420, and 11200219], CityU SRG-Fd fund 7005300,
and the support from the CityU-JD Digits Laboratory in Financial Technology
and Engineering, HK Institute of Data Science. The work described in this
paper was partially supported by the InnoHK initiative, The Government of the
HKSAR, and the Laboratory for AI-Powered Financial Technologies.

## Proof of Proposition 1 and 2

###### Proof.

We begin with the analysis over the explicit form of the bivariate copula
distance $CD_{\mathcal{H}}(\mathbf{X,Y})$ when $\mathcal{H}$ is taken to be
different divergence measures. Note that the copula distance between
multivariate distributions is defined in terms of bi-variate sub-
distributions. Thus, it is enough to prove the boundedness and monotonicity of
$CD_{\mathcal{H}}(\mathbf{X,Y})$ between any two bivariate random vectors
$\mathbf{X}$, $\mathbf{Y}\in\mathbb{R}^{2}$. Suppose that the Gaussian copula
parameters for $\mathbf{X}$ and $\mathbf{Y}$ are $\Sigma^{\mathbf{X}}$ and
$\Sigma^{\mathbf{Y}}$, respectively. Their copula density functions are:

$\displaystyle
c^{\mathbf{X}}(u_{1},u_{2})=|\Sigma^{\mathbf{X}}|^{-\frac{1}{2}}\exp\Big{(}-\dfrac{1}{2}\mathbf{x}^{T}\big{(}(\Sigma^{\mathbf{X}})^{-1}-I\big{)}\mathbf{x}\Big{)},$
(6) $\displaystyle
c^{\mathbf{Y}}(u_{1},u_{2})=|\Sigma^{\mathbf{Y}}|^{-\frac{1}{2}}\exp\Big{(}-\dfrac{1}{2}\mathbf{x}^{T}\big{(}(\Sigma^{\mathbf{Y}})^{-1}-I\big{)}\mathbf{x}\Big{)},$

where $\mathbf{x}:=[x_{1},x_{2}]^{T}=[\Phi^{-1}(u_{1}),\Phi^{-1}(u_{2})]^{T}$
with $\Phi$ being the CDF of the standard normal distribution.

The first divergence class is $\phi$-divergence (see [41] for the detailed
descriptions of the $\phi$-divergence family). Given a convex function
$\phi(x)$ such that $\phi(1)=0$, the $\phi$ divergence between two
distributions $P^{\mathbf{X}}$ and $P^{\mathbf{Y}}$ is defined by
$\mathcal{H}_{\phi}(P^{\mathbf{X}},P^{\mathbf{Y}})=\int\phi(\frac{dP^{\mathbf{X}}}{dP^{\mathbf{Y}}})dP^{\mathbf{Y}}$.
With the following proposition, we prove that the copula distance between
bivariate random vectors $\mathbf{X}$ and $\mathbf{Y}$,
$CD_{\mathcal{H}_{\phi}}(\mathbf{X,Y})$, can be fully characterized by the
copula density functions $c^{\mathbf{X}}$ and $c^{\mathbf{Y}}$.

######  Proposition 4.

For any bivariate random vector $\mathbf{X}\in\mathbb{R}^{2}$, the
$\phi$-divergence between the probability distribution $P^{\mathbf{X}}$ and
the product of marginal distributions $P^{\mathbf{X}}_{1}P^{\mathbf{X}}_{2}$
is,

$\small\mathcal{H}_{\phi}(P^{\mathbf{X}},P^{\mathbf{X}}_{1}P^{\mathbf{X}}_{2})=\int_{0}^{1}\int_{0}^{1}\phi\big{(}c^{\mathbf{X}}(u_{1},u_{2})\big{)}du_{1}du_{2}.$

For any two bivariate random vectors $\mathbf{X}$,
$\mathbf{Y}\in\mathbb{R}^{2}$, the copula distance between $\mathbf{X}$ and
$\mathbf{Y}$ when $\mathcal{H}$ takes $\phi$-divergence is

$\displaystyle CD_{\mathcal{H}_{\phi}}(\mathbf{X,Y})$ $\displaystyle=$
$\displaystyle|\int_{0}^{1}\int_{0}^{1}\phi\big{(}c^{\mathbf{X}}(u_{1},u_{2})\big{)}-\phi\big{(}c^{\mathbf{Y}}(u_{1},u_{2})\big{)}du_{1}du_{2}|.$

###### Proof.

We denote the two marginal density functions for the bivariate random vector
$\mathbf{X}$ as $p_{1}^{\mathbf{X}}(\cdot)$ and $p_{2}^{\mathbf{X}}(\cdot)$.
By the definition of copula density function, the $\phi$-divergence between
$P^{\mathbf{X}}$ and $P^{\mathbf{X}}_{1}P^{\mathbf{X}}_{2}$ is,

$\displaystyle\mathcal{H}_{\phi}(P^{\mathbf{X}},P^{\mathbf{X}}_{1}P^{\mathbf{X}}_{2})$
$\displaystyle=$
$\displaystyle\int\phi\big{(}\frac{p_{1}^{\mathbf{X}}(x_{1})p_{2}^{\mathbf{X}}(x_{2})c^{\mathbf{X}}(P_{1}^{\mathbf{X}}(x_{1}),P_{2}^{\mathbf{X}}(x_{2}))}{p_{1}^{\mathbf{X}}(x_{1})p_{2}^{\mathbf{X}}(x_{2})}\big{)}dP_{1}^{\mathbf{X}}(x_{1})dP_{2}^{\mathbf{X}}(x_{2})$
$\displaystyle=$
$\displaystyle\int\phi\big{(}c^{\mathbf{X}}(P_{1}^{\mathbf{X}}(x_{1}),P_{2}^{\mathbf{X}}(x_{2}))\big{)}dP_{1}^{\mathbf{X}}(x_{1})dP_{2}^{\mathbf{X}}(x_{2}).$

With change of variables $u_{1}=P_{1}^{\mathbf{X}}(x_{1})$ and
$u_{2}=P_{2}^{\mathbf{X}}(x_{2})$, we finally have
$\mathcal{H}_{\phi}(P^{\mathbf{X}},P^{\mathbf{X}}_{1}P^{\mathbf{X}}_{2})=\int_{0}^{1}\int_{0}^{1}\phi(c^{\mathbf{X}}(u_{1},u_{2}))du_{1}du_{2}$.
For the bivariate random vector $\mathbf{Y}$, we similarly have
$\mathcal{H}_{\phi}(P^{\mathbf{Y}},P^{\mathbf{Y}}_{1}P^{\mathbf{Y}}_{2})=\int_{0}^{1}\int_{0}^{1}\phi(c^{\mathbf{Y}}(u_{1},u_{2}))du_{1}du_{2}$.
The copula distance between $\mathbf{X}$ and $\mathbf{Y}$ is defined as the
absolute difference between
$\mathcal{H}_{\phi}(P^{\mathbf{X}},P^{\mathbf{X}}_{1}P^{\mathbf{X}}_{2})$ and
$\mathcal{H}_{\phi}(P^{\mathbf{Y}},P^{\mathbf{Y}}_{1}P^{\mathbf{Y}}_{2})$.
That completes the proof. ∎

With Proposition 4, we can directly obtain the results in Section 3 in the
main paper:

  * -

When $\phi(x)=x^{2}-1$, the resulting $\phi$-divergence is a $\chi^{2}$
distance. Thus,
$\mathcal{H}_{\chi^{2}}(P_{ij},P_{i}P_{j})=\int_{0}^{1}\int_{0}^{1}(c_{ij}^{2}(u_{i},u_{j})-1)\mathrm{d}u_{i}\mathrm{d}u_{j}$.

  * -

When $\phi(x)=(\sqrt{x}-1)^{2}$, it corresponds to Hellinger distance. So we
have
$\mathcal{H}_{H}(P_{ij},P_{i}P_{j})=\int_{0}^{1}\int_{0}^{1}[\sqrt{c_{ij}(u_{i},u_{j})}-1]^{2}\mathrm{d}u_{i}\mathrm{d}u_{j}$.

  * -

When $\phi(x)=\frac{x(1-x^{-(\alpha+1)/2})}{1-\alpha^{2}}$, it results in
$\alpha$-divergence. Thus,
$\mathcal{H}_{\alpha}(P_{ij},P_{i}P_{j})=\frac{1}{1-\alpha^{2}}\int_{0}^{1}\int_{0}^{1}[1-{c_{ij}(u_{i},u_{j})}^{-\frac{\alpha+1}{2}}]c_{ij}(u_{i},$
$u_{j})\mathrm{d}u_{i}\mathrm{d}u_{j}$.

Proposition 4 states that the copula distance defined by the $\phi$-divergence
is a function of the copula densities. Also, it proves that the
$\phi$-divergence between the joint distribution and the product of marginals
is purely a function of the copula densities. That is to say, when the
divergence metric is taken to be $\phi$-divergence, the calculation of the
copula distance has nothing to do with the marginal distributions. Thus, in
the following proofs, when calculating the copula distance between any two
random vectors $\mathbf{X}$ and $\mathbf{Y}$, we will assume the marginals are
both standard normal distributions that has mean 1 and variance 0. That will
greatly simplify our calculation. We can just calculate the copula distance
between two bivariate Gaussian vectors with copula densities
$c^{\mathbf{X}}(\mathbf{u})$ and $c^{\mathbf{Y}}(\mathbf{u})$, respectively.
Furthermore, given that $\mathbf{X}$ is Gaussian with standard normal
marginals, we know that their Gaussian copula parameter $\Sigma^{\mathbf{X}}$
is exactly the correlation matrix ([42]). In the following proof, we will
write $\Sigma^{\mathbf{X}}:=\begin{gathered}\begin{pmatrix}1&\rho\\\
\rho&1\end{pmatrix}\end{gathered}$, with $\rho\in[-1,1]$. Now we are ready to
provide the explicit forms of the copula distance for various choices of the
divergence measures.

KL divergence. When $\phi(x)=x\log x$, the corresponding $\phi$-divergence is
KL divergence. By definition, we have

$\footnotesize\mathcal{H}_{\textup{KL}}(P^{\mathbf{X}},P^{\mathbf{X}}_{1}P^{\mathbf{X}}_{2})=\iint
p^{\mathbf{X}}(x_{1},x_{2})\log\frac{p^{\mathbf{X}}(x_{1},x_{2})}{p^{\mathbf{X}}_{1}(x_{1})p^{\mathbf{X}}_{2}(x_{2})}dx_{1}dx_{2}.$

Given that
$p^{\mathbf{X}}(x_{1},x_{2})=p^{\mathbf{X}}(x_{1})p^{\mathbf{X}}(x_{2})c^{\mathbf{X}}(u_{1},u_{2})$,
with Eq. (6), we have

$\displaystyle\mathcal{H}_{\textup{KL}}(P^{\mathbf{X}},P^{\mathbf{X}}_{1}P^{\mathbf{X}}_{2})$
$\displaystyle=$
$\displaystyle\iint\frac{p^{\mathbf{X}}(x_{1},x_{2})([x_{1},x_{2}](\mathbf{I}-(\Sigma^{\mathbf{X}})^{-1})[x_{1},x_{2}]^{T}-\log|\Sigma^{\mathbf{X}}|)}{2}dx_{1}dx_{2}$
$\displaystyle=$
$\displaystyle\frac{1}{2}\mathbb{E}_{p^{\mathbf{X}}}([x_{1},x_{2}](\mathbf{I}-(\Sigma^{\mathbf{X}})^{-1})[x_{1},x_{2}]^{T}-\log|\Sigma^{\mathbf{X}}|)$
$\displaystyle=$ $\displaystyle\frac{1}{2}(-\log|\Sigma^{\mathbf{X}}|+2-2)$
$\displaystyle=$ $\displaystyle-\frac{1}{2}\log|\Sigma^{\mathbf{X}}|.$

The third equality comes from [43], where it proves that
$\mathbb{E}_{p^{\mathbf{X}}}([x_{1},x_{2}](\Sigma^{\mathbf{X}})^{-1}[x_{1},x_{2}]^{T})=2.$
Finally, with the definition of the copula distance, we have:

$CD_{\mathcal{H}_{\textup{KL}}}(\mathbf{X,Y})=\frac{1}{2}|\log|\Sigma^{\mathbf{X}}|-\log|\Sigma^{\mathbf{Y}}||.$

$\chi^{2}$ distance. When $\phi(x)=x^{2}-1$, the resulting $\phi$-divergence
is a $\chi^{2}$ distance. By definition, we have

$\displaystyle\mathcal{H}_{\chi^{2}}(P^{\mathbf{X}},P^{\mathbf{X}}_{1}P^{\mathbf{X}}_{2})$
$\displaystyle=$
$\displaystyle\iint(\frac{p^{\mathbf{X}}(x_{1},x_{2})}{p^{\mathbf{X}}_{1}(x_{1})p^{\mathbf{X}}_{2}(x_{2})})^{2}p^{\mathbf{X}}_{1}(x_{1})p^{\mathbf{X}}_{2}(x_{2})dx_{1}dx_{2}-1.$

Since
$\frac{p^{\mathbf{X}}(x_{1},x_{2})}{p^{\mathbf{X}}_{1}(x_{1})p^{\mathbf{X}}_{2}(x_{2})}=c^{\mathbf{X}}(u_{1},u_{2})$
and
$p^{\mathbf{X}}_{1}(x)=p^{\mathbf{X}}_{2}(x)=\frac{1}{\sqrt{2\pi}}\exp{(-\frac{x^{2}}{2})}$,
we can further simplify the calculation as:

$\displaystyle\mathcal{H}_{\chi^{2}}(P^{\mathbf{X}},P^{\mathbf{X}}_{1}P^{\mathbf{X}}_{2})$
$\displaystyle=$
$\displaystyle\iint\frac{\exp\Big{(}[x_{1},x_{2}]\big{(}\frac{I}{2}-(\Sigma^{\mathbf{X}})^{-1}\big{)}[x_{1},x_{2}]^{T}\Big{)}}{2\pi|\Sigma^{\mathbf{X}}|}dx_{1}dx_{2}-1$
$\displaystyle=$ $\displaystyle|\Sigma^{\mathbf{X}}|^{-1}-1.$

The last equality comes from the following fact:
$2(\Sigma^{\mathbf{X}})^{-1}-I$ is positive definite with determinant $1$.
That gives:
$\iint\exp\big{(}[x_{1},x_{2}]\big{(}\frac{I}{2}-(\Sigma^{\mathbf{X}})^{-1}\big{)}[x_{1},x_{2}]^{T}\big{)}dx_{1}dx_{2}=2\pi.$
Finally, we have:

$CD_{\mathcal{H}_{\chi^{2}}}(\mathbf{X,Y})=||\Sigma^{\mathbf{X}}|^{-1}-|\Sigma^{\mathbf{Y}}|^{-1}|.$

It is not hard to derive more results about the copula distance for other
$\phi$-divergence. So we omit them here and turn to the derivation for
Wasserstein-2 distance and MMD distance.

Wasserstein-2 distance. Assume that the marginal distributions of
$\mathbf{X,Y}$ are standard normals. By directly applying the conclusion in
Proposition 7 in [44], we have

$\mathcal{H}_{\textup{W}}^{2}(P^{\mathbf{X}},P^{\mathbf{X}}_{1}P^{\mathbf{X}}_{2})=4-2\textup{Tr}\big{(}(\Sigma^{\mathbf{X}})^{\frac{1}{2}}\big{)}=4-2\sqrt{2+2\sqrt{|\Sigma^{\mathbf{X}}|}}.$

Thus,

$\displaystyle CD_{\mathcal{H_{\textup{W}}}}(\mathbf{X,Y})$ $\displaystyle=$
$\displaystyle\Big{|}\sqrt{4-2\sqrt{2+2\sqrt{|\Sigma^{\mathbf{X}}|}}}-\sqrt{4-2\sqrt{2+2\sqrt{|\Sigma^{\mathbf{Y}}|}}}\Big{|}.$

Gaussian MMD distance. Assume that the marginal distributions of
$\mathbf{X,Y}$ are standard normals. For ease of calculation, we take the
simplest kernel function
$k(\mathbf{X},\mathbf{Y})=e^{-||\mathbf{X}-\mathbf{Y}||_{2}^{2}}$. But we
emphasize that, the following calculation applies to all Gaussian kernels.
Using the kernel trick, the squared MMD distance can be computed as the
expectation of kernel functions:

$\displaystyle\mathcal{H}_{\textup{MMD}}^{2}(P^{\mathbf{X}},P^{\mathbf{X}}_{1}P^{\mathbf{X}}_{2})$
(7) $\displaystyle=$
$\displaystyle\mathbb{E}_{\mathbf{X},\mathbf{X}}k(\mathbf{X},\mathbf{X})+\mathbb{E}_{\dot{\mathbf{X}},\dot{\mathbf{X}}}k(\dot{\mathbf{X}},\dot{\mathbf{X}})-2\mathbb{E}_{\mathbf{X},\dot{\mathbf{X}}}k(\mathbf{X},\dot{\mathbf{X}}),$

where $\dot{\mathbf{X}}\in\mathbb{R}^{2}$ is a random Gaussian vector with CDF
$P^{\dot{\mathbf{X}}}(x_{1},x_{2})=P_{1}^{\mathbf{X}}(x_{1})P_{2}^{\mathbf{X}}(x_{2})$
and the Gaussian copula parameter
$\Sigma^{\dot{\mathbf{X}}}=\begin{gathered}\begin{pmatrix}1&0\\\
0&1\end{pmatrix}\end{gathered}.$

From Eq. (7), we know that to calculate the squared MMD distance
$\mathcal{H}_{\textup{MMD}}^{2}(P^{\mathbf{X}},P^{\mathbf{X}}_{1}P^{\mathbf{X}}_{2})$,
we need to calculate the three expectations on the right-hand-side of this
equation. We begin with the calculation of
$\mathbb{E}_{\mathbf{X},\dot{\mathbf{X}}}k(\mathbf{X},\dot{\mathbf{X}})$. By
definition, we have:

$\displaystyle\mathbb{E}_{\mathbf{X},\dot{\mathbf{X}}}k(\mathbf{X},\dot{\mathbf{X}})$
$\displaystyle=$
$\displaystyle\iint\frac{k(\mathbf{X},\mathbf{Y})\exp\big{(}-\frac{1}{2}\mathbf{X}^{T}(\Sigma^{\mathbf{X}})^{-1}\mathbf{X}-\frac{1}{2}\mathbf{Y}^{T}(\Sigma^{\dot{\mathbf{X}}})^{-1}\mathbf{Y}\big{)}}{4\pi^{2}\sqrt{|\Sigma^{\mathbf{X}}\Sigma^{\dot{\mathbf{X}}}|}}d\mathbf{X}d\mathbf{Y}$
$\displaystyle=$
$\displaystyle\iiiint\frac{\exp(-\dfrac{1}{2}[x_{1},x_{2},y_{1},y_{2}]A[x_{1},x_{2},y_{1},y_{2}]^{T})}{4\pi^{2}\sqrt{|\Sigma^{\mathbf{X}}\Sigma^{\dot{\mathbf{X}}}|}}dx_{1}dx_{2}dy_{1}dy_{2}$
$\displaystyle=$
$\displaystyle\dfrac{1}{4\pi^{2}\sqrt{|\Sigma^{\mathbf{X}}\Sigma^{\dot{\mathbf{X}}}|}}\times(2\pi)^{2}|A|^{-\frac{1}{2}}$
$\displaystyle=$
$\displaystyle\frac{1}{\sqrt{|2\Sigma^{\mathbf{X}}+2\Sigma^{\dot{\mathbf{X}}}+I|}}=\frac{1}{\sqrt{21+4|\Sigma^{\mathbf{X}}|}}.$

Here, the matrix
$A:=\begin{gathered}\begin{pmatrix}(\Sigma^{\mathbf{X}})^{-1}+2I&-2I\\\
-2I&(\Sigma^{\dot{\mathbf{X}}})^{-1}+2I\end{pmatrix}\end{gathered}\in\mathbb{R}^{4\times
4}$ is positive semidefinite with determinant
$\frac{|2\Sigma^{\mathbf{X}}+2\Sigma^{\dot{\mathbf{X}}}+I_{2}|}{|\Sigma^{\mathbf{X}}\Sigma^{\dot{\mathbf{X}}}|}$.
Similarly, we have:

$\displaystyle\mathbb{E}_{\mathbf{X},\mathbf{X}}k(\mathbf{X},\mathbf{X})$
$\displaystyle=\frac{1}{\sqrt{|4\Sigma^{\mathbf{X}}+I|}}=\frac{1}{\sqrt{9+16|\Sigma^{\mathbf{X}}|}},$
$\displaystyle\mathbb{E}_{\dot{\mathbf{X}},\dot{\mathbf{X}}}k(\dot{\mathbf{X}},\dot{\mathbf{X}})$
$\displaystyle=\frac{1}{\sqrt{|4\Sigma^{\dot{\mathbf{X}}}+I|}}=\frac{1}{5}.$

Organizing the three terms together, we have the squared MMD distance:

$\mathcal{H}^{2}_{\textup{MMD}}(P^{\mathbf{X}},P^{\mathbf{X}}_{1}P^{\mathbf{X}}_{2})=\frac{1}{\sqrt{9+16|\Sigma^{\mathbf{X}}|}}+\frac{1}{5}-\frac{2}{\sqrt{21+4|\Sigma^{\mathbf{X}}|}},$

and the copula distance

$\displaystyle CD_{\mathcal{H}_{\textup{MMD}}}(\mathbf{X},\mathbf{Y})$
$\displaystyle=\Big{|}\sqrt{\frac{1}{\sqrt{9+16|\Sigma^{\mathbf{X}}|}}+\frac{1}{5}-\frac{2}{\sqrt{21+4|\Sigma^{\mathbf{X}}|}}}$
$\displaystyle-\sqrt{\frac{1}{\sqrt{9+16|\Sigma^{\mathbf{Y}}|}}+\frac{1}{5}-\frac{2}{\sqrt{21+4|\Sigma^{\mathbf{Y}}|}}}\Big{|}.$

Boundedness. Given that $\Sigma^{\mathbf{X}}$ and $\Sigma^{\mathbf{Y}}$ for
Gaussian random vectors are in essence the correlation matrix, we know that
$|\Sigma^{\mathbf{X}}|\leq 1$ and $|\Sigma^{\mathbf{Y}}|\leq 1$. Thus, it is
easy to verify that when $\mathcal{H}$ is Wasserstein-2 distance or Gaussian
MMD distance, the copula distance is bounded. Furthermore, we know that the
divergence measures (including the total variation distance, Hellinger
distance, Jensen-Shannon divergence, etc.) are bounded by definition.
Consequently, the corresponding copula distance is bounded.

Monotonicity. We fix the Gaussian copula parameter $\Sigma^{\mathbf{Y}}$ and
express $CD_{\mathcal{H}}(\mathbf{X,Y})$ as a function of
$\Sigma_{12}^{\mathbf{X}}=\rho$. A simple observation is that, if a function
$f(x)$ is monotonically increasing with respect to $x$, then given $y$ fixed,
$|f(x)-f(y)|$ is monotonically increasing with respect to $|x-y|$. So if
$\mathcal{H}(P^{\mathbf{X}},P^{\mathbf{X}}_{1}P^{\mathbf{X}}_{2})$ is
increasing with respect to $\rho^{2}$, we can conclude that the corresponding
copula distance is monotonically increasing with
$|(\Sigma_{12}^{\mathbf{X}})^{2}-(\Sigma_{12}^{\mathbf{Y}})^{2}|$. We check
them one by one.

  * -

$\mathcal{H}_{\text{KL}}(P_{12},P_{1}P_{2})=-\frac{1}{2}(1-\rho^{2})$
monotonically increases with $\rho^{2}$.

  * -

$\mathcal{H}_{\chi^{2}}(P_{12},P_{1}P_{2})=\frac{1}{1-\rho^{2}}-1$
monotonically increases with $\rho^{2}$.

  * -

$\mathcal{H}_{\textup{W}}^{2}(P^{\mathbf{X}},P^{\mathbf{X}}_{1}P^{\mathbf{X}}_{2})=4-2\sqrt{2+2\sqrt{1-\rho^{2}}}$
monotonically increases with $\rho^{2}$.

  * -

$\mathcal{H}^{2}_{\textup{MMD}}(P^{\mathbf{X}},P^{\mathbf{X}}_{1}P^{\mathbf{X}}_{2})=\frac{1}{\sqrt{25-16\rho^{2}}}+\frac{1}{5}-\frac{2}{\sqrt{25-4\rho^{2}}}$.

From the above calculations, we conclude that
$\mathcal{H}(P^{\mathbf{X}},P^{\mathbf{X}}_{1}P^{\mathbf{X}}_{2})$ is
increasing with respect to $\rho^{2}$ when $\mathcal{H}$ is KL divergence,
$\chi^{2}$ distance and Wasserstein-2 distance. For Gaussian MMD distance, we
consider the function
$f(x)=\frac{1}{\sqrt{25-16x}}-\frac{2}{\sqrt{25-4x}},x\in[0,1]$. The first
derivative $f^{\prime}(x)=8(25-16x)^{-3/2}-4(25-4x)^{-3/2}>0$, suggesting that
$\mathcal{H}_{\textup{MMD}}(P^{\mathbf{X}},P^{\mathbf{X}}_{1}P^{\mathbf{X}}_{2})$
is increasing with respect to $\rho^{2}$. ∎

## Proof of Proposition 3

###### Proof.

Without loss of generality, we assume that the probability density function
$p(x_{1},x_{2})$ of $[X_{1},X_{2}]$ is continuous. Consider the area
$A_{\delta}=\\{(x_{1},x_{2},\widetilde{x}_{1},\widetilde{x}_{2}):|x_{1}-\widetilde{x}_{1}|\leq\delta\textup{
or }|x_{2}-\widetilde{x}_{2}|\leq\delta\\}\subset\mathbb{R}^{4}$. Define
$c_{\delta}:=\iiiint_{A_{\delta}}p(x_{1},x_{2})p(\widetilde{x}_{1},\widetilde{x}_{2})dx_{1}dx_{2}d\widetilde{x}_{1}d\widetilde{x}_{2}$.
It always holds that,

$\displaystyle\rho(X_{1},X_{2};a)=\mathbb{E}[\tanh\big{(}a(X_{1}-\widetilde{X}_{1})(X_{2}-\widetilde{X}_{2})\big{)}]$
$\displaystyle=$
$\displaystyle(\iiiint_{A_{\delta}}+\iiiint_{\mathbb{R}^{4}-A_{\delta}})\tanh\big{(}a(x_{1}-\widetilde{x}_{1})(x_{2}-\widetilde{x}_{2})\big{)}$
$\displaystyle\times
p(x_{1},x_{2})p(\widetilde{x}_{1},\widetilde{x}_{2})d\widetilde{x}_{2}d\widetilde{x}_{1}dx_{2}dx_{1}.$

Outside $A_{\delta}$, we have
$|a(x_{1}-\widetilde{x}_{1})(x_{2}-\widetilde{x}_{2})|\geq a\delta^{2}$, and

$\displaystyle\lim\limits_{a\to\infty}\iiiint_{\mathbb{R}^{4}-A_{\delta}}\tanh\big{(}a(x_{1}-\widetilde{x}_{1})(x_{2}-\widetilde{x}_{2})\big{)}$
$\displaystyle\times
p(x_{1},x_{2})p(\widetilde{x}_{1},\widetilde{x}_{2})dx_{1}dx_{2}d\widetilde{x}_{1}d\widetilde{x}_{2}$
$\displaystyle=$
$\displaystyle\lim\limits_{a\to\infty}\iiiint_{\mathbb{R}^{4}-A_{\delta}}\textup{sign}\big{(}a(x_{1}-\widetilde{x}_{1})(x_{2}-\widetilde{x}_{2})\big{)}$
$\displaystyle\times
p(x_{1},x_{2})p(\widetilde{x}_{1},\widetilde{x}_{2})dx_{1}dx_{2}d\widetilde{x}_{1}d\widetilde{x}_{2}\quad\text{(dominated
convergence theorem)}$ $\displaystyle=$
$\displaystyle\rho_{\tau}(X_{1},X_{2})-\lim\limits_{a\to\infty}\iiiint_{A_{\delta}}\textup{sign}\big{(}a(x_{1}-\widetilde{x}_{1})(x_{2}-\widetilde{x}_{2})\big{)}$
$\displaystyle\times
p(x_{1},x_{2})p(\widetilde{x}_{1},\widetilde{x}_{2})dx_{1}dx_{2}d\widetilde{x}_{1}d\widetilde{x}_{2}.$

Notice that inside $A_{\delta}$, $|\tanh(x)|\leq 1$. Thus, $\forall\delta$,

$\displaystyle|\lim\limits_{a\to\infty}\rho(X_{1},X_{2};a)-\rho_{\tau}(X_{1},X_{2})|$
$\displaystyle\leq$
$\displaystyle\lim\limits_{a\to\infty}\iiiint_{A_{\delta}}\Big{(}|\textup{sign}\big{(}a(x_{1}-\widetilde{x}_{1})(x_{2}-\widetilde{x}_{2})\big{)}|+$
$\displaystyle|\tanh\big{(}a(x_{1}-\widetilde{x}_{1})(x_{2}-\widetilde{x}_{2})\big{)}|\Big{)}p(x_{1},x_{2})p(\widetilde{x}_{1},\widetilde{x}_{2})dx_{1}dx_{2}d\widetilde{x}_{1}d\widetilde{x}_{2}$
$\displaystyle\leq$ $\displaystyle
2\iiiint_{A_{\delta}}p(x_{1},x_{2})p(\widetilde{x}_{1},\widetilde{x}_{2})dx_{1}dx_{2}d\widetilde{x}_{1}d\widetilde{x}_{2}=2c_{\delta}.$

Indeed, it is easy to verify that $c_{0}=0$, $c_{\delta}$ is finite and is
continuous with respect to $\delta$. Thus, we let $\delta$ approach zero and
get the desired result that
$\textup{lim}_{a\to\infty}\rho(X_{1},X_{2};a)=\rho_{\tau}(X_{1},X_{2}).$ ∎

## Implementation Details

All experimental models were trained using the Adam optimizer implemented by
Pytorch with initial learning rate 0.01. All activation functions were taken
to be the ReLU.

###  -A Toy problem

The MLP model is a simple neural network with 2 hidden layers of 8 and 4
neurons respectively.

In the DAN model, the feature extractor is a neural network with 2 hidden
layers of 8 and 4 neurons respectively. The domain discrepancy is evaluated by
the MMD distance and the discriminator is the final output layer.

The CORAL model has the same architecture with the DAN model, except for that
the domain discrepancy is measured by the Frobenius norm.

The CDAN model differs with the CORAL model only in the measurement of the
domain discrepancy.

###  -B Retail credit classification

The MLP model is a simple neural network with 2 hidden layers of 128 and 64
neurons respectively.

In the DAN model, the feature extractor is a neural network with 3 hidden
layers of 64, 32 and 16 neurons respectively. The domain discrepancy is
evaluated by the MMD distance of the 16-dimensional feature representations
between the source and the target domains. The discriminator is the final
output layer.

The AFN model contains 3 hidden layers with 128, 64 and 64 neurons
respectively. The output tensors of the second hidden layer are aligned to a
scale vector, which is pre-determined according to [37].

In MCD model, the neural network consists of 3 hidden layers with 128, 64 and
64 neurons respectively. For output tensors of the second hidden layer from
the target domain, two additional networks are used to construct the
regularization term of [38].

The CORAL model is essentially the same model as the DAN model, except that
the domain discrepancy is evaluated by the Frobenius norm of the covariance
matrices of the 16-dimensional feature representations.

In the CDAN model, the feature extractor is a neural network with 6 hidden
layers of 128, 128, 128, 128, 64 and 8 neurons respectively. The marginal
divergence is calculated by the Gaussian MMD distance of each dimensional
representations among the 8-dimensional features. The copula distance is with
respect to the KL divergence. We run the model for 100 trials, and each trial
costs about 2-3 minutes.

###  -C Intra-day equity price regression

After separating the historical prices of 22 stocks into two domains, we slice
the data into pieces with length 12 and package them into batches of size
1024. For each stock in either domain, we use the MinMaxScaler to normalize
its price. For each model, we use an LSTM layer as the feature extractor, and
conduct the batch normalization for each Linear layer. We train each model for
at most 100 epochs, and the early-stopping threshold is set to be 20 epochs.
We tune the hyperparameters by grid search, and we also fine-tune the network
parameters (including but not limited to number of layers, number of units,
etc.).

The LSTM (RNN resp.) model consists of one LSTM (RNN resp.) layer of hidden
size 64, and two Linear layers of size 64 and 32 respectively.

The DANN model consists of 3 neural networks, namely a feature extractor, a
discriminator and a regressor. The feature extractor contains one LSTM layer
of hidden size 64 and a Linear layer of size also 64. The discriminator is a
binary classifier, which consists of three Linear layers of size 64, 32 and 16
respectively. The regressor consists of two Linear layers of size 64 and 32
respectively.

The CORAL model consists of one LSTM layer of hidden size 64, and 3 Linear
layers of size 64, 32, 16 respectively. After extracting the features by LSTM,
we calculate the regularization term according to [17].

The DAN model consist of one LSTM layer of hidden size 64, and 3 Linear layers
of size 64, 32, 16 respectively. After extracting the features by LSTM, we
calculate the MMD with a two-Gaussian-kernel function.

Our model CDAN consists of one LSTM layer of hidden size 64, and two Linear
layers of size 64 and 32 respectively. After extracting the features by LSTM,
we calculate the divergence between marginal distributions by a two-Gaussian-
kernel MMD, and calculate the copula distance with respect to KL divergence.
We run the model for 100 trials, and each trial costs about 10-20 minutes.

## References

  * [1] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky, “Domain-adversarial training of neural networks,” _The journal of machine learning research_ , vol. 17, no. 1, pp. 2096–2030, 2016.
  * [2] E. Tzeng, J. Hoffman, K. Saenko, and T. Darrell, “Adversarial discriminative domain adaptation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 7167–7176.
  * [3] H. Zhao, S. Zhang, G. Wu, J. M. Moura, J. P. Costeira, and G. J. Gordon, “Adversarial multiple source domain adaptation,” _Advances in neural information processing systems_ , vol. 31, pp. 8559–8570, 2018.
  * [4] H. Liu, M. Long, J. Wang, and M. Jordan, “Transferable adversarial training: A general approach to adapting deep classifiers,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 4013–4022.
  * [5] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola, “A kernel two-sample test,” _The Journal of Machine Learning Research_ , vol. 13, no. 1, pp. 723–773, 2012.
  * [6] M. Long, Y. Cao, J. Wang, and M. Jordan, “Learning transferable features with deep adaptation networks,” in _International conference on machine learning_. PMLR, 2015, pp. 97–105.
  * [7] M. Long, H. Zhu, J. Wang, and M. I. Jordan, “Deep transfer learning with joint adaptation networks,” in _International conference on machine learning_. PMLR, 2017, pp. 2208–2217.
  * [8] N. Courty, R. Flamary, D. Tuia, and A. Rakotomamonjy, “Optimal transport for domain adaptation,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 39, no. 9, pp. 1853–1865, 2016.
  * [9] J. Shen, Y. Qu, W. Zhang, and Y. Yu, “Wasserstein distance guided representation learning for domain adaptation,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2018.
  * [10] M. Sklar, “Fonctions de repartition an dimensions et leurs marges,” _Publ. inst. statist. univ. Paris_ , vol. 8, pp. 229–231, 1959.
  * [11] S. Zhao, X. Yue, S. Zhang, B. Li, H. Zhao, B. Wu, R. Krishna, J. E. Gonzalez, A. L. Sangiovanni-Vincentelli, S. A. Seshia _et al._ , “A review of single-source deep unsupervised visual domain adaptation,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 33, no. 2, pp. 473–493, 2020.
  * [12] L. Duan, D. Xu, and I. W.-H. Tsang, “Domain adaptation from multiple sources: A domain-dependent regularization approach,” _IEEE Transactions on neural networks and learning systems_ , vol. 23, no. 3, pp. 504–518, 2012.
  * [13] H. Zhao, R. T. Des Combes, K. Zhang, and G. Gordon, “On learning invariant representations for domain adaptation,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 7523–7532.
  * [14] S. Chen, L. Han, X. Liu, Z. He, and X. Yang, “Subspace distribution adaptation frameworks for domain adaptation,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 31, no. 12, pp. 5204–5218, 2020.
  * [15] Z. Wang, B. Du, and Y. Guo, “Domain adaptation with neural embedding matching,” _IEEE transactions on neural networks and learning systems_ , vol. 31, no. 7, pp. 2387–2397, 2019.
  * [16] L. Zhang, J. Fu, S. Wang, D. Zhang, Z. Dong, and C. P. Chen, “Guide subspace learning for unsupervised domain adaptation,” _IEEE transactions on neural networks and learning systems_ , vol. 31, no. 9, pp. 3374–3388, 2019.
  * [17] B. Sun, J. Feng, and K. Saenko, “Return of frustratingly easy domain adaptation,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2016.
  * [18] B. Sun and K. Saenko, “Deep coral: Correlation alignment for deep domain adaptation,” in _European conference on computer vision_. Springer, 2016, pp. 443–450.
  * [19] Z. Zhang, M. Wang, Y. Huang, and A. Nehorai, “Aligning infinite-dimensional covariance matrices in reproducing kernel hilbert spaces for domain adaptation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 3437–3445.
  * [20] C. Chen, Z. Fu, Z. Chen, S. Jin, Z. Cheng, X. Jin, and X.-S. Hua, “Homm: Higher-order moment matching for unsupervised domain adaptation,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2020, pp. 3422–3429.
  * [21] Y. Zhao and M. Udell, “Missing value imputation for mixed data via gaussian copula,” in _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 2020, pp. 636–646.
  * [22] G. Marti, S. Andler, F. Nielsen, and P. Donnat, “Exploring and measuring non-linear correlations: Copulas, lightspeed transportation and clustering,” in _NIPS 2016 Time Series Workshop_. PMLR, 2017, pp. 59–69.
  * [23] C. D. Tran, O. O. Rudovic, and V. Pavlovic, “Unsupervised domain adaptation with copula models,” in _2017 IEEE 27th International Workshop on Machine Learning for Signal Processing (MLSP)_. IEEE, 2017, pp. 1–6.
  * [24] D. Lopez-Paz, J. Hernandez-Lobato, and B. Schölkopf, “Semi-supervised domain adaptation with copulas,” in _26th Annual Conference on Neural Information Processing Systems (NIPS 2012)_ , 2012, pp. 674–682.
  * [25] N. A. Letizia and A. M. Tonello, “Segmented generative networks: Data generation in the uniform probability space,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 33, no. 3, pp. 1338–1347, 2020.
  * [26] J. Xuan, J. Lu, G. Zhang, R. Y. Da Xu, and X. Luo, “Doubly nonparametric sparse nonnegative matrix factorization based on dependent indian buffet processes,” _IEEE transactions on neural networks and learning systems_ , vol. 29, no. 5, pp. 1835–1849, 2017.
  * [27] H. Quan, A. Khosravi, D. Yang, and D. Srinivasan, “A survey of computational intelligence techniques for wind power uncertainty quantification in smart grids,” _IEEE transactions on neural networks and learning systems_ , vol. 31, no. 11, pp. 4582–4599, 2019.
  * [28] S. Nowozin, B. Cseke, and R. Tomioka, “f-gan: Training generative neural samplers using variational divergence minimization,” in _Advances in Neural Information Processing Systems_ , D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, Eds., vol. 29. Curran Associates, Inc., 2016. [Online]. Available: https://proceedings.neurips.cc/paper/2016/file/cedebb6e872f539bef8c3f919874e9d7-Paper.pdf
  * [29] Y. Pantazis, D. Paul, M. Fasoulakis, Y. Stylianou, and M. A. Katsoulakis, “Cumulant gan,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2022.
  * [30] C. Donnelly and P. Embrechts, “The devil is in the tails: Actuarial mathematics and the subprime mortgage crisis,” _ASTIN Bulletin_ , vol. 40, no. 1, pp. 1–33, 2010.
  * [31] Y. Fang and L. Madsen, “Modified gaussian pseudo-copula: Applications in insurance and finance,” _Insurance: Mathematics and Economics_ , vol. 53, no. 1, pp. 292–301, 2013.
  * [32] W. Y. Wang and Z. Hua, “A semiparametric gaussian copula regression model for predicting financial risks from earnings calls,” in _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2014, pp. 1155–1165.
  * [33] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” _Advances in neural information processing systems_ , vol. 27, 2014.
  * [34] S. Ben-David, J. Blitzer, K. Crammer, F. Pereira _et al._ , “Analysis of representations for domain adaptation,” _Advances in neural information processing systems_ , vol. 19, p. 137, 2007.
  * [35] D. Ruppert and D. S. Matteson, _Statistics and data analysis for financial engineering_. Springer, 2011, vol. 13.
  * [36] W. Xiao, Z. Ding, and H. Liu, “Implicit semantic response alignment for partial domain adaptation,” _Advances in Neural Information Processing Systems_ , vol. 34, 2021.
  * [37] R. Xu, G. Li, J. Yang, and L. Lin, “Larger norm more transferable: An adaptive feature norm approach for unsupervised domain adaptation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 1426–1435.
  * [38] K. Saito, K. Watanabe, Y. Ushiku, and T. Harada, “Maximum classifier discrepancy for unsupervised domain adaptation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 3723–3732.
  * [39] D. Belov and R. Armstrong, “Distributions of the kullback-leibler divergence with applications,” _The British journal of mathematical and statistical psychology_ , vol. 64, pp. 291–309, 05 2011.
  * [40] P. Cortez, A. Cerdeira, F. Almeida, T. Matos, and J. Reis, “Modeling wine preferences by data mining from physicochemical properties,” _Decision support systems_ , vol. 47, no. 4, pp. 547–553, 2009.
  * [41] I. Sason and S. Verdú, “$f$ -divergence inequalities,” _IEEE Transactions on Information Theory_ , vol. 62, no. 11, pp. 5973–6006, 2016.
  * [42] L. Dalla Valle, “Bayesian copulae distributions, with application to operational risk management,” _Methodology and Computing in Applied Probability_ , vol. 11, no. 1, pp. 95–115, 2009.
  * [43] K. B. Petersen and M. S. Pedersen, “The matrix cookbook,” nov 2012, version 20121115\. [Online]. Available: http://www2.compute.dtu.dk/pubdb/pubs/3274-full.html
  * [44] C. R. Givens and R. M. Shortt, “A class of Wasserstein metrics for probability distributions.” _Michigan Mathematical Journal_ , vol. 31, no. 2, pp. 231 – 240, 1984. [Online]. Available: https://doi.org/10.1307/mmj/1029003026