# A Framework for Supervised Heterogeneous Transfer Learning using Dynamic
Distribution Adaptation and Manifold Regularization

Md Geaur Rahman<EMAIL_ADDRESS>Md Zahidul Islam<EMAIL_ADDRESS>School
of Computing, Mathematics and Engineering, Charles Sturt University, Australia

###### Abstract

Transfer learning aims to learn classifiers for a target domain by
transferring knowledge from a source domain. However, due to two main issues:
feature discrepancy and distribution divergence, transfer learning can be a
very difficult problem in practice. In this paper, we present a framework
called TLF that builds a classifier for the target domain having only few
labeled training records by transferring knowledge from the source domain
having many labeled records. While existing methods often focus on one issue
and leave the other one for the further work, TLF is capable of handling both
issues simultaneously. In TLF, we alleviate feature discrepancy by identifying
shared label distributions that act as the pivots to bridge the domains. We
handle distribution divergence by simultaneously optimizing the structural
risk functional, joint distributions between domains, and the manifold
consistency underlying marginal distributions. Moreover, for the manifold
consistency we exploit its intrinsic properties by identifying $k$ nearest
neighbors of a record, where the value of $k$ is determined automatically in
TLF. Furthermore, since negative transfer is not desired, we consider only the
source records that are belonging to the source pivots during the knowledge
transfer. We evaluate TLF on seven publicly available natural datasets and
compare the performance of TLF against the performance of eleven state-of-the-
art techniques. We also evaluate the effectiveness of TLF in some challenging
situations. Our experimental results, including statistical sign test and
Nemenyi test analyses, indicate a clear superiority of the proposed framework
over the state-of-the-art techniques.

###### keywords:

Transfer learning, heterogeneous domain adaptation, manifold regularization,
random forest, dynamic distribution adaptation

††journal: Artificial Intelligence

##  1 Introduction

Nowadays, Transfer Learning (TL) has made breakthroughs in solving real-world
problems, including activity recognition, sentiment classification, document
analysis and indoor localization [1, 2]. Traditional Machine Learning (ML)
methods cannot handle these problems due to various reasons, including
insufficient data for building a model and distribution mismatch between
training and test datasets [3, 4].

TL algorithms build a model for a domain (often referred to as the target
domain), which has insufficient data, by transferring knowledge from single or
multiple auxiliary domains (referred to as source domains) [1, 2]. However,
commonly used TL methods assume that the source domain dataset (SDD) and
target domain dataset (TDD) have the same set of attributes [3, 5]. In
addition, traditional supervised TL methods assume that the datasets have the
same set of labels [3, 4, 6]. The methods may not perform well if the datasets
have different set of attributes and labels [2]. The domains of such datasets
often known as heterogeneous domains [2].

(a) Source domain: A toy dataset (Product) of a hypothetical multinational
company “ABC”

(b) Target domain: A toy dataset (Product) of a hypothetical Australian
company “XYZ”

Figure 1: Sample source and target domains using toy datasets.

A hypothetical heterogeneous setting is illustrated in Fig. 1, where Fig. 1a
and Fig. 1b show two toy datasets related to products, sales and customer
feedback information of a renowned multinational electronics company “ABC” and
a newly established Australian superstore “XYZ”, respectively. The datasets
are heterogeneous in terms of attributes and labels. The “XYZ” company aims to
evaluate the products based on the customer feedback. Since the dataset of
“XYZ” company contains only few labeled records, it may not be possible to
build a good model by applying an ML algorithm on the dataset. Besides, a good
model for the “XYZ” company can be built by transferring records in a
compatible way from the dataset of “ABC” company which has many records.
However, traditional TL methods, such as ARTL [3] and MEDA [4] are not capable
of transferring knowledge from source to target domain since the datasets have
different sets of features and labels.

Figure 2: Basic concept of bridging between source and target domains.

For dealing with such settings, a number of heterogeneous transfer learning
(HTL) methods have been proposed, and have recently received significant
attention in the literature [2, 7]. An existing HTL method called SHDA [2]
which deals with such setting by transferring records of SDD to TDD. SHDA
projects source records for TDD by identifying similar label distributions
that act as the bridge between SDD and TDD. The concept of bridging the
domains is illustrated in Fig 2. Using the SDD (Fig. 2a), SHDA builds a
decision forest (in short, forest) which consists of a set of decision trees
(in short, trees) as shown in Fig. 2b. A tree has a set of rules, where a rule
is used to create a leaf (see Fig. 3) containing records that satisfy the
conditions of the rule. Using the records of a leaf, SHDA calculates label
distributions of the leaf. The hypothetical label distributions of all leaves
for the source domain are shown in Fig. 2c. Similarly, for the TDD (Fig. 2g),
SHDA builds a forest (Fig. 2f) and then calculates the label distributions of
all leaves (Fig. 2e). After that SHDA finds similar label distributions (see
Fig. 2d) between the domains. The similar label distributions act as the
pivots to bridge the domains and help achieving accurate transfer of data.

For both source and target forests, SHDA then calculates attribute
contributions of the leaves that are associated with the pivots (discussed in
Section 3). The contribution of an attribute $A_{j}$ is calculated as
$(\frac{1}{2})^{l}$, where $l$ is the level of the node where $A_{j}$ is
contributed. As an example, for Leaf 3 of the sample tree (see Fig. 3) the
contribution of attributes “Item Group” and “Sales” are 0.625 (=0.500+0.125)
and 0.250 respectively. The attribute contributions are then used in Eq. (3)
to project the records of SDD into TDD. Combining the projected records with
the TDD, SHDA builds the final model as the target model which is then used to
classify the test data (discussed in Section 5.2.1).

Figure 3: A sample decision tree.

In HTL, a good method should address two challenging issues [7]: distribution
divergence and feature discrepancy. However, existing methods often focus on
one issue and leave the other one for the further work. For example, ARTL [3]
and MEDA [4] focus on minimizing distribution divergence while SHDA [2]
attends to learning shared label distributions.

ARTL [3] and MEDA [4] have minimized the conditional distribution divergence
by using maximum mean discrepancy (MMD) [8] and marginal distribution
divergence by exploiting intrinsic properties of manifold consistency [9].
However, to identify the nearest neighbors of a record during manifold
regularization, the methods use a user-defined parameter $k$ which may fail to
exploit the properties accurately (explained later in Section 4.1 and Fig. 5)
resulting in a low classification accuracy. Moreover, the methods are unable
to handle the feature discrepancy.

The feature discrepancy is handled in SHDA [2] by identifying pivots
(discussed earlier) between SDD and TDD, the distribution divergence, however,
is not addressed. Moreover, it provides higher weights to parent nodes than
their children while calculating attribute contributions. For a leaf, the
importance of a test attribute in classifying the records belonging to the
leaf does not necessarily depend on the position/level of the attribute in the
rule. It also transfers all records from SDD to TDD and may result in negative
transfer [7].

The existing methods therefore have room for further improvement. In this
paper, we present a novel transfer learning framework called TLF that builds a
classifier for TDD by transferring knowledge from SDD. TLF is capable of
handling the two issues. The main contributions of TLF are summarized as
follows.

  * 1.

We eliminate the feature discrepancy by identifying pivots between SDD and TDD
by following the SHDA’s approach. However, we calculate attribute
contributions using the leaf’s centroids (attribute wise mean plus standard
deviation) instead of considering the node’s level in the tree path from the
root to the leaf used in SHDA (discussed in Section 4.1).

  * 2.

We minimize distribution divergence by using MMD and manifold simultaneously
where the intrinsic property of the manifold consistency is exploited
automatically (discussed in Section 4.1).

  * 3.

We minimize negative transfer, during the projection, by considering only the
source records that are belonging to the source pivots (discussed in Section
4.1).

We evaluate TLF on seven publicly available real datasets by comparing its
performance with the performance of eleven high quality existing techniques
including two baseline algorithms. Our experimental results and statistical
analyses indicate that TLF performs significantly better than the existing
methods.

The rest of the paper is organized as follows. Section 2 presents problem
formulation and assumptions, and Section 3 presents a background study on
transfer learning. Our proposed transfer learning framework is presented in
Section 4. Section 5 presents experimental results, Section 6 presents the
effectiveness of TLF in some challenging situations, and Section 7 gives
concluding remarks.

##  2 Problem Formulation

In this section, we define the problem setting, assumptions and goal for
supervised heterogeneous transfer learning. The notations that we use
frequently in this paper, are presented in Table 1.

Table 1: Notations and their meanings Notation | Meaning | Notation | Meaning
---|---|---|---
$D_{s}$ | Source domain | $D_{t}$ | Target domain
$n_{s}$ | Size of $D_{s}$ | $n_{t}$ | Size of $D_{t}$
$d_{s}$ | Attributes of $D_{s}$ | $d_{t}$ | Attributes of $D_{t}$
$X_{s}$ | Records of $D_{s}$ | $X_{t}$ | Records of $D_{t}$
$Y_{s}$ | Labels of $D_{s}$ | $Y_{t}$ | Labels of $D_{t}$
$C$ |  # shared labels | K | Kernel matrix
$\sigma$ | Ridge regularization | $\mu$ | MMD adaptive factor
$\lambda$ | MMD regularization | M | MMD matrix
$\gamma$ | Manifold regularization | L | Graph Laplacian matrix

######  Definition 1.

Domain [7, 3]: A domain $D$ is formed by a feature space $X$ having $d$ number
of attributes and a marginal probability distribution $P(x)$, i.e.,
$D={X,P(x)}$, where $x\in X$.

######  Definition 2.

Task [7, 3]: For a domain $D$, a task $T$ is formed by a set of labels $Y$ and
a classification function $f(x)$, i.e., $T=\\{Y,f(x)\\}$, where $y\in Y$, and
$f(x)=Q(y|x)$ can be considered as the conditional probability distribution.

Let $D_{s}$ and $D_{t}$ be the source domain and target domain, respectively.
If the domains have different feature spaces or marginal distributions, i.e.,
$X_{s}\neq X_{t}\vee P_{s}(x_{s})\neq P_{t}(x_{t})$ then the domains can be
considered as different i.e., $D_{s}\neq D_{t}$. Also, let $T_{s}$ and $T_{t}$
be the source task and target task, respectively. If the tasks have different
labels or conditional distributions, i.e., $Y_{s}\neq Y_{t}\vee
Q_{s}(y_{s}|x_{s})\neq Q_{t}(y_{t}|x_{t})$ then the tasks can be considered as
different i.e., $T_{s}\neq T_{t}$.

######  Definition 3.

Supervised Heterogeneous Transfer Learning (SHTL): Given a source domain
having labels i.e.
$D_{s}=\left\\{(x_{s_{1}},y_{s_{1}}),...,(x_{s_{n_{s}}},y_{s_{n_{s}}})\right\\}$
and a target domain having labels i.e.
$D_{t}=\left\\{(x_{t_{1}},y_{t_{1}}),...,(x_{t_{n_{t}}},y_{t_{n_{t}}})\right\\}$,
the goal of SHTL is to build classifier $f$ on $D_{t}$ by learning a mapping
$P:D_{s}\mapsto D_{t}$, under the assumptions $n_{s}\gg n_{t}$, $X_{s}\cap
X_{t}=\emptyset$, $d_{s}\cap d_{t}=d$, $Y_{s}\cap Y_{t}=C$, $P_{s}(x_{s})\neq
P_{t}(x_{t})$, and $Q_{s}(y_{s}|x_{s})\neq Q_{t}(y_{t}|x_{t})$.

##  3 Related Work

A number of methods have been proposed for heterogeneous transfer learning
(HTL) [2, 10, 11]. HTL methods can broadly be classified into 3 categories:
Unsupervised HTL, Semi-supervised HTL and Supervised HTL as shown in Table 2.
Our proposed framework, TLF falls under Supervised HTL category.

Table 2: Heterogeneous transfer learning (HTL) categories. Category | $Y_{s}$ | $Y_{t}$ | Examples
---|---|---|---
Unsupervised HTL | No | No | STC [12]
Semi-supervised HTL | Yes | No | ARTL [3], MEDA [4], DASVM [5]
Supervised HTL | Yes | Yes | SHDA [2], TLF [ours]

_Unsupervised HTL_ methods deal with the setting where both source and target
domains have insufficient or no label records, and are generally applied to
tasks that are special and unique [1]. An example of unsupervised HTL method
is called STC [12] that performs clustering on target domain, $D_{t}$ having
small unlabeled records, by utilizing the knowledge obtained from target
domain, $D_{s}$ having sufficient unlabeled records. STC first transforms both
$D_{s}$ and $D_{t}$ in to a common feature space and then uses $D_{s}$ to
cluster $D_{t}$. However, the methods are found to be difficult to apply to
real-world tasks [1].

_Semi-supervised HTL_ methods consider the scenario where $D_{s}$ has labeled
records but $D_{t}$ has no label data [13, 4]. The methods initially build a
model from $D_{s}$ and then apply the model on $D_{t}$ to obtain pseudo-labels
for the records of $D_{t}$. After that the methods iteratively minimize
distribution differences between $D_{s}$ and $D_{t}$, and update the pseudo-
labels until convergence criteria are met.

An existing method called DASVM [5] initially builds a classifier by applying
the support vector machine (SVM) [14] on $D_{s}$. At the first iteration, the
records of $D_{s}$ are considered as the training set. The classifier is then
used to estimate the labels of the records of $D_{t}$. The training set is
then iteratively updated by adding $m$ records, that are classified with high
confidence, from $D_{t}$ and removing $m$ records, that have less influence on
the classifier, of $D_{s}$. In each iteration, the classifier is also
retrained based on the current training set. Since DASVM ultimately builds the
final classifier by considering only the records of $D_{t}$, a good classifier
may not be built if $D_{t}$ contains insufficient records. Moreover, only the
structural risk functional (SRF) is minimized in DASVM.

In addition to SRF, ARTL [3] and MEDA [4] minimize the difference between
joint distributions of $D_{s}$ and $D_{t}$ by using maximum mean discrepancy
(MMD) [8], and maximize manifold consistency [9] by considering the domains
marginal distributions. However, a user requires to provide a value for $k$,
to exploit the intrinsic property of the manifold consistency, which can be a
difficult task of the user in real-world applications. Moreover, in both
$D_{s}$ and $D_{t}$, the methods require same set of features and expect same
set of labels.

_Supervised HTL_ methods deal with the setting where both $D_{s}$ and $D_{t}$
have labeled data and $n_{s}\gg n_{t}$. Since our proposed framework, TLF
belongs to this category, we discuss a recent existing method called SHDA [2]
in detail as follows.

SHDA first builds a forest (say 100 trees, a sample tree is shown in Fig. 3)
by applying the Random Forest (RF) [15] on $D_{s}$ and obtains a set of rules
$R_{s}$ that produces $L_{s}$ leaves. For every rule from the root to a leaf
node, it calculates the contribution of each attribute of $D_{s}$. The
contribution of an attribute $d_{s_{j}}\in d_{s}$ is calculated as
$W_{s_{j}}=\sum^{v}_{i=1}{(\frac{1}{2})}^{v(i)}$ where $v$ is a list contains
the level numbers at which the attribute $d_{s_{j}}$ is considered as the
candidate split. Thus for all rules, it produces a contribution matrix
$W_{s}^{[L_{s}\times d_{s}]}$. After that, for each leaf SHDA calculates the
distributions of labels. If the size of the class attribute of $D_{s}$ is
$C_{s}$ then SHDA produces a label distribution matrix, $Q_{s}^{[L_{s}\times
C_{s}]}$.

SHDA then refines $Q_{s}$ by removing the duplicate rows of the label
distribution. For every duplicate row, the corresponding rows of the
attributes contribution matrix $W_{s}$ are averaged. Similar to $D_{s}$, SHDA
produces the attributes contribution matrix $W_{t}^{[L_{t}\times d_{t}]}$ and
label distribution matrix, $Q_{t}^{[L_{t}\times C_{t}]}$ by building a forest
from $D_{t}$, where $R_{t}$ is number of rules and $C_{t}$ is the size of the
class attribute of $D_{t}$. The label distribution $L_{t}$ is also refined by
removing the duplicate rows, and then the corresponding rows of the attributes
contribution matrix $W_{t}$ are also averaged.

Using $Q_{s}$ and $Q_{t}$, SHDA identifies the identical or similar label
distributions across $D_{s}$ and $D_{t}$. Let $N_{p}$ be the number of
identical or similar label distributions across the source and target domains.
These $N_{p}$ rows of $Q_{s}$ and $Q_{t}$ are called the pivots that act as
the bridge between $D_{s}$ and $D_{t}$. The concept of bridging $D_{s}$ and
$D_{t}$ is illustrated in Fig. 2.

Since the attribute contribution matrices $W_{s}$ and $W_{t}$ are also
refined, both of them have $N_{p}$ rows. The source projection matrix $P_{s}$
is obtained as [2]:

$\displaystyle\begin{array}[]{l
l}\displaystyle\min_{P_{s}}{\frac{1}{N_{p}}}{\sum_{i=1}^{N_{p}}{\left\|W_{t}-W_{s}\cdot
P_{s}\right\|_{2}^{2}}+\sum_{i=1}^{d_{t}}{\lambda\left\|P_{s_{i}}\right\|_{1}}}&\\\
\text{such that}\quad P_{S_{i}}\geq 0&\end{array}$ (3)

Using $P_{s}$, SHDA calculates the projected source data ($D_{s}\times P_{s}$)
which are then appended to $D_{t}$ to build the final classifier. It is
reported that SHDA outperforms existing state-of-the-art methods [2]. However,
since SHDA projects the whole $D_{s}$ into $D_{t}$, the final classifier may
suffer from negative transfer resulting in a low classification accuracy if
the number of pivots between $D_{s}$ and $D_{t}$ is very low.

The existing methods therefore have room for further improvement. TLF, is
capable of transferring knowledge from $D_{s}$ and $D_{t}$ by using dynamic
distribution adaptation (DDA) and manifold regularization (MR) even if
$d_{s}\neq d_{t}$, $Y_{s}\neq Y_{t}$ and $n_{s}\gg n_{t}$. A comparison on the
properties of TLF and some existing methods is presented in Table 3.

Table 3: Comparison of TLF and some existing methods, where DDA and MR denote dynamic distribution adaptation and manifold regularization, respectively. Method [Ref.] | DDA? | MR? |  $d_{s}\neq d_{t}$? |  $Y_{s}\neq Y_{t}$? |  $n_{s}\gg n_{t}$?
---|---|---|---|---|---
DASVM [5] | No | No | No | No | No
ARTL [3] | No | Yes | No | No | No
MEDA [4] | Yes | Yes | No | No | No
BDA [16] | Yes | Yes | No | No | No
SHDA [2] | No | No | Yes | Yes | Yes
TLF [ours] | Yes | Yes | Yes | Yes | Yes

##  4 Proposed Framework TLF

The motivation of TLF is presented in the introduction section. We now present
our proposed a novel framework called TLF for supervised heterogeneous
transfer learning (HTL). We then analyze the target domain error bound and
computational complexity of TLF.

###  4.1 General Framework

TLF consists of five main steps as follows (also shown in Algorithm 1).

Figure 4: Overall flow diagram of our proposed transfer learning framework,
TLF.

Step-1: Build forests and find label distributions and centroids from both
source $D_{s}$ and target $D_{t}$ domains.

Step-2: Remove duplicate label distributions and find the aggregated values of
the corresponding centroids.

Step-3: Find pivots between source and target domains.

Step-4: Calculate projection matrix between source and target domains.

Step-5: Find and transfer projected source data to the target domain and build
the final classifier.

We also present an overall flow diagram of TLF as shown in Fig. 4. We now
discuss the steps as follows.

Step-1: Build forests and find label distributions and centroids from both
source $D_{s}$ and target $D_{t}$ domains.

In TLF, we take two datasets, namely source domain dataset $D_{s}$ and target
dataset $D_{t}$ as input. $D_{s}$ has $n_{s}$ rows, $d_{s}$ attributes and
$C_{s}$ labels, and $D_{t}$ has $n_{t}$ rows, $d_{t}$ attributes and $C_{t}$
labels. In this step, we build two forests $F_{s}$ and $F_{t}$ (where each
forest has $\tau$ trees) by applying a decision forest algorithm such as
Random Forest (RF) [15] on $D_{s}$ and $D_{t}$, respectively as shown in the
Step 1 of Algorithm 1 and Fig. 4. Using $F_{s}$, we find the leaves $L_{s}$
(see Fig. 3) where a leaf $l^{i}_{s}\in L_{s}$ is represented by the $i$th
rule of $F_{s}$. For the $i$th leaf $l^{i}_{s}$, we find records (from
$D_{s}$) that belong to $l^{i}_{s}$. We then calculate the label distribution
$v^{i}_{s}$, the centroid $w^{i}_{s}$ and centroid’s label $r^{i}_{s}\in
C_{s}$. The $v^{ij}_{s}$ is the probability of the $j$th label in $l^{i}_{s}$.
If the $j$th attribute is a numerical attribute then the value of $w^{ij}_{s}$
is calculated as follows.

$\displaystyle w^{ij}_{s}=u^{ij}_{s}+log(std)$ (4)

where $u^{ij}_{s}$ and $std$ are the $j$th attribute mean and standard
deviation, respectively, of the records that belong to leaf $l^{i}_{s}$. For
categorical attributes, we use the mode values. Thus, for all leaves of
$F_{s}$ we obtain a label distribution matrix $V^{[L_{s}\times C_{s}]}_{s}$, a
centroid matrix $W^{[L_{s}\times d_{s}]}_{s}$ and a matrix for centroids label
$R^{[L_{s}\times 1]}_{s}$. Similarly, using $D_{t}$ and $F_{t}$, we find a
label distribution matrix $V^{[L_{t}\times C_{t}]}_{t}$, a centroid matrix
$W^{[L_{t}\times d_{t}]}_{t}$ and a matrix for centroids label
$R^{[L_{t}\times 1]}_{t}$. The centroids are considered as the attribute
contributions which are calculated differently than SHDA [2].

Step-2: Remove duplicate label distributions and find the aggregated values of
the corresponding centroids.

Since label distribution of a leaf is calculated based on the number of
records, of each label, belonging to the leaf, the existence of duplicate
label distributions at the leaves is common. For example, Leaf 2 and Leaf 4 of
the sample tree (in Fig. 3) have the same label distribution. In this step, we
remove duplicates from $V_{s}$ and $V_{t}$ as shown in Step 2 of Algorithm 1.
For every duplicate entry in $V_{s}$ and $V_{t}$, the corresponding centroids
in $W_{s}$ and $W_{t}$ are aggregated. For numerical and categorical
attributes we take the average and mode values, respectively. This procedure
helps to keep attribute contributions for the aggregated vectors. Let
$L^{\prime}_{s}$ and $L^{\prime}_{t}$ be the number of entries remained after
the aggregation. Now, the matrices are:
$V^{[{L^{\prime}_{s}}\times{C_{s}}]}_{s}$,
$V^{[{L^{\prime}_{t}}\times{C_{t}}]}_{t}$,
$W^{[{L^{\prime}_{s}}\times{d_{s}}]}_{s}$,
$W^{[{L^{\prime}_{t}}\times{d_{t}}]}_{t}$, $R^{[{L^{\prime}_{s}}\times
1]}_{s}$ and $R^{[{L^{\prime}_{t}}\times 1]}_{t}$.

Step-3: Find pivots between source and target domains.

In most real-world HTL scenarios, the feature discrepancy is a challenging
issue since the correspondences between $D_{s}$ and $D_{t}$ are not available.
To minimize the discrepancy, some common information between $D_{s}$ and
$D_{t}$ can be identified to bridge the domains. Since we assume that both
$D_{s}$ and $D_{t}$ share some label space, we exploit the shared label space
to identify correspondences between domains. Thus, in Step 3, we determine the
leaves with similar label distributions across the domains as shown in Step 3
of Algorithm 1 and Fig. 4.

Let $S^{L^{\prime}_{s}\times L^{\prime}_{t}}$ be the similarity matrix of
$V^{[{L^{\prime}_{s}}\times{d_{s}}]}_{s}$ and
$V^{[{L^{\prime}_{t}}\times{d_{t}}]}_{t}$, where each value varies between 0
and 1. A lower value indicates a higher similarity. The similarity of two
label distributions is computed using the Jensen Shannon divergence (JSD)
method [17]. A source label distribution $V^{i}_{s}$ is said to be similar to
a target label distribution $V^{k}_{t}$ if the divergence between them is less
than 10%.

Since $D_{t}$ generally contains small labeled data, it is common to have a
small number of common labels. So, we my have only a small number of similar
labeled distributions across the domains. Each shared labeled distribution
(i.e. $(V^{i}_{s},V^{k}_{t})$) is considered as a pivot that acts as the
bridge between $D_{s}$ and $D_{t}$. Let $N_{p}$ be the number of pivots
between $D_{s}$ and $D_{t}$.

For each pivotal label distribution, we also have the attribute contribution
which is calculated in Step 1 and refined in Step 2. Based on the similar
source and target label distributions, we then determine the attribute
contribution matrices $W^{[N_{p}\times d_{s}]}_{s}$ and $W^{[N_{p}\times
d_{t}]}_{t}$, and the associated label matrices $R^{[N_{p}\times 1]}_{s}$ and
$R^{[N_{p}\times 1]}_{t}$.

Input : Source dataset $D_{s}$, target dataset $D_{t}$.

Output : Classifier $F$.

Step 1:

Build two decision forests $F_{s}$ and $F_{t}$ by applying an existing
decision forest algorithm such as RF [15] on $D_{s}$ and $D_{t}$,
respectively, where $|F_{s}|$=$|F_{t}|$=$\tau$ trees;

Find source leaves, $L_{s}\leftarrow findLeaves(F_{s})$;

Find source label distributions, $V_{s}\leftarrow
findClassDistribution(D_{s},L_{s})$;

Find source centroids, $W_{s}\leftarrow findCentroids(D_{s},L_{s})$;

Find source centroid’s labels, $R_{s}\leftarrow
findCentroidLabel(D_{s},L_{s})$;

Find target leaves, $L_{t}\leftarrow findLeaves(F_{t})$;

Find target label distributions, $V_{t}\leftarrow
findClassDistribution(D_{t},L_{t})$;

Find target centroids, $W_{t}\leftarrow findCentroids(D_{t},L_{t})$;

Find target centroid’s labels, $R_{t}\leftarrow
findCentroidLabel(D_{t},L_{t})$;

end

Step 2:

Remove duplicates from $V_{s}$ and $V_{t}$. For every duplicate entry in
$V_{s}$ and $V_{t}$, the corresponding centroids in $W_{s}$ and $W_{t}$, and
centroid labels $R_{s}$ and $R_{t}$ are aggregated.

end

Step 3:

Calculate similarity matrix, $S\leftarrow
calculateSimilarityByJSD(V_{s},V_{t})$ between source and target label
distributions;

Find source pivots, $V_{s}\leftarrow findPivots(V_{s},S)$ and associated
centroids $W_{s}$ and labels $R_{s}$;

Find target pivots, $V_{t}\leftarrow findPivots(V_{t},S)$ and associated
centroids $W_{t}$ and labels $R_{t}$;

end

Step 4:

Find projection matrix, $P_{s}\leftarrow
findProjectionMatrix(W_{s},R_{S},W_{t},R_{t})$ by considering ridge, maximum
mean discrepancy (MMD) and manifold regularizations as shown in Eq. (38)

end

Step 5:

Find transferrable source data, $D^{\prime}_{s}\leftarrow
findProjectedSourceData(V_{s},D_{s},P_{s})$;

Projected source data, $D^{\prime\prime}_{s}\leftarrow D^{\prime}_{s}\times
P_{s}$;

New target dataset, $D^{\prime}_{t}\leftarrow
merge(D^{\prime\prime}_{s},D_{t})$;

Build the final decision forest, $F$ by applying the RF on $D^{\prime}_{t}$;

Return $F$.

end

Algorithm 1 LearnTLF()

Step-4: Calculate projection matrix between source and target domains.

In this step, we calculate a projection matrix $P_{s}$ for $D_{s}$ so that the
data can be projected to $D_{t}$ as shown in Step 4 of Algorithm 1 and Fig. 4.
We calculate $P_{s}$ by applying structural risk minimization (SRM) principle
and regularization theory [14] on the attribute contribution matrices
$W^{[N_{p}\times d_{s}]}_{s}$ and $W^{[N_{p}\times d_{t}]}_{t}$. Let $J_{s}$
and $J_{t}$ be the source and target joint probability distributions,
respectively and $P_{s}$ and $P_{t}$ be the source and target marginal
distributions, respectively. To calculate $P_{s}$ we particularly, aim to
optimize the following three objective functions:

  1. 1.

To minimize structural risk function on $W_{s}$ and $W_{t}$;

  2. 2.

To minimize distribution discrepancy between $J_{s}$ and $J_{t}$; and

  3. 3.

To maximize manifold consistency underlying $P_{s}$ and $P_{t}$.

Let $D_{c}$ be the combine dataset of $(W_{s},R_{s})$ and $(W_{t},R_{t})$ i.e.
$D_{c}=\left\\{(w_{s_{1}},r_{s_{1}}),...,(w_{s_{N_{p}}},r_{s_{N_{p}}}),(w_{t_{1}},r_{t_{1}}),\allowbreak...,(w_{t_{N_{p}}},r_{t_{N_{p}}})\right\\}$.
By ignoring source and destination subscripts, we can simply write $D_{c}$ as
$\left\\{(w_{1},r_{1}),...,(w_{z},r_{z})\right\\}$, where $z=2N_{p}$. Also let
$f=\beta^{T}\phi(w)$ be the classifier having parameters $w$ and feature
mapping function $\phi:w\mapsto\mathcal{H}$ that projects the actual feature
vector to a Hilbert space $\mathcal{H}$. Using the SRM principle and
regularization theory, the framework of TLF can be formulated as:

$f=\operatorname*{argmin}_{f\in\mathcal{H_{K}}}\sum^{z}_{i=1}{l(f(w_{i}),r_{i})}+\sigma\left\|f\right\|^{2}_{K}\\\
+\lambda D_{f,K}(J_{s},J_{t})+\gamma M_{f,K}(P_{s},P_{t})$ (5)

where $K$ is the kernel function represented by $\phi$ such that
$\left\langle\phi(w_{i}),\phi(w_{i})\right\rangle=K(w_{i},w_{j})$, and
$l(.,.)$, $\left\|f\right\|^{2}_{K}$ and $D_{f,K}(.,.)$ represent loss
function, ridge regularization and dynamic distribution adaptation (DDA),
respectively. Moreover, we introduce $M_{f,K}(P_{s},P_{t})$ as a Laplacian
regularization to further exploit intrinsic properties of nearest points in
Manifold $\mathcal{G}$ [9]. The symbols $\sigma$, $\lambda$, and $\gamma$ are
positive regularization parameters for ridge, DDA, and Manifold, respectively.

In Eq. (5), the loss function $l(.)$ measures the fitness of $f$ for
classifying labels on $D_{c}$. We use SVM [14] to solve $l(.)$. Moreover,
since the kernel mapping $\phi:w\mapsto\mathcal{H}$ may have infinite
dimensions, we solve Eq. (5) by reformulating it with the following
Representer Theorem.

######  Theorem 1.

(Representer Theorem) [3, 14] If $K$ is a kernel function induced by $\phi$
then the optimization problem in Eq. (5) for $D_{c}$, can be represented by

$\displaystyle
f(w)=\sum^{z}_{i=1}{\alpha_{i}{K(w_{i},w)}}\text{~{}and~{}}\beta=\sum^{z}_{i=1}{\alpha_{i}{\phi(w_{i})}}$
(6)

where $\alpha_{i}$ is a coefficient.

Using the SRM principle and equations (6), we can reformulate loss function,
ridge regularization of Eq. (5) as follows.

$f=\operatorname*{argmin}_{f\in\mathcal{H}}\left\|R-\alpha^{T}K\right\|^{2}_{F}+\sigma(tr(\alpha^{T}{K}\alpha))$
(7)

We now discuss the process of minimizing distribution divergence through
dynamic distribution adaptation and manifold regularization as follows.

Dynamic Distribution Adaptation (DDA): Since the joint distributions $J_{s}$
and $J_{t}$ are different, the loss function $l(.)$ with only the ridge
regularization in Eq. (5) may not generalize well to $W_{t}$ [3]. Thus, we aim
to minimize joint distribution divergence 1) between the marginal
distributions $P_{s}$ and $P_{t}$, and 2) between the conditional
distributions $Q_{s}$ and $Q_{t}$, simultaneously.

Following ARTL [3] and MEDA [4], we adopt MMD [8] to handle the distribution
divergence. Based on the samples drawn from two domains, MMD is capable of
determining whether two distributions are different or not. The MMD between
marginal distributions $P_{s}$ and $P_{t}$ is computed as

$\displaystyle
MMD^{2}_{\mathcal{H}}(W_{s},W_{t})=\left\|\frac{1}{N_{p}}\sum^{N_{p}}_{i=1}{\phi(w_{i})}-\frac{1}{N_{p}}\sum^{z}_{j=N_{p}}{\phi(w_{j})}\right\|^{2}_{\mathcal{H}}$
(8)

where $\phi:w\mapsto\mathcal{H}$ is a feature mapping. The MMD can be
regularized for the classifier $f$ as

$\displaystyle
D_{f,K}(P_{s},P_{t})=\left\|\frac{1}{N_{p}}\sum^{N_{p}}_{i=1}{f(w_{i})}-\frac{1}{N_{p}}\sum^{z}_{j=N_{p}}{f(w_{j})}\right\|^{2}_{\mathcal{H}}$
(9)

where $f=\beta^{T}\phi(w)$.

Since we have labeled data in both domains, we now minimize the distance
$D_{f,K}(P_{s},P_{t})$ between conditional distributions $Q_{s}$ and $Q_{t}$.
Based on the intra-class centroids of two distributions $Q_{s}(w_{s}|r_{s})$
and $Q_{t}(w_{t}|r_{t})$, the MMD of a class $c\in C$ is computed as

$\displaystyle
D^{(c)}_{f,K}(Q_{s},Q_{t})=\left\|\frac{1}{|W^{(c)}_{s}|}\sum_{w_{i}\in
W^{(c)}_{s}}{f(w_{i})}-\frac{1}{|W^{(c)}_{t}|}\sum_{w_{j}\in
W^{(c)}_{t}}{f(w_{j})}\right\|^{2}_{\mathcal{H}}$ (10)

where $W^{(c)}_{s}=\left\\{w_{i}:w_{i}\in W_{s}\wedge r(w_{i})=c\right\\}$ is
the set records associated to class $c$ in $D_{s}$ and
$W^{(c)}_{t}=\left\\{w_{j}:w_{j}\in W_{t}\wedge r(w_{j})=c\right\\}$ is the
set records associated to class $c$ in $D_{t}$. If $|W^{(c)}_{k}|=0$ then we
ignore the part of $|W_{k}|$ in Eq. (10).

In real-world scenarios, the importance of marginal ($P$) and conditional
($Q$) are different [4]. Thus, the divergence for joint joint distributions
$J_{s}$ and $J_{t}$ is adapted dynamically by integrating Eq. (9) and Eq. (10)
as follows.

$\displaystyle
D_{f,K}(J_{s},J_{t})=(1-\mu)D_{f,K}(P_{s},P_{t})+\mu\sum^{C}_{c=1}{D^{(c)}_{f,K}(Q_{s},Q_{t})}$
(11)

where $\mu\in[0,1]$ is the adaptive factor which is calculated dynamically by
following MEDA [4]. $\mu\rightarrow 0$ indicates the distribution distance
between the domains is large, and thus, the marginal distribution adaptation
is important. Besides, $\mu\rightarrow 1$ indicates the class distribution is
dominant than feature distribution, and thus, the conditional distribution
adaptation is important.

Using the Representer Theorem (see Theorem 1 and Eq. (6)), Eq. (11) becomes

$\displaystyle D_{f,K}(J_{s},J_{t})=tr(\alpha^{T}{K}{M}{K}\alpha)$ (12)

where $M^{[z\times z]}=(1-\mu)M_{0}+\mu\sum^{C}_{c=1}{M_{c}}$ is the MMD
matrix. The components of the MMD matrix are calculated as follows.

$\displaystyle(M_{0})_{ij}=\left\\{\begin{array}[]{l
l}\frac{1}{N^{2}_{p}}&\text{$w_{i},w_{j}\in{W_{s}}\vee
w_{i},w_{j}\in{W_{t}}$}\\\ \frac{-1}{N^{2}_{p}}&\text{otherwise}\\\
\end{array}\right.;\forall{i,j}$ (15)

$\displaystyle(M_{c})_{ij}=\left\\{\begin{array}[]{l
l}\frac{1}{n^{2}_{c}}&\text{$w_{i},w_{j}\in{W^{(c)}_{s}}$}\\\
\frac{1}{m^{2}_{c}}&\text{$w_{i},w_{j}\in{W^{(c)}_{t}}$}\\\
\frac{-1}{{n^{2}_{c}}{m^{2}_{c}}}&\text{$w_{i},w_{j}\in{W^{(c)}_{s}}\vee
w_{i},w_{j}\in{W^{(c)}_{t}}$}\\\ 0&\text{otherwise}\\\
\end{array}\right.;\forall{i,j}$ (20)

where $n_{c}=|W^{(c)}_{s}|$ and $m_{c}=|W^{(c)}_{t}|$.

Manifold Regularization: In addition to DDA, we further exploit intrinsic
properties of the marginal distributions $P_{s}$ and $P_{t}$ by manifold
regularization [9] due to its effectiveness in mitigating influence of feature
distortion [3, 4]. The features of a domain can be transformed into manifold
space which is suitable for exploiting more detailed property and structure of
the domain. Considering the manifold assumption [9], the manifold
regularization is computed as

$\displaystyle\begin{array}[]{l l}\displaystyle
M_{f,K}(P_{s},P_{t})&=\sum^{z}_{i,j=1}{{(f(w_{i})-f(w_{j}))}^{2}{B_{ij}}}\\\
&=\sum^{z}_{i,j=1}{{f(w_{i})}{L_{ij}}{f(w_{j})}}\end{array}$ (23)

where $L^{[z\times z]}$ is the normalized Laplacian regularization matrix and
$B$ is the graph affinity matrix which is calculated as

$\displaystyle B_{ij}=\left\\{\begin{array}[]{l l}\cos(w_{i},w_{j})&\text{if
$w_{i}\in{k(w_{j})}\vee w_{j}\in{k(w_{i})}$}\\\ 0&\text{otherwise}\\\
\end{array}\right.$ (26)

where $\cos(.,.)$ is a similarity function which is used to calculate distance
between two records $w_{i}$ and $w_{j}$, and $k(w_{j})$ represents the $k$
nearest-neighbors (kNN) of the record $w_{j}$. While existing methods
including ARTL [3] and MEDA [4] use a user-defined value for $k$, we propose a
novel approach to determine $k$ automatically so that the intrinsic properties
of a record can be exploited more accurately resulting in a high transfer
performance.

In TLF, we determine $k$ NN of a record $w_{i}$ having label $r_{i}$ as
follows. We first calculate distance between $w_{i}$ and other records one by
one and then sort the records based on the distance in ascending order. We set
the minimum of value of $k$ to 4. That is, the first four records are added to
kNN set. Following the order, we continue to add a record $w_{u}$ to kNN set
iteratively until $r_{i}\neq{r_{u}};\forall u$. Fig. 5 illustrates the
difference between user-defined and automatic $k$, the automatic one finds the
nearest-neighbors of a record more accurately.

(a) User-defined $k$

(b) Automatic $k$

Figure 5: Justification of determining $k$ value automatically in manifold
regularization.

We test the influences of automatic $k$ on a real-world dataset Reuters-215782
(RT) [18] which has 3 cross-domain pairs as follows: orgs$\mapsto$people,
orgs$\mapsto$place and people$\mapsto$place (see Section 5.1 for details). We
first apply TLF on each pair for different user-defined $k$ values such as 4,
8, 16, 32 and 64, and then calculate classification accuracy for each $k$.
After that we apply TLF on each pair where $k$ is determined automatically and
then calculate classification accuracy. We present the results of the pairs in
Fig. 6. The figure clearly shows the effectiveness of automatic $k$ since TLF
achieves the best results when we determine $k$ automatically.

(a) orgs$\mapsto$people

(b) orgs$\mapsto$place

(c) people$\mapsto$place

Figure 6: Justification of automatic determination of k value on RT dataset in
terms of classification accuracy.

The Laplacian matrix, $L$ of Eq. (23) is calculated as

$\displaystyle L=I-D^{-\frac{1}{2}}BD^{-\frac{1}{2}}$ (27)

where $D$ is a diagonal matrix with each element
$D_{ii}=\sum^{z}_{j=1}{B_{ij}}$. Thus, by incorporating Eq. (6) into Eq. (23),
we get

$\displaystyle M_{f,K}(P_{s},P_{t})=tr(\alpha^{T}{K}{L}{K}\alpha)$ (28)

Using equations (7), (12) and (28), we can reformulate $f$ in Eq. (5) as
follows.

$\displaystyle
f=\operatorname*{argmin}_{f\in\mathcal{H}}\left\|R-\alpha^{T}K\right\|^{2}_{F}+\sigma(tr(\alpha^{T}{K}\alpha))$
(29)
$\displaystyle+\lambda(tr(\alpha^{T}{K}{M}{K}\alpha))+\gamma(tr(\alpha^{T}{K}{L}{K}\alpha))$
(30)

Setting derivative $\frac{\partial f}{\partial\alpha}=0$, we obtain the
solution as

$\displaystyle\alpha=\sigma{I}+(\lambda{M}+\gamma{L})K$ (31)

We now transform the attribute contribution matrices $W^{[N_{p}\times
d_{s}]}_{s}$ and $W^{[N_{p}\times d_{t}]}_{t}$ into $G^{[z\times d_{s}]}_{s}$
and $G^{[z\times d_{t}]}_{t}$, respectively as follows.

$\displaystyle(G_{s})_{ij}=\left\\{\begin{array}[]{l l}e_{ij}&\text{if
$e_{ij}=w_{ij}\wedge w_{i}\in{W_{s}}$}\\\ 0&\text{otherwise}\\\
\end{array}\right.;\forall i,j$ (34)

$\displaystyle(G_{t})_{ij}=\left\\{\begin{array}[]{l l}e_{ij}&\text{if
$e_{ij}=w_{ij}\wedge w_{i}\in{W_{t}}$}\\\ 0&\text{otherwise}\\\
\end{array}\right.;\forall i,j$ (37)

Using equations (31), (34) and (37), we obtain the projection matrix $P_{s}$
as follows.

$\displaystyle P_{s}=G^{T}_{s}{\alpha}{G_{t}}$ (38)

Step-5: Find and transfer projected source data to the target domain and build
the final classifier.

In this step, we first identify transferable data $D^{\prime}_{s}$ from
$D_{s}$ by using the projection matrix $P_{s}$ (obtained in Eq. (38)) and
source pivots $V_{s}$ (obtained in Step 3). We then project $D^{\prime}_{s}$
into $D^{\prime\prime}_{s}$ i.e.
$D^{\prime{\prime}}_{s}=D^{\prime}_{s}\times{P_{s}}$. After that we merge
$D^{\prime\prime}_{s}$ with $D_{t}$ and obtain $D^{\prime}_{t}$ (see Step 5 of
Algorithm 1 and Fig. 4). We finally build the final forest $F$ by applying RF
on $D^{\prime}_{t}$. The final classifier is then used to classify unlabeled
target data.

###  4.2 Generalization Bound Analysis

Following the approaches in [3, 4, 19], we analyze the generalization error
bound of TLF on the target domain based on the structural risk on the source
domain. Let $y=h(x)$ and $\hat{y}=f(x)$ be the true labeling function and
prediction function, respectively. Also let $l(x)$ be a continuous loss
function $l(x)=|h(x)-f(x)|$ and $e$ be the expected error $e=l(x)$. The
expected error $e_{t}(f)$ of the prediction function $f(x)$, where a record
$x$ is drawn from the target domain $D_{t}$, is defined as

$\displaystyle e_{t}(f)=E_{x\sim p_{t}}\left[|h(x)-f(x)|\right]=E_{x\sim
p_{t}}\left[l(x)\right]$ (39)

Similarly, the expected error $e_{s}(f)$ of the prediction function $f(x)$,
where a record $x$ is drawn from the source domain $D_{s}$, is defined as

$\displaystyle e_{s}(f)=E_{x\sim p_{s}}\left[|h(x)-f(x)|\right]=E_{x\sim
p_{s}}\left[l(x)\right]$ (40)

Let $\hat{e}_{s}(f)$ and $\hat{e}_{t}(f)$ be the empirical errors of $f$ in
$D_{s}$ and $D_{t}$, respectively and
$\hat{e}(f)=\hat{e}_{s}(f)+\hat{e}_{t}(f)$ be the total empirical error of
both $D_{s}$ and $D_{t}$. Following the approaches in [3, 19], now we present
the error bound for the target domain in terms of the source risk in Theorem
2.

######  Theorem 2.

[3, 19] If $d$ be the Vapnik-Chervonenkis (VC) dimension [14] of a prediction
function $f$ for the target domain $D_{t}$ then the expected error $e_{t}(f)$
of $f$ in $D_{t}$ is bounded with probability at least $1-\delta$ by

$\displaystyle e_{t}(f)\leq\hat{e}_{s}(f)+\sqrt{\frac{4}{n}\left(d\cdot
log(\frac{2\cdot e\cdot n}{d})+log(\frac{4}{\delta})\right)}$ (41)
$\displaystyle+D_{f,K}(J_{s},J_{t})+\Omega$ (42)

where $n$ is the number of records, $e$ is the base of natural logarithm and
$\Omega=inf_{f\in H_{K}}\left[e_{s}(f)+e_{t}(f)\right]$.

###### Proof.

Following the $VC$ theory [14], the expected error $e_{s}(f)$ of $f$ in
$D_{s}$ for a VC dimension $d$ is bounded with high probability $1-\delta$ by

$\displaystyle e_{s}(f)\leq\hat{e}_{s}(f)+\sqrt{\frac{4}{n}\left(d\cdot
log(\frac{2\cdot e\cdot n}{d})+log(\frac{4}{\delta})\right)}$ (43)

Now we can write Eq. (39) as

$\displaystyle e_{t}(f)=E_{x\sim p_{t}}\left[l(x)\right]$ (44)

By adding an error term $\hat{e}_{t}(f)\geq 0$, Eq. (44) can be expressed as

$\displaystyle\begin{array}[]{l l}e_{t}(f)&\leq\hat{e}_{t}(f)+E_{x\sim
p_{t}}\left[l(x)\right]\\\ &\leq\hat{e}_{t}(f)+E_{x\sim
p_{s}}\left[l(x)\right]+\left[E_{x\sim p_{t}}\left[l(x)\right]-E_{x\sim
p_{s}}\left[l(x)\right]\right]\end{array}$ (47)

Again, let $D_{f,K}(J_{s},J_{t})$ be the distribution difference between the
joint probability distributions $J_{s}$ and $J_{t}$, where
$D_{f,K}(J_{s},J_{t})=\left[E_{x\sim p_{t}}\left[l(x)\right]-E_{x\sim
p_{s}}\left[l(x)\right]\right]$. Thus, Eq. (47) can be expressed as

$\displaystyle\begin{array}[]{l l}e_{t}(f)&\leq\hat{e}_{t}(f)+E_{x\sim
p_{s}}\left[l(x)\right]+D_{f,K}(J_{s},J_{t})\\\
&\leq\hat{e}_{t}(f)+\hat{e}_{s}(f)+e_{s}(f)+D_{f,K}(J_{s},J_{t})\\\
&\leq\Omega+e_{s}(f)+D_{f,K}(J_{s},J_{t})\end{array}$ (51)

Thus by substituting the value of $e_{s}(f)$ from Eq. (43), Eq. (51) can be
expressed as

$\displaystyle e_{t}(f)\leq\hat{e}_{s}(f)+\sqrt{\frac{4}{n}\left(d\cdot
log(\frac{2\cdot e\cdot n}{d})+log(\frac{4}{\delta})\right)}$ (52)
$\displaystyle+D_{f,K}(J_{s},J_{t})+\Omega$ (53)

∎

From Theorem 2, the expected error $e_{t}(f)$ in $D_{t}$ is bounded if we can
simultaneously minimize 1) the empirical error of labeled data in $D_{s}$,
i.e., $\hat{e}_{s}(f)$, 2) the distribution distance between $D_{s}$ and
$D_{t}$ in $\mathcal{H}$, i.e., $D_{f,K}(J_{s},J_{t})$, and 3) the
adaptability of the true function $h$ in terms of hypothesis space $H_{K}$,
i.e., $\Omega$.

In TLF framework, i.e., Eq. (5), $\hat{e}_{s}(f)$ is explicitly minimized by
structural risk minimization in Eq. (7); $D_{f,K}(J_{s},J_{t})$ is explicitly
minimized by distribution adaptation in Eq. (12); $\Omega$ is implicitly
minimized by manifold regularization in Eq. (28).

###  4.3 Computational Complexity

We now calculate the complexity of TLF (as shown in Algorithm 1). As input,
TLF takes two datasets: $D_{s}$ has $n_{s}$ records and $d_{s}$ attributes,
and $D_{t}$ has $n_{t}$ records and $d_{t}$ attributes. Let $D_{s}$ and
$D_{t}$ have $C$ common labels and $n(=n_{s}+n_{t})$ records.

In Step 1, we build two forests $F_{s}$ and $F_{t}$ having $\tau$ trees each.
Since $n_{s}\gg n_{t}$, the complexity of building forests is
$O({\tau}{d_{s}}{n^{2}_{s}}{log(n_{s})})$ [2]. We then identify shared label
distributions that has a complexity $O({\tau}^{2}{n_{s}}{n_{t}}C)$.

The complexity of Step 2 is $C(l^{2}_{s}+l^{2}_{t})$ where $l_{s}$ and $l_{t}$
are the leaves of $F_{s}$ and $F_{t}$, respectively. In Step 3, $N_{p}$ common
label distributions across the domains are identified. The complexity of Step
3 is $O(N_{p}(d_{s}+d_{t}))$.

The complexity of Step 4 consists of three parts: (i) solving the linear
system of Eq. (7) using LU decomposition that has a complexity $O((z)^{3})$
[3] for TLF, where $z=2N_{p}$, (ii) for constructing the graph Laplacian
matrix L, TLF needs $O((d_{s}+d_{t}){z^{2}})$, which is performed once, and
(iii) for constructing the kernel matrix $K$ and aggregate MMD matrix $M$, TLF
requires $O(C{z^{2}})$. In Step 5, TLF needs $O({n_{s}}{d_{s}}{d_{t}})$ for
transferring the source dataset to target domain. Finally, in Step 6, TLF
builds the final decision forest which requires
$O(\tau{d_{t}}{n^{2}}{log(n)})$. Typically, $n_{t}$, $N_{p}$, $C$, $z$,
$l_{s}$ and $l_{t}$ are very small, especially compared to $n$. Therefore, the
overall complexity of TLF is
$O({n_{s}}{d_{s}}{d_{t}}+\tau{d_{t}}{n^{2}}log(n))$.

##  5 Experiments

###  5.1 Data Preparation

We apply TLF and eleven existing methods on seven real datasets which are
shown at a glance in Table 4.

Table 4: datasets at a glance. dataset [Ref.] |  #Records |  #Attributes |  #Classes | Sub-domain | Area
---|---|---|---|---|---
PIE [20] | 11554 | 1024 | 68 | PIE5, PIE7, PIE9, PIE27, PIE29 | Face recognition
Office+Caltech (OC) [20] | 2533 | 800 | 10 | amazon, Caltech10, dslr, webcam | Object recognition
20-Newsgroups (NG) [18] | 40252 |  10000 | 2 | C, S, R, T | Text Classification
VLSC [20] | 18070 | 4096 | 5 | C, I, L, S, V | Image classification
Reuters-21578 (RT) [18] | 6570 |  4000 | 3 | Orgs,People,Places | Text Classification
MNIST+USPS (MU) [20] | 3800 | 256 | 10 | MUS, MUT, UMS, UMT | Object recognition
COIL20 (COIL) [20] | 1440 | 1024 | 20 | C1S, C1T, C2S, C2T | Object recognition

PIE [20] has 11554 handwritten digits from 5 different subcategories: PIE5,
PIE7, PIE9, PIE27 and PIE29. We generate 20 combinations of source-target
pairs as follows: PIE5$\mapsto$PIE7, PIE5$\mapsto$PIE9, PIE5$\mapsto$PIE27,
PIE5$\mapsto$PIE29, PIE7$\mapsto$PIE5, and so on (see Table 5).

Office+Caltech (OC) [20] has 4 different subcategories: amazon, Caltech10,
dslr and webcam. We generate 12 combinations of source-target pairs as
follows: amazon$\mapsto$Caltech10, amazon$\mapsto$dslr, amazon$\mapsto$webcam,
Caltech10$\mapsto$amazon, and so on (see Table 6). 20-Newsgroups (NG) [18] has
approximately 20,000 documents distributed evenly in 4 different
subcategories: C, S, R, and T. Following the approach in ARTL [3], we have 5
combinations of source-target pairs as follows: CS-src (CSS)$\mapsto$CS-tgt
(CST), CT-src (CTS)$\mapsto$CT-tgt (CTT), RS-src (RSS)$\mapsto$RS-tgt (RST),
RT-src (RTS)$\mapsto$RT-tgt (RTT) and ST-src (STS)$\mapsto$ST-tgt (STT) (see
Table 6). VLSC [20] consists of 18070 images from 5 different subcategories:
Caltech101 (C), ImageNet (I), LabelMe (L), SUN09 (S), and VOC2007 (V). We
generate 20 combinations of source-target pairs as follows:
Caltech101$\mapsto$ImageNet, Caltech101$\mapsto$LabelMe,
Caltech101$\mapsto$SUN09, Caltech101$\mapsto$VOC2007,
ImageNet$\mapsto$Caltech101, and so on (see Table 6).

Similarly, MNIST+USPS (MU) and COIL [20] have 4 different subcategories from
which we generate 12 combinations of source-target pairs, separately (see
Table 6). Besides, Reuters-215782 (RT) [18] has three top categories orgs,
people, and place. Following the approach in ARTL [3], we can construct 3
cross-domain pairs as follows: orgs$\mapsto$people, orgs$\mapsto$place and
people$\mapsto$place (see Table 7).

###  5.2 Experimental Setup

####  5.2.1 Target and Test datasets Preparation

From each source$\mapsto$target pair, we use the (original) target dataset to
create test dataset to evaluate the performance of TLF and existing methods.
We divide the (original) target dataset into 2 datasets, namely (new) target
and test datasets. Since TLF assumes small labeled data in the target domain,
the newly created target and test datasets contain 5% and 95% of records,
respectively, of the original target dataset. The size of the target dataset
is determined based on empirical evaluation on VLSC and COIL datasets.

For instance, Fig. 7 shows the classification accuracies of RF-S, RF-T and TLF
on VLSC and COIL dataset. For RF-S, we build a forest by applying RF on the
source data and use the forest to classify the test data. Similarly, for RF-T,
we build a forest by applying RF on the target data and use the forest to
classify the test data. The x-axis of the figure shows the size of new target
dataset and corresponding ratio (% of records) of the original target dataset.

We can see from Fig. 7a that when target dataset size is 50% (i.e. 1810
records), both TLF and RF-T perform well indicating that the (new) target
dataset is sufficient to build a good forest. On the other hand, none of them
perform well when the target dataset size is less than 5%. It is worth noting
from the figure that a model does not perform well when it is built on the
source dataset only. The empirical evaluation clearly demonstrates the need
for transfer learning across domains. Therefore, we choose 5% and 95% (of
original target dataset) as the size of (new) target and test datasets,
respectively to evaluate the techniques properly. We also get similar results
on the COIL dataset as shown in Fig. 7b.

(a) VLSC dataset

(b) COIL dataset

Figure 7: Justification of target dataset size on COIL and VLSC datasets in
terms of classification accuracy.

####  5.2.2 Baseline Methods

We compare the performance of TLF with eleven state-of-the-art machine
learning and transfer learning techniques, comprising nine transfer learning
and two baseline (non-transfer learning) algorithms, namely SHDA [2], MEDA [4,
21], TCA [22], BDA [16], CORAL [23], EasyTL [6], JDA [24], ARTL [3], RF [15],
and SysFor [25].

We implement TLF in the Java programming language using the Weka APIs [26].
The source code of TLF is available at GitHub
(https://github.com/grahman20/TLF). We also implement an existing technique
called SHDA [2]. For implementing the RF and SysFor, we use the Java code from
the Weka platform [26]. All other methods are already available in the MEDA
framework [21]. We use the default settings of the methods while running the
experiment.

####  5.2.3 Implementation Details

In our experiment, we use a number of parameters for TLF and existing
techniques. Forest algorithms require two common parameters, namely number of
trees ($\tau$) and minimum leaf size. $\tau$ is set to 10 and the minimum leaf
size for large datasets (i.e. the size of the dataset is greater than 10000)
is set to 50, otherwise it is set to 20. For TLF, we use $\lambda=0.01$,
$\theta=5$ and $\gamma=0.001$ for manifold, MMD and ridge regularizations,
respectively. In our experimentation we use the majority voting [27] method to
calculate final output of a forest. In case of tie, we consider the prediction
distribution of class values to break the tie by following the WEKA [26] and
MOA [28] approaches.

###  5.3 Experimental Results

We present the performances of TLF and existing methods on 20 pairs of SDD and
TDD that are created from PIE dataset in Table 5.

For each method with suffix ’-S’, we first build a classifier by applying the
method on the SDD. We then calculate the classification accuracy by applying
the classifier on the test data. For example, for combination 1
(“PIE05$\mapsto$PIE07”) the classification accuracy of RF-S is 0.052 which is
reported in the first row (under the RF-S column) in Table 5. Similarly, for
each method with suffix ’-T’, we first build a classifier by applying the
method on the TDD and then calculate the classification accuracy by applying
the classifier on the test data. For example, for combination 1
(“PIE05$\mapsto$PIE07 ”) the classification accuracy of RF-T is 0.064 which is
reported in the first row (under the RF-T column) in Table 5.

For each transfer learning method, we take both SDD and TDD as input of the
method and get a classifier as output. We then calculate the classification
accuracy by applying the classifier on the test data. For example, for
combination 1 (“PIE05$\mapsto$PIE07 ”) the classification accuracy of TLF is
0.338 which is reported in the first row (under the TLF column) in Table 5.
Bold values in the table indicate that TLF performs the best in all
combinations.

For each method, we also calculate the average classification accuracy from
the classification accuracies of 20 combinations as shown in the last row of
the table. The average classification accuracy of TLF is 0.472. It is clear
from the experimental results that TLF outperforms other techniques on PIE
dataset.

Table 5: Classification accuracies of TLF and existing methods on PIE dataset. Domains | Baseline methods | Existing transfer learning methods | TLF
---|---|---|---
Source | Target | RF-S | RF-T | SysFor-S | SysFor-T | EasyTL | ARTL | GFK | TCA | JDA | CORAL | BDA | MEDA | SHDA
PIE05 | PIE07 | 0.052 | 0.064 | 0.025 | 0.045 | 0.098 | 0.265 | 0.233 | 0.274 | 0.214 | 0.111 | 0.306 | 0.269 | 0.298 | 0.338
PIE05 | PIE09 | 0.041 | 0.050 | 0.023 | 0.051 | 0.124 | 0.378 | 0.294 | 0.322 | 0.265 | 0.127 | 0.335 | 0.282 | 0.224 | 0.381
PIE05 | PIE27 | 0.056 | 0.090 | 0.024 | 0.087 | 0.337 | 0.540 | 0.354 | 0.493 | 0.236 | 0.368 | 0.510 | 0.485 | 0.343 | 0.584
PIE05 | PIE29 | 0.039 | 0.060 | 0.015 | 0.048 | 0.092 | 0.172 | 0.219 | 0.242 | 0.214 | 0.097 | 0.246 | 0.174 | 0.224 | 0.249
PIE07 | PIE05 | 0.066 | 0.152 | 0.022 | 0.085 | 0.226 | 0.321 | 0.234 | 0.232 | 0.265 | 0.225 | 0.238 | 0.219 | 0.253 | 0.386
PIE07 | PIE09 | 0.121 | 0.050 | 0.037 | 0.051 | 0.133 | 0.492 | 0.488 | 0.340 | 0.265 | 0.137 | 0.494 | 0.427 | 0.224 | 0.526
PIE07 | PIE27 | 0.181 | 0.090 | 0.080 | 0.087 | 0.358 | 0.482 | 0.612 | 0.514 | 0.377 | 0.354 | 0.362 | 0.484 | 0.343 | 0.662
PIE07 | PIE29 | 0.086 | 0.060 | 0.022 | 0.048 | 0.105 | 0.327 | 0.312 | 0.295 | 0.215 | 0.108 | 0.338 | 0.271 | 0.224 | 0.364
PIE09 | PIE05 | 0.049 | 0.152 | 0.046 | 0.085 | 0.218 | 0.323 | 0.357 | 0.297 | 0.265 | 0.228 | 0.389 | 0.224 | 0.353 | 0.392
PIE09 | PIE07 | 0.095 | 0.064 | 0.051 | 0.045 | 0.123 | 0.428 | 0.253 | 0.366 | 0.356 | 0.128 | 0.252 | 0.434 | 0.298 | 0.529
PIE09 | PIE27 | 0.147 | 0.090 | 0.104 | 0.087 | 0.356 | 0.533 | 0.628 | 0.547 | 0.451 | 0.360 | 0.366 | 0.536 | 0.343 | 0.682
PIE09 | PIE29 | 0.073 | 0.060 | 0.054 | 0.048 | 0.127 | 0.325 | 0.392 | 0.361 | 0.315 | 0.126 | 0.362 | 0.331 | 0.224 | 0.411
PIE27 | PIE05 | 0.077 | 0.152 | 0.025 | 0.085 | 0.361 | 0.538 | 0.576 | 0.253 | 0.526 | 0.386 | 0.355 | 0.480 | 0.430 | 0.605
PIE27 | PIE07 | 0.156 | 0.064 | 0.057 | 0.045 | 0.192 | 0.590 | 0.466 | 0.560 | 0.652 | 0.189 | 0.701 | 0.591 | 0.298 | 0.740
PIE27 | PIE09 | 0.188 | 0.050 | 0.082 | 0.051 | 0.211 | 0.666 | 0.734 | 0.625 | 0.457 | 0.218 | 0.475 | 0.664 | 0.224 | 0.769
PIE27 | PIE29 | 0.098 | 0.060 | 0.043 | 0.048 | 0.149 | 0.438 | 0.248 | 0.449 | 0.365 | 0.151 | 0.422 | 0.374 | 0.224 | 0.481
PIE29 | PIE05 | 0.046 | 0.152 | 0.020 | 0.085 | 0.148 | 0.212 | 0.251 | 0.202 | 0.215 | 0.187 | 0.256 | 0.126 | 0.244 | 0.292
PIE29 | PIE07 | 0.058 | 0.064 | 0.021 | 0.045 | 0.091 | 0.253 | 0.296 | 0.256 | 0.251 | 0.099 | 0.284 | 0.257 | 0.230 | 0.302
PIE29 | PIE09 | 0.034 | 0.050 | 0.022 | 0.051 | 0.108 | 0.280 | 0.233 | 0.286 | 0.214 | 0.123 | 0.232 | 0.284 | 0.224 | 0.312
PIE29 | PIE27 | 0.050 | 0.090 | 0.022 | 0.087 | 0.261 | 0.312 | 0.391 | 0.359 | 0.322 | 0.270 | 0.432 | 0.319 | 0.343 | 0.437
Average | 0.086 | 0.083 | 0.040 | 0.063 | 0.191 | 0.394 | 0.379 | 0.364 | 0.322 | 0.200 | 0.368 | 0.362 | 0.278 | 0.472

Table 6 presents the classification accuracies of TLF and the existing methods
on Office+Caltech (OC), 20-Newsgroups (NG) and VLSC datasets. For OC dataset,
we present the performances of the techniques of 12 combinations of SDD and
TDD in the top 12 rows of the table. Bold values in the table indicate that
TLF performs the best in all combinations. For each method, we also calculate
the average classification accuracy from the classification accuracies of 12
combinations as shown in the 13-th row of the table. The average
classification accuracy of TLF is 0.400. The experimental results clearly
indicate that TLF outperforms other methods on OC dataset.

The next 5 rows of Table 6 presents the performances of the techniques of 5
combinations of SDD and TDD that are created from the NG dataset. The best
performances are indicated with bold values. The table shows that TLF performs
the best in all combinations.

We present the performances of the techniques of 20 combinations of SDD and
TDD that are created from the VLSC dataset in the last 21 rows of Table 6. For
VLSC dataset, TLF performs the best in 15 (out of 20) cases. For the remaining
5 cases, TLF performs the second best. However, in terms of average
performance (as shown in the last row of the table), TLF outperforms existing
techniques.

Table 6: Classification accuracies of TLF and existing methods on OC, NG and VLSC datasets. Dataset | Domains | Baseline methods | Existing transfer learning methods | TLF
---|---|---|---|---
Source | Target | RF-S | RF-T | SysFor-S | SysFor-T | EasyTL | ARTL | GFK | TCA | JDA | CORAL | BDA | MEDA | SHDA
OC | amazon | Caltech10 | 0.153 | 0.207 | 0.134 | 0.229 | 0.331 | 0.268 | 0.233 | 0.264 | 0.214 | 0.249 | 0.263 | 0.219 | 0.277 | 0.354
amazon | dslr | 0.107 | 0.087 | 0.060 | 0.067 | 0.048 | 0.295 | 0.280 | 0.309 | 0.242 | 0.105 | 0.242 | 0.228 | 0.282 | 0.313
amazon | webcam | 0.132 | 0.232 | 0.104 | 0.186 | 0.093 | 0.286 | 0.294 | 0.307 | 0.296 | 0.085 | 0.321 | 0.279 | 0.268 | 0.325
Caltech10 | amazon | 0.207 | 0.254 | 0.092 | 0.273 | 0.248 | 0.117 | 0.237 | 0.165 | 0.170 | 0.180 | 0.175 | 0.317 | 0.360 | 0.367
Caltech10 | dslr | 0.107 | 0.087 | 0.060 | 0.067 | 0.204 | 0.248 | 0.269 | 0.289 | 0.215 | 0.204 | 0.282 | 0.288 | 0.295 | 0.369
Caltech10 | webcam | 0.129 | 0.232 | 0.071 | 0.186 | 0.090 | 0.246 | 0.229 | 0.275 | 0.221 | 0.205 | 0.250 | 0.229 | 0.293 | 0.329
dslr | amazon | 0.197 | 0.254 | 0.092 | 0.273 | 0.313 | 0.271 | 0.223 | 0.282 | 0.247 | 0.323 | 0.135 | 0.237 | 0.193 | 0.354
dslr | Caltech10 | 0.169 | 0.207 | 0.102 | 0.229 | 0.368 | 0.288 | 0.229 | 0.284 | 0.240 | 0.403 | 0.267 | 0.238 | 0.217 | 0.429
dslr | webcam | 0.196 | 0.232 | 0.136 | 0.186 | 0.160 | 0.646 | 0.466 | 0.466 | 0.479 | 0.180 | 0.682 | 0.643 | 0.336 | 0.686
webcam | amazon | 0.149 | 0.254 | 0.092 | 0.273 | 0.328 | 0.175 | 0.236 | 0.112 | 0.195 | 0.329 | 0.232 | 0.209 | 0.219 | 0.358
webcam | Caltech10 | 0.153 | 0.207 | 0.116 | 0.229 | 0.230 | 0.240 | 0.219 | 0.213 | 0.188 | 0.234 | 0.243 | 0.189 | 0.218 | 0.294
webcam | dslr | 0.094 | 0.087 | 0.094 | 0.067 | 0.186 | 0.591 | 0.475 | 0.490 | 0.544 | 0.108 | 0.557 | 0.550 | 0.503 | 0.627
Average | 0.149 | 0.195 | 0.096 | 0.189 | 0.216 | 0.306 | 0.283 | 0.288 | 0.271 | 0.217 | 0.304 | 0.302 | 0.289 | 0.400
NG | CSS | CST | 0.428 | 0.478 | 0.360 | 0.448 | 0.528 | 0.588 | 0.566 | 0.561 | 0.569 | 0.289 | 0.579 | 0.594 | 0.570 | 0.616
CTS | CTT | 0.376 | 0.479 | 0.374 | 0.479 | 0.376 | 0.480 | 0.573 | 0.549 | 0.553 | 0.515 | 0.605 | 0.549 | 0.574 | 0.632
RSS | RST | 0.328 | 0.478 | 0.326 | 0.482 | 0.413 | 0.619 | 0.570 | 0.530 | 0.549 | 0.434 | 0.547 | 0.524 | 0.555 | 0.627
RST | RTT | 0.325 | 0.434 | 0.327 | 0.382 | 0.412 | 0.605 | 0.573 | 0.574 | 0.567 | 0.526 | 0.569 | 0.584 | 0.538 | 0.628
STS | STT | 0.422 | 0.424 | 0.365 | 0.380 | 0.461 | 0.653 | 0.517 | 0.520 | 0.530 | 0.424 | 0.524 | 0.535 | 0.565 | 0.695
Average | 0.376 | 0.458 | 0.350 | 0.434 | 0.438 | 0.589 | 0.560 | 0.547 | 0.554 | 0.438 | 0.565 | 0.557 | 0.560 | 0.640
VLSC | Caltech101 | ImageNet | 0.293 | 0.197 | 0.294 | 0.163 | 0.446 | 0.651 | 0.285 | 0.206 | 0.316 | 0.329 | 0.317 | 0.543 | 0.482 | 0.680
Caltech101 | LabelMe | 0.280 | 0.395 | 0.280 | 0.389 | 0.106 | 0.601 | 0.416 | 0.410 | 0.389 | 0.328 | 0.400 | 0.531 | 0.511 | 0.556
Caltech101 | SUN09 | 0.294 | 0.214 | 0.294 | 0.183 | 0.162 | 0.603 | 0.372 | 0.376 | 0.383 | 0.329 | 0.316 | 0.516 | 0.500 | 0.620
Caltech101 | VOC2007 | 0.346 | 0.264 | 0.346 | 0.266 | 0.268 | 0.522 | 0.470 | 0.441 | 0.266 | 0.346 | 0.428 | 0.482 | 0.476 | 0.571
ImageNet | Caltech101 | 0.317 | 0.205 | 0.113 | 0.243 | 0.592 | 0.784 | 0.589 | 0.167 | 0.433 | 0.411 | 0.296 | 0.748 | 0.761 | 0.824
ImageNet | LabelMe | 0.078 | 0.096 | 0.046 | 0.061 | 0.208 | 0.460 | 0.130 | 0.309 | 0.306 | 0.205 | 0.308 | 0.508 | 0.526 | 0.556
ImageNet | SUN09 | 0.094 | 0.127 | 0.054 | 0.101 | 0.210 | 0.603 | 0.218 | 0.304 | 0.201 | 0.354 | 0.316 | 0.602 | 0.438 | 0.620
ImageNet | VOC2007 | 0.309 | 0.305 | 0.215 | 0.131 | 0.371 | 0.522 | 0.380 | 0.132 | 0.131 | 0.215 | 0.128 | 0.373 | 0.467 | 0.571
LabelMe | Caltech101 | 0.008 | 0.236 | 0.165 | 0.228 | 0.149 | 0.844 | 0.217 | 0.404 | 0.228 | 0.165 | 0.731 | 0.790 | 0.412 | 0.838
LabelMe | ImageNet | 0.308 | 0.208 | 0.311 | 0.218 | 0.195 | 0.651 | 0.217 | 0.169 | 0.218 | 0.311 | 0.170 | 0.514 | 0.452 | 0.680
LabelMe | SUN09 | 0.256 | 0.271 | 0.267 | 0.238 | 0.317 | 0.603 | 0.306 | 0.283 | 0.383 | 0.267 | 0.380 | 0.530 | 0.487 | 0.620
LabelMe | VOC2007 | 0.204 | 0.340 | 0.240 | 0.381 | 0.170 | 0.522 | 0.268 | 0.381 | 0.381 | 0.240 | 0.373 | 0.490 | 0.415 | 0.571
SUN09 | Caltech101 | 0.024 | 0.295 | 0.021 | 0.283 | 0.539 | 0.843 | 0.300 | 0.309 | 0.283 | 0.302 | 0.199 | 0.607 | 0.452 | 0.838
SUN09 | ImageNet | 0.211 | 0.215 | 0.320 | 0.218 | 0.174 | 0.651 | 0.210 | 0.195 | 0.218 | 0.320 | 0.517 | 0.582 | 0.526 | 0.680
SUN09 | LabelMe | 0.042 | 0.289 | 0.102 | 0.207 | 0.313 | 0.601 | 0.138 | 0.228 | 0.368 | 0.302 | 0.365 | 0.517 | 0.478 | 0.556
SUN09 | VOC2007 | 0.047 | 0.232 | 0.091 | 0.228 | 0.386 | 0.522 | 0.300 | 0.126 | 0.228 | 0.091 | 0.207 | 0.378 | 0.321 | 0.571
VOC2007 | Caltech101 | 0.228 | 0.224 | 0.044 | 0.247 | 0.674 | 0.784 | 0.645 | 0.312 | 0.247 | 0.044 | 0.544 | 0.713 | 0.542 | 0.804
VOC2007 | ImageNet | 0.711 | 0.248 | 0.406 | 0.289 | 0.303 | 0.651 | 0.431 | 0.201 | 0.289 | 0.406 | 0.291 | 0.633 | 0.417 | 0.680
VOC2007 | LabelMe | 0.193 | 0.188 | 0.234 | 0.215 | 0.342 | 0.560 | 0.277 | 0.327 | 0.215 | 0.234 | 0.514 | 0.533 | 0.453 | 0.556
VOC2007 | SUN09 | 0.199 | 0.150 | 0.285 | 0.138 | 0.334 | 0.603 | 0.254 | 0.310 | 0.375 | 0.285 | 0.161 | 0.532 | 0.415 | 0.620
Average | 0.222 | 0.235 | 0.206 | 0.221 | 0.313 | 0.629 | 0.321 | 0.280 | 0.293 | 0.274 | 0.348 | 0.556 | 0.477 | 0.651

Table 7 presents the classification accuracies of TLF and the existing methods
on Reuters-215782 (RT), MNIST+USPS (MU) and COIL datasets. For RT dataset, we
present the performances of the techniques of 3 combinations of SDD and TDD in
the top 3 rows of the table. Bold values in the table indicate that TLF
performs the best in all combinations. For each method, we also calculate the
average classification accuracy from the classification accuracies of 3
combinations as shown in the 4-th row of the table. The average classification
accuracy of TLF is 0.615. The experimental results clearly indicate that TLF
outperforms other methods on RT dataset.

The next 13 rows of Table 7 presents the performances of the techniques of 12
combinations of SDD and TDD that are created from the NG dataset. The best
performances are indicated with bold values. The table shows that TLF performs
the best in all combinations. We also present the performances of the
techniques of 12 combinations of SDD and TDD that are created from the COIL
dataset in the last 13 rows of Table 7. For COIL dataset, TLF outperforms all
existing techniques in all combinations.

Table 7: Classification accuracies of TLF and existing methods on RT, MU and COIL datasets. Dataset | Domains | Baseline methods | Existing transfer learning methods | TLF
---|---|---|---|---
Source | Target | RF-S | RF-T | SysFor-S | SysFor-T | EasyTL | ARTL | GFK | TCA | JDA | CORAL | BDA | MEDA | SHDA
RT | Orgs | People | 0.337 | 0.376 | 0.320 | 0.369 | 0.427 | 0.552 | 0.469 | 0.533 | 0.539 | 0.513 | 0.534 | 0.513 | 0.515 | 0.596
Orgs | Places | 0.322 | 0.406 | 0.319 | 0.395 | 0.510 | 0.584 | 0.464 | 0.501 | 0.454 | 0.510 | 0.431 | 0.542 | 0.527 | 0.639
People | Places | 0.336 | 0.372 | 0.333 | 0.362 | 0.412 | 0.554 | 0.455 | 0.470 | 0.507 | 0.431 | 0.467 | 0.560 | 0.510 | 0.610
Average | 0.332 | 0.385 | 0.324 | 0.375 | 0.450 | 0.563 | 0.463 | 0.501 | 0.500 | 0.485 | 0.477 | 0.538 | 0.517 | 0.615
MU | MUS | MUT | 0.100 | 0.195 | 0.081 | 0.081 | 0.286 | 0.497 | 0.313 | 0.311 | 0.315 | 0.321 | 0.313 | 0.492 | 0.312 | 0.563
MUS | UMS | 0.099 | 0.195 | 0.083 | 0.131 | 0.290 | 0.506 | 0.308 | 0.312 | 0.312 | 0.313 | 0.326 | 0.483 | 0.430 | 0.551
MUS | UMT | 0.189 | 0.208 | 0.175 | 0.163 | 0.290 | 0.485 | 0.325 | 0.399 | 0.397 | 0.357 | 0.397 | 0.462 | 0.418 | 0.522
MUT | MUS | 0.084 | 0.210 | 0.070 | 0.192 | 0.388 | 0.526 | 0.417 | 0.311 | 0.377 | 0.418 | 0.313 | 0.451 | 0.317 | 0.549
MUT | UMS | 0.192 | 0.196 | 0.184 | 0.196 | 0.370 | 0.506 | 0.351 | 0.383 | 0.397 | 0.362 | 0.399 | 0.575 | 0.421 | 0.615
MUT | UMT | 0.108 | 0.211 | 0.069 | 0.210 | 0.404 | 0.447 | 0.319 | 0.392 | 0.314 | 0.413 | 0.314 | 0.410 | 0.428 | 0.522
UMS | MUS | 0.108 | 0.210 | 0.070 | 0.192 | 0.388 | 0.526 | 0.317 | 0.311 | 0.314 | 0.410 | 0.313 | 0.411 | 0.417 | 0.549
UMS | MUT | 0.192 | 0.207 | 0.184 | 0.196 | 0.420 | 0.497 | 0.325 | 0.380 | 0.397 | 0.365 | 0.395 | 0.457 | 0.417 | 0.631
UMS | UMT | 0.151 | 0.211 | 0.069 | 0.210 | 0.413 | 0.447 | 0.319 | 0.392 | 0.414 | 0.312 | 0.314 | 0.410 | 0.428 | 0.522
UMT | MUS | 0.188 | 0.312 | 0.175 | 0.293 | 0.463 | 0.456 | 0.353 | 0.398 | 0.396 | 0.365 | 0.397 | 0.463 | 0.423 | 0.549
UMT | MUT | 0.100 | 0.195 | 0.081 | 0.181 | 0.386 | 0.497 | 0.329 | 0.411 | 0.315 | 0.365 | 0.413 | 0.509 | 0.431 | 0.631
UMT | UMS | 0.151 | 0.195 | 0.083 | 0.191 | 0.390 | 0.506 | 0.385 | 0.412 | 0.412 | 0.422 | 0.413 | 0.514 | 0.430 | 0.615
Average | 0.138 | 0.212 | 0.111 | 0.186 | 0.374 | 0.491 | 0.338 | 0.368 | 0.363 | 0.369 | 0.359 | 0.470 | 0.406 | 0.568
COIL | C1S | C1T | 0.164 | 0.439 | 0.171 | 0.369 | 0.509 | 0.707 | 0.627 | 0.676 | 0.692 | 0.496 | 0.621 | 0.592 | 0.566 | 0.735
C1S | C2S | 0.193 | 0.476 | 0.171 | 0.263 | 0.503 | 0.705 | 0.685 | 0.708 | 0.681 | 0.522 | 0.575 | 0.597 | 0.604 | 0.733
C1S | C2T | 0.137 | 0.408 | 0.128 | 0.368 | 0.637 | 0.728 | 0.587 | 0.595 | 0.654 | 0.635 | 0.700 | 0.670 | 0.670 | 0.774
C1T | C1S | 0.124 | 0.335 | 0.117 | 0.336 | 0.497 | 0.651 | 0.685 | 0.575 | 0.581 | 0.486 | 0.575 | 0.660 | 0.616 | 0.737
C1T | C2S | 0.124 | 0.476 | 0.119 | 0.263 | 0.642 | 0.690 | 0.654 | 0.597 | 0.600 | 0.640 | 0.600 | 0.680 | 0.680 | 0.775
C1T | C2T | 0.177 | 0.408 | 0.164 | 0.368 | 0.481 | 0.637 | 0.681 | 0.689 | 0.579 | 0.481 | 0.575 | 0.650 | 0.638 | 0.727
C2S | C1S | 0.124 | 0.335 | 0.117 | 0.336 | 0.497 | 0.651 | 0.585 | 0.675 | 0.581 | 0.486 | 0.575 | 0.650 | 0.616 | 0.737
C2S | C1T | 0.147 | 0.439 | 0.129 | 0.369 | 0.645 | 0.690 | 0.547 | 0.596 | 0.600 | 0.630 | 0.685 | 0.676 | 0.676 | 0.772
C2S | C2T | 0.177 | 0.408 | 0.164 | 0.368 | 0.481 | 0.637 | 0.581 | 0.689 | 0.579 | 0.481 | 0.675 | 0.680 | 0.620 | 0.727
C2T | C1S | 0.107 | 0.335 | 0.109 | 0.336 | 0.638 | 0.693 | 0.654 | 0.596 | 0.599 | 0.622 | 0.654 | 0.694 | 0.693 | 0.774
C2T | C1T | 0.106 | 0.439 | 0.107 | 0.369 | 0.509 | 0.707 | 0.583 | 0.576 | 0.579 | 0.496 | 0.582 | 0.592 | 0.592 | 0.735
C2T | C2S | 0.193 | 0.476 | 0.171 | 0.263 | 0.503 | 0.705 | 0.585 | 0.571 | 0.581 | 0.522 | 0.675 | 0.599 | 0.604 | 0.733
Average | 0.148 | 0.414 | 0.139 | 0.334 | 0.545 | 0.683 | 0.621 | 0.628 | 0.609 | 0.541 | 0.624 | 0.645 | 0.631 | 0.746

We present the average classification accuracies (see Table 8) of the TLF and
existing techniques on all datasets. Bold values in the tables indicate the
best results. From Table 8, we can see that TLF outperforms the other
techniques in terms of classification accuracy for all datasets.

Table 8: Overall classification accuracies of TLF and existing methods on all datasets. Dataset | Baseline methods | Existing transfer learning methods | TLF
---|---|---|---
RF-S | RF-T | SysFor-S | SysFor-T | EasyTL | ARTL | GFK | TCA | JDA | CORAL | BDA | MEDA | SHDA
PIE | 0.086 | 0.083 | 0.040 | 0.063 | 0.191 | 0.394 | 0.379 | 0.364 | 0.322 | 0.200 | 0.368 | 0.362 | 0.278 | 0.472
OC | 0.149 | 0.195 | 0.096 | 0.189 | 0.216 | 0.306 | 0.283 | 0.288 | 0.271 | 0.217 | 0.304 | 0.302 | 0.289 | 0.400
NG | 0.376 | 0.458 | 0.350 | 0.434 | 0.438 | 0.589 | 0.560 | 0.547 | 0.554 | 0.438 | 0.565 | 0.557 | 0.560 | 0.640
VLSC | 0.222 | 0.235 | 0.206 | 0.221 | 0.313 | 0.629 | 0.321 | 0.280 | 0.293 | 0.274 | 0.348 | 0.556 | 0.477 | 0.651
RT | 0.332 | 0.385 | 0.324 | 0.375 | 0.450 | 0.563 | 0.463 | 0.501 | 0.500 | 0.485 | 0.477 | 0.538 | 0.517 | 0.615
MU | 0.138 | 0.212 | 0.111 | 0.186 | 0.374 | 0.491 | 0.338 | 0.368 | 0.363 | 0.369 | 0.359 | 0.470 | 0.406 | 0.568
COIL | 0.148 | 0.414 | 0.139 | 0.334 | 0.545 | 0.683 | 0.621 | 0.628 | 0.609 | 0.541 | 0.624 | 0.645 | 0.631 | 0.746
Average | 0.207 | 0.283 | 0.181 | 0.258 | 0.361 | 0.522 | 0.423 | 0.425 | 0.416 | 0.360 | 0.435 | 0.490 | 0.451 | 0.585

###  5.4 Statistical Analysis of the Results

We now analyze the results by using a statistical non-parametric sign test
[29] and Nemenyi [30] test for all datasets in order to evaluate the
statistical significance of the superiority of TLF over the existing methods.

We carry out sign test on the results of TLF with other methods (one by one)
at the right-tailed by considering significance level $\alpha=0.025$ (i.e.
97.5% significance level). While we compare TLF with EasyTL (i.e. TLF vs
EasyTL) we obtain a z-value (test static value) which is used to determine the
significant improvement of TLF over EasyTL. Similarly we obtain z-values for
other comparisons based on all datasets as shown in Fig. 8(a). In the figure,
each bar represents the z-value of the comparison between TLF and an existing
method, whereas the line represents the z-ref value. The z-ref value for
$\alpha=0.025$ is 1.96. The performance of TLF is significantly better than a
method if the z-value of TLF vs the method is greater than the z-ref which is
1.96 in this case for $\alpha=0.025$, as the z-ref value can be obtained from
a table [29]. The sign test results (see Fig. 8(a)) indicate that both TLF
performs significantly better than the other methods (at $z>1.96$, $p<0.025$,
and right-tailed) on all datasets.

(a) Statistical Sign test analysis

(b) Statistical Nemenyi test analysis

Figure 8: Statistical significance analysis based on Sign test and Nemenyi
test on all datasets.

Similarly, we carry out the Nemenyi test on the results of TLF with other
methods (one by one) at the right-tailed by considering significance level
$\alpha=0.025$ (i.e. 97.5% significance level) as shown in Fig. 8(b). The
Nemenyi test results (see Fig. 8(b)) indicate that TLF performs significantly
better than the other techniques (at $p<0.025$) on all datasets except one
case marked as down arrow.

###  5.5 Effectiveness Verification of TLF Components

Our proposed TLF framework consists of three main components, namely ridge,
MMD and Manifold regularizations. We now evaluate the effectiveness of each
component in terms of classification accuracy by carrying out experiments on
the PIE and OC datasets. The experimental results are shown in Fig. 9 where
the first bar represents the classification accuracy of TLF with all
components, the second bar represents the classification accuracy of TLF
without ridge regularization, the third represents the classification accuracy
of TLF without MMD regularization and the fourth represents the classification
accuracy of TLF without manifold regularization. From the figures, we can see
the importance of all components as the classification accuracies of TLF with
all components (first bar of the figures) are higher than other settings.

(a) PIE dataset

(b) OC dataset

Figure 9: Effectiveness of the use of ridge, MMD and Manifold regularizations
in TLF is demonstrated in terms of classification accuracy on PIE and OC
datasets.

##  6 Analysis of the Effectiveness of TLF in Some Challenging Situations

After presenting the main experimental results and evaluation in Section 5, we
now explore the effectiveness of TLF in some challenging situations, such as
when datasets have missing values, as follows.

###  6.1 Handling Datasets with Missing Values

Similar to SHDA [2] and ARTL [3], in TLF we consider the datasets without
missing values. However, in some real applications, due to many reasons
datasets may contain missing values [31]. In RF [15], missing values are dealt
by either (i) using an approach called SRD that simply deletes the records
having missing values, or (ii) by imputing missing values with the attribute
mean (for numerical) or mode (for categorical) values. If a dataset contains
missing values, we suggest to (i) impute the missing values by using a state-
of-the-art technique such as SiMI [31], and (ii) consider the imputed dataset
as input to TLF.

We examine the effectiveness of TLF in handling missing values on PIE and OC
datasets as shown in Fig. 10. For PIE dataset, we artificially create missing
values in all source and target datasets. The missing values are created
randomly with 5 missing ratios: 10%, 20%, 30%, 40% and 50%, where x% missing
ratio means x% of the total records of a dataset have contained one or more
missing values. For example, if a dataset has 100 records then a missing ratio
of 50% means that 50 records (out of 100) have at least one missing value. The
records are chosen randomly. Once a record is selected, y% of total attribute
values of the record are randomly made missing. The value of y is chosen
randomly between 1 and 50. For example, if the value of y is 10 and the
dataset has 100 attributes then 10 attribute values of the records are
randomly made missing.

For the SRD approach, we first delete the records having missing values from
both source and target datasets and then build a classifier for the target
domain by applying TLF on the datasets without missing values. The classifier
is then used to classify the test data and calculate the classification
accuracy. This approach is called “SRD-TLF” in Fig. 10. For an imputation
approach, we impute the missing values of both source and target datasets
using SiMI [31] and then build a classifier by applying TLF on the datasets
without missing values. We then use the classifier to classify the test data
and calculate the classification accuracy. This approach is called “SiMI-TLF”
in Fig. 10. The average accuracies of all source and target datasets for SRD-
TLF and SiMI-TLF on PIE dataset are presented in Fig. 10(a) where the average
accuracy of “Original-TLF” is obtained from the TLF column of Table 8.
Similarly, for OC dataset, we calculate the average classification accuracies
of Original-TLF, SRD-TLF and SiMI-TLF and present them in Fig. 10(b). It is
clear from the figures that if a dataset contains missing values, TLF performs
better if the missing values are imputed instead of deleting the records
having missing values. For small missing ratios, SiMI-TLF achieves almost the
accuracy of TLF for a dataset without any missing values.

(a) PIE dataset

(b) OC dataset

Figure 10: Classification accuracies of Original-TLF, SRD-TLF and SiMI-TLF on
PIE and OC datasets.

###  6.2 Handling Class Imbalanced Datasets

Since it is possible to integrate any decision forest algorithm into TLF
framework, TLF is capable of handling class imbalanced datasets by integrating
an existing algorithm called CSForest [32] which is designated to handle class
imbalances problems and we call this variant TLF-C. For implementing the
CSForest, we use the Java code from the Weka platform [26]. In our experiment,
the default cost metrics of CSForest are set to: true-positive=1, true-
negative=0, false-positive=1, and false-negative=5.

We investigate the effectiveness of TLF-C, TLF, CSForest and 3 existing
transfer learning methods, namely SHDA, MEDA and ARTL on two class imbalanced
datasets, namely segment0 and yeast1 that are publicly available at Keel
repository [33]. We create source, target and test datasets from each original
dataset as follows. We first create source dataset by randomly selecting 70%
records of the original dataset and then create target dataset with the
remaining records of the original dataset. After that we divide the target
dataset in to test and target datasets by randomly choosing 95% and 5%,
respectively, records of the actual target dataset. We repeat this process for
10 runs. For each run, we find the result of a method. We calculate the
average result of the 10 runs and present it in Table 9. Since the
classification accuracy is not a useful metric for evaluating classifiers
built from class imbalanced datasets, we calculate Precision, Recall and
F-measure (F1) for each test dataset.

Following the similar process discussed in Section 5.3, we obtain results for
the methods, and present in Table 9. Bold values in the table indicate the
best results. For the cost metric, TLF-C achieves higher precision, recall and
F1 than the other methods. It is clear from table that TLF has the ability to
handle class imbalanced datasets by integrating an algorithm such as CSForest
[32].

Table 9: Overall Precision, Recall and F1 of TLF and existing methods on two class imbalanced datasets. Dataset | Precision (higher the better) | Recall (higher the better) |  $F_{1}$ (higher the better)
---|---|---|---
CSForest-S | CSForest-T | SHDA | MEDA | ARTL | TLF | TLF-C | CSForest-S | CSForest-T | SHDA | MEDA | ARTL | TLF | TLF-C | CSForest-S | CSForest-T | SHDA | MEDA | ARTL | TLF | TLF-C
segment0 | 0.854 | 0.792 | 0.715 | 0.754 | 0.778 | 0.812 | 0.914 | 0.715 | 0.701 | 0.642 | 0.675 | 0.688 | 0.700 | 0.846 | 0.778 | 0.744 | 0.676 | 0.712 | 0.730 | 0.752 | 0.879
yeast1 | 0.541 | 0.481 | 0.425 | 0.445 | 0.451 | 0.501 | 0.654 | 0.488 | 0.479 | 0.408 | 0.412 | 0.415 | 0.452 | 0.605 | 0.513 | 0.480 | 0.416 | 0.428 | 0.432 | 0.475 | 0.629

##  7 Conclusion

This paper introduced TLF, a supervised heterogeneous transfer learning
framework, which learns a classifier for the target domain having only few
labeled training records by transferring knowledge from the source domain
having many labeled records. Differing form existing methods, in TLF we
simultaneously address two key issues: feature discrepancy and distribution
divergence.

To alleviate feature discrepancy, TLF identifies shared label distributions
that act as the pivots to bridge the domains (see Step 3 of Algorithm 1 and
Fig. 2). To handle distribution divergence, TLF simultaneously optimizes the
structural risk functional, joint distributions between domains, and the
manifold consistency underlying marginal distributions (see Step 4 of
Algorithm 1 and Section 4.1). In TLF, we propose a novel approach to determine
nearest neighbors of a record automatically so that intrinsic properties of
manifold consistency can be exploited in detail resulting in a high transfer
performance (see Section 4.1 and Fig. 6). Moreover, to avoid negative
transfer, we identify transferable records of the source dataset by using the
pivots (see Step 3 of Algorithm 1 and Section 4.1).

The effectiveness of TLF is also reflected in the experimental results. We
evaluate TLF on seven publicly available natural datasets and compare the
performance of TLF against the performance of eleven state-of-the-art
techniques including ARTL [3], SHDA [2] and MEDA [4]. From Table 8, we can see
that in all datasets TLF outperforms the other techniques in terms of
classification accuracy. Statistical sign test and Nemenyi test analyses (see
Section 5.4) indicate that TLF performs significantly better than the other
methods at $z>1.96$, $p<0.025$ on all datasets.

From the experiments we highlighted the strengths and weaknesses of TLF as
follows. The main strengths are: (1) TLF achieves the best classification
performance on all datasets over the eleven state-of-the-art techniques (see
Table 8), (2) TLF is capable of handling both feature discrepancy and
distribution divergence issues simultaneously, (3) TLF has the ability to
handle imbalanced datasets (see Section 6.2), and (5) TLF is capable of
dealing with datasets having missing values (see Section 6.1). However, like
existing methods including ARTL [3], SHDA [2] and MEDA [4], TLF has a
limitation on processing multiple source domains. TLF is now suitable for
handling only a single source domain at a time.

In future, we plan to explore the effectiveness of combining records from
multiple source domains to learn classifiers for the target domain.

## References

  * [1] S. Niu, Y. Liu, J. Wang, H. Song, A decade survey of transfer learning (2010–2020), IEEE Transactions on Artificial Intelligence 1 (2) (2020) 151–166.
  * [2] S. Sukhija, N. C. Krishnan, Supervised heterogeneous feature transfer via random forests, Artificial Intelligence 268 (2019) 30–53.
  * [3] M. Long, J. Wang, G. Ding, S. J. Pan, S. Y. Philip, Adaptation regularization: A general framework for transfer learning, IEEE Transactions on Knowledge and Data Engineering 26 (5) (2014) 1076–1089.
  * [4] J. Wang, Y. Chen, W. Feng, H. Yu, M. Huang, Q. Yang, Transfer learning with dynamic distribution adaptation, ACM Transactions on Intelligent Systems and Technology (TIST) 11 (1) (2020) 1–25.
  * [5] L. Bruzzone, M. Marconcini, Domain adaptation problems: A dasvm classification technique and a circular validation strategy, IEEE transactions on pattern analysis and machine intelligence 32 (5) (2009) 770–787.
  * [6] J. Wang, Y. Chen, H. Yu, M. Huang, Q. Yang, Easy transfer learning by exploiting intra-domain structures, in: IEEE International Conference on Multimedia & Expo (ICME), 2019.
  * [7] S. J. Pan, Q. Yang, A survey on transfer learning, IEEE Transactions on knowledge and data engineering 22 (10) (2010) 1345–1359.
  * [8] S. J. Pan, J. T. Kwok, Q. Yang, et al., Transfer learning via dimensionality reduction., in: AAAI, Vol. 8, 2008, pp. 677–682.
  * [9] M. Belkin, P. Niyogi, V. Sindhwani, Manifold regularization: A geometric framework for learning from labeled and unlabeled examples., Journal of machine learning research 7 (11).
  * [10] W. Li, L. Duan, D. Xu, I. W. Tsang, Learning with augmented features for supervised and semi-supervised heterogeneous domain adaptation, IEEE transactions on pattern analysis and machine intelligence 36 (6) (2013) 1134–1148.
  * [11] J. T. Zhou, I. W. Tsang, S. J. Pan, M. Tan, Heterogeneous domain adaptation for multiple classes, in: Artificial intelligence and statistics, 2014, pp. 1095–1103.
  * [12] W. Dai, Q. Yang, G.-R. Xue, Y. Yu, Self-taught clustering, in: Proceedings of the 25th international conference on Machine learning, 2008, pp. 200–207.
  * [13] Y. Lu, W. Wang, C. Yuan, X. Li, Z. Lai, Manifold transfer learning via discriminant regression analysis, IEEE Transactions on Multimedia.
  * [14] V. Vapnik, Statistical Learning Theory, John Wiley, New York, 1998.
  * [15] L. Breiman, Random forests, Machine learning 45 (1) (2001) 5–32.
  * [16] J. Wang, Y. Chen, S. Hao, W. Feng, Z. Shen, Balanced distribution adaptation for transfer learning, in: The IEEE International conference on data mining (ICDM), 2017, pp. 1129–1134.
  * [17] B. Saha, S. Gupta, D. Phung, S. Venkatesh, Multiple task transfer learning with small sample sizes, Knowledge and information systems 46 (2) (2016) 315–342.
  * [18] E. Zhong, Transfer learning resources, https://www.cse.ust.hk/TL/, accessed April 25, 2021 (2020).
  * [19] S. Ben-David, J. Blitzer, K. Crammer, F. Pereira, Analysis of representations for domain adaptation, Advances in neural information processing systems 19 (2006) 137–144.
  * [20] J. Wang, et al., Everything about transfer learning and domain adapation, http://transferlearning.xyz, accessed May 29, 2021 (2020).
  * [21] J. Wang, W. Feng, Y. Chen, H. Yu, M. Huang, P. S. Yu, Visual domain adaptation with manifold embedded distribution alignment, in: Proceedings of the 26th ACM international conference on Multimedia, 2018, pp. 402–410.
  * [22] S. J. Pan, I. W. Tsang, J. T. Kwok, Q. Yang, Domain adaptation via transfer component analysis, IEEE Transactions on Neural Networks 22 (2) (2011) 199–210.
  * [23] B. Sun, J. Feng, K. Saenko, Return of frustratingly easy domain adaptation, AAAI 6 (7).
  * [24] M. Long, J. Wang, G. Ding, J. Sun, P. S. Yu, Transfer feature learning with joint distribution adaptation, in: Proceedings of the IEEE international conference on computer vision, 2013, pp. 2200–2207.
  * [25] M. Z. Islam, H. Giggins, Knowledge discovery through sysfor: a systematically developed forest of multiple decision trees, in: Proceedings of the Ninth Australasian Data Mining Conference-Volume 121, Australian Computer Society, Inc., 2011, pp. 195–204.
  * [26] E. Frank, M. A. Hall, I. H. Witten, The weka workbench. online appendix for ”data mining: Practical machine learning tools and techniques” (2016).
  * [27] N. Jamali, C. Sammut, Majority voting: Material classification by tactile sensing using surface texture, IEEE Transactions on Robotics 27 (3) (2011) 508–521.
  * [28] A. Bifet, G. Holmes, R. Kirkby, B. Pfahringer, MOA: massive online analysis, J. Mach. Learn. Res. 11 (2010) 1601–1604.
  * [29] R. D. Mason, D. A. Lind, W. G. Marchal, Statistics: an introduction, Duxbury Press, 1994.
  * [30] J. Demšar, Statistical comparisons of classifiers over multiple data sets, Journal of Machine learning research 7 (Jan) (2006) 1–30.
  * [31] M. G. Rahman, M. Z. Islam, Missing value imputation using decision trees and decision forests by splitting and merging records: Two novel techniques, Knowledge-Based Systems 53 (2013) 51 – 65.
  * [32] M. J. Siers, M. Z. Islam, Novel algorithms for cost-sensitive classification and knowledge discovery in class imbalanced datasets with an application to nasa software defects, Information Sciences 459 (2018) 53–70.
  * [33] J. Alcalá-Fdez, A. Fernández, J. Luengo, J. Derrac, S. García, L. Sánchez, F. Herrera, Keel data-mining software tool: data set repository, integration of algorithms and experimental analysis framework., Journal of Multiple-Valued Logic & Soft Computing 17.