# Transfer Learning Extensions for the Probabilistic Classification Vector
Machine 111The final authenticated publication is available at
https://doi.org/10.1016/j.neucom.2019.09.104

Christoph Raab<EMAIL_ADDRESS>Frank-Michael Schleif
<EMAIL_ADDRESS>University for Applied Science Würzburg-Schweinfurt,
Sanderheinrichsleitenweg 20, Würzburg, Germany

###### Abstract

Transfer learning is focused on the reuse of supervised learning models in a
new context. Prominent applications can be found in robotics, image processing
or web mining. In these fields, the learning scenarios are naturally changing
but often remain related to each other motivating the reuse of existing
supervised models. Current transfer learning models are neither sparse nor
interpretable. Sparsity is very desirable if the methods have to be used in
technically limited environments and interpretability is getting more critical
due to privacy regulations. In this work, we propose two transfer learning
extensions integrated into the sparse and interpretable probabilistic
classification vector machine. They are compared to standard benchmarks in the
field and show their relevance either by sparsity or performance improvements.

###### keywords:

Transfer Learning, Probabilistic Classification Vector Machine, Transfer
Kernel Learning, Nyström Approximation, Basis Transfer, Sparsity

PCVM

    Probabilistic Classification Vector Machine
PCTKVM

    Probabilistic Classification Transfer Kernel Vector Machine
BT

    Basis-Transfer
TKL

    Transfer Kernel Learning
TCA

    Transfer Component Analysis
SVM

    Support Vector Machine
JDA

    Joint Distribution Adaptation
SVD

    Singular Value Decomposition
EVD

    Eigen Value Decomposition
RKHS

    Reproducing Kernel Hilbert Space
PCA

    Principal Component Analysis
NBT

    Nyström Basis Transfer
SA

    Subspace Alignment
SVM

    Support Vector Machine
NTVM

    Nyström Transfer Vector Machine
SURF

    Speeded Up Robust Features Extraction

##  1 Introduction

Supervised learning and in particular classification is an important task in
machine learning with a broad range of applications. The obtained models are
used to predict the label of unseen test samples. In general, it is assumed
that the underlying domain of interest is not changing between training and
test samples. If the domain is changing from one task to a related but
different task, one would like to reuse the available learning model. Domain
differences are quite common in real-world scenarios and eventually lead to
substantial performance drops [1].

A transfer learning example is the classification of web pages: A classifier
is trained in the domain of university web pages with a word distribution
according to universities and in the test scenario, the domain has changed to
non-university web pages, where the word distribution may not be similar to
training distribution. Figure 1 shows a toy example of a traditional and a
transfer classification task with clearly visible domain differences.

(a) Traditional Problem

(b) Transfer Problem

Figure 1: Toy example showing a comparison of a traditional classification
task with two classes shown in red and blue. Left figure shows a traditional
classification task with one domain and right side shows transfer learning
classification task with two domains, which are indicated as shapes. Transfer
learning aims to extract common information in one domain to help a model in
another domain.

More formally, let $\mathbf{Z}=\\{\mathbf{z}_{1},\dots,\mathbf{z}_{n}\\}$ be
source data sampled from the source domain distribution $p(z)$ and let
$\mathbf{X}=\\{\mathbf{x}_{1},\dots,\mathbf{x}_{m}\\}$ be target data from the
target domain distribution $p(x)$. Traditional machine learning assumes
similar distributions, i.e. $p(z)\sim p(x)$, but transfer learning assumes
different distributions, i.e. $p(z)\neq p(x)$, and appears in the web page
example where $\mathbf{Z}$ could be features of university websites and
$\mathbf{X}$ are features of non-university websites.

In general, transfer learning aims to solve the divergence between domain
distributions by reusing information in one domain to help to learn a target
prediction function in a different domain of interest [2]. However, despite
the definition, the proposed solutions implicitly solve differences by linear
transformations, detailed in section 4.1 and 4.2. Multiple transfer learning
methods have been already proposed, following different strategies and
improving prediction performance of underlying classification algorithms in
test scenarios [1][2]. In this paper, we focus on sparse models, which are not
yet covered sufficiently by transfer learning approaches.

The Probabilistic Classification Vector Machine (PCVM) [3] is a sparse
probabilistic kernel classifier pruning unused basis functions during
training, found to be very effective [3][4] with competitive performance to
Support Vector Machine (SVM) [5]. The PCVM is naturally sparse and creates
interpretable models as needed in many application domains of transfer
learning. The original PCVM is not well suited for transfer learning due to
their focus on stationary Gaussian distribution and is equipped within this
work with two transfer learning approaches.

The contributions are detailed in the following:
We integrate Transfer Kernel Learning (TKL) [6] into the PCVM to retain its
sparsity. Inspired by the Basis-Transfer (BT) [7], a subspace transfer
learning approach is proposed and is also combined with PCVM, boosting
prediction performance significantly compared to the baseline. This is
enhanced by Nyström techniques, which reduces computational complexity
compared to BT. Finally, a data augmentation strategy is proposed, making the
approach independent of a certain domain adaptation task, which is a drawback
of BT. The proposed solutions are tested against other commonly used transfer
learning approaches on common datasets in the field.

The rest of the paper is organized as follows: An overview of related work is
given in section 2. The mathematical preliminaries of PCVM and Nyström
approximation are introduced in section 3. The proposed transfer learning
extensions following in section 4. An experimental part is given in section 5,
addressing the classification performance, the sparsity and the computational
time of the approaches. A summary and a discussion of open issues are provided
in the conclusion.

##  2 Related Work

The area of transfer learning provides a broad range of transfer strategies
with many competitive approaches [1][2]. In the following, we briefly name
these strategies and discuss the key approaches used herein.

The instance transfer methods try to align the distribution by re-weighting
some source data, which can directly be used with target data in the training
phase [1].

Approaches implementing the symmetric feature transfer [1] are trying to find
a common latent subspace for source and target domain with the goal to reduce
distribution differences, such that the underlying structure of the data is
preserved in the subspace. An example of a symmetric feature transfer method
is the Transfer Component Analysis (TCA) [8].

The asymmetric feature transfer approaches try to transform source domain data
in the target (subspace) domain. This should be done in a way that the
transformed source data will match the target distribution. In comparison to
the symmetric feature transfer approaches, there is no shared subspace
available, but only the target space [1]. An example is given by the Joint
Distribution Adaptation (JDA) [9], which solves divergences in distributions
similar to TCA, but aligning conditional distributions with pseudo-labeling
techniques. Pseudo-labeling is performed by assigning labels to unlabeled
target data by a baseline classifier, e.g. SVM, resulting in a target
conditional distribution, followed by matching it to the source conditional
distribution of the ground truth source label [9]. The Subspace Alignment (SA)
[10] algorithm is another asymmetric transfer learning approach. It computes a
target subspace representation where source and target data are aligned but is
only evaluated on image domain adaptation data. We included SA in the
experimental study also containing non-image data.

The relational-knowledge transfer aims to find some relationship between
source and target data [1]. Transfer Kernel Learning (TKL) [6] is a recent
approach, which approximates a kernel of training data $K(\mathbf{Z})$ with
kernel of test data $K(\mathbf{X})$ via the Nyström kernel approximation. It
only considers discrepancies in distributions and further claims it is
sufficient to approximate a training kernel via test kernel, i.e.
$K(\mathbf{Z})\approx K(\mathbf{X})$, for effective knowledge transfer [6].
Note that the restriction to kernels does not apply to the Nyström transfer
learning extension (section 4.2) because it is completely in Euclidean space.

All the considered methods have approximately a complexity of
$\mathcal{O}(n^{2})$ where $n$ is the most significant number of samples
concerning test or training [11, 9, 6, 8]. According to the definition of
transfer learning [2], these algorithms do transductive transfer learning,
because some unlabeled test data must be available at training time. These
transfer-solutions cannot be directly used as predictors, but instead are
wrappers for classification algorithms. The baseline classifier is most often
the SVM. Note that discussed approaches are only tested with a classification
task and may limited to this task.

##  3 Preliminaries

###  3.1 Probabilistic Classification Vector Machine

The Probabilistic Classification Vector Machine (PCVM) [3] uses a
probabilistic kernel model

$f(\mathbf{x};\mathbf{w},b)=\Psi\left(\sum_{i=1}^{n}w_{i}\phi_{i}(\mathbf{x})+b\right)=\Psi\left(\Phi(\mathbf{x})^{\top}\mathbf{w}+b\right)\in\mathbb{R}$
(1)

with a link function $\Psi(\cdot)$, $w_{i}$ being the weights of the basis
functions $\phi_{i}(\mathbf{x})$ and $b$ as bias term. The class assignment
$c(\mathbf{x})$ of given data $\mathbf{x}$ is given by [3]

$c(x)=\begin{cases}1&\Phi(\mathbf{x})^{\top}\mathbf{w}+b\geq 0\\\
-1&else.\end{cases}$ (2)

In PCVM the basis functions $\phi_{i}$ are defined explicitly as part of the
model design. In (1) the standard kernel trick can be applied [12]. The
probabilistic output of PCVM [3] is calculated by using the probit link
function, i.e.

$\Psi(\mathbf{x})=\int_{-\infty}^{x}\mathcal{N}(t|0,1)dt,$ (3)

where $\Psi(\mathbf{x})$ is the cumulative distribution of the normal
distribution $\mathcal{N}(0,1)$. The PCVM [3] uses the Expectation-
Maximization algorithm for learning the model. However, the PCVM is not
restricted to EM and other optimization approaches like Monte Carlo techniques
are also possible [13]. The underlying sparsity framework within the
optimization prunes unused basis functions, independent of the optimization
approach, and is, therefore, a sparse probabilistic learning machine. In PCVM
we will use the standard RBF-kernel with a Gaussian width $\theta$.

In [4] a PCVM with linear costs was suggested, which makes use of the Nyström
approximation and could be additionally used to improve the run-time and
memory complexity. Further details can be found in [3] and [4].

###  3.2 Nyström Approximation

The computational complexity of calculating kernels or eigensystems scales
with $\mathcal{O}(n^{3})$ where $n$ is the sample size [14]. Therefore, low-
rank approximations and dimensionality reductions of data matrices are popular
methods to speed up computational processes [15]. The Nyström approximation
[15] is a reliable technique to approximate a kernel matrix $\mathbf{K}$ by a
low-rank representation, without computing the eigendecomposition of the whole
matrix:
By the Mercer theorem, kernels $k(\mathbf{x},\mathbf{x^{\prime}})$ can be
expanded by orthonormal eigenfunctions $f_{i}$ and non-negative eigenvalues
$\lambda_{i}$ in the form

$k(\mathbf{x},\mathbf{x^{\prime}})=\sum_{i=1}^{\infty}\lambda_{i}f_{i}(\mathbf{x})f_{i}(\mathbf{x^{\prime}}).$
(4)

The eigenfunctions and eigenvalues of a kernel are defined as solutions of the
integral equation

$\int
k(\mathbf{x^{\prime}},\mathbf{x})f_{i}(\mathbf{x})p(\mathbf{x})d\mathbf{x}=\lambda_{i}f_{i}(\mathbf{x^{\prime}}),$
(5)

where $p(\mathbf{x})$ is a probability density over the input space. This
integral can be approximated based on the Nyström technique by an i.i.d.
sample $\\{\mathbf{x}_{n}\\}_{k=1}^{s}$ from $p(\mathbf{x})$

$\frac{1}{s}\sum_{k=1}^{s}k(\mathbf{x^{\prime}},\mathbf{x}_{n})f_{i}(\mathbf{x}_{n})\approx\lambda_{i}f_{i}(\mathbf{x^{\prime}}).$
(6)

Using this approximation we denote with ${\mathbf{K}}^{(s)}$ the corresponding
$s\times s$ Gram sub-matrix and get the corresponding matrix eigenproblem
equation as

$\frac{1}{s}{\mathbf{K}}^{(s)}{\mathbf{U}}^{(s)}={\mathbf{U}}^{(s)}\boldsymbol{\Lambda}^{(s)},$
(7)

with ${\mathbf{U}}^{(s)}\in\mathbb{R}^{s\times s}$ is column orthonormal and
$\boldsymbol{\Lambda}^{(s)}$ is a diagonal matrix. Now we can derive the
approximations for the eigenfunctions and eigenvalues of the kernel $k$

$\boldsymbol{\lambda_{i}}\approx\frac{\boldsymbol{\lambda}_{i}^{(s)}\cdot
N}{s},\quad
f_{i}(\mathbf{x^{\prime}})\approx\frac{\sqrt{s/N}}{\boldsymbol{\lambda}_{i}^{(s)}}\mathbf{k}_{x}^{\prime,\top}\mathbf{u}_{i}^{(s)},$
(8)

where $\mathbf{u}_{i}^{(s)}$ is the $i$th column of $\mathbf{U}^{(s)}$. Thus,
we can approximate $f_{i}$ at an arbitrary point $\mathbf{x^{\prime}}$ as long
as we know the vector
$\mathbf{k}_{x}^{\prime}=(k(\mathbf{x}_{1},\mathbf{x^{\prime}}),...,k(\mathbf{x}_{s},\mathbf{x^{\prime}})).$
For a given $N\times N$ Gram matrix ${\mathbf{K}}$ one may randomly chose $s$
rows and $s$ columns. The corresponding indices are called landmarks, and
should be chosen such that the data distribution is sufficiently covered.
Strategies how to chose the landmarks have recently been addressed in [16, 17,
18, 19]. The approximation is exact if the sample size is equal to the rank of
the original matrix and the rows of the sample matrix are linear independent.

###  3.3 Nyström Matrix Form

The technique just introduced can be simplified by rewriting it in matrix form
[20], where scaling factors like in equation (8) are neglected. We will use
the matrix formulation throughout the remaining paper. Again, given a Gram
matrix $\mathbf{K}\in\mathbb{R}^{n\times n}$, it can be decomposed to

$\mathbf{K}=\begin{bmatrix}\mathbf{A}&\mathbf{B}\\\ \mathbf{C}&\mathbf{D}\\\
\end{bmatrix},$ (9)

with $\mathbf{A}\in\mathbb{R}^{s\times s}$,
$\mathbf{B}\in\mathbb{R}^{s\times(n-s)}$,
$\mathbf{C}\in\mathbb{R}^{(n-s)\times s}$ and
$\mathbf{D}\in\mathbb{R}^{(n-s)\times(n-s)}$. The submatrix $\mathbf{A}$ is
called the landmark matrix containing $s$ randomly chosen rows and columns
from $\mathbf{K}$ and has the Eigen Value Decomposition (EVD)
$\mathbf{A}=\mathbf{U}\boldsymbol{\Lambda}\mathbf{U}^{-1}$ as in equation (7),
where eigenvectors are $\mathbf{U}\in\mathbb{R}^{s\times s}$ and eigenvalues
are on the diagonal of $\boldsymbol{\Lambda}\in\mathbb{R}^{s\times s}$. The
remaining approximated eigenvectors $\hat{\mathbf{U}}$ of $\mathbf{K}$, i.e.
$\mathbf{C}$ or $\mathbf{B}^{T}$, are obtained by the Nyström method with
$\hat{\mathbf{U}}\boldsymbol{\Lambda}=\mathbf{CU}$ as in equation (8).
Combining $\mathbf{U}$ and $\hat{\mathbf{U}}$ the full approximated
eigenvectors of $\mathbf{K}$ are

$\tilde{\mathbf{U}}=\begin{bmatrix}\mathbf{U}\\\
\hat{\mathbf{U}}\end{bmatrix}=\begin{bmatrix}\mathbf{U}\\\
\mathbf{C}\mathbf{U}\boldsymbol{\Lambda}^{-1}\end{bmatrix}\in\mathbb{R}^{n\times
s}.$ (10)

The eigenvectors of $\mathbf{K}$ can be inverted by computing (Note
$\mathbf{C}=\mathbf{B}^{T}$):

$\tilde{\mathbf{V}}=\begin{bmatrix}\mathbf{U}^{-1}&\boldsymbol{\Lambda}^{-1}\mathbf{U}^{-1}\mathbf{B}\end{bmatrix}.$
(11)

Combining equation (10), equation (11) and $\boldsymbol{\Lambda}$, the matrix
$\mathbf{K}$ is approximated by

$\tilde{\mathbf{K}}=\tilde{\mathbf{U}}\boldsymbol{\Lambda}\tilde{\mathbf{V}}=\begin{bmatrix}\mathbf{U}\\\
\mathbf{CU}\boldsymbol{\Lambda}^{-1}\end{bmatrix}\boldsymbol{\Lambda}\begin{bmatrix}\mathbf{U}^{-1}&\boldsymbol{\Lambda}^{-1}\mathbf{U}^{-1}\mathbf{B}\end{bmatrix}.$
(12)

The Nyström approximation error is given by the Frobenius Norm between ground
truth and reconstructed matrices, i.e.
$error_{ny}=\left\lVert\tilde{\mathbf{K}}-\mathbf{K}\right\rVert_{F}$.

###  3.4 Kernel Approximation

The Nyström approximation [15] speeds up kernel computations, because kernel
evaluation must not be done over all points, but only a fraction of it. Let
$\mathbf{K}\in\mathbb{R}^{n\times n}$ be a Mercer kernel matrix with
decomposition as in equation (9). Again we pick $s$ samples with $s\ll n$,
leading to $\mathbf{A}$ as defined before. An approximated kernel is
constructed by combining equation (12) and (9).

$\tilde{\mathbf{K}}=\begin{bmatrix}\mathbf{A}&\mathbf{B}\\\
\mathbf{C}&\mathbf{C}\mathbf{A}^{-1}\mathbf{B}\end{bmatrix}=\begin{bmatrix}\mathbf{A}\\\
\mathbf{C}\end{bmatrix}\mathbf{A}^{-1}\begin{bmatrix}\mathbf{A}\mathbf{B}\end{bmatrix}=\mathbf{K}_{n,s}\mathbf{K}_{s,s}^{-1}\mathbf{K}_{s,n},$
(13)

with $\mathbf{K}_{n,s}$ being a sub-matrix of $\mathbf{K}$ incorporating all
$n$ rows and $s$ landmark columns and $\mathbf{K}_{s,s,}$ as landmarks matrix.
Based on the definition of kernel matrices,
$\mathbf{K}_{s,n}=\mathbf{K}_{n,s}^{T}$ is valid and, therefore, only
$\mathbf{K}_{n,s}$ and $\mathbf{K}_{s,s}$ must be computed. Note that the
kernel approximation is used in section 4.1.

###  3.5 General Matrix Approximation

Despite its initial restriction to kernel matrices, recent research expanded
the Nyström technique to approximate a Singular Value Decomposition (SVD)
[20].
Nyström-SVD generalizes the concept of matrix decomposition with the
consequence that respective matrices must not be square.
Let $\mathbf{G}\in\mathbb{R}^{n\times d}$ be a rectangular matrix with
decomposition as in equation (9). The SVD of the landmark matrix is given by
$\mathbf{A}=\mathbf{L}\mathbf{S}\mathbf{R}^{T}$ where $\mathbf{L}$ are left
and $\mathbf{R}$ are right singular vectors. $\mathbf{S}$ are positive
singular values. The decomposition matrices have the same size as in section
3.2. Similarly to EVD in section 3.2 the left and right singular vectors for
the non-symmetric part $\mathbf{C}$ and $\mathbf{B}$ are obtained via Nyström
techniques [20] and are defined as
$\hat{\mathbf{L}}=\mathbf{C}\mathbf{R}\mathbf{S}^{-1}$ and
$\hat{\mathbf{R}}=\mathbf{B}^{T}\mathbf{L}\mathbf{S}^{-1}$ respectively.
Applying the same principals as for Nyström-EVD, $\mathbf{G}$ is approximated
by

$\tilde{\mathbf{G}}=\tilde{\mathbf{L}}\mathbf{S}\tilde{\mathbf{R}}^{T}=\begin{bmatrix}\mathbf{L}\\\
\hat{\mathbf{L}}\end{bmatrix}\mathbf{S}\begin{bmatrix}\mathbf{R}&\hat{\mathbf{R}}\end{bmatrix}=\begin{bmatrix}\mathbf{L}\\\
\mathbf{C}\mathbf{R}\mathbf{S}^{-1}\end{bmatrix}\mathbf{S}\begin{bmatrix}\mathbf{R}&\mathbf{S}^{-1}\mathbf{L}^{T}\mathbf{B}\end{bmatrix}.$
(14)

Note that for non-gram matrices like $\mathbf{G}$, $C=B^{T}$ is no longer
valid. The matrix approximation (equation (14)), which is described in this
section, is used in the performance extension in section 4.2.

##  4 Nyström Transfer Learning

###  4.1 Transfer Kernel Sparsity-Extension

The domain invariant TKL [6] technique is part of the first transfer learning
extension. Based on experimental results shown in section 5 the use of TKL
retains model sparsity of PCVM. Given the categories of transfer learning [1]
introduced in section 2, the approach can be seen as relational-knowledge-
transfer. It approximates the source kernel by the target kernel via the
Nyström method. It aims to find a source kernel close to target distribution
and simultaneously searches an eigensystem of the source kernel, which
minimizes the squared Frobenius norm between ground truth and approximated
kernel [6].

Let $\mathbf{Z}=\\{\mathbf{z}_{i}\\}_{i=1}^{n}\in\mathbb{R}^{d}$ be training
data sampled from $p(z)$ with labels
$\mathbf{Y}_{Z}=\\{y_{i}\\}_{i=1}^{n}\in\mathcal{Y}=\\{1,2,..,C\\}$ and
$\mathbf{X}=\\{\mathbf{x}_{j}\\}_{j=1}^{m}\in\mathbb{R}^{d}$ be test data
sampled from $p(x)$ with labels
$\mathbf{Y}_{X}=\\{y_{j}\\}_{j=1}^{m}\in\mathcal{Y}$. Note that we assume
$p(z)\neq p(x)$. We obtain the associated kernel matrices $\mathbf{K}_{Z}$ for
training and $\mathbf{K}_{X}$ for testing by evaluating an appropriate kernel
function (e.g. the RBF-kernel). For clarity, we rewrite equation (9) in kernel
notation

$\mathbf{K}_{A}=\begin{bmatrix}\mathbf{K}_{X}\>\>\>\>\mathbf{K}_{ZX}\\\
\mathbf{K}_{XZ}\>\>\>\>\mathbf{K}_{Z}\end{bmatrix},$ (15)

with $\mathbf{K}_{Z}$ as train-, $\mathbf{K}_{X}$ as test- and
$\mathbf{K}_{ZX}=\mathbf{K}_{XZ}^{T}$ as cross-domain-kernel between domains.
Revisiting the original Nystrom approach, it uses randomly chosen columns and
rows from $\mathbf{K}_{A}$, however, in TKL, the target kernel
$\mathbf{K}_{X}$ is seen as a landmark matrix and is used for approximating
the training kernel $\mathbf{K}_{Z}$. Hence, landmarks are not randomly picked
from $\mathbf{K}_{A}$, but $\mathbf{K}_{X}$ is used as landmark matrix and is
the complete target set, therefore the approximation uses $m$ landmarks.
$\mathbf{K}_{Z}$ is not used in the landmark selection. This differs from the
original Nyström approach.
The TKL approach assumes that the distributions differences are sufficiently
aligned if $\mathbf{K}_{Z}\simeq\mathbf{K}_{X}$, which also leads to
$p(K(\mathbf{Z}))\simeq p(K(\mathbf{X}))$ [6].
Rewriting equation (13), we can create an approximated training kernel by

$\mathbf{K}_{Z}=\mathbf{U}_{Z}\boldsymbol{\Lambda}_{X}\mathbf{U}_{Z}^{T}=\mathbf{K}_{ZX}\mathbf{K}_{X}^{-1}\mathbf{K}_{XZ},$
(16)

where $\mathbf{U}_{Z}$ are eigenvalues from source kernel and
$\boldsymbol{\Lambda}$ are eigenvalues from target kernel. This new kernel is
not based on the eigensystem of $\mathbf{K}_{X}$ and should therefore not
reduce distribution differences sufficiently. Hence, the eigenvectors of
$\mathbf{K}_{Z}$ are constructed by the target eigensystem

$\bar{\mathbf{U}}_{Z}=\mathbf{K}_{ZX}\mathbf{U}_{X}\boldsymbol{\Lambda}_{X}^{-1}.$
(17)

To fully approximate the eigensystem of the training kernel, the eigenvalues
$\boldsymbol{\Lambda}=\\{\lambda_{i}\\}_{i=1}^{m}$ are defined as model
parameters of TKL, leading to approximated kernel
$\bar{\mathbf{K}}_{Z}=\bar{\mathbf{U}}_{Z}\boldsymbol{\Lambda}\bar{\mathbf{U}}_{Z}^{T}$.
These new parameters must be well-chosen to reduce domain differences and
while keeping original training information. The following optimization
problem was suggested in [6] to solve this issue

$\begin{gathered}\min_{\boldsymbol{\Lambda}}||\bar{\mathbf{K}}_{Z}-\mathbf{K}_{Z}||^{2}_{F}=||\bar{\mathbf{U}}_{Z}\boldsymbol{\Lambda}\bar{\mathbf{U}}_{Z}^{T}-\mathbf{K}_{Z}||^{2}_{F}\\\
\lambda_{i}\geq\zeta\lambda_{i+1},i=1,\dots,m-1\quad\lambda_{i}\geq
0,i=0,\dots,m\end{gathered}$ (18)

with $\zeta\geq 1$ as eigenspectrum dumping factor. The obtained kernel is
domain invariant [6] and can be used in any kernel machine. The complexity of
the TKL algorithm can be given with $\mathcal{O}((d+r)(n+m)^{2})$, where $r$
denotes the number of used eigenvectors and $d$ refers to the dimensionality
of data [6]. TKL in combination with PCVM is called Probabilistic
Classification Transfer Kernel Vector Machine (PCTKVM).

####  4.1.1 Properties of PCTKVM algorithm

The TKL kernel
$\bar{\mathbf{K}}_{Z}=\bar{\mathbf{U}}_{Z}\boldsymbol{\Lambda}\bar{\mathbf{U}}_{Z}^{T}$
is used to train the PCVM. In general, an RBF-kernel with in-place optimized
distribution-width parameter $\theta$ is used in PCVM. Accordingly, a simple
replacement of the standard RBF-kernel by a kernel obtained with TKL will be
inefficient. In PCVM the kernel is recalculated in each iteration, based on
the optimized $\theta$ from the previous iteration. Consequently, we would
have to recalculate the entire transfer kernel too. The complexity of the
standard PCVM is $\mathcal{O}(l^{3})$ where $l$ is the number of basis
functions and $l=n$ at the beginning of the training and before pruning basis
functions. The complexity of TKL is $\mathcal{O}((d+r)(n+m)^{2})$. Combining
them, we would end up with a computational complexity of
$\mathcal{O}(n^{3}(d+r)(n+m)^{2})$.

However, the performance of PCTKVM strongly depends on the quality of
$\theta$. Hence, some reasonable $\theta$ must be obtained via grid search,
because in-place optimization is infeasible. The PCTKVM using a fixed $\theta$
has the complexity of $\mathcal{O}(n^{3}+m^{2})$. Note that TKL and PCTKVM are
restricted to kernels, but are independent of the kernel type.

Predictions are made with the PCVM prediction function, but by employing
$\bar{\mathbf{K}}_{XZ}=\mathbf{U}_{X}\mathbf{\Lambda}\bar{\mathbf{U}}_{Z}^{T}$
as kernel for test data [6]. However, in case of the SVM as baseline
classifier, the kernel with size $n\times m$ is used and restricted to the
respective support vectors. The prediction function for the SVM has the form
$\hat{\mathbf{y}}=\bar{\mathbf{K}}_{ZX}(\mathbf{\alpha}\cdot\mathbf{y}_{\mathbf{Z}})+b$,
where $\mathbf{\alpha}$ are the Lagrange multipliers [6].

Because of the sparsity of the PCVM the number of basis functions used in the
decision function is typically small222The decision function may only be
constructed by $10$ samples.. If we consider that our model has $l$ non-zero
weight vectors with $l\ll n$ and because the PCVM uses only kernel
rows/columns corresponding to the non-zero weight vector index, our final
kernel $\bar{\mathbf{K}}_{LX}$ for prediction has size $(l\times m)$.
Therefore, the prediction function of the PCTKVM has the form:
$\hat{\mathbf{y}}=\bar{\mathbf{K}}_{LX}\mathbf{w}+b.$ The probabilistic output
is calculated with the probit link function used in the PCVM. Pseudo code of
sparsity extension is shown in algorithm 1.

Algorithm 1 Probabilistic Classification Transfer Kernel Vector Machine

1:$\mathbf{K}=[\mathbf{Z};\mathbf{X}]$ as $N$ sized training and $M$ sized
test set; $\mathbf{Y}$ as $N$ sized training label vector; kernel(-type) ker;
eigenspectrum dumping factor $\zeta$; $\theta$ as kernel parameter.

2:Weight Vector $\mathbf{w}$; bias $b$, kernel parameter $\theta$; transfer
kernel $\bar{\mathbf{K}}_{\mathcal{A}}$.

3:$\mathbf{D}$ = calculate_dissimilarity_matrix($\mathbf{K}$);

4:$\bar{\mathbf{K}}_{\mathcal{A}}$ =
transfer_kernel_Learning($\mathbf{D}$,ker,$\theta$,$\zeta$); $\triangleright$
According to equation (18)

5:[$\mathbf{w}$,$b$] = pcvm_training($\bar{\mathbf{K}}_{\mathcal{Z}}$);
$\triangleright$ According to section 3.1

However, the sparsity extension performs similar to common approaches shown in
the experiments in section 5, but to fully capture the advantages of Nyström
and PCVM the following section shows a transfer learning approach focused on
performance.

###  4.2 Nyström Basis Transfer Performance-Extension

It is a reasonable strategy in TKL to align kernel matrices rather than kernel
distributions in Reproducing Kernel Hilbert Space (RKHS), since distributions
alignments are non-trivial in RKHS [6]. Hence, TKL modifies the kernel
explicitly to reduce the difference between two kernel matrices. Similar
source and target kernels must be obtained, because the underlying classifier
is kernel-based and has no transfer learning.

In [21] is shown that if source and target datasets are similar, they follow
similar distributions, i.e. if $\mathbf{Z}\simeq\mathbf{X}$ then $p(z)\simeq
p(x)$, and further have similar kernel distributions and similar kernels.
Therefore, we propose a transfer learning approach operating in Euclidean
space rather than RKHS, because it does not limit approaches to kernel
classifiers. Further, the obtained kernels after transfer of data also follow
similar distributions. A recent study [7] already showed great transfer
capabilities and performance, by aligning $\mathbf{X}$ and $\mathbf{Z}$ with a
small error in terms of the Frobenius norm. However, this requires same
samples sizes of $\mathbf{Z}$ and $\mathbf{X}$ and is assumed in the following
with size $m\times d$. The study considered the following optimization problem

$\min\limits_{\mathbf{M},\mathbf{T}}\left\lVert\mathbf{M}\mathbf{Z}\mathbf{T}-\mathbf{X}\right\rVert_{F}^{2}$
(19)

where $\mathbf{M}\in\mathbb{R}^{m\times m}$ and
$\mathbf{T}\in\mathbb{R}^{d\times d}$ are transformation matrices drawing the
data closer together. A solution is found analytically, summarized in three
steps [7]: First, normalize data to standard mean and variance. This will
align marginal distributions in Euclidean space without considering label
information [7]. Second, compute an SVD of source and target data, i.e.
$\mathbf{Z}=\mathbf{L}_{Z}\mathbf{S}_{Z}\mathbf{R}_{Z}^{T}$ and
$\mathbf{X}=\mathbf{L}_{X}\mathbf{S}_{X}\mathbf{R}_{X}^{T}$. Next, the
approach assumes $\mathbf{S}_{Z}\sim\mathbf{S}_{X}$ in terms of Frobenius norm
due to normalization with zero mean and variance one, reducing the scaling
factor of singular values to the same range. Finally, compute a solution for
equation (19) by solving the linear equations. One obtains
$\mathbf{M}=\mathbf{L}_{X}\mathbf{L}_{Z}^{-1}$ and
$\mathbf{T}=\mathbf{R}_{Z}^{-1}\mathbf{R}_{X}^{T}$. Note that
$\mathbf{L}_{Z}\mathbf{L}_{Z}^{-1}=\mathbf{R}_{Z}\mathbf{R}_{Z}^{-1}=\mathbf{I}$.
Apply the transfer operation and approximate the source matrix by using target
basis information

$\tilde{\mathbf{Z}}=\mathbf{M}\mathbf{Z}\mathbf{T}=\mathbf{L}_{X}\mathbf{L}_{Z}^{-1}\mathbf{L}_{Z}\mathbf{S}_{Z}\mathbf{R}_{Z}\mathbf{R}_{Z}^{-1}\mathbf{R}_{X}^{T}=\mathbf{L}_{X}\mathbf{S}_{Z}\mathbf{R}_{X}^{T},$
(20)

with $\tilde{\mathbf{Z}}\in\mathbb{R}^{m\times d}$ as approximated source
data, used for training. The three-step process is shown in figure 2 as
geometrical interpretation demonstrated with a toy example created by Gaussian
random sampling.

(a) Data unnormalized

(b) Data after standard normalization

(c) Data after Basis-Transfer

Figure 2: Process of Basis-Transfer with samples from two domains. Class
information is given by color (red/blue) and domain is indicated by shape
(domain one - $o$, domain two - $\ast$). First (a), the non-normalized data
with a knowledge gap. Second (b), a normalized feature space. Third (c),
Basis-Transfer approximation is applied, correcting the samples, i.e. shapes
with same color are aligned, and training data is usable for learning a
classification model. Best viewed in color.

In the following, the work [7] is continued, and we propose a Nyström based
version with three main improvements: Reduction of computational complexity
via Nyström, implicit dimensionality reduction and neglecting sample size
requirements of BT. Further, we introduce a data augmentation strategy that
eliminates the restriction [7] to the task of text transfer learning.

Recap equation (19) and consider a slightly changed optimization problem

$\min\limits_{\mathbf{M}}||\mathbf{M}\mathbf{Z}-\mathbf{X}||_{F}^{2},$ (21)

where $\mathbf{Z},\mathbf{X}\in\mathbb{R}^{m\times d}$, a transformation
matrix $\mathbf{M}\in\mathbb{R}^{m\times m}$ must be found, which is again
obtained analytically. Because we apply a dimensionality reduction technique,
just the left-sided transformation matrix must be determined, which is derived
in the following: Based on the relationship between SVD and EVD, the Principal
Component Analysis (PCA) can be rewritten in terms of SVD. Consider the target
matrix with SVD

$\mathbf{X}^{T}\mathbf{X}=(\mathbf{R}_{X}\mathbf{S}_{X}\mathbf{L}_{X}^{T})(\mathbf{L}_{X}\mathbf{S}_{X}\mathbf{R}_{X}^{T})=\mathbf{R}_{X}\mathbf{S}_{X}^{2}\mathbf{R}_{X}^{T},$
(22)

where $\mathbf{R}_{X}\in\mathbb{R}^{d\times s}$ as eigenvectors and
$\mathbf{S}_{X}^{2}\in\mathbb{R}^{s\times s}$ as eigenvalues of
$\mathbf{X}^{T}\mathbf{X}$. By choosing only the biggest $s$ eigenvalues and
corresponding eigenvectors the dimensionality of $\mathbf{X}$ is reduced by

$\mathbf{X}_{s}=\mathbf{X}\mathbf{R}_{s}=\mathbf{L}_{s}\mathbf{S}_{s}\mathbf{R}_{s}^{T}\mathbf{R}_{s}=\mathbf{L}_{s}\mathbf{S}_{s},$
(23)

where $\mathbf{L}_{s}\in\mathbb{R}^{m\times s}$,
$\mathbf{S}_{s}\in\mathbb{R}^{s\times s}$ and
$\mathbf{X}_{s}\in\mathbb{R}^{m\times s}$ is the reduced target matrix. Hence,
only a left sided transformation in equation (21) is required, because right
sided transformation is omitted in equation (23).

The computational complexity of BT and PCA is decreased by applying Nyström-
SVD: Let $\mathbf{Z}$ and $\mathbf{X}$ have a decomposition given as in
equation (9). Note for clarity the Nyström notation is used as in section 3.5.
For a Nyström-SVD we chose from both matrices $s$ columns/rows obtaining
landmarks matrices
$\mathbf{A}_{Z}=\mathbf{L}_{Z}\mathbf{S}_{Z}\mathbf{R}_{Z}^{T}\in\mathbb{R}^{s\times
s}$ and
$\mathbf{A}_{X}=\mathbf{L}_{X}\mathbf{S}_{X}\mathbf{R}_{X}^{T}\in\mathbb{R}^{s\times
s}$. Based on Nyström-SVD in equation (14), the dimensionality is reduced as
in equation (23) keeping only most relevant data structures

$\mathbf{X}_{s}=\tilde{\mathbf{L}}_{X}\mathbf{S}_{X}=\begin{bmatrix}\mathbf{L}_{X}\\\
\hat{\mathbf{L}}_{X}\end{bmatrix}\mathbf{S}_{X}=\begin{bmatrix}\mathbf{L}_{X}\\\
\mathbf{C}_{X}\mathbf{R}_{X}\mathbf{S}_{X}^{-1}\end{bmatrix}\mathbf{S}_{X}\in\mathbb{R}^{m\times
s},$ (24)

Hence, it is sufficient to compute an SVD of $\mathbf{A}_{X}$ instead of
$\mathbf{X}$ with $s\ll\\{m,d\\}$ and therefore is considerably lower in
computational complexity. Analogy, we approximate source data by
$\mathbf{Z}_{s}=\tilde{\mathbf{L}}_{Z}\mathbf{S}_{Z}\in\mathbb{R}^{n\times
s}$. Since we again assume $\mathbf{S}_{Z}\sim\mathbf{S}_{Z}$ due to data
normalization, solving the linear equation as a possible solution for equation
(21), leads to $\mathbf{M}=\tilde{\mathbf{L}}_{X}\tilde{\mathbf{L}}_{Z}^{-1}$.
Plugging it back we obtain

$\mathbf{Z}_{s}=\tilde{\mathbf{L}}_{X}\tilde{\mathbf{L}}_{Z}^{-1}\tilde{\mathbf{L}}_{Z}\mathbf{S}_{Z}=\tilde{\mathbf{L}}_{X}\mathbf{S}_{Z}\in\mathbb{R}^{m\times
s}$ (25)

where again the basis of target data transfers structural information into the
training domain. The matrix $\mathbf{Z}_{s}$ is used for training and
$\mathbf{X}_{s}$ is used for testing. According to [1], it is an asymmetric
transfer approach. Further, it is transductive [2] and does not need labeled
target data. For further references, we call the approach Nyström Basis
Transfer (NBT) and in combination with PCVM, Nyström Transfer Vector Machine
(NTVM).

####  4.2.1 Properties of Nyström Basis Transfer

We showed that NBT is a valid PCA approximation by equation (23). It follows
by definition of SVD that
$\tilde{\mathbf{L}}_{Z}\tilde{\mathbf{L}}_{Z}^{T}=\tilde{\mathbf{L}}_{X}\tilde{\mathbf{L}}_{X}^{-1}=\mathbf{I}$
and $\tilde{\mathbf{L}}_{X}$ is an orthogonal basis. Therefore, equation (24)
and equation (25) are orthogonal transformations. In particular equation (25)
transforms the source data into the target subspace and projects it onto the
principal components of $\mathbf{X}$. If data matrices $\mathbf{X}$ and
$\mathbf{Z}$ are standard normalized333Experimental data are standard
normalized to mean zero and variance one in the preprocessing., the geometric
interpretation is a rotation of source data w.r.t to angles of the target
basis, already shown in figure 2.
The computational complexity of NBT is given by the complexity of Nyström-SVD
and the calculation of the respective landmark matrices $\mathbf{A}_{Z}$ and
$\mathbf{A}_{X}$ with complexity $\mathcal{O}(s^{2})$. The matrix inversion of
diagonal matrix $\mathbf{S}_{X}^{-1}$ in equation (24) can be neglected.
Remaining matrix multiplications are $\mathcal{O}(ns^{2})$, contributing to an
overall complexity of NBT, which is $O(s^{2})$ with $s\ll\\{n,m\\}$. This
makes NBT the fastest transfer learning solution in terms of computational
complexity in comparison to the discussed methods in section 2.
The approximation error is similar to the original Basis-Transfer [7] error:

$error_{nbt}=\left\lVert\mathbf{X}_{s}-\mathbf{Z}_{s}\right\rVert_{F}=\left\lVert\tilde{\mathbf{L}}_{X}\mathbf{S}_{X}-\tilde{\mathbf{L}}_{X}\mathbf{S}_{Z}\right\rVert_{F}=\left\lVert\mathbf{S}_{X}-\mathbf{S}_{Z}\right\rVert_{F}$
(26)

####  4.2.2 Data Augmentation

In BT [7], sample sizes of data matrices must be aligned. This is not required
in NBT as seen in equation (24) and equation (25), because differences in size
are aligned during transformation. However, the original dataset $\mathbf{Z}$
has an $n\times 1$ sized label vector with $n\neq m$, which does not
correspond to $\mathbf{Z}_{s}$ and this label-vector should not be transformed
into the new size because semantic label information does not correspond with
transformed data. Hence, sample sizes must still be the same, i.e. $m=n$, but
is not required by definition of NBT. We propose a data augmentation strategy
for solving different sample sizes, applied before doing knowledge transfer.
Data augmentation is common in machine or deep learning and has a variety of
applications [22, 23]. However, source and target data should have a
reasonable size to proper encode domain knowledge.

In general, there are two cases, first $n<m$, meaning there is not enough
source data. This is augmented via sampling from a class-wise multivariate
Gaussian distribution $\mathcal{N}_{d}$ harmonizing the number of samples per
class of source data. The other case is $n>m$ and is solved by uniform random
removal of source data from the largest class, i.e. $\mathcal{U}(1,|c_{max}|)$
with $|c_{max}|$ as number of class samples and $c_{max}=\\{c_{i}\mid\forall
c_{i}\>\>max(|c_{i}|)\\}$ as label with most samples, in the source set
$(\mathbf{Z},\mathbf{Y}_{Z})$. The approach reduces source data to size $m$.
This is somewhat counter-intuitive because one does not want to reduce the
source set. However, we have no class information of the target set at
training time, and we would be guessing class labels of target data when
adding new artificial samples. The data augmentation strategy is summarized as

$\\{\mathbf{Z},\mathbf{Y}_{Z}\\}=\begin{cases}\\{\mathbf{Z},\mathbf{Y}_{Z}\\}\cup\\{\mathbf{z}=\mathcal{N}_{d}(\boldsymbol{\mu}_{c_{i}},\boldsymbol{\sigma}_{c_{i}}),y=\\{c_{i}\mid\forall
c_{i}\>\>min(|c_{i}|)\\}\\}&n<m\\\
\\{\mathbf{Z},\mathbf{Y}_{Z}\\}\setminus\\{\mathbf{z},y|\mathbf{z}\in\mathbf{Z}\cap
p(\mathbf{z})=\frac{1}{|c_{max}|}\cap
y(\mathbf{z}_{i})=c_{max}\\}&m<n\end{cases}$ (27)

where
$\boldsymbol{\mu}_{c_{i}}=\frac{1}{|c_{i}|}\sum_{y(\mathbf{z}_{i})=c_{i}}\mathbf{z}_{i}$
is class-wise mean,
$\boldsymbol{\sigma}_{c_{i}}=\frac{1}{|c_{i}|}\sum_{y(\mathbf{z}_{i})=c_{i}}(\mathbf{z}_{i}-\boldsymbol{\mu}_{c_{i}})^{2}$
is class-wise variance. The function $y(\cdot)$ maps a training sample to the
ground truth label $c_{i}\in\mathcal{Y}$ and $|c_{i}|$ is the number of class
sample occurrences.

Pseudo code of Data Augmentation, NBT and PCVM summarized as NTVM is shown in
algorithm 2.

Algorithm 2 Nyström Transfer Vector Machine (NTVM)

1:$\mathbf{Z}$ as $n$ sized training; $\mathbf{X}$ as $m$ sized test set;
$\mathbf{Y}$ as $n$ sized training label vector; $s$ as number of landmarks
parameter; $\theta$ as RBF-kernel parameter.

2:Weight Vector $\mathbf{w}$; bias $b$;

3:$[\mathbf{Z}_{n},\mathbf{X}_{n}]$=standard_normalization($\mathbf{Z}$,$\mathbf{X}$)
$\triangleright$ Similar as in Fig. 2(b)

4:$\mathbf{Z}_{a},\mathbf{Y}_{a}$ =
data_augmentation($\mathbf{Z}_{n}$,$\mathbf{Y}$) $\triangleright$ According to
equation (27)

5:$\mathbf{A}_{Z}$ = matrix_decomposition($\mathbf{Z}_{a}$,$s$)
$\triangleright$ According to equation (9)

6:$\mathbf{A}_{X},\mathbf{C}_{X}$ = matrix_decomposition($\mathbf{X}_{n}$,$s$)
$\triangleright$ According to equation (9)

7:$\mathbf{S}_{Z}$ = $SVD(\mathbf{A}_{Z})$; $\triangleright$ Singular Values
of landmark matrix of $\mathbf{Z}$

8:$\mathbf{L}_{X},\mathbf{S}_{X},\mathbf{R}_{X}$ = $SVD(\mathbf{A}_{X})$;
$\triangleright$ SVD of landmark matrix of $\mathbf{X}$

9:$\tilde{\mathbf{L}}_{X}$ =
$\begin{bmatrix}\mathbf{L}_{X}&\mathbf{C}_{X}\mathbf{R}_{X}\mathbf{S}_{X}^{-1}\end{bmatrix}^{T}$
$\triangleright$ According to equation (24)

10:$\mathbf{X}_{s}=\tilde{\mathbf{L}}_{X}\mathbf{S}_{X}$ $\triangleright$
According to equation (24)

11:$\mathbf{Z}_{s}=\tilde{\mathbf{L}}_{X}\mathbf{S}_{Z}$ $\triangleright$
According to equation (25) and similar to Fig 2(c)

12:[$\mathbf{w}$,$b$] = pcvm_training($\mathbf{Z}_{s}$,$\mathbf{Y}$,$\theta$);
$\triangleright$ According to [3]

##  5 Experiments

We follow the experimental design typical for transfer learning algorithms [6,
24, 9, 2, 8]. A crucial characteristic of datasets for transfer learning is
that domains for training and testing are different but related. This relation
exists because the train and test classes have the same top category or
source. The classes themselves are subcategories or subsets. The parameters
for respective methods444Source code, parameters and datasets obtainable via
https://github.com/ChristophRaab/ntvm are determined for best performance in
terms of accuracy via grid search evaluated on source data.

###  5.1 Dataset Description

The study consists of $24$ benchmark datasets and are already preprocessed.
Reuters from [25], 20-Newsgroup from [26] and Caltech-Office from [24]. A
summary of image and text datasets is shown in table 1 and table 2. Respective
datasets are detailed in the following.

####  5.1.1 Image Datatsets

Caltech-256555https://people.eecs.berkeley.edu/j̃hoffman/domainadapt/#datasets_code\-
Office: The first, Caltech (C) is an extensive dataset of images and initially
contains of 30607 images within 257 categories. However, in this setting, only
1123 images are used to be related to the Office dataset. We adopt the
sampling scheme from [24]. The Office dataset is a collection of images drawn
from three sources, which are from Amazon (A), digital SLR camera DSLR (D) and
webcam (W). They vary regarding camera, light situation and size, but having
31 object categories, e.g. computer or printer, in common. Duplicates are
removed, as well as images, which have more than 15 similar Scale Invariant
Feature Transform (SIFT) in common.
To get an overall collection of the four image sets, which are considered as
domains, categories with the same description are taken. From the Caltech and
Office dataset, ten similar categories are extracted: backpack, touring-bike,
calculator, head-phones, computer-keyboard, laptop-101, computer-monitor,
computer mouse, coffee-mug, and projector. They are the class labels from one
to ten.
With this, a classifier should be trained in the training domain, e.g. on
projector images (Class One) from amazon (Domain A), and should be able to
classify the test image to the corresponding image category, e.g. projector
(Class One) images from Caltech (Domain C) against other image types like
head-phones (Class Two). The final feature extraction is done with Speeded Up
Robust Features Extraction (SURF) and encoded with 800-bin histograms.
Finally, the twelve combination of domain datasets are designed to be trained
and tested against each other by the ten labels [24]. An overview of the image
dataset is given in table 1.

Dataset | Subsets | #Samples | #Features | #Classes
---|---|---|---|---
Caltech-256 | Caltech (C) | 1123 | 800 | 10
Office | Amazon (A) | 958
| DSLR (D) | 295
| Webcam(W) | 157

Table 1: Dataset characteristics of image domain adaptation datasets
containing datasets and corresponding subsets, numbers of samples, features
and classes.

####  5.1.2 Text Datasets

In the following, the text datasets are discussed. The arranging of the text
domain adaption datasets is different from the image datasets. The text
datasets are structured into top categories and subcategories. These top
categories are regarded as labels and the subcategories are used for training
and testing. The variation of subcategories between training and testing
creates a transfer problem. The difference to image datasets is that at the
image datasets the (sub) -categories are labels and the difference in the top
category (source of images) between training and testing, e.g Caltech to
Amazon, creates the transfer problem. An overview of the text datasets is
given in table 2.

Reuters-21578666http://www.daviddlewis.com/resources/testcollections/reuters21578:
A collection of Reuters news-wire articles collected in 1987 with a
hierarchical structure given as top-categories and subcategories to organize
the articles. The three top categories Organization (Orgs), Places and People
are used in our experiments. The category Orgs has 56 subcategories, Places
has 176 and People has 269. In the category Places, all articles about USA are
removed making the three nearly even distributed in terms of articles.

We follow the sampling scheme from [25], which will be discussed in the
following. Note that the top categories, which are just mentioned, are the
labels of the datasets.

All subcategories of a top category are randomly divided into two parts of
subcategories with about the same number of articles. This selection is fixed
for all experiments. For a top category $O$ this creates two parts $o1$ and
$o2$ of subcategories and for another top category $P$ this creates the parts
$p1$ and $p2$. The top category $O$ is regarded as a class and $P$ as another
one.

The transfer problem is created by using $o1$ and $p1$ for training and $o2$
and $p2$ is used for testing. This is a two class problem, because of the two
top categories $O$ and $P$. Such a configuration is called $O$ vs $P$. If the
second part is used for training, i.e. $o2$ and $p2$, and the first part for
testing, i.e. $o1$ and $p1$, it is regarded as $P$ vs $O$. The individual
subcategories have different distribution but are related by the top category,
creating a change in distribution between training and testing.

Based on this six datasets are generated: Orgs vs. Places, Orgs vs. People,
People vs. Places, Places vs. Orgs, People vs. Places and Places vs. People.
The articles are converted to lower case, words are stemmed and stopwords are
removed. With the Document Frequency (DF)-Threshold of 3, the numbers of
features are cut down. The features are generated with Term-Frequency Inverse-
Document-Frequency (TFIDF). For a detailed choice of subcategories see [25].

Top Category (Names) | Subcategory | #Samples | #Features | #Classes
---|---|---|---|---
Comp | comp.graphics | 970 | 25804 | 2
| comp.os.ms-windows.misc | 963
| comp.sys.ibm.pc.hardware | 979
| comp.sys.mac.hardware | 958
Rec | rec.autos | 987
| rec.motorcycles | 993
| rec.sport.baseball | 991
| rec.sport.hokey | 997
Sci | sci.crypt | 989
| sci.electronics | 984
| sci.med | 987
| sci.space | 985
Talk | talk.politics.guns | 909
| talk.politics.mideast | 940
| talk.politics.misc | 774
| talk.religion.misc | 627
Orgs | 56 subcategories | 1237 | 4771 | 2
People | 269 subcategories | 1208
Places | 176 subcategories | 1016

Table 2: Dataset characteristics of text domain adaptation datasets containing
top-categories and corresponding subcategories, numbers of samples, features
and labels. Horizontal line separates dataset in 20-Newsgroup (upper half) and
Reuters (lower half) [6]. At reuters there are many subcategories, therefore
we only show the number of subcategories.

20-Newsgroup777http://qwone.com/~jason/20Newsgroups/: The original collection
has approximately 20000 text documents from 20 newsgroups. The four top
categories are comp, rec, talk and sci and containing four subcategories each.
We follow a data sampling scheme introduced by [6] and generate 216 cross
domain datasets based on subcategories: Again the top categories are the
labels and the sub categories are varied between training and testing to
create a transfer problem.

Let $C$ be a top category and $C1,C2,C3,C4$ are subcategories of $C$ and
another top category with $K$ and $K1,K2,K3,K4$ are subcategories of $K$. A
dataset is constructed by selecting two subcategories for each top-category,
e.g. $C1$, $C2$, $K1$, and $K2$, for training and select another four, e.g.
$C3$, $C4$, $K3$, and $K4$ for testing. The top categories $C$ and $K$ are
respective classes.

For two top categories every permutation is used and therefore $C^{2}_{4}\cdot
K^{2}_{4}=36$ datasets are generated. By combining each top category with each
other there are 216 dataset combinations. The datasets are summarized as mean
per top category combination, e.g. C vs K, which are comp vs rec, comp vs
talk, comp vs sci, rec vs sci, rec vs talk and sci vs talk. The transfer
problem is created by training and testing on different subcategories analogy
to Reuters. This version of 20-Newsgroup has 25804 TF-IDF features within
15033 documents [6].

Note to reproduce the results below, one should use the linked version of
datasets with same choice of subcategories. Regardless of dataset, features
have been normalized to standard mean and variance. The samples for training
and testing the classifiers are drawn with $5\times 2$-fold sampling scheme
suggested by [27], with a transfer learning adapted data sampling scheme as
suggested in [24].

###  5.2 Comparison of Prediction Performance

The results of the experiments are summarized in table 3 and showing mean
errors of the cross-validation study per dataset. To determine statistically
significant differences, we follow [3], using the Friedman Test [28] with a
confidence level of $5\%$ and Bonferroni-Dunn Post-Hoc correction. The $\ast$
marks statistical significance against NTVM. The PCTKVM and NTVM are compared
to baseline classifier to standard transfer learning methods and non-transfer
learning baseline methods, i.e. SVM and PCVM. The PCTKVM has overall
comparable performance to PCVM, however, is worse at Newsgroup, showing
negative transfer [1]. This should be investigated in future work.

The NTVM method has excellent performance and outperforms every other
algorithm by far. In the overall comparison, the NTVM is significantly better
compared to the other approaches, except SA.

Dataset | SVM | PCVM | TCA | JDA | TKL | SA |  | PCTKVM
---
Our Work
| NTVM
---
Our Work
Orgs vs People | 23.0 | 31.1 | 23.8 | 25.7 | 18.6 | 6.3 | 20.7 | 3.1
People vs Orgs | 21.1 | 28.1 | 20.3 | 24.9 | 13.0 | 5.7 | 23.2 | 3.1
Orgs vs Place | 30.8 | 33.4 | 28.7 | 27.4 | 22.7 | 6.4 | 33.9 | 2.6
Place vs Orgs | 35.8 | 35.7 | 34.1 | 34.1 | 17.5 | 6.6 | 40.6 | 2.4
People vs Place | 38.8 | 41.3 | 37.2 | 41.3 | 30.6 | 8.3 | 36.3 | 2.6
Place vs People | 41.3 | 41.2 | 43.4 | 43.3 | 34.0 | 11.5 | 39.7 | 2.5
Reuters Mean | 31.8 $\ast$ | 35.1 $\ast$ | 31.3 $\ast$ | 32.8 $\ast$ | 22.7 | 7.5 | 32.4 $\ast$ | 2.7
Comp vs Rec | 12.7 | 17.9 | 8.1 | 7.8 | 3.0 | 1.8 | 37.3 | 0.5
Comp vs Sci | 24.5 | 29.1 | 26.3 | 27.1 | 9.5 | 4.8 | 33.7 | 0.8
Comp vs Talk | 5.1 | 6.2 | 2.9 | 4.2 | 2.4 | 0.9 | 14.9 | 7.2
Rec vs Sci | 23.7 | 36.6 | 17.3 | 23.9 | 5.1 | 1.6 | 29.3 | 0.3
Rec vs Talk | 18.7 | 27.8 | 13.6 | 15.2 | 5.6 | 1.8 | 33.4 | 3.1
Sci vs Talk | 21.7 | 30.9 | 20.1 | 26.1 | 14.6 | 2.9 | 30.2 | 6.9
Newsgroup Mean | 17.8 $\ast$ | 24.7 $\ast$ | 14.7 | 17.4 $\ast$ | 6.7 | 2.3 | 29.8 $\ast$ | 3.1
C vs A | 48.0 | 55.9 | 49.7 | 49.0 | 49.1 | 38.0 | 52.8 | 19.9
C vs W | 53.8 | 57.0 | 55.1 | 53.5 | 53.1 | 68.1 | 55.2 | 59.5
C vs D | 62.2 | 65.9 | 65.2 | 58.9 | 59.4 | 68.3 | 60.9 | 47.4
A vs C | 54.8 | 59.5 | 53.8 | 54.6 | 53.9 | 42.4 | 56.6 | 37.5
A vs W | 61.0 | 66.1 | 58.6 | 58.8 | 57.2 | 67.5 | 59.7 | 55.6
A vs D | 62.6 | 64.8 | 66.5 | 59.2 | 59.2 | 64.9 | 61.6 | 48.9
D vs C | 68.6 | 72.5 | 62.4 | 61.1 | 65.2 | 66.1 | 68.7 | 19.9
D vs A | 69.0 | 72.4 | 65.5 | 65.8 | 64.6 | 66.2 | 67.0 | 35.6
D vs W | 40.6 | 62.5 | 28.2 | 30.1 | 27.4 | 30.7 | 45.8 | 47.9
W vs C | 65.9 | 67.8 | 62.1 | 65.3 | 63.2 | 59.4 | 63.9 | 16.1
W vs A | 67.6 | 69.0 | 63.3 | 67.0 | 63.0 | 64.7 | 68.4 | 38.5
W vs C | 23.1 | 41.1 | 21.6 | 22.1 | 27.4 | 32.5 | 45.5 | 59.8
Image Mean | 56.4 $\ast$ | 62.9 $\ast$ | 54.3 | 53.8 | 53.6 | 55.7 $\ast$ | 58.8 $\ast$ | 40.6
Overall Mean | 35.3 $\ast$ | 40.9 $\ast$ | 33.4 $\ast$ | 34.6 $\ast$ | 27.7 $\ast$ | 21.8 | 40.4 $\ast$ | 15.5

Table 3: Result of cross-validation test shown in mean error per dataset. Mean
over dataset group at the end of each section. Bold marks winner. $\ast$ marks
statistical differences with significance level of $0.05$ against NTVM. The
study shows that none of the listed algorithms is statistically significantly
better as NTVM.

Especially at Reuters, NTVM convinces with stable and best performance over
multiple datasets. Table 3 shows that NTVM is significantly better in terms of
mean at Reuters except TKL and SA.

The NTVM also outperforms most of the time at image datasets, showing the
capability to tackle multi-class problems and their independence from a
certain domain adaptation task, unlike previous work [7]. Further, in terms of
mean error on image, the NTVM outperforms SVM, PCVM, PCTKVM and SA with
statistical significant differences.

The NTVM is also very good at Newsgroup, but not that outstanding. It is
overall little worse than SA, but not statistically significant. Further, it
is best at half of the datasets and convinces with error performances under
one percent.

Note that the standard deviation is not shown, because it will not provide
more insights into the performance. It is overall very similar and small,
because the underlying classifier is the same.

(a) Office - Caltech

(b) Reuters

Figure 3: Relationship between number of landmarks and classification error as
mean over all Office vs Caltech datasets, shown in left figure, and mean over
all Reuters datasets, shown in right figure. Number of landmarks for datasets
smaller as maximum rank are determined via $min({n,d})$.

The sensitivity of the number of landmarks on prediction error as the only
parameter of NBT is demonstrated in figure 3. It shows a comparison of the
number of landmarks and the mean of classification error over Reuters and
Office - Caltech datasets. The plot indicates that the error decreases to a
global minimum with an increasing number of landmarks, which supports the
Nyström error expectation. However, further increasing to the maximum number
of landmarks, i.e. all samples, the error starts to increase. We assume this
indicates that only a subset of features is relevant for classification and
remaining features are noise. Further, this subset is drawn randomly, hence by
choosing various landmark matrices, other features becoming relevant or non-
relevant, as features correlate with certain features and again other features
correlate with others.

###  5.3 Comparison of Model Complexity

Dataset | PCVM | SVM | TCA | JDA | SA | TKL | PCTKVM | NTVM
---|---|---|---|---|---|---|---|---
Reuters(1153.66) | 49.07 | 441.78 | 168.51 | 201.87 | 100.87 | 351.21 | 1.97 | 329.51
Image(633.25) | 62.87 | 284.37 | 231.65 | 264.38 | 238.44 | 262.64 | 46.63 | 27.59
20 Newsgroup(3758.30) | 74.23 | 1247.10 | 269.75 | 252.49 | 211.57 | 1046.26 | 92.89 | 74.70
Overall Mean | 62.06 | 640.17 | 223.30 | 245.31 | 183.60 | 553.27 | 47.16 | 143.93

Table 4: Average mean number of model vectors of a classifier for Reuters,
Image and Newsgroup datasets. The average number of examples in the datasets
are shown on the right side of the dataset name in brackets.

We measured the model complexity with the number of model vectors, e.g.
support vectors. The result of our experiment is shown as mean summarizing a
dataset group in table 4. We see that the transfer learning models of the
PCTKVM provide very sparse models while having good performance. The sparsity
of NTVM is also very competitive. However, the overall sparsity is worse in
comparison to PCTKVM and PCVM. In comparison to all non-PCVM methods, the
PCTKVM outperforms the respective methods by far.

The difference in model complexity is exemplarily shown in figure 4. It shows
a sample result of classification of PCTKVM and TKL-SVM on the text dataset
Orgs vs People with the settings from above. The PCTKVM error is $21\%$ with
three model vectors and the error of TKL-SVM is $19\%$ with $339$ support
vectors.
PCTKVM achieves sustain performance by a small model complexity and provides a
way to interpret the model. Note that the algorithms are trained in the
original feature space and the models are plotted in a reduced space, using
the t-distributed stochastic neighbor embedding algorithm [29]. Note that the
Kullback-Leibler divergence of the data shown in figure 4 between input
distribution (original space) and output distribution (reduced space) is 0.92.

Figure 4: Sample run on Orgs vs People (Text dataset). Red colors for the
class Orgs and blue for the class People. This plot includes training and
testing data. Model complexity of PCTKVM on the left and SVM on the right. The
PCTKVM uses three vectors and achieves an error of $21\%$. The TKL-SVM needs
$339$ vectors and has an error of $19\%$. The black circled points are used
model vectors. Reduced with t-SNE [29]. Best viewed in color.

(a) PCTKVM

(b) TKL-SVM

###  5.4 Time Comparison

The mean time results in seconds of the cross validation study per data set
group are shown in table 5. Note that SVM and PCVM are underlying classifier
for compared approaches and are presented for the baseline and not marked as
winners in the table. They are also included in the time measurement of
transfer learning approaches. Overall the SVM is the fastest algorithm,
because it is baseline and uses the LibSVM implementation. The overall fastest
transfer learning approach is TKL, but JDA is also promising and fastest at
Reuters and Newsgroup.
The PCVM is overall by far the slowest classifier. By integration of TKL and
NBT the time performance of resulting PCTKVM and NTVM are a big magnitude
faster. We assume that the PCVM converges faster with transfer learning
resulting in less computational time. Overall NTVM is slightly faster and
lower in time at Reuters and Newsgroup in comparison to PCTKVM, which supports
the discussion about computational complexity in section 4.2.1. In comparison
to other transfer learning approaches, both approaches are slower than other
transfer learning approaches. But the reason for this should be the PCVM as
underlying classifier, because TKL is the fastest transfer approach with SVM.
Further work should aim to measure the time with same classifier to make
results more comparable.

Dataset |  | SVM
---
Baseline
TCA | JDA | TKL | SA |  |  | PCVM
---
Baseline
PCTKVM | NTVM
Reuters | 0.06 | 0.86 | 0.36 | 0.40 | 0.87 |  | 543.71 | 16.20 | 8.78
Newsgroup | 1.35 | 21.39 | 4.79 | 2.80 | 59.70 |  | 1501.7 | 5.06 | 25.29
Image | 0.02 | 0.29 | 0.16 | 0.44 | 0.08 |  | 258.78 | 17.69 | 3.41
Overall | 0.48 | 7.51 | 1.77 | 1.21 | 20.22 |  | 768.06 | 12.98 | 12.49

Table 5: Result of cross-validation test shown in mean time in seconds per
dataset group. The baseline classifiers are SVM is underlying classifier of
TCA, JDA, TKL and SA and the PCVM is underlying classifier of PCTKVM and NTVM.
Note that the SVM time is naturally lower to transfer learning approaches.

##  6 Conclusion

Summarizing, we proposed two transfer learning extensions for the PCVM,
resulting in PCTKVM and NTVM. The first shows the best overall sparsity and
comparable performance to common transfer learning approaches. The NTVM has an
outstanding performance, both in absolute values and statistical significance.
It has competitive sparsity and lowest computational complexity compared to
discussed solutions. NBT is an enhancement of previous versions of Basis
Transfer via Nyström methods and is no longer limited to specific domain
adaptation tasks. The dimensionality reduction paired with projection of
source data into the target subspace via NBT showed its reliability and
robustness in this study. Proposed solutions are tested against standard
benchmarks in the field in terms of algorithms and datasets. In future work,
deep transfer learning, different baseline classifiers and real-world or
different domain adaptation datasets should be integrated. Further, smart
sampling techniques for landmark selection should be tackled.

## References

  * [1] K. Weiss, T. M. Khoshgoftaar, D. D. Wang, A survey of transfer learning, Vol. 3, Springer International Publishing, 2016.
  * [2] S. J. Pan, Q. Yang, A Survey on Transfer Learning, IEEE Transactions on Knowledge and Data Engineering 22 (10) (2010) 1345–1359.
  * [3] H. Chen, P. Tiňo, X. Yao, Probabilistic classification vector machines, IEEE Transactions on Neural Networks 20 (6) (2009) 901–914.
  * [4] F. M. Schleif, H. Chen, P. Tino, Incremental probabilistic classification vector machine with linear costs, Proceedings of the International Joint Conference on Neural Networks 2015-September (4).
  * [5] C. Cortes, V. Vapnik, Support vector network, Machine Learning 20 (1995) 1–20.
  * [6] M. Long, J. Wang, J. Sun, P. S. Yu, Domain invariant transfer kernel learning, IEEE Transactions on Knowledge and Data Engineering 27 (6) (2015) 1519–1532.
  * [7] C. Raab, F.-M. Schleif, Sparse Transfer Classification for Text Documents, in: F. Trollmann, A.-Y. Turhan (Eds.), KI 2018: Advances in Artificial Intelligence, Springer International Publishing, 2018, pp. 169–181.
  * [8] S. J. Pan, I. W. Tsang, J. T. Kwok, Q. Yang, Domain adaptation via transfer component analysis, IEEE Transactions on Neural Networks 22 (2) (2011) 199–210.
  * [9] M. Long, J. Wang, G. Ding, J. Sun, P. S. Yu, Transfer feature learning with joint distribution adaptation, Proceedings of the IEEE International Conference on Computer Vision (2013) 2200–2207.
  * [10] B. Fernando, A. Habrard, M. Sebban, T. Tuytelaars, Unsupervised visual domain adaptation using subspace alignment, Proceedings of the IEEE International Conference on Computer Vision (2013) 2960–2967.
  * [11] W. Dai, Q. Yang, G.-R. Xue, Y. Yu, Boosting for transfer learning, Proceedings of the 24th international conference on Machine learning - ICML ’07 (2007) 193–200.
  * [12] B. Scholkopf, A. J. Smola, Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond, MIT Press, Cambridge, MA, USA, 2001\.
  * [13] H. Chen, P. Tino, X. Yao, Efficient probabilistic classification vector machine with incremental basis function selection, IEEE Transactions on Neural Networks and Learning Systems 25 (2) (2014) 356–369.
  * [14] F.-M. Schleif, A. Gisbrecht, P. Tiño, Supervised low rank indefinite kernel approximation using minimum enclosing balls, Neurocomputing 318 (2018) 213–226.
  * [15] C. K. I. Williams, M. Seeger, Using the Nyström Method to Speed Up Kernel Machines, in: T. K. Leen, T. G. Dietterich, V. Tresp (Eds.), Advances in Neural Information Processing Systems 13, MIT Press, 2001, pp. 682–688.
  * [16] K. Zhang, J. T. Kwok, Clustered nyström method for large scale manifold learning and dimension reduction, IEEE Transactions on Neural Networks 21 (10) (2010) 1576–1587.
  * [17] S. Kumar, M. Mohri, A. Talwalkar, Sampling methods for the Nyström method, Journal of Machine Learning Research 13 (2012) 981–1006.
  * [18] A. Gittens, M. W. Mahoney, Revisiting the nystrom method for improved large-scale machine learning, Journal of Machine Learning Research 17 (2016) 117:1–117:65.
  * [19] K. D. Brabanter, J. D. Brabanter, J. A. K. Suykens, B. D. Moor, Optimized fixed-size kernel models for large data sets, Computational Statistics & Data Analysis 54 (6) (2010) 1484–1504.
  * [20] A. Nemtsov, A. Averbuch, A. Schclar, Matrix Compression using the Nystroöm Method, Intelligent Data Analysis 20 (5) (2013) 997–1019.
  * [21] B. Schölkopf, A. Smola, K. Müller, Nonlinear Component Analysis as a Kernel Eigenvalue Problem, Neural Computation 10 (5) (1998) 1299–1319.
  * [22] C. Zhang, S. Bengio, M. Hardt, B. Recht, O. Vinyals, Understanding deep learning requires rethinking generalization, arXiv e-prints (2016) arXiv:1611.03530.
  * [23] S. Hauberg, O. Freifeld, A. B. L. Larsen, J. W. F. III, L. K. Hansen, Dreaming More Data: Class-dependent Distributions over Diffeomorphisms for Learned Data Augmentation, in: Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, AISTATS 2016, Cadiz, Spain, May 9-11, 2016, 2016, pp. 342–350.
  * [24] B. Gong, K. Grauman, F. Sha, Geodesic flow kernel and landmarks: kernel methods for unsupervised domain adaptation, Advances in Computer Vision and Pattern Recognition (9783319583464) (2017) 59–79.
  * [25] W. Dai, G.-R. Xue, Q. Yang, Y. Yu, Co-clustering based classification for out-of-domain documents, in: Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Jose, California, USA, August 12-15, 2007, 2007, pp. 210–219.
  * [26] M. Long, J. Wang, G. Ding, S. J. Pan, P. S. Yu, Adaptation regularization: A general framework for transfer learning, IEEE Transactions on Knowledge and Data Engineering 26 (5) (2014) 1076–1089.
  * [27] E. Alpaydm, Combined 5 $\times$ 2 cv F Test for Comparing Supervised Classification Learning Algorithms, Neural Computation 11 (8) (1999) 1885–1892.
  * [28] J. Demšar, Statistical Comparisons of Classifiers over Multiple Data Sets, Journal of Machine Learning Research 7 (2006) 1–30.
  * [29] L. van der Maaten, G. Hinton, Visualizing Data using t-SNE, Journal of Machine Learning Research 9 (2008) 2579–2605.

  *[PCVM]: Probabilistic Classification Vector Machine
  *[SVM]: Support Vector Machine
  *[TKL]: Transfer Kernel Learning
  *[BT]: Basis-Transfer
  *[TCA]: Transfer Component Analysis
  *[JDA]: Joint Distribution Adaptation
  *[SA]: Subspace Alignment
  *[EVD]: Eigen Value Decomposition
  *[SVD]: Singular Value Decomposition
  *[PCTKVM]: Probabilistic Classification Transfer Kernel Vector Machine
  *[RKHS]: Reproducing Kernel Hilbert Space
  *[PCA]: Principal Component Analysis
  *[NBT]: Nyström Basis Transfer
  *[NTVM]: Nyström Transfer Vector Machine
  *[SURF]: Speeded Up Robust Features Extraction