# Conformal Transformation of Kernels: A Geometric Perspective on Text
Classification

IOANA RĂDULESCU (LĂZĂRESCU)  Interdisciplinary Doctoral School, Transilvania
University of Braşov, Romania<EMAIL_ADDRESS>ALEXANDRA BĂICOIANU
Department of Mathematics and Computer Science, Faculty of Mathematics and
Computer Science, Transilvania University of Braşov, Romania,
<EMAIL_ADDRESS>ADELA MIHAI Corresponding author<EMAIL_ADDRESS>Interdisciplinary Doctoral School, Transilvania University of Braşov, Romania,
<EMAIL_ADDRESS>Department of Mathematics and Computer Science,
Technical University of Civil Engineering Bucharest, Romania
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

In this article we investigate the effects of conformal transformations on
kernel functions used in Support Vector Machines. Our focus lies in the task
of text document categorization, which involves assigning each document to a
particular category. We introduce a new Gaussian Cosine kernel alongside two
conformal transformations. Building upon previous studies that demonstrated
the efficacy of conformal transformations in increasing class separability on
synthetic and low-dimensional datasets, we extend this analysis to the high-
dimensional domain of text data. Our experiments, conducted on the Reuters
dataset on two types of binary classification tasks, compare the performance
of Linear, Gaussian, and Gaussian Cosine kernels against their conformally
transformed counterparts. The findings indicate that conformal transformations
can significantly improve kernel performance, particularly for sub-optimal
kernels. Specifically, improvements were observed in 60% of the tested
scenarios for the Linear kernel, 84% for the Gaussian kernel, and 80% for the
Gaussian Cosine kernel. In light of these findings, it becomes clear that
conformal transformations play a pivotal role in enhancing kernel performance,
offering substantial benefits.

Keywords and Phrases: Text classification, Support Vector Machines, Riemannian
geometry, kernels, conformal transformation.

2020 AMS Mathematics Subject Classification: 62B11, 53B12, 62H30

##  1 Introduction

The volume of information available worldwide in electronic format grows
exponentially, requiring effective organization and retrieval methods.
Therefore, text classification, the task of assigning documents to predefined
categories, finds applications across multiple domains. These include topic
detection, spam email filtering, SMS spam filtering, author identification,
web page classification and sentiment analysis [TextClassApp2,
TextClassApp4uysal2016, rill2014App5Topic, gunal2006App6SpamEmail,
idris2014App7EmailSpam, uysal2013AppSMSSpam, zhang2014AppAuthId].

A wide variety of techniques have been proposed for text classification, as it
can be seen in the extensive survey by Aggarwal and Zhai [TextClassAlgSurvey].
We will focus on the Support Vector Machine (SVM), a powerful technique for
pattern recognition [joachims1998text, VapnikSVM]. SVM is a type of machine
learning algorithm that can handle high-dimensional data. It is efficient
because its decision boundary depends only on a subset of the training
samples, called support vectors. SVM is effective for both linearly and non-
linearly separable data. By mapping input data into a higher-dimensional space
and utilizing the kernel, SVM can surpass the limitations of linear
classifiers and handle more complex or non-linear datasets. The strength of
the SVM lies in the fact that we do not need to compute the mapped patterns
explicitly, and only need to know the inner products between the mapped
patterns. This method is called the ”kernel trick” [scholkopf2000kernelTrick,
cristianini2002supportTrick].

The effectiveness of text classification significantly depends on the document
representation and the choice of kernel. This article delves into the
exploration of these aspects, focusing on a geometric perspective. We consider
representing text documents as draws from a multinomial distribution. A kernel
for the multinomial distribution was introduced by Lafferty and Lebanon in
[diffusion] based on the heat equation on a Riemannian manifold defined by the
Fisher information metric associated with a statistical model. We considered
this kernel for our study, however we found that the Diffusion kernel does not
induce a Riemannian metric. Therefore we propose a kernel that we call the
Gaussian Cosine (GC) kernel, based on the cosine distance, that exhibits the
desired property of inducing a Riemannian metric and is appropriate for the
multinomial geometry. Amari and Wu explored in [amariInfGeo, AmariWu1999,
WuAmari] the Riemannian geometry induced by a kernel and proposed an efficient
method to increase the separability of classes by a conformal transformation
of the kernel. This article applies these techniques inspired by information-
geometry to text classification. An appropriate conformal transformation of
the Linear and Gaussian kernel is proposed. Additionally, a specific
transformation dedicated to multinomial geometry is analyzed for the GC
kernel.

The remaining sections are organized as follows. In Section 2 we present the
SVM classifier, focusing on the Riemannian structure induced by a kernel
function. The conformal transformation technique that enlarges the separation
between classes is presented. The geometry of the multinomial distribution is
presented, together with an appropriate distance that will be used in the
proposed conformal transformation. In Section 3 we propose the new GC kernel
and we give proof that it is a Mercer kernel. We propose two conformal
transformations. One is based on the cosine distance and will be applied to
the Linear, Gaussian, and GC kernels. The second one is based on the geodesic
distance on the sphere and will be applied to the GC kernel. We derive the
formula for the induced Riemannian metric and for the one induced by the
conformal transformation for the GC kernel. We present the algorithm that will
be used in our experiments. In Section 4 we describe the dataset, the pre-
processing steps and the setup of our experiments. The results of the proposed
method applied on text documents categorization using the Reuters dataset are
detailed in Section 5. The accuracy of the original and modified kernels are
discussed. We end with conclusions in Section 6.

##  2 Preliminaries

###  2.1 Support Vector Machines

We consider a binary classification problem. We start from a set of training
points $\\{(x_{i},y_{i})\\}$, $i=\overline{1...n}$, where $x_{i}$ is a vector
in the input space $\mathcal{X}\subset\mathbb{R}^{n}$ and $y_{i}$ are the
labels, with $y_{i}\in\\{-1,1\\}$, corresponding to one of the two classes.
SVM classifies a point into one of the two classes according to the sign of
the decision function

$f(\bm{x})=\bm{w}\cdot\bm{x}+b.$ (1)

The parameters $\bm{w}$ and $b$ are found by maximizing the margin to the
hyperplane separating the two classes, which can be solved using Lagrange
multipliers. We refer the reader to [smola2004tutorial, burges1998tutorial]
for details. This leads to a decision function of the form

$f(\bm{x})=\sum_{i\in SV}\alpha_{i}y_{i}\bm{x}_{i}\cdot\bm{x}+b,$ (2)

with $\alpha_{i}>0$ real constants. In the above equation (2) the summation
goes over all support support vectors. For a simplified notation, we wrote
$\sum_{i\in SV}$ which means $\sum_{x_{i}\in\text{ \\{Support Vectors\\} }}$.
For linearly separable data, the points which are closest to the separating
hyperplane satisfy $f(\bm{x})=\pm 1$ and are called the support vectors. It is
important to notice the function from Equation (2) depends only on the support
vectors. The constants $\alpha_{i}$ are non-zero only for the support vectors
and zero otherwise.

When data is not linearly separable, the SVM maps the input data into a higher
dimensional space $\mathbb{R}^{m},m>n$, where $m$ can even be infinite, using
a non-linear mapping $\bm{\phi}$. In this new space, the problem becomes
linearly separable in the form of

$f(\bm{x})=\bm{w}\cdot\bm{\phi(x)}+b.$ (3)

The embedding functions $\bm{\phi}$ do not need to be computed. SVM uses a
method called the ”kernel trick” [scholkopf2000kernelTrick,
cristianini2002supportTrick] to avoid the difficulty of finding good embedding
functions. Considering the inner product after the embedding,

$K(\bm{x},\bm{x}^{\prime})=\langle\bm{\phi(x),\phi(x^{\prime})\rangle},$ (4)

the optimal output function for the SVM becomes

$f(\bm{x})=\sum\alpha_{i}y_{i}K(\bm{x}_{i},\bm{x})+b.$ (5)

When the problem is considered too costly to separate, a bound can be imposed
on the multipliers $\alpha_{i}$ by the condition $\alpha_{i}\leq C$, for some
constant $C>0$, called the soft-margin parameter. In order to express a kernel
as an inner product in the form of Equation (4), it needs to be a Mercer
kernel, for which we give the following characterization theorem (Mercer’s
Theorem [mercer1909, vapnik1999]):

######  Theorem 2.1.

Let $\mathcal{X}$ be a subset of $\mathbb{R}^{n}$,
$\mathcal{X}\subset\mathbb{R}^{n}$. The function
$k:\mathcal{X}^{2}\to\mathbb{R}$ is a Mercer kernel function (also known as
kernel function) if:

  1. 1.

it is symmetric: $k(\bm{x},\bm{y})=k(\bm{y},\bm{x})$,

  2. 2.

and its corresponding kernel matrix for any finite sequence
$S=\\{\bm{x}_{1},\bm{x}_{2},\dots,\bm{x}_{m}\\},\bm{x}_{i}\in\mathcal{X},i=\overline{1,m}$
, given by

$\bm{K}(i,j)=k(\bm{x}_{i},\bm{x}_{j}),\qquad\forall i,j\in\\{1,\dots,m\\}$ (6)

is positive semi-definite: $\bm{K}\succcurlyeq 0$.

The corresponding kernel matrix of a Mercer kernel is called a Mercer kernel
matrix.

In order to check that we have a valid Mercer kernel, we need to prove it
satisfies the properties from Theorem 2.1.

The following proposition gives some important kernel properties
[ShaweTaylorCristianini2004].

######  Proposition 2.2.

Let $\mathcal{X}$ be a subset of $\mathbb{R}^{n}$,
$\mathcal{X}\subset\mathbb{R}^{n}$. Let $k_{1}$ and $k_{2}$ be Mercer kernels
over $\mathcal{X}^{2},k_{1},k_{2}:\mathcal{X}^{2}\to\mathbb{R}$,
$a\in\mathbb{R}_{+}$ a positive constant, $f$ a real-valued function on
$\mathcal{X}$, $\bm{x}\in\mathcal{X},\bm{z}\in\mathcal{X}$. Then the following
functions are Mercer kernels:

  1. 1.

$k(\bm{x},\bm{z})=k_{1}(\bm{x},\bm{z})+k_{2}(\bm{x},\bm{z})$,

  2. 2.

$k(\bm{x},\bm{z})=ak_{1}(\bm{x},\bm{z})$,

  3. 3.

$k(\bm{x},\bm{z})=k_{1}(\bm{x},\bm{z})k_{2}(\bm{x},\bm{z})$,

  4. 4.

$k(\bm{x},\bm{z})=f(\bm{x})f(\bm{z})$,

  5. 5.

$k(\bm{x},\bm{z})=\exp\\{k_{1}(\bm{x},\bm{z})\\}$.

We give the following property that we will use in the proof that the proposed
GC kernel is a Mercer kernel.

######  Proposition 2.3.

A kernel matrix whose element from the $i$-th row and $j$-th column is given
by the inner product of a transformation applied to elements of any finite
sequence
$S=\\{\bm{x}_{1},\bm{x}_{2},\dots,\bm{x}_{m}\\},\bm{x}_{i},\bm{x}_{j}\in\mathcal{X}\subset\mathbb{R}^{n},\forall
i,j\in\\{1,\dots,m\\}$:

$\bm{K}(i,j)=\langle\phi(\bm{x}_{i}),\phi(\bm{x}_{j})\rangle$ (7)

is positive semi-definite.

###### Proof.

The matrix $\bm{K}$ is symmetric, so for any non-zero vector
$\bm{z}\in\mathbb{R}^{n}$ we have:

$\bm{z}^{T}\bm{K}\bm{z}=\langle\bm{K}\bm{z},\bm{z}\rangle=\langle\bm{A}^{T}\bm{A}\bm{z},\bm{z}\rangle=\|\bm{A}\bm{z}\|^{2}\geq
0$, so it is positive semi-definite. ∎

###  2.2 Geometry of the SVM

It has been shown by Amari and Wu in [amariInfGeo, AmariWu1999, WuAmari] that
a Riemannian metric is induced by the embedding of the original input space
$\mathbb{R}^{n}$ in a higher dimensional space $\mathbb{R}^{m}$, where $m\gg
n$ and can be infinite. This metric can be expressed in terms of the kernel as
a matrix $G=(g_{ij})$ with elements given by:

$g_{ij}(\bm{x})=\pdv*{k(\bm{x},\bm{y})}{x_{i},y_{j}}\Bigg{\rvert}_{\bm{y}=\bm{x}}$
(8)

It is also known that the volume element at point $\bm{x}$ is given by

$dV(\bm{x})=\sqrt{|g_{ij}(\bm{x})|}dx_{1}\cdots dx_{n},$ (9)

where $|g_{ij}(x)|$ is the determinant of the Riemannian matrix $(g_{ij})$.
The scaling factor $\sqrt{|g_{ij}(x)|}$, called the magnification factor in
[AmariWu1999], shows how the volume can be contracted or expanded around a
point $\bm{x}$ under the mapping $\bm{\phi}$. Since the output function of SVM
given by (5) depends only on the support vectors, Amari and Wu propose to
expand the neighborhood around support vectors with the purpose of obtaining a
better separation between samples [amariInfGeo, AmariWu1999, WuAmari]. The
idea is to modify a given kernel by

$\tilde{k}(\bm{x},\bm{y})=D(\bm{x})D(\bm{y})k(\bm{x},\bm{y})\qquad,$ (10)

called a conformal transformation of a kernel, where $D(x)$ is a positive
scalar function.

######  Proposition 2.4.

The kernel given by the conformal transformation from equation (10) is a
Mercer kernel.

###### Proof.

The proof follows immediately knowing that $k(\bm{x},\bm{y})$ is a Mercer
kernel and applying the Properties 4 and 3 from Proposition 2.2. ∎

This transformation leads to a change of the metric to

$\displaystyle\tilde{g}_{ij}(\bm{x})$
$\displaystyle=D(\bm{x})^{2}g_{ij}(\bm{x})+D_{i}(\bm{x})D_{j}(\bm{x})k(\bm{x},\bm{x})$
$\displaystyle+D(\bm{x})\\{D_{i}(\bm{x})k_{j}(\bm{x},\bm{x})+D_{j}(\bm{x})k_{i}(\bm{x},\bm{x})\\}$
(11)

where

$D_{i}(\bm{x})\overset{\mathrm{not}}{=}\pdv{}{x_{i}}D(\bm{x}),\qquad
k_{i}(\bm{x},\bm{x})\overset{\mathrm{not}}{=}\pdv{}{x_{i}}k(\bm{x},\bm{y})\Big{\rvert}_{\bm{y}=\bm{x}}.$
(12)

The function $D(\bm{x})$ needs to be chosen in such a way that it has large
values around the boundaries, and small values far from the boundary. A
problem that arises from this is that the location of the boundary is unknown
initially. The procedure proposed by Amari and Wu [AmariWu1999, WuAmari]
involves a first step of applying the SVM with a kernel to get a first
estimation of the support vectors which are known to be mostly located around
the boundary, then apply the conformal transformation to the original kernel
and apply SVM a second time with the modified kernel.

Popular choices of kernels for SVM, which we will also use in our experiments,
include the Linear kernel given by

$k_{L}(\bm{x},\bm{y})=\langle\bm{x},\bm{y}\rangle.$ (13)

and the Gaussian kernel given by

$k_{G}(\bm{x},\bm{y})=e^{-\gamma{\|\bm{x}-\bm{y}\|^{2}}}.$ (14)

###  2.3 Geometry of the Multinomial Distributions

Consider a family of probability distributions which depends on a finite
number of parameters. If the parameters vary smoothly, it can be seen as a
parameterized surface [calin, amariMethods].

######  Definition 2.5.

Denote a set of probability distributions on that depends on $n$ parameters
$\theta=\left(\theta_{1},\dots,\theta_{n}\right)\in\mathrm{\Theta}\subset\mathbb{R}^{n}$
by

$\mathcal{S}=\\{p_{\theta}=p(x;\theta)\\}.$ (15)

$\mathcal{S}$ is a subset of
$\mathcal{P}(\chi)=\left\\{f:\chi\to\mathbb{R}:f\geq
0,\displaystyle\int_{\chi}fdx=1\right\\}$. If the mapping $\theta\to
p_{\theta}$ is an immersion, then the set $\mathcal{S}$ is a statistical model
of dimension $n$.

######  Definition 2.6.

The log-likelihood function is given by

$\ell(\theta)=\ell(p_{\theta})(x)=\ln p_{\theta}(x).$ (16)

One of the concepts that plays a central role in information geometry is the
Fisher information metric which was first introduced by Rao in [Rao].

######  Definition 2.7.

The formula for the Fisher information matrix is

$g_{ij}(\theta)=\mathbf{E}\left[\frac{\partial\ell(\theta)}{\partial\theta_{i}}\cdot\frac{\partial\ell(\theta)}{\partial\theta_{j}}\right],\quad\forall
i,j\in\\{1,\dots,n\\},$ (17)

where
$\theta=(\theta_{1},\dots,\theta_{n})\in\mathrm{\Theta}\subset\mathbb{R}^{n}$
and $\mathbf{E}$ is the expectation of a random variable.

It was shown in [calin] that for any statistical model, the Fisher information
matrix is a Riemannian metric. As a consequence, the pair $(\mathcal{S},g)$ is
a Riemannian manifold. The Fisher information matrix can also be written as

$g_{ij}(\theta)=-\mathbf{E}_{\theta}[\partial{\theta_{i}}\partial{\theta_{j}}\ell(\theta)].$
(18)

We present the geometry of the multinomial probability distribution, taken
from [calin]. Consider $m$ independent, identical trials with $n$ possible
outcomes. Consider the statistical family given by equation (15), with

$p(x;\theta)=\frac{m!}{x_{1}+x_{2}+\cdots+x_{n+1}}\prod_{i=1}^{n+1}\theta_{i}^{x_{i}},$
(19)

where $\theta=(\theta_{i}),\theta_{i}\in(0,1),i=\\{1,\dots,n\\}$. The
parameter space is the $n-$dimensional simplex $\mathcal{P}_{n}$

$\mathcal{P}_{n}=\\{\theta\in\mathbb{R}^{n+1}:\sum_{i=1}^{n+1}\theta_{i}=1,\theta_{i}>0\\}.$
(20)

If we consider the transformation $z_{i}=2\sqrt{\theta_{i}}$, then
$\sum_{i=1}^{n+1}z_{i}^{2}=4$ and the statistical manifold of multinomial
probability distributions can be identified with the positive portion of the
$n$-dimensional sphere of radius $2$:

$\mathbb{S}^{n}_{2,+}=\\{z=(z_{1},\dots,z_{n+1})\in\mathbb{R}^{n+1}:\sum_{i=1}^{n+1}z_{i}^{2}=4\\}.$
(21)

Since the transformation given by $z_{i}$ is an isometry (see [diffusion]),
the geodesic distance between two distributions $\theta$ and $\theta^{\prime}$
on the multinomial can be computed as the shortest curve on
$\mathbb{S}^{n}_{2,+}$ connecting $z_{\theta}$ and $z_{\theta^{\prime}}$. The
angle $\alpha$ between unit vectors $\frac{z_{\theta}}{2}$ and
$\frac{z_{\theta^{\prime}}}{2}$ satisfies

$\cos{\alpha}=\left\langle\frac{z_{\theta}}{2},\frac{z_{\theta^{\prime}}}{2}\right\rangle.$
(22)

Since the distance on the sphere is the product between the radius and the
central angle we have from [calin]:

$d(\theta,\theta^{\prime})=2\arccos{\left(\sum_{i=1}^{n+1}\sqrt{\theta_{i}\theta^{\prime}_{i}}\right)}.$
(23)

We will use this distance in the conformal transformation that we propose in
Section 3.

##  3 Proposed Method

###  3.1 Gaussian Cosine Kernel

In the following section, we propose a kernel that we will use for
classification with SVM. This kernel is an adaptation of the Gaussian kernel
considering points lying on the $n-$dimensional sphere, which makes it more
suitable for text classification. Instead of the Euclidean distance which is
used in Gaussian kernel, we propose to use the cosine distance, which is
strongly related with the cosine similarity of two vectors [Manning2009]. The
cosine similarity is widely used in natural language processing as it provides
a measure of the similarity of two documents.

######  Definition 3.1.

Given two vectors $\bm{x},\bm{y}\in\mathbb{R}^{n}$, the cosine similarity is
defined as:

$k_{sim}(\bm{x},\bm{y})=\frac{\langle\bm{x},\bm{y}\rangle}{\|\bm{x}\|\|\bm{y}\|},$
(24)

where $\langle,\rangle$ is the inner product and $||\cdot||$ is the $L^{2}$
norm.

######  Definition 3.2.

Given two vectors $\bm{x},\bm{y}\in\mathbb{R}^{n}$, the cosine distance is
defined as:

$k_{dist}(\bm{x},\bm{y})=1-\frac{\langle\bm{x},\bm{y}\rangle}{\|\bm{x}\|\|\bm{y}\|},$
(25)

where $\langle,\rangle$ is the inner product and $||\cdot||$ is the $L^{2}$
norm.

We propose the following GC kernel:

$k_{GC}(\bm{x},\bm{y})=e^{-\gamma\left(1-\frac{\langle\bm{x},\bm{y}\rangle}{\|\bm{x}\|\|\bm{y}\|}\right)},\qquad\gamma\in\mathbb{R_{+}}.$
(26)

######  Proposition 3.3.

The GC kernel is a Mercer kernel.

###### Proof.

In order to prove this, we will first show that the cosine similarity
function, given by Equation (24) is a Mercer kernel. First, we see that the
kernel matrix $\bm{K}$ given by associated to the cosine similarity kernel
function is symmetric, from the definition of the kernel function.

$\displaystyle\bm{K}(i,j)=k_{sim}(\bm{x}_{i},\bm{x}_{j})=\frac{\langle\bm{x}_{i},\bm{x}_{j}\rangle}{\|\bm{x}_{i}\|\|\bm{x}_{j}\|}=\frac{\langle\bm{x}_{j},\bm{x}_{i}\rangle}{\|\bm{x}_{j}\|\|\bm{x}_{i}\|}=k_{sim}(\bm{x}_{j},\bm{x}_{i}).$

Next, we need to show that the matrix $\bm{K}$ is positive semi-definite. For
this, we will use Proposition 2.3, which means we need to show that the kernel
matrix for equation (24) can be written in the form of equation (7). Taking
any finite sequence
$S=\\{\bm{x}_{1},\bm{x}_{2},\dots,\bm{x}_{m}\\},\bm{x}_{i},\bm{x}_{j}\in\mathcal{X}\subset\mathbb{R}^{n},\forall
i,j\in\\{1,\dots,m\\}$, we have:

$\displaystyle\bm{K}(i,j)$
$\displaystyle=\frac{\langle\bm{x}_{i},\bm{x}_{j}\rangle}{\|\bm{x}_{i}\|\|\bm{x}_{j}\|}=\frac{\sum_{k=1}^{n}x_{ik}x_{jk}}{\|\bm{x}_{i}\|\|\bm{x}_{j}\|}=\frac{x_{i1}x_{j1}}{\|\bm{x}_{i}\|\|\bm{x}_{j}\|}+\frac{x_{i2}x_{j2}}{\|\bm{x}_{i}\|\|\bm{x}_{j}\|}+\cdots+\frac{x_{in}x_{jn}}{\|\bm{x}_{i}\|\|\bm{x}_{j}\|}$
$\displaystyle=\frac{x_{i1}}{\|\bm{x}_{i}\|}\cdot\frac{x_{j1}}{\|\bm{x}_{j}\|}+\frac{x_{i2}}{\|\bm{x}_{i}\|}\cdot\frac{x_{j2}}{\|\bm{x}_{j}\|}+\cdots+\frac{x_{in}}{\|\bm{x}_{i}\|}\cdot\frac{x_{jn}}{\|\bm{x}_{j}\|}$
$\displaystyle=\left\langle\frac{\bm{x}_{i}}{\|\bm{x}_{i}\|},\frac{\bm{x}_{j}}{\|\bm{x}_{j}\|}\right\rangle=\langle\phi(\bm{x}_{i}),\phi(\bm{x}_{j})\rangle,$

where we made the notation
$\frac{\bm{x}_{i}}{\|\bm{x}_{i}\|}\overset{\mathrm{not}}{=}\left(\frac{x_{i1}}{\|\bm{x}_{i}\|},\frac{x_{i2}}{\|\bm{x}_{i}\|},\cdots\frac{x_{in}}{\|\bm{x}_{i}\|}\right)$
and $\phi(\bm{x}_{i})=\frac{x_{i}}{\|x_{i}\|}.$ Since we wrote the kernel
matrix $\bm{K}(i,j)$ in the form of (7), from Proposition 2.3 we can conclude
that it is a positive semi-definite matrix. Since it satisfies both properties
required by Mercer’s theorem, the kernel $k_{sim}(\bm{x},\bm{y})$ is a Mercer
kernel.
The last part of the proof is a direct consequence of the kernel’s properties.
We rewrite the GC kernel in a more convenient form:

$\displaystyle k_{GC}(\bm{x},\bm{y})$
$\displaystyle=e^{-\gamma(1-k_{sim}(\bm{x},\bm{y}))}=e^{\gamma(k_{sim}(\bm{x},\bm{y})-1)}=$
$\displaystyle=e^{\gamma
k_{sim}(\bm{x},\bm{y})}\cdot\frac{1}{e^{\gamma}}=a\cdot e^{\gamma
k_{sim}(\bm{x},\bm{y})},$

where we made the notation $a=\frac{1}{e^{\gamma}},a>0$. By using properties
from Proposition 2.2 in the following order, we see that:

  * •

From property 2, we have that $\gamma k_{sim}(\bm{x},\bm{y})$ is a Mercer
kernel.

  * •

Then, using property 5, we have that $e^{\gamma k_{sim}(\bm{x},\bm{y})}$ is a
Mercer kernel.

  * •

And finally, by using property 2 again, we have that $a\cdot e^{\gamma
k_{sim}(\bm{x},\bm{y})}$ is a Mercer kernel, which ends the proof.

∎

###  3.2 Conformal Transformation

Following the idea of Amari and Wu in [AmariWu1999, WuAmari] to apply a
conformal transformation in order to improve the performance of a given
kernel, we propose using a conformal transformation for the GC kernel and
analyze its performance for text classification. We also perform experiments
with conformal transformation of the Linear and Gaussian kernels given by
Equations (13) and (14).

In order to conduct our experiments, we analyzed the conformal transformations
proposed by Amari and Wu on synthetic datasets. The first one was proposed by
Amari and Wu in 1999 in [AmariWu1999]:

$D_{1}(\bm{x})=\sum_{i\in
SV}\alpha_{i}e^{-\frac{\|\bm{x}-\bm{x}_{i}\|^{2}}{2\tau^{2}}},$ (27)

where $\tau$ is a free parameter that needs to be adjusted, $\alpha_{i}$ are
given by the output function of SVM (5) and summation runs over all support
vectors.

In 2002, Wu and Amari proposed an improved version of 27 in [WuAmari] that
replaces the unique parameter $\tau$ with a set of parameters $\\{\tau_{i}\\}$
that accounts for irregular distributions of support vectors. Also, the term
$\alpha_{i}$ was dropped to avoid double counting in the training set, leading
to:

$D_{2}(\bm{x})=\sum_{i\in
SV}e^{-\frac{\|\bm{x}-\bm{x}_{i}\|^{2}}{2\tau_{i}^{2}}},$ (28)

where

$\tau_{i}=\frac{1}{M}\sum_{j\in\\{M\subset
SV\\}}\|\bm{x}_{j}-\bm{x}_{i}\|^{2}.$ (29)

The summation in (29) runs over $M$ support vectors that are nearest to
support vector $\bm{x}_{i}$. $M$ is a parameter that needs to be adjusted.

In 2005, Williams et al. [williams2005] proposed another version:

$D_{3}(\bm{x})=e^{-\kappa f(\bm{x})^{2}},$ (30)

where $f(\bm{x})$ is given by (5). We did not consider 27, since 28 is already
an improved version of it. We analyzed the performance of 28 and 30 on
synthetic datasets separated by nonlinear boundaries. We generated datasets in
the region $[-0.5,0.5]\times[-0.5,0.5]$ with the boundary given by
$y=0.5\sin(2\pi x)$ as described in [WuAmari] and another set in
$[-1,1]\times[-1,1]$ with the boundary given by

$y=2e^{-4x^{2}}-1$ as described in [williams2005]. We aimed to reproduce the
results described in the mentioned articles. We performed $10000$ experiments
for each of the proposed transformations, each time using a different training
set of $100$ samples and a test set of $10000$ samples for (28) and $1000$
samples for (30), generated randomly and uniformly distributed, as described
in the experiments done in [WuAmari, williams2005]. The performance
improvement for the modified kernel is computed by comparing the test error
for the original model $E_{o}$ vs. the modified one $E_{m}$, averaged over
$10000$ experiments. The error decrease is expressed in percentage and is
computed as

$E_{d}=\frac{E_{o}-E_{m}}{E_{o}}\cdot 100$ (31)

Negative values indicate a decrease in performance. The conformal
transformation (28) showed superior results over transformation (30) as it can
be seen in Table 2 compared to Table 1, so we considered it for application to
text classification.

$\sigma$ |  | Error decrease
---
(%)
0.05 | 17.817
0.3 | -17.761
0.4 | -65.65
0.5 | -106.054
0.7 | -60.005
1 | -28.485
2 | -28.504
Table 1: Result of SVM with Gaussian kernel for different values of $\sigma$ using conformal transformation given by (30). $\sigma$ |  | Error decrease
---
(%)
0.05 | -9.117
0.3 | 7.707
0.4 | 6.646
0.5 | 7.626
2 | 15.528
Table 2: Result of SVM with Gaussian kernel for different values of $\sigma$
using conformal transformation given by (28).

However, the conformal transformation proposed in [WuAmari] uses the Euclidean
distance, which is inappropriate for text documents due to the high
dimensionality and sparsity of data [huang2008similarity]. We want to use a
distance that has the properties required by the conformal transformation that
it should be large only around the support vectors and should decrease with
the distance to the support vectors.

We propose the following conformal transformation:

$D_{cos}(\bm{x})=\sum_{s\in
SV}e^{-\frac{1}{\tau_{s}}\left(1-\frac{\langle\bm{x},\bm{x_{s}}\rangle}{\|\bm{x}\|\|\bm{x_{s}}\|}\right)},\quad$
(32)

with

$\tau_{s}=\frac{1}{M}\sum_{j\in\\{M\subset
SV\\}}\left(1-\frac{\langle\bm{x_{j}},\bm{x_{s}}\rangle}{\|\bm{x_{j}}\|\|\bm{x_{s}}\|}\right),$
(33)

where $s$ in (32) runs over all support vectors $\bm{x}_{s}$ and the summation
in (33) goes over $M$ support vectors which are closest to support vector
$\bm{x}_{s}$. We did not use square the distances in the GC kernel, nor in the
conformal transformation. The reason is that the cosine distance is based on
the orientation of vectors and not on magnitude. If we would use the square
cosine distance, points that are opposed (dissimilar) to a given point would
be treated the same as points that are very similar. This is not desired for
text classification.

For concise expressions, we will make the following notations for two
$n$-dimensional vectors $\bm{x},\bm{y}\in\mathcal{X}\subset\mathbb{R}^{n}$

$\displaystyle\sqrt{\bm{x}}\qquad$
$\displaystyle\overset{\mathrm{not}}{=}\left(\sqrt{x_{1}},\sqrt{x_{2}},\cdots,\sqrt{x_{n}}\right),$
$\displaystyle\langle\bm{\sqrt{x}},\bm{\sqrt{y}}\rangle$
$\displaystyle\overset{\mathrm{not}}{=}\sum_{i=1}^{n}{\sqrt{x_{i}y_{i}}}.$

For the GC kernel, in addition to (25) we also consider the geodesic distance
on the sphere given by Equation (23), so we have the conformal transformation:

$D_{arc}(\bm{x})=\sum_{s\in
SV}e^{-\frac{2\arccos{\left\langle\sqrt{\bm{x}},\sqrt{\bm{\bm{x}_{s}}}\right\rangle}}{\tau_{s}}},$
(34)

with

$\tau_{s}=\frac{1}{M}\sum_{j\in
SV}2\arccos{\left\langle\sqrt{\bm{x}_{j}},\sqrt{\bm{x}_{s}}\right\rangle}.$
(35)

Similar to the previous transformation, $s$ in (34) runs overall support
vectors $\bm{x}_{s}$ and the summation in (35) goes over $M$ support vectors
which are closest to support vector $\bm{x}_{s}$. $M$ is a free parameter that
needs to be adjusted.

###  3.3 Induced Riemannian Metric

Next, we will calculate the Riemannian Metric induced by GC kernel. We recall
that this can be calculated with the formula:

$g_{ij}(\bm{x})=\pdv*{k(\bm{x},\bm{y})}{x_{i},y_{j}}\Bigg{\rvert}_{\bm{y}=\bm{x}}.$

We make a note here that we also considered the Diffusion kernel introduced in
[diffusion] on the multinomial manifolds for our experiments, given by:

$K_{t}(\bm{x},\bm{y})=(4\pi
t)^{-\frac{n}{2}}\exp{\left(-\frac{1}{t}\arccos^{2}\left(\sum_{i=1}^{n+1}\sqrt{\bm{x}_{i}\bm{y}_{i}}\right)\right)}.$
(36)

We notice, however, that this kernel does not induce a Riemannian metric. Due
to the derivative of the $\arccos$ function and the fact that our data lies on
the multinomial and satisfies Equation (20), the second derivative of the
kernel (39) cannot be evaluated at the point $\bm{y}=\bm{x}$.
Therefore, we consider the GC kernel more appropriate for this specific type
of data lying on the multinomial manifold.

We compute the first partial derivative of the GC kernel
$k(\bm{x},\bm{y})=e^{-\gamma\left(1-\frac{\langle\bm{x},\bm{y}\rangle}{\|\bm{x}\|\|\bm{y}\|}\right)}$.

$\displaystyle\pdv{k(\bm{x},\bm{y})}{x_{i}}$ $\displaystyle=\gamma
e^{-\gamma(1-\frac{\langle\bm{x},\bm{y}\rangle}{\|\bm{x}\|\|\bm{y}\|})}\frac{y_{i}\|\bm{x}\|\|\bm{y}\|-\sum_{i}^{n}x_{i}y_{i}\|\bm{y}\|\frac{2x_{i}}{2\sqrt{\sum_{i}^{n}x_{i}^{2}}}}{\|\bm{x}\|^{2}\|\bm{y}\|^{2}}$
$\displaystyle=\gamma
k(\bm{x},\bm{y})\frac{x_{i}\|\bm{x}\|^{2}\|\bm{y}\|-x_{i}\|\bm{y}\|\langle\bm{x},\bm{y}\rangle}{\|\bm{x}\|^{3}\|\bm{y}\|^{2}}$
$\displaystyle=\gamma
k(\bm{x},\bm{y})\frac{y_{i}\|x\|^{2}-x_{i}\langle\bm{x},\bm{y}\rangle}{\|\bm{x}\|^{3}\|\bm{y}\|}$
(37)

For computing the second partial derivative, we first do a preliminary
computation and we see that:

$\displaystyle\pdv*{}{y_{j}}\left(\frac{y_{i}\|\bm{x}\|^{2}-x_{i}\langle\bm{x},\bm{y}\rangle}{\|\bm{x}\|^{3}\cdot\|\bm{y}\|}\right)=\pdv*{}{y_{j}}\left(\frac{y_{i}}{\|\bm{x}\|\|\bm{y}\|}-\frac{x_{i}\langle\bm{x},\bm{y}\rangle}{\|\bm{x}\|^{3}\|\bm{y}\|}\right)$

with

$\displaystyle\pdv*{}{y_{j}}\left(\frac{y_{i}}{\|\bm{x}\|\|\bm{y}\|}\right)$
$\displaystyle=\frac{1}{\sqrt{\sum_{i}x_{i}^{2}}}\cdot\frac{\delta_{ij}\sqrt{\sum_{i}y_{i}^{2}}-y_{i}\frac{1}{2\sqrt{\sum_{i}y_{i}^{2}}}\cdot
2y_{j}}{\sum_{i}y_{i}^{2}}$
$\displaystyle=\frac{1}{\|\bm{x}\|}\cdot\frac{\delta_{ij}\|\bm{y}\|-\frac{y_{i}y_{j}}{\|\bm{y}\|}}{\|\bm{y}\|}=\frac{1}{\|\bm{x}\|}\left(\delta_{ij}-\frac{y_{i}y_{j}}{\|\bm{y}\|^{2}}\right)$

and

$\displaystyle\pdv*{}{y_{j}}\left(\frac{x_{i}\langle\bm{x},\bm{y}\rangle}{\|\bm{x}\|^{3}\|\bm{y}\|}\right)$
$\displaystyle=\frac{x_{i}}{\|\bm{x}\|^{3}}\cdot\pdv*{}{y_{j}}\left(\frac{\sum_{i}x_{i}y_{i}}{\sqrt{\sum_{i}y_{i}^{2}}}\right)$
$\displaystyle=\frac{x_{i}}{\|\bm{x}\|^{3}}\cdot\frac{x_{j}\sqrt{\sum_{i}y_{i}^{2}}-\sum_{i}x_{i}y_{i}\frac{1}{2\sqrt{\sum_{i}y_{i}^{2}}}\cdot
2y_{j}}{\sum_{i}y_{i}^{2}}$
$\displaystyle=\frac{x_{i}}{\|\bm{x}\|^{3}}\frac{x_{j}\|\bm{y}\|^{2}-y_{j}\langle\bm{x},\bm{y}\rangle}{\|\bm{y}\|^{3}}.$

We compute the second mixed partial derivative:

$\displaystyle\pdv{k(\bm{x},\bm{y})}{x_{i},y_{j}}=\gamma$
$\displaystyle\Bigg{\\{}\gamma
k(\bm{x},\bm{y})\frac{x_{j}\|\bm{y}\|^{2}-y_{j}\langle\bm{x},\bm{y}\rangle}{\|\bm{y}\|^{3}\|\bm{x}\|}\cdot\frac{y_{i}\|\bm{x}\|^{2}-x_{i}\langle\bm{x},\bm{y}\rangle}{\|\bm{x}\|^{3}\|\bm{y}\|}+$
$\displaystyle+k(\bm{x},\bm{y})\Big{[}\frac{1}{\|\bm{x}\|}\left(\delta_{ij}-\frac{y_{i}y_{j}}{\|\bm{y}\|^{2}}\right)-\frac{x_{i}}{\|\bm{x}\|^{3}}\cdot\frac{x_{j}\|\bm{y}\|^{2}-y_{j}\langle\bm{x},\bm{y}\rangle}{\|\bm{y}\|^{3}}\Big{]}\Bigg{\\}}$

Now we need to evaluate the second derivative at $\bm{y}=\bm{x}$. We first
notice that:

$k(\bm{x},\bm{x})=e^{-\gamma(1-\frac{\langle\bm{x},\bm{x}\rangle}{\|\bm{x}\|^{2}})}=e^{0}=1.$
(38)

This leads to:

$\displaystyle
g_{ij}(\bm{x})=\pdv{k(\bm{x},\bm{y})}{x_{i},y_{j}}\Bigg{\rvert}_{\bm{y}=\bm{x}}=\frac{\gamma}{\|\bm{x}\|}\left(\delta_{ij}-\frac{x_{i}x_{j}}{\|\bm{x}\|^{2}}\right).$
(39)

###  3.4 Metric Induced by the Conformal Transformation

In order to compute the metric induced by the conformal transformation, we
first notice from Equation (3.3) that:

$\displaystyle k_{i}(\bm{x},\bm{x})=0.$ (40)

By replacing (40) and (38) in (2.2) we have:

$\displaystyle\tilde{g}_{ij}(\bm{x})$
$\displaystyle=D(\bm{x})^{2}g_{ij}(\bm{x})+D_{i}(\bm{x})D_{j}(\bm{x}).$ (41)

For the GC kernel, using the conformal transformation given by Equation (32),
we have:

$\displaystyle D_{i}(\bm{x})=\pdv{D_{cos}(\bm{x})}{x_{i}}=\sum_{s\in
SV}e^{-\frac{1}{\tau_{s}}\left(1-\frac{\langle\bm{x},\bm{x_{s}}\rangle}{\|\bm{x}\|\|\bm{x_{s}}\|}\right)}\cdot\frac{x_{s}\|\bm{\bm{x}}\|^{2}-x_{i}\langle\bm{x},\bm{x_{s}}\rangle}{\|\bm{x}\|^{3}\|\bm{x_{s}}\|}.$
(42)

Then the metric induced by the conformal transformation becomes:

$\displaystyle\tilde{g}_{ij}(\bm{x})$
$\displaystyle=D_{cos}(\bm{x})^{2}\frac{\gamma}{\|\bm{x}\|}\left(\delta_{ij}-\frac{x_{i}x_{j}}{\|\bm{x}\|^{2}}\right)+$
$\displaystyle+\sum_{s\in
SV}\frac{1}{\tau_{s}}e^{-\frac{1}{\tau_{s}}\left(1-\frac{\langle\bm{x},\bm{x_{s}}\rangle}{\|\bm{x}\|\|\bm{x_{s}}\|}\right)}\frac{x_{s}\|\bm{\bm{x}}\|^{2}-x_{i}\langle\bm{x},\bm{x_{s}}\rangle}{\|\bm{x}\|^{3}\|\bm{x_{s}}\|}\cdot\sum_{s\in
SV}\frac{1}{\tau_{s}}e^{-\frac{1}{\tau_{s}}\left(1-\frac{\langle\bm{x},\bm{x_{s}}\rangle}{\|\bm{x}\|\|\bm{x_{s}}\|}\right)}\frac{x_{s}\|\bm{\bm{x}}\|^{2}-x_{j}\langle\bm{x},\bm{x_{s}}\rangle}{\|\bm{x}\|^{3}\|\bm{x_{s}}\|}=$
$\displaystyle=\left(\sum_{s\in
SV}e^{-\frac{1}{\tau_{s}}\left(1-\frac{\langle\bm{x},\bm{x_{s}}\rangle}{\|\bm{x}\|\|\bm{x_{s}}\|}\right)}\right)^{2}\frac{\gamma}{\|\bm{x}\|}\left(\delta_{ij}-\frac{x_{i}x_{j}}{\|\bm{x}\|^{2}}\right)+$
$\displaystyle+\left(\sum_{s\in
SV}\frac{1}{\tau_{s}}e^{-\frac{1}{\tau_{s}}\left(1-\frac{\langle\bm{x},\bm{x_{s}}\rangle}{\|\bm{x}\|\|\bm{x_{s}}\|}\right)}\frac{x_{s}\|\bm{\bm{x}}\|^{2}-x_{i}\langle\bm{x},\bm{x_{s}}\rangle}{\|\bm{x}\|^{3}\|\bm{x_{s}}\|}\right)\cdot\left(\sum_{s\in
SV}\frac{1}{\tau_{s}}e^{-\frac{1}{\tau_{s}}\left(1-\frac{\langle\bm{x},\bm{x_{s}}\rangle}{\|\bm{x}\|\|\bm{x_{s}}\|}\right)}\frac{x_{s}\|\bm{\bm{x}}\|^{2}-x_{j}\langle\bm{x},\bm{x_{s}}\rangle}{\|\bm{x}\|^{3}\|\bm{x_{s}}\|}\right).$

If we consider the conformal transformation given by (34), we have:

$\displaystyle D_{i}(\bm{x})=\pdv{D_{arc}(\bm{x})}{x_{i}}=\sum_{s\in
SV}\frac{1}{\tau_{s}}e^{-\frac{2\arccos{\left\langle\sqrt{\bm{x}},\sqrt{\bm{x_{s}}}\right\rangle}}{\tau_{s}}}\frac{1}{\sqrt{1-\langle\bm{\sqrt{x}},\bm{\sqrt{x_{s}}}\rangle^{2}}}\frac{\sqrt{x_{si}}}{\sqrt{x_{i}}}$

and

$\displaystyle\tilde{g}_{ij}(\bm{x})=D_{arc}(\bm{x})^{2}\frac{\gamma}{\|\bm{x}\|}\left(\delta_{ij}-\frac{x_{i}x_{j}}{\|\bm{x}\|^{2}}\right)+$
$\displaystyle+$ $\displaystyle\sum_{s\in
SV}\frac{1}{\tau_{s}}e^{-\frac{2\arccos{\left\langle\sqrt{\bm{x}},\sqrt{\bm{x_{s}}}\right\rangle}}{\tau_{s}}}\frac{1}{\sqrt{1-\langle\bm{\sqrt{x}},\bm{\sqrt{x_{s}}}\rangle^{2}}}\frac{\sqrt{x_{si}}}{\sqrt{x_{i}}}\cdot$
$\displaystyle\cdot\sum_{s\in
SV}\frac{1}{\tau_{s}}e^{-\frac{2\arccos{\left\langle\sqrt{\bm{x}},\sqrt{\bm{x_{s}}}\right\rangle}}{\tau_{s}}}\frac{1}{\sqrt{1-\langle\bm{\sqrt{x}},\bm{\sqrt{x_{s}}}\rangle^{2}}}\frac{\sqrt{x_{sj}}}{\sqrt{x_{j}}}=$
$\displaystyle=\left(\sum_{s\in
SV}e^{-\frac{2\arccos{\left\langle\sqrt{\bm{x}},\sqrt{\bm{x_{s}}}\right\rangle}}{\tau_{s}}}\right)^{2}\frac{\gamma}{\|\bm{x}\|}\left(\delta_{ij}-\frac{x_{i}x_{j}}{\|\bm{x}\|^{2}}\right)+$
$\displaystyle+\left(\sum_{s\in
SV}\frac{1}{\tau_{s}}e^{-\frac{2\arccos{\left\langle\sqrt{\bm{x}},\sqrt{\bm{x_{s}}}\right\rangle}}{\tau_{s}}}\frac{1}{\sqrt{1-\langle\bm{\sqrt{x}},\bm{\sqrt{x_{s}}}\rangle^{2}}}\frac{\sqrt{x_{si}}}{\sqrt{x_{i}}}\right)\cdot$
$\displaystyle\cdot\left(\sum_{s\in
SV}\frac{1}{\tau_{s}}e^{-\frac{2\arccos{\left\langle\sqrt{\bm{x}},\sqrt{\bm{x_{s}}}\right\rangle}}{\tau_{s}}}\frac{1}{\sqrt{1-\langle\bm{\sqrt{x}},\bm{\sqrt{x_{s}}}\rangle^{2}}}\frac{\sqrt{x_{sj}}}{\sqrt{x_{j}}}\right).$

Using the formula for the metric induced by the conformal transformation, one
can compute the magnification factor given by (9). By comparing the
magnification factor for different kernels, one could choose the kernel that
yields the highest value, hence leading to an optimal separation between
classes. However, due to the extensive nature of these computations, we have
omitted them in this discussion. From a practical perspective, considering the
high-dimensionality of the analyzed dataset, it is more efficient to implement
and apply the conformal transformation of the kernels and assess their
performance rather than computing the magnification factor upfront.

###  3.5 Experimental Procedure

These are the three kernels that are used in the current experiments:

  * •

Linear kernel, given by Equation (13).

  * •

Gaussian kernel, given by Equation (14).

  * •

GC kernel, given by Equation (26)

The experimental procedure we propose is based on a slightly modified version
of [WuAmari], where we dropped the iterative approach.

  1. 1.

Apply SVM using the original kernel in order to extract an initial information
about the support vectors. In this experiment, we used the Linear, Gaussian,
and GC kernels, defined by equations (13), (14), (26).

  2. 2.

Modify the kernel using the conformal transformation given by Equation (32).
In addition, for the GC kernel, conformal transformation given by (34) was
used. The value for $M$ in (33) and (35) was chosen to be $M=3$, a value that
was empirically validated by Wu and Amari in [WuAmari].

  3. 3.

Apply SVM with the modified kernel.

##  4 Experiments

###  4.1 Text Documents Representation

In this section, we present the application of conformal transformation of the
Linear, Gaussian, and GC kernels to text document classification. For
representing documents we consider the ”bag of words” approach
[joachims1997BagOfWords, Zhang2010BagOfWords], a technique often used in
natural language processing and information retrieval. This implies that word
order and grammatical structure are discarded and the focus is on word
occurrence. This is a simple way of transforming text into numerical data for
processing purposes. Assuming a vocabulary $V$ of size $n+1$, a document is
represented as a sequence of words over the vocabulary. Each document can be
seen as an outcome of a multinomial distribution [MultinomialEventModel,
diffusion]. When we say a document is a draw from the multinomial
distribution, we mean that the words in the document are drawn independently
from a distribution over the vocabulary, where the probability of drawing each
word is proportional to its frequency in the document.

We denote by $x_{w}$ the number of times the word $w$ appears in the document.
Then $\\{x_{w}\\}_{w\in V}$ is the sample space of the multinomial
distribution. As indicated in [diffusion], we can embed documents in the
multinomial simplex using an embedding function
$\hat{\theta}:\mathbb{Z}_{+}^{n+1}\to\mathcal{P}_{n}$. We consider some of the
most common embeddings used in text processing and information retrieval
[salton1988termTFIDF], the term frequency (TF) and term frequency-inverse
document frequency (TFIDF). The term frequency of a term $w$ in a document $d$
is defined as the number of times $w$ appears in $d$, which we denoted by
$x_{w}$. The document frequency $df_{w}$ of a term $w$ is defined as the
number of documents in which the term $w$ appears at least once. We denote by
$N$ the total number of documents in the corpus. The inverse document
frequency $idf$ is $N/df$. For a large corpus of documents, this value can
become very large, so a log version is used. In practical applications, like
the one we use in this experiment, smoothing is applied and division by zero
is avoided by considering (see [scikit-learn]):

$\displaystyle idf_{w}=\log{\frac{N+1}{df_{w}+1}}+1.$ (43)

The term frequency-inverse document frequency (TFIDF) is defined as:

$tfidf_{w}=tf_{w}\cdot idf_{w}=x_{w}(\log{\frac{N+1}{df_{w}+1}}+1).$ (44)

In text classification, the TF and TFIDF representations are known to be
normalized to unit length using $L^{1}$ or $L^{2}$ norm. When viewed in the
context of the multinomial distribution, the observed term frequencies in the
TF or TFIDF representation can be used to estimate the probabilities of
different terms occurring within the document. The embedding of documents
using $L^{1}$ normalization is:

$\hat{\Theta}_{tf}(\bm{x})=\left(\frac{x_{1}}{\sum_{i}x_{i}},\dots,\frac{x_{n+1}}{\sum_{i}x_{i}}\right).$
(45)
$\hat{\Theta}_{tfidf}(\bm{x})=\left(\frac{x_{1}idf_{1}}{\sum_{i}x_{i}idf_{i}},\dots,\frac{x_{n+1}idf_{n+1}}{\sum_{i}x_{i}idf_{i}}\right).$
(46)

Similar formulas can be given using the $L^{2}$ norm. When using the GC
kernel, only $L^{1}$ norm will be used in order to obtain a valid embedding in
the simplex and ensure that the sum of term frequencies adds up to one, which
is a fundamental property of a probability distribution. We chose to only use
$L^{1}$ such that we can consider the mapping of the multinomial to the sphere
as presented in Section 2 and use the geodesic distance on the sphere in the
conformal transformation (34). For the Linear and Gaussian kernels, both
$L^{1}$ and $L^{2}$ will be used in the experiments. We analyze the
performance of the SVM algorithm using the original and modified kernel for
each of the three kernels. This comparison is quantified using the F1 score, a
harmonic mean of precision and recall, providing a robust metric for model
performance evaluation (see details in Subsection 4.4).

###  4.2 Dataset

In our experiments, we used the Reuters-21578 dataset 111Available at
http://www.nltk.org/nltk_data, which has become a benchmark in text
classification tasks. This is a collection of $10788$ news documents, having
$1.3$ million words in total. They are categorized into more than $135$
topics, but some of the topics contain less than $100$ documents. In this
experiment we chose only the topics with more than $500$ documents, as
indicated in [diffusion]. Those topics are ”earn”, ”acq”, ”money-Fx”, ”grain”
and ”crude” having $3964$, $2369$, $717$, $582$ and $578$ documents,
respectively.

This dataset is available through the Natural Language Toolkit (NLTK), a
platform designed for building Python programs to work with human language
data. It offers easy to use interfaces to $50$ text corpora along with
libraries for text parsing, stemming, processing and classification 222
https://www.nltk.org/.

We conducted two binary classification experiments. For the first experiment
we labeled one topic as the ”positive” topic and all the remaining topics as
”negative”, with the intention of distinguishing one topic against the others.
For the second experiment we labeled one topic as ”positive” and chose another
topic as the ”negative” example, and we did this for all pairs of two topics.
We did not reverse the classes, meaning if we chose topic $A$ to be positive
and $B$ to be negative, we did not repeat the experiment with $A$ as negative
and $B$ as positive.

###  4.3 Data preparation

The role of data preparation in a classification process is crucial for
ensuring the quality and efficiency of the results. Therefore, before training
the SVM classifiers, we applied some common preprocessing steps on the
documents in order to reduce the dimensionality of data and improve
classification accuracy [TextProcessing]. The manner in which preprocessing is
conducted is intricately influenced by the specific attributes, content, and
structure of the dataset under consideration. The following steps are suitable
for the Reuters dataset and help remove noise and irrelevant information from
the dataset:

  * •

We converted all words to lowercase.

  * •

We removed stop-words like: ”a”, ”and”, ”the”. These are words that have a
high frequency in documents and do not bring much information in
classification tasks.

  * •

We considered valid words those that contain only letters, so we removed all
words that contain other characters.

  * •

We removed all words smaller than three letters. Words like ”to”, ”be”, ”on”
are common in most of the documents in our dataset and bring no useful
information for the classification task.

  * •

We applied stemming, a common technique in text processing, that removed
suffixes or prefixes from words, keeping only their root form. We used
PortStemmer as stemming tool [willett2006PortStem], offered by NLTK333
https://www.nltk.org/howto/stem.html.

  * •

We discarded all documents that became empty after the above steps.

After this data preparation step we ended up with $8167$ documents and a
dictionary containing more than $16000$ words.

###  4.4 Methodology

The setup for this experiment was inspired by [diffusion]. In this article, we
employed the Support Vector Machine (SVM) algorithm for the classification
task. The SVM model was implemented using the scikit-learn library [scikit-
learn], a popular machine learning library in Python. In our study, we split
the dataset into training and test subsets using a stratified $k$-fold cross-
validation strategy, where we empirically chose $k=20$, as suggested in
[diffusion]. This strategy ensured that the ratio of positive to negative
instances was preserved across all folds in both the training and test sets.

In our experiments we had to tune several hyper-parameters: the soft-margin
$C$ parameter had to be adjusted in all experiments, since it is a parameter
of the SVM algorithm that controls the trade-off between the training error
and the margin. We tested $C$ in the set $\\{1,10,100,1000\\}$, keeping the
same value for the original and modified kernel in order to make sure we only
measure the influence of the conformal transformation. For the Gaussian and GC
kernels we tested the $\gamma$ parameter in the set
$\\{0.0001,0.001,0.01,0.1\\}$. For all three kernels we tested both TF and
TFIDF representations. We used $L^{1}$ and $L^{2}$ norms for the Linear and
Gaussian kernels. For the GC kernel we tested only the $L^{1}$ norm, since
only this norm ensures a valid embedding into the probability simplex as
mentioned in [diffusion].

We used the TfidfVectorizer from the skikit-learn library with the use_idf
parameter on false, respectively on true for obtaining the TF and TFIDF
representation. TfidfVectorizer also does normalization and allows specifying
the norm. We left all other parameters of TfidfVectorizer at their default
settings.

We evaluate a model’s performance by looking at the F1 score, rather than the
accuracy, since there is an imbalance in the distribution of positive and
negative examples in the dataset [goutte2005F1]. The F1 score is an adequate
measure in this case. Labeling all examples as negative would lead to a high
accuracy score, which would distort the perception of the model’s performance.
We recall the formula for the F1 score:

$F1=\frac{2P\cdot R}{P+R},$ (47)

where

$\displaystyle P=\frac{{TP}}{{TP+FP}},\quad R=\frac{TP}{TP+FN},$ (48)

where $P$ is the precision, $R$ is the recall, $TP$ represents the number of
true positive samples and $FP$ (resp. $FN$) represents the number of false
positive (resp. false negative) samples identified by the model. The F1 score
is computed as the mean value over the $20$-fold cross-validation.

We have also documented the $p-$value statistic for the F1 score to emphasize
where the change in performance is statistically significant according to the
paired $t$-test at the $0.05$ level. If the $p-$value is less than $0.05$, we
consider that the improvement in accuracy is statistically significant.
Another interesting property to notice is the number of support vectors in the
original versus the modified kernel model. A smaller number of support vectors
leads to simpler models and faster prediction times [NrSupVec]. A simpler
model is also known to generalize better and reduce the chances of over-
fitting. We consider an improvement the cases where the decrease F1 is not
statistically significant but we see a decrease in the number of support
vectors. This type of compromise between accuracy and model complexity can be
acceptable in some scenarios. This combination of metrics suggests a nuanced
evaluation where the overall impact on model effectiveness needs to be
carefully assessed, taking into account factors such as computational
efficiency, interpretability, and the specific requirements of the problem
domain.

##  5 Results Analysis and Interpretation

Using the subset of the Reuters dataset and the combination of parameters
described in Section 4 we present the results for one-vs.-rest and the one-
vs.-one binary classification tasks. We compare the results of SVM using the
Linear, Gaussian and GC kernels with the modified kernels obtained by
transformations Dcos (32) and Darc (34) using the algorithm described in the
Section 3.5. We consider an improvement in the following cases:

  * •

F1 metric has a higher value for the modified model compared to the original
model.

  * •

F1 metric for the modified kernel has a smaller value compared to the original
kernel, but it is statistically non-significant ($p-$value $\geq 0.05$).
Additionally, the number of support vectors in the modified model is smaller
compared to the original model.

###  5.1 One vs. Rest Task

For this task, we compared the performance of the original Linear, Gaussian
and GC kernels with their modified counterparts by the conformal
transformations Dcos (32) and Darc (34) for GC only, when employed in the SVM
algorithm. For the Linear kernel (13), compared to the modified kernel
obtained by transformation Dcos (32), we notice that the procedure is
especially successful in the cases where the original kernel does not perform
well. For this kernel, we noticed an increase in accuracy, measured by the F1
metric, in $38\%$ of the studied scenarios. If we also consider the cases
where the F1 metric decreased, but not significantly ($p-$value $\geq 0.05$),
and the number of support vectors is reduced, we see an improvement in $61\%$
of the cases. For the Gaussian kernel (14), compared to the modified kernel
obtained by transformation $D_{cos}$ (32), we notice an increase in accuracy
in $71\%$ of the tested scenarios. If we consider also the cases where the F1
metric decreased but not significantly, and the number of support vectors is
reduced, we see an improvement in $84\%$ of the cases.

For the GC kernel (26) we only considered $L^{1}$ normalization in order to
obtain a valid embedding in the simplex and to compare the results using the
conformal transformations given by Equations (32) and (34). For both conformal
transformations, in $62.5\%$ of the tested scenarios, we noticed an increase
in accuracy, measured by the F1 metric. If we also consider the cases where
the F1 metric decreased but not significantly, and the number of support
vectors is reduced, we see an improvement in $78\%$ of the cases for (32) and
$75\%$ of the cases for (34). These findings are summarized in Table 3. In the
Accuracy Increase column, we present the percentage of cases where an increase
in accuracy using the F1 metric was observed. In the Efficiency Increase
column besides an increase in F1 metric we additionally consider the cases
where a non-significant decrease in F1 was observed accompanied by a decrease
in the number of support vectors.

Model |  | Accuracy
---
Increase
(% of tested
scenarios)
| Efficiency
---
Increase
(% of tested
scenarios)
| Linear
---
with Dcos
38% | 61%
| Gauss
---
with Dcos
72% | 84%
| GC
---
with Dcos
62.5% | 78%
| GC
---
with Darc
62.5% | 75%
Table 3: Percentage of tested scenarios where an improvement was observed in
the conformally transformed kernel compared to the original kernel.

Discussion and Implications

For the one-vs.-rest task, the conformal transformation technique proves
effective for kernels with poor performance. The technique has the biggest
impact on the Gaussian kernel, improving the performance in $72\%$ of the
tested scenarios by increasing accuracy or $84\%$ of the cases by increasing
model efficiency.

If we consider the original kernels’ best accuracy (maximum F1 values), we see
that applying the conformal transformation did not lead to further improvement
on any of the kernels. This is a contradiction with the original experiments
performed by Amari and Wu in [AmariWu1999, WuAmari], where they first selected
the best original model and then applied the conformal transformation on it
which led to further improvement. A possible explanation for this might be the
different dataset properties. In [AmariWu1999, WuAmari] the authors used a
balanced dataset for training. In [WuAmari] we see the number of training
samples ($100$) was less than the number of test samples ($10000$), while in
the current experiment, the number of training samples is $19$ times higher
than the number of test samples, due to the chosen stratified $k-$fold
validation technique. We present these results in Table 4. These results are
the maximum values obtained for the F1 metric for each of the original
kernels. These results were obtained using the TF representation. The
conformal transformation could not further improve the F1 metric, it only
reduced the number of support vectors. However, the decrease in accuracy in
these cases is statistically significant, indicated by the $p-$values, which
are $<0.05$.

Model | Task | Norm | $\gamma$ | C | F1 | #S.V. |  P-value for F1
---|---|---|---|---|---|---|---
original | custom | original | custom
| Linear
---
orig. vs. Dcos
| earn
---
vs. rest
$L^{2}$ | - | 1 | 0.98340 | 0.97732 | 876 | 820 | 0.018809
| Gaussian
---
orig. vs. Dcos
| money-fx
---
vs. rest
$L^{2}$ | 0.001 | 1000 | 0.97706 | 0.96351 | 416 | 378 | 0.000201
| GC
---
orig. vs. Dcos
| earn
---
vs. rest
$L^{1}$ | 0.001 | 1000 | 0.98340 | 0.97757 | 875 | 819 | 0.020046
| GC
---
orig. vs. Darc
| earn
---
vs. rest
$L^{1}$ | 0.001 | 1000 | 0.98340 | 0.97722 | 875 | 819 | 0.014901

Table 4: Maximum values obtained for the F1 metric for each of the original
kernels.

Analyzing the best accuracy of all transformed models, we notice that for $3$
out of $7$ models, we have an increase in F1 compared to the original kernels.
In $4$ out of $7$ models we have a decrease in F1 which is not statistically
significant. In $2$ out of the latter $4$, we see a decrease in the number of
support vectors. These results are summarized in Table 5. Results from this
table were obtained using the TF representation and $L^{1}$ norm. The
$p-$values which are less than $0.05$ are marked in bold, showing the cases
where the change in the F1 metric is statistically significant.

Model | Task | $\gamma$ | C | F1 | #S.V. |  P-value for F1
---|---|---|---|---|---|---
original | custom | original | custom
| Linear
---
orig. vs. Dcos
| earn
---
vs. rest
- | 10 | 0.97555 | 0.97751 | 1081 | 775 | 0.40255
| Gaussian
---
orig. vs. Dcos
| grain
---
vs. rest
0.0001 | 10 | 0.00000 | 0.98101 | 1062 | 388 | 1.98E-32
| GC
---
orig. vs. Dcos
| earn
---
vs. rest
0.1 | 1 | 0.96925 | 0.97874 | 1716 | 872 | 0.001473
| GC
---
orig. vs. Dcos
| earn
---
vs. rest
0.1 | 10 | 0.98263 | 0.97874 | 903 | 879 | 0.084303
| GC
---
orig. vs. Dcos
| earn
---
vs. rest
0.1 | 100 | 0.97980 | 0.97874 | 848 | 882 | 0.080378
| GC
---
orig. vs. Dcos
| earn
---
vs. rest
0.1 | 1000 | 0.97876 | 0.97874 | 870 | 884 | 0.971844
| GC
---
orig. vs. Darc
| earn
---
vs. rest
0.1 | 10 | 0.98263 | 0.97851 | 903 | 879 | 0.070112

Table 5: Maximum values obtained for the F1 metric for each of the transformed
kernels.

Considering all the analyzed models, the best performance overall for the one-
vs.-rest task is achieved by the Linear and GC kernels with an F1 score of
$0.98340$, in their original versions without any transformation applied. See
Table 6. In bold we marked the highest F1 value across all models. The F1
score is the same for the Linear original and GC original models.

Model | F1
---|---
Linear orig. | 0.98340
Gaussian orig. | 0.97706
GC orig. | 0.98340
Linear Dcos | 0.97751
Gauss Dcos | 0.98101
GC Dcos | 0.97874
GC Darc | 0.97851
Table 6: Maximum value for F1 metric of each kernel for the one-vs.-rest task.

An important result that we want to emphasize is the scenarios where we
obtained the highest increase in performance for each kernel. These results
confirm previous results obtained by Amari and Wu in [AmariWu1999, WuAmari]
that show the efficiency of the conformal transformation technique in
improving a bad kernel. These results can be seen in table 7. These results
were obtained using the $L^{1}$ norm.

Model | Task | TFIDF | $\gamma$ | C | F1 | #S.V. |  P-value for F1
---|---|---|---|---|---|---|---
original | custom | original | custom
| Linear
---
orig. vs. Dcos
| grain
---
vs. rest
TRUE | - | 1 | 0.58656 | 0.96701 | 1185 | 638 | 8.05E-10
| Gaussian
---
orig. vs. Dcos
| grain
---
vs. rest
FALSE | 0.0001 | 10 | 0.00000 | 0.98101 | 1062 | 388 | 1.98E-32
| GC
---
orig. vs. Dcos
| earn
---
vs. rest
FALSE | 0.0001 | 1 | 0.00000 | 0.97820 | 7410 | 831 | 3.79E-37
| GC
---
orig. vs. Darc
| earn
---
vs. rest
FALSE | 0.0001 | 1 | 0.00000 | 0.97721 | 7410 | 816 | 3.19E-37

Table 7: Highest increase in performance obtained by the conformal
transformation for each kernel.

The outcome of our study suggests that the GC kernel is on par with the Linear
kernel in terms of performance for this specific task. The Linear kernel is
well known to achieve good performance for high-dimensional text
classification problems because most of them are linearly separable
[joachims1998text, hasan2022classificationF1Score]. While the conformal
transformation method can enhance a kernel’s accuracy, it does not outperform
well-known kernels used in machine learning. In the cases where an increase in
performance was observed, there was an imbalance in the number of positive and
negative examples, with the number of positive examples being fewer. For
example, considering the ”grain” topic as the positive label, there are about
13 times more negative samples. For this topic, the Linear and Gaussian showed
poor accuracy in some cases and could be significantly improved by the
conformal transformation. This shows that the technique might prove effective
for imbalanced datasets with a small number of positive samples for this
particular task.

###  5.2 One vs. One Task

For this task, we compared the performance of the SVM classification algorithm
using the Linear, Gaussian and GC kernels and their transformed counterparts.
We notice again that the procedure is especially successful in the cases where
the original kernel does not perform well. For the Linear kernel, considering
the Dcos transformation given by (32), we see an increase in F1 metric in
$42\%$ of the cases. If we also consider the cases where we have a non-
significant decrease in F1 ($p-$value $\geq 0.05$) and a decrease in the
number of support vectors, we see an improvement in $60\%$ of the cases. For
the Gaussian kernel with Dcos transformation given by (32) applied we notice
an increase of the F1 score in $70\%$ of the cases. If we also consider the
cases where we have a non-significant decrease in F1 and a decrease in the
number of support vectors, we see an improvement in $84\%$ of the cases.

For the GC kernel using transformations Dcos and Darc given by Equations (32)
and (34) we noticed an increase of the F1 score in $64\%$ of the cases, for
both transformations. If we also consider the cases where we have a non-
significant decrease in F1 and a decrease in the number of support vectors, we
see an improvement in $81\%$ and $80\%$ of the cases for Dcos and Darc
respectively. A summary of these results can be seen in Table 8. In the
Accuracy Increase column we present the percentage of cases where an increase
of accuracy using F1 metric was observed. In the Efficiency Increase column
besides an increase in F1 metric we additionally consider the cases where a
non-significant decrease in F1 was observed accompanied by a decrease in the
number of support vectors.

Model |  | Accuracy
---
Increase
(% of tested
scenarios)
| Efficiency
---
Increase
(% of tested
scenarios)
| Linear
---
with Dcos
42% | 60%
| Gauss
---
with Dcos
70% | 84%
| GC
---
with Dcos
64% | 81%
| GC
---
with Darc
64% | 80%
Table 8: Percentage of tested scenarios where an improvement was observed in
the conformally transformed kernel compared to the original kernel.

Discussion and Implications

For the one-vs.-one task, we see that the conformal transformation technique
proves effective for kernels with a bad performance. It has the biggest impact
on the Gaussian kernel, improving accuracy in $70\%$ of the tested scenarios
and model efficiency in $84\%$ of the tested scenarios as seen in Table 8.

If we consider the best F1 accuracy over all original models we see that these
could not be further improved by the conformal transformation technique for
any of the kernels. Again, this contradicts the original experiments performed
in [AmariWu1999, WuAmari]. Similar to the one-vs.-rest task, this could be
explained by a difference in the setup of the original experiment: a more
balanced dataset and fewer training samples compared to the test samples.

The current experiment achieved the best performance for the ”acq vs money-fx”
task for all analyzed original kernels. For this combination of topics, the
number of positive samples is about three times higher than the number of
negative samples. This could indicate that the technique is not appropriate
for this particular task for datasets where the positive samples outnumber the
negative samples. We notice that for the Linear kernel, the performance
obtained by the conformal transformation matches the one of the original
kernel but with a higher number of support vectors. For the Gaussian and GC
kernels we notice a negligible degradation in accuracy, not statistically
significant and a decrease in the number of support vectors which can be seen
as an improvement in model efficiency. An overview is presented in Table 9.
Results from this table were obtained using $L^{1}$ norm and $C=1000$.

Model | Task | TFIDF | $\gamma$ | F1 | #S.V. |  P-value for F1
---|---|---|---|---|---|---
original | custom | original | custom
| Linear
---
orig. vs. Dcos
| acq
---
vs. money-fx
TRUE | 1 | 0.99874 | 0.99874 | 477 | 488 | -
| Gaussian
---
orig. vs. Dcos
| acq
---
vs. money-fx
FALSE | 0.01 | 0.99916 | 0.99852 | 360 | 278 | 0.082822
| GC
---
orig. vs. Dcos
| acq
---
vs. money-fx
FALSE | 0.001 | 0.99832 | 0.99831 | 321 | 299 | 0.990841
| GC
---
orig. vs. Darc
| acq
---
vs. money-fx
FALSE | 0.001 | 0.99832 | 0.99831 | 321 | 306 | 0.995401

Table 9: Maximum values of F1 metric for each original kernel.

Analyzing the best F1 score of all transformed kernels, we notice in $6$ out
of $8$ cases an improvement in F1 metric compared to the original kernel. Out
of these $6$, $3$ are statistically significant. In the remaining $2$ out of
the $8$ scenarios, we notice a non-significant decrease in F1 but a smaller
number of support vectors, which could still indicate an improvement in model
efficiency. These results are summarized in Table 10. Results from this table
were obtained using the $L^{1}$ norm.

Model | Task | TFIDF | Gamma | C | F1 | #S.V. |  P-value for F1
---|---|---|---|---|---|---|---
original | custom | original | custom
| Linear
---
orig. vs. Dcos
| acq vs.
---
money-fx
TRUE | 1 | 1 | 0.93697 | 0.99874 | 1196 | 482 | 4.08E-13
| Linear
---
orig. vs. Dcos
| acq vs.
---
money-fx
TRUE | 1 | 10 | 0.99726 | 0.99874 | 667 | 484 | 0.110155
| Linear
---
orig. vs. Dcos
| acq vs.
---
money-fx
TRUE | 1 | 100 | 0.99852 | 0.99874 | 479 | 487 | 0.329877
| Gaussian
---
orig. vs. Dcos
| acq vs.
---
money-fx
FALSE | 0.1 | 1000 | 0.99873 | 0.99895 | 276 | 284 | 0.579204
| GC
---
orig. vs. Dcos
| acq vs.
---
money-fx
FALSE | 0.001 | 1000 | 0.99832 | 0.99831 | 321 | 299 | 0.990841
| GC
---
orig. vs. Darc
| acq vs.
---
money-fx
FALSE | 0.001 | 1000 | 0.99832 | 0.99831 | 321 | 306 | 0.995401
| GC
---
orig. vs. Darc
| acq vs.
---
money-fx
FALSE | 0.01 | 1 | 0.87619 | 0.99831 | 1275 | 302 | 8.80E-26
| GC
---
orig. vs. Darc
| acq vs.
---
money-fx
FALSE | 0.01 | 10 | 0.99517 | 0.99831 | 710 | 301 | 0.001578

Table 10: Maximum values of F1 metric for each transformed kernel.

The best overall F1 score for the one-vs.-one task was achieved by the
Gaussian original kernel, at $0.99916$. This could not be surpassed by the
conformal transformation, as we can see in Table 11. In bold we marked the
highest F1 value across all models.

Model | F1
---|---
Linear orig. | 0.99874
Gaussian orig. | 0.99916
GC orig. | 0.99832
Linear Dcos | 0.99874
Gauss Dcos | 0.99895
GC Dcos | 0.99831
GC Darc | 0.99831
Table 11: Maximum value of F1 metric for each kernel for the one-vs.-one task.

An important result that we want to emphasize is the scenarios where we
obtained the highest increase in performance for each kernel. These results
confirm previous results obtained by Amari and Wu in [AmariWu1999, WuAmari]
that show the efficiency of the conformal transformation technique in
improving a bad kernel. These results can be seen in Table 12.

Model | Task | Norm | TFIDF | $\gamma$ | C | F1 | #S.V. |  P-value for F1
---|---|---|---|---|---|---|---|---
original | custom | original | custom
| Linear
---
orig. vs. Dcos
| grain
---
vs. earn
$L^{2}$ | TRUE | - | 1 | 0.67719 | 0.99737 | 1024 | 416 | 1.10E-10
| Gaussian
---
orig. vs. Dcos
| grain
---
vs. earn
$L^{1}$ | FALSE | 0.0001 | 100 | 0.00000 | 0.99743 | 1072 | 232 | 1.70E-43
| GC
---
orig. vs. Dcos
| crude
---
vs. money-fx
$L^{1}$ | FALSE | 0.0001 | 1 | 0.00000 | 0.99646 | 1064 | 205 | 2.80E-42
| GC
---
orig. vs. Darc
| money-fx
---
vs. earn
$L^{1}$ | FALSE | 0.0001 | 1 | 0.00000 | 0.99611 | 1250 | 248 | 8.42E-39

Table 12: Highest increase in F1 metric obtained by the conformal
transformation of each kernel.

The outcome of our study suggests that the Gaussian kernel without the
conformal transformation achieved the best performance for this specific task.
While the conformal transformation method can enhance a kernel’s accuracy, it
does not outperform the Gaussian kernel which is a well known for its
flexibility and ability to handle non-linear data. For the topic combinations
where the highest increase of performance was observed, ”grain vs. earn”,
”crude vs. money-fx” and ”money-fx vs. earn” we see that the number of
positive samples is smaller than the number of negative samples (up to five
times), indicating that this technique can be useful for this particular task
when dealing with this type of imbalanced data.

##  6 Conclusions and Future Work

In this article we have studied the effect of conformal transformation of
kernels for two types of binary classification tasks, one-vs.-rest and one-
vs.-one. We analyzed the performance of this technique for some well known
kernels, the Linear and Gaussian kernels using the cosine distance in the
conformal transformation which is appropriate for text documents. By
considering text documents as points on a multinomial statistical manifold
that can be isometrically mapped on the sphere, we proposed a new GC kernel
and a conformal transformation using the geodesic distance on the sphere. The
proposed method is inspired by information geometry. These experiments show
that this type of transformation can increase the performance of a given
kernel, for both types of binary classification tasks especially if the
original kernel performs badly. The procedure proves effective when applied to
the Linear, Gaussian and GC kernels, increasing or matching the performance of
the original kernel in $60\%$ of cases for the Linear kernel, $84\%$ for the
Gaussian kernel and $78\%$ for the GC kernel. However, the best performance
achieved by the transformed kernels could not surpass the best performance
achieved by the original kernels.

Since there is no general solution for all classification tasks and the best
solution depends on the problem to solve and the dataset properties, we want
to note that the proposed method has both advantages and drawbacks. The
primary benefit lies in its ability to shorten the time needed for
hyperparameter tuning and the search for the optimal kernel, bypassing time-
consuming techniques such as cross-validation. With this method, one can begin
experimenting with any kernel, even those that may initially perform poorly,
as the technique has the capacity to ”correct” them. In fact we have showed
the method works especially well on bad kernels. There is no need for extra
tuning of parameters compared to the original kernel. Another advantage is
that it performs well on imbalanced datasets where negative samples outnumber
positive ones. As disadvantages, we observe the computational time. The method
requires a first pass using any kernel, followed by a second step with the
transformed kernel, which will increase the computational time. Another
disadvantage is the complexity. It is a custom kernel, so it needs to be
implemented as opposed to using an already available ”off-the-shelf” solution.
From the experiments on both synthetic and real datasets, the technique seems
more effective on lower-dimensional non-linear data than on high-dimensional
data like text documents, for which mapping into an even higher dimensional
space does not necessarily yield substantial performance gains. As a
consequence, it could not exceed the performance of already established
benchmarks set by traditional kernels, like the Linear or Gaussian kernels in
the context of text documents classification.

A potential future direction would be to consider other representation schemes
for the documents that capture the semantic meaning of words or phrases. An
overview of such techniques can be found in [pal2018semantic]. We can explore
more the geometry of the multinomial statistical manifold and consider the use
of the Hellinger distance in the conformal transformation, which is related to
the geodesic distance (23) by Equation 49.

$d_{H}(\bm{\theta},\bm{\theta}^{\prime})=2\sin{d(\bm{\theta},\bm{\theta}^{\prime})/4}$
(49)

and when $\bm{\theta}$ and $\bm{\theta}^{\prime}$ are close,
$d_{H}(\bm{\theta},\bm{\theta}^{\prime})\approx\frac{1}{2}{d(\bm{\theta},\bm{\theta}^{\prime})}$
[diffusion, calin]. Another possibility is to consider the averaged Kullback-
Leibler divergence that gives a symmetric measure that can be used as a
distance [huang2008similarity]. The use of the Diffusion kernel on the
hypersphere could be further explored in the approximated form given in
[diffusion] or [zhang2005text], or using the exact formula proposed in
[zhao2018exact].

The classification results obtained in the current study on the Reuters
dataset partially confirm the findings reported by Amari and Wu [AmariWu1999,
WuAmari]. While these earlier studies, conducted on synthetic and lower-
dimensional data, demonstrated that conformal transformations can
significantly enhance the performance of a kernel, our experiments indicate
that the efficacy of such transformations is somewhat dependent on the nature
of the dataset. Specifically, when applied to the high-dimensional and sparse
data characteristic of the Reuters corpus, the improvements provided by
conformal transformations were less pronounced than those observed in lower-
dimensional settings. This suggests that while conformal transformations
remain a powerful tool for kernel enhancement, their impact may be moderated
by factors intrinsic to the data, such as dimensionality and sparsity.

## References