# CO-SNE: Dimensionality Reduction and Visualization for Hyperbolic Data

Yunhui Guo Haoran Guo Stella X. Yu
UC Berkeley / ICSI

###### Abstract

Hyperbolic space can naturally embed hierarchies that often exist in real-
world data and semantics. While high-dimensional hyperbolic embeddings lead to
better representations, most hyperbolic models utilize low-dimensional
embeddings, due to non-trivial optimization and visualization of high-
dimensional hyperbolic data.

We propose CO-SNE, which extends the Euclidean space visualization tool,
t-SNE, to hyperbolic space. Like t-SNE, it converts distances between data
points to joint probabilities and tries to minimize the Kullback-Leibler
divergence between the joint probabilities of high-dimensional data $X$ and
low-dimensional embedding $Y$. However, unlike Euclidean space, hyperbolic
space is inhomogeneous: A volume could contain a lot more points at a location
far from the origin. CO-SNE thus uses hyperbolic normal distributions for $X$
and hyperbolic Cauchy instead of t-SNE’s Student’s t-distribution for $Y$, and
it additionally seeks to preserve $X$’s individual distances to the Origin in
$Y$.

We apply CO-SNE to naturally hyperbolic data and supervisedly learned
hyperbolic features. Our results demonstrate that CO-SNE deflates high-
dimensional hyperbolic data into a low-dimensional space without losing their
hyperbolic characteristics, significantly outperforming popular visualization
tools such as PCA, t-SNE, UMAP, and HoroPCA which is also designed for
hyperbolic data.

##  1 Introduction

Datasets with hierarchical structures are ubiquitous. Social networks [10] and
complex networks [1] are representative examples of hierarchical data.
Euclidean space cannot embed entities in such hierarchical datasets without
distortion. Hyperbolic space, a non-Euclidean space with constant negative
curvature, has been widely used for embedding hierarchical data since
hyperbolic metric can closely approximate the tree metric. Hyperbolic space
has thus been used in the representation learning of word embeddings [23]
(Figure 1) and visual inputs [13, 18]. Several algorithms also directly
operate on hyperbolic space [7, 6, 32].

Figure 1: Our CO-SNE method deflates a high-dimensional hyperbolic
representation while preserving their global hierarchy and local similarities.
We generate the Poincaré embeddings [23] of the $mammal$ subtree from WordNet
in a five-dimensional hyperbolic space. We apply the standard t-SNE [31], the
recently proposed HoroPCA [5] and our proposed CO-SNE method to visualize the
embeddings in a two-dimensional Euclidean (in t-SNE) or hyperbolic space (in
HoroPCA and CO-SNE). CO-SNE preserves the global hierarchy (root node is in
the center and leaf nodes are close to boundary) and the local similarity
(sibling nodes are close) in the two-dimensional embeddings.

Higher embedding dimension can generally lead to better hyperbolic
representation quality [23, 11]. However, learning with two-dimensional
hyperbolic space is prevalent [23, 11]. One of the reasons is the ease of
visualization. There are several isometrically equivalent models for
representing hyperbolic space. The Poincaré ball model is arguably the most
widely used model in hyperbolic representation learning [23, 11, 9]. With the
Poincaré ball model, we can easily visualize two-dimensional hyperbolic
embeddings within a unit Euclidean circle. However, to visualize high-
dimensional hyperbolic data is not easy, as most visualization methods assume
the data exist in Euclidean space.

Embeddings in Poincaré ball have two notable properties: 1) The embeddings
have a global hierarchical structure. Root nodes are in the center of the ball
and leaf nodes are close to the boundary of the ball. 2) The embeddings
possess a local similarity structure. Sibling nodes should be close in the
embedding space.

t-SNE [31] is a popular visualization tool for visualizing high-dimensional
Euclidean data. However, t-SNE cannot preserve the global hierarchy of the
hyperbolic embeddings. HoroPCA [5] is recently proposed as an extension of PCA
on hyperbolic space. However, HoroPCA cannot preserve the local similarity of
the hyperbolic embeddings. In this paper, we propose CO-SNE which can preserve
both the global hierarchy and local similarity of high-dimensional hyperbolic
embeddings in a low- dimensional hyperbolic space (1). In Figure 1, the
$mammal$ subtree from WordNet [21] is embedded in a five-dimensional
hyperbolic space via Poincaré embeddings [23]. We use t-SNE [31], HoroPCA [5]
and CO-SNE to visualize the embeddings in a two-dimensional space. CO-SNE can
preserve the structure of the data well.

In CO-SNE, for maintaining local similarity structure, we adopt the same idea
as in the standard t-SNE to minimize the KL-divergence between the high-
dimensional similarities and the low-dimensional similarities. We adopt
hyperbolic versions of the normal distribution and Cauchy distribution to
compute the similarities. To maintain the global hierarchical structure, we
adopt a distance loss function which seeks to preserve the individual
distances of high-dimensional hyperbolic embeddings to the Origin in low-
dimensional hyperbolic space.

In summary, we make the following contributions,

  * •

We propose CO-SNE which can represent high-dimensional hyperbolic datapoints
in a low-dimensional hyperbolic space while maintaining the local similarity
and the global hierarchical structure.

  * •

We propose to use hyperbolic Cauchy distribution for computing low-dimensional
similarities which is crucial for producing good visualization in hyperbolic
space.

  * •

We apply CO-SNE to visualize synthetic hyperbolic data, hierarchical
biological datasets and hyperbolic embeddings learned by supervised and
unsupervised learning methods to better understand high-dimensional hyperbolic
data. Across all the cases, CO-SNE produces much better visualization than the
baselines.

##  2 Related Work

Large-margin Classification [6] in hyperbolic space was proposed by changing
the distance function from Euclidean to hyperbolic. Robust large-margin
classification [32] in hyperbolic space is also proposed and has the first
theoretical guarantees for learning a hyperbolic classifier.

Hyperbolic Neural Networks (HNNs) [7] rewrite multinomial logistic regression
(MLR), fully connected layers and Recurrent Neural Networks for hyperbolic
embeddings via gyrovector space operations [30]. In a follow-up work,
Hyperbolic Neural Networks++ [28] introduces hyperbolic convolutional layers.
Hyperbolic attention networks [8] are proposed by extending attention
operations to hyperbolic space in a manner similar to [7]. Hyperbolic graph
neural networks [17] are further proposed by altering the geometry of Graph
Neural Networks (GNNs) [34] to hyperbolic space. Hyperbolic neural networks
have also been used for visual inputs and achieve better results than
Euclidean neural networks on tasks such as few-shot classification and person
re-identification [13].

Unsupervised Learning Methods in Hyperbolic Space are also proposed recently.
Poincaré embeddings are proposed to embed words and relations in hyperbolic
space [23]. [22] proposed a generalized version of the normal distribution on
hyperbolic space called wrapped normal distribution. The proposed wrapped
normal distribution is used as the latent space for constructing hyperbolic
variational autoencoders (VAEs) [14]. A similar idea is also adopted in [18]
to construct Poincaré VAEs. Unsupervised 3D segmentation [11] and instance
segmentation [33] are achieved in hyperbolic space via hierarchical triplet
loss.

Data Visualization is the process of generating low-dimensional representation
of each high-dimensional datapoint. A good visualization should maintain the
interesting structure of the data presented in high-dimensional space.
t-distributed stochastic neighbor embedding (t-SNE) [31] is arguably the most
widely used tool for data visualization. t-SNE attempts to maintain the local
similarities of the high-dimensional datapoints in the low-dimensional space.
More recently, UMAP [19] is proposed as a manifold learning technique for
dimensionality reduction and visualization. Compared with t-SNE, UMAP can
better preserve the global structure of the high-dimensional data.

Dimensionality Reduction methods also can be used for data visualization.
Principal Component Analysis (PCA [12] is a commonly used dimensionality
reduction technique. PCA aims at maintaining the maximum amount of variation
of information in the low-dimensional space. Isomap [29] is a non-linear
dimensionality reduction method which attempts to preserve local structures.
LLE [27] is another popular non-linear dimensionality reduction method that
can produce neighborhood preserving embeddings of high-dimensional data.

Notably, no existing data visualization methods are designed for visualizing
high-dimensional hyperbolic data. Our proposed CO-SNE method can be used to
visualize high-dimensional hyperbolic embeddings.

Figure 2: The comparison of Euclidean distance and hyperbolic distance as we
move towards the boundary of the Poincaré ball.

##  3 CO-SNE

We present the proposed CO-SNE method which can faithfully represent high-
dimensional hyperbolic data in a low-dimensional hyperbolic space. In Section
3.1, we review the basics of Poincaré ball model for hyperbolic space. In
Section 3.2, we review t-distributed Stochastic neighbor embedding (t-SNE). In
Section 3.3 and 3.4, we introduce the hyperbolic version of normal
distribution and Student’s t-distribution. In Section 3.5, we discuss the
issue of using hyperbolic Student’s t-distribution for computing low-
dimensional similarities and propose to use hyperbolic Cauchy distribution. In
Section 3.6, we present the distance loss for maintaining global hierarchy. In
Section 3.7, we present the optimization details of CO-SNE.

###  3.1 Poincaré Ball Model for Hyperbolic Space

A hyperbolic space is a Riemannian manifold with a constant negative
curvature. There are several isometrically equivalent models for representing
hyperbolic space. The Poincaré ball model is the most commonly used one in
hyperbolic representation learning [23, 7]. The $n$-dimensional Poincaré ball
model is defined as $(\mathbb{B}^{n},\mathfrak{g}_{\mathbf{x}})$, where
$\mathbb{B}^{n}$ =
$\\{\mathbf{x}\in\mathbb{R}^{n}:\lVert\mathbf{x}\rVert<1\\}$ and
$\mathfrak{g}_{\mathbf{x}}=(\gamma_{\mathbf{x}})^{2}I_{n}$ is the Riemannian
metric tensor. $\gamma_{\mathbf{x}}=\frac{2}{1-\lVert\mathbf{x}\rVert^{2}}$ is
the conformal factor and $I_{n}$ is the Euclidean metric tensor. Given two
points $\bm{u}\in\mathbb{B}^{n}$ and $\bm{v}\in\mathbb{B}^{n}$, the hyperbolic
distance between them is defined as,

$d_{\mathbb{B}^{n}}(\bm{u},\bm{v})=\operatorname{arcosh}\left(1+2\frac{\lVert\bm{u}-\bm{v}\rVert^{2}}{(1-\lVert\bm{u}\rVert^{2})(1-\lVert\bm{v}\rVert^{2})}\right)$
(1)

where $\operatorname{arcosh}$ is the inverse hyperbolic cosine function and
$\lVert\cdot\rVert$ is the usual Euclidean norm. Different from Euclidean
distance, hyperbolic distance grows exponentially fast as we move the points
towards the boundary of the Poincaré ball as in Figure 2.

###  3.2 t-SNE

t-SNE [31] begins by mapping high-dimensional distances between datapoints to
similarity values. The similarity values are either conditional or joint
probabilities based on the probability a point will pick another as its
neighbor if neighbors are picked in proportion to the probability density of a
distribution centered at that point. t-SNE defines the conditional probability
$p_{j|i}$, the probability that the point $\mathbf{x}_{i}$ will pick a point
$\mathbf{x}_{j}$ as its neighbor, using a normal distribution centered at the
point $\mathbf{x}_{i}$. t-SNE then defines the joint probability distribution
$P$, by setting $p_{ij}=\frac{p_{i|j}+p_{j|i}}{2m}$ as a way to increase the
cost contribution of outlier points, where $m$ is the number of high-
dimensional datapoints. The conditional probability density $p_{j|i}$ is,

$p_{j|i}=\frac{\exp(-d(\mathbf{x}_{i},\mathbf{x}_{j})^{2}/2\sigma_{i}^{2})}{\sum_{k\neq
i}\exp(-d(\mathbf{x}_{i},\mathbf{x}_{k})^{2}/2\sigma_{i}^{2})}$ (2)

where $d(\mathbf{x}_{i},\mathbf{x}_{j})$ is the distance between
$\mathbf{x}_{i}$ and $\mathbf{x}_{j}$. In the low-dimensional space, Student’s
t-distribution is used for modeling the joint probability distribution $Q$
between embeddings, and $q_{ij}$ is defined as,

$q_{ij}=\frac{(1+d(\mathbf{y}_{i},\mathbf{y}_{j})^{2})^{-1}}{\sum_{k\neq
l}(1+d(\mathbf{y}_{k},\mathbf{y}_{l})^{2})^{-1}}$ (3)

where $\mathbf{y}_{i}$ is the corresponding low-dimensional embedding of
$\mathbf{x}_{i}$. In t-SNE, to maintain the local similarities, the cost
function to minimize is the Kullback-Leibler divergence between the
probability distributions $P$ and $Q$:

$\mathcal{C}=KL(P||Q)=\sum_{i}\sum_{j}p_{ij}\log{\frac{p_{ij}}{q_{ij}}}$ (4)

One direct extension of t-SNE is to replace Euclidean version of normal
distribution and Student’s t-distribution with hyperbolic normal distribution
and hyperbolic Student’s t-distribution. We call such a direct extension as
HT-SNE. In Section 3.3 and Section 3.4, we show how to generalize the
distributions to hyperbolic space.

|  |  |
---|---|---|---
a) Normal and Cauchy Dist. | b) t-SNE | c) HT-SNE | d) CO-SNE
Figure 3: Hyperbolic Student’s t-distribution is not heavy-tailed which leads
to strong attraction forces in HT-SNE. a) The probability density function of
the hyperbolic normal distribution, hyperbolic Cauchy distribution with
$\gamma=0.1$ and hyperbolic Cauchy distribution with $\gamma=1.0$. Hyperbolic
Student’s t-distribution (hyperbolic Cauchy distribution with $\gamma=1.0$) is
not heavy-tailed. b) The gradients of the standard t-SNE as a function of low-
dimensional and high-dimensional Euclidean distance. c) The gradients of the
HT-SNE as a function of low-dimensional and high-dimensional hyperbolic
distance. d) The gradients of CO-SNE as a function of low-dimensional and
high-dimensional hyperbolic distance. There is strong repulsion when
dissimilar high-dimensional datapoints are projected close.

###  3.3 Hyperbolic Normal Distribution

To define the conditional probability in the high-dimensional hyperbolic
space, we need to generalize the normal distribution to hyperbolic space. One
natural generalization is called the Riemannian normal distribution which is
the maximum entropy probability distribution given an expectation and a
variance [26]. Given the Fréchet mean $\bm{\mu}\in\mathbb{B}^{n}_{c}$ and a
dispersion parameter $\sigma>0$, the Riemannian normal distribution is defined
as,

$\mathcal{N}_{\mathbb{B}^{n}}(\bm{x}|\bm{\mu},\sigma^{2})=\frac{1}{\bm{Z}}\exp(-\frac{d_{\mathbb{B}^{n}}(\bm{\mu},\bm{x})^{2}}{2\sigma^{2}})$
(5)

where $\bm{Z}$ is the normalization constant. There are other generalizations
of the normal distribution in hyperbolic space [18]. We use the Riemannian
normal distribution for simplicity. Thereafter, we refer to Riemannian normal
distribution as hyperbolic normal distribution.

###  3.4 Hyperbolic Student’s t-Distribution

One way to define the Student’s t-distribution is to express the random
variable $t$ as,

$t=\frac{u}{\sqrt{v/n}}$ (6)

where $u$ is a random variable sampled from a standard normal distribution and
$v$ is a random variable sampled from a $\chi^{2}$-distribution of $n$ degrees
of freedom. In particular, t-SNE adopts a Student’s t-distribution with one
degree of freedom and the probability density function is defined as,

$f(t;t_{0})=\frac{1}{\pi(1+(t-t_{0})^{2})}$ (7)

To extend the Student’s t-distribution to hyperbolic space, we derive the
probability density function as,

$f_{\mathbb{B}^{n}}(t;t_{0})=\frac{1}{\pi(1+d_{\mathbb{B}^{n}}(t,t_{0})^{2})}$
(8)

The details can be found in the Supplementary.

###  3.5 Hyperbolic t-Distribution is Not Enough

The motivation to use Student’s t-distribution in the standard t-SNE is that
Student’s t-distribution has heavier tails than the normal distribution. This
causes a repulsion force between dissimilar high-dimensional datapoints which
helps mitigate the “Crowding Problem” [31]. However, hyperbolic Student’s
t-distribution is not heavy-tailed since the hyperbolic distance grows
exponentially fast.

Figure 3 plots the hyperbolic normal distribution and hyperbolic Student’s
t-distribution. It can be noted that the tails of hyperbolic Student’s
t-distribution diminish as fast as hyperbolic normal distribution. In
contrast, the standard Euclidean Student’s t-distribution has heavier tails
than the normal distribution. The consequence is that the repulsion force and
the attraction force with hyperbolic Student’s t-distribution behave
drastically different from that of using Euclidean Student’s t-distribution.

Similar to [31], we plot the gradients between two low-dimensional embeddings
$\mathbf{y}_{i}$ and $\mathbf{y}_{j}$ as a function of their pairwise
Euclidean (hyperbolic) distance in the low-dimensional Euclidean (hyperbolic)
and the pairwise Euclidean (hyperbolic) distance of the corresponding
datapoints in the high-dimensional Euclidean (hyperbolic) space. Positive
values indicate attraction and negative values indicate repulsion between the
embeddings $\mathbf{y}_{i}$ and $\mathbf{y}_{j}$ respectively. Here we only
consider the initial stage when all the low-dimensional embeddings are close.
The results are shown in Figure 3. We have two observations. First, for the
standard t-SNE, there is strong repulsion when dissimilar high-dimensional
datapoints are projected closely. Second, with the hyperbolic Student’s
t-distribution in HT-SNE, there is little or no repulsion force which causes
the low-dimensional embeddings tend to cluster together.

To remedy this issue, we seek to replace the hyperbolic Student’s
t-distribution to create more repulsion forces between dissimilar high-
dimensional datapoints. Consider the hyperbolic Cauchy distribution which has
the probability density function,

$f(t;t_{0},\gamma)=\frac{1}{\pi\gamma}[\frac{\gamma^{2}}{d_{\mathbb{B}^{n}}(t,t_{0})^{2}+\gamma^{2}}]$
(9)

where $\gamma$ is the scale parameter. Notice that the Student’s
t-distribution is a special case of Cauchy distribution with $\gamma=1.0$.
With a small $\gamma$, Cauchy distribution has a higher peak (Figure 3 (a)).
Thus, if the high-dimensional datapoints are modeled with close low-
dimensional embeddings, the density value is much larger than the
corresponding density in the high-dimensional space. This produces a strong
repulsion when dissimilar high-dimensional datapoints are modeled with close
low-dimensional embeddings. More details can be found in the Supplementary. We
show the corresponding gradients in Figure 3 (d). We can observe that there is
a strong repulsion force when high-dimensional dissimilar datapoints are
projected closely in CO-SNE. Such repulsion is particularly important in the
initial stage of training since all the low-dimensional embeddings are close.

###  3.6 Distance Loss

Hyperbolic space can be naturally used for embedding tree structured data. The
datapoint which is close to the center of the Poincaré ball can be viewed as
the root node and datapoints which are close to the boundary of the Poincaré
ball can be viewed as leaf nodes. The criterion of t-SNE cannot preserve the
tree-structure of the high-dimensional datapoints in the low-dimensional
space. One natural way to characterize the level of the datapoint in the
underlying tree is the norm of the datapoint. Thus, we attempt to keep the
norm invariant after the high-dimensional datapoint is projected. This leads
to the following loss function which minimizes the difference between the
norms of the high-dimensional datapoint $\mathbf{x}_{i}$ and the corresponding
low-dimensional embedding $\mathbf{y}_{i}$,

$\mathcal{H}=\frac{1}{m}\sum_{i=1}^{m}(\lVert\mathbf{x}_{i}\rVert^{2}-\lVert\mathbf{y}_{i}\rVert^{2})^{2}$
(10)

With the distance loss, we can preserve the global hierarchy of the high-
dimensional hyperbolic embeddings.

| Metric | High/low-dimensional Dist. | Losses
---|---|---|---
t-SNE | Euclidean | Normal/t-distribution | KL-div
HT-SNE | Hyperbolic | Normal/t-distribution | KL-div
CO-SNE | Hyperbolic | Normal/Cauchy | KL-div + Distance
Table 1: Our CO-SNE extends t-SNE by adopting hyperbolic normal distribution
and hyperbolic Cauchy distribution. Compared with t-SNE, CO-SNE assumes the
high-dimensional and low-dimensional space are hyperbolic. t-SNE cannot
maintain the global hierarchy of the hyperbolic embeddings. Compared with HT-
SNE, CO-SNE adopts hyperbolic Cauchy distribution and an additional distance
loss. HT-SNE cannot push dissimilar high-dimensional points away in the low-
dimensional space.

###  3.7 Optimization of CO-SNE

Criterion. The criterion of CO-SNE is composed of the KL-divergence for
maintaining local similarity and the distance loss for maintaining the global
hierarchy,

$\mathcal{L}=\lambda_{1}\mathcal{C}+\lambda_{2}\mathcal{H}$ (11)

where $\lambda_{1}$ and $\lambda_{2}$ are hyperparameters. Table 1 shows a
comparison of t-SNE, HT-SNE and CO-SNE.

Gradients. The cost function is optimized by gradient descent with respect to
the low-dimensional embeddings. The gradient of the KL-divergence
$\mathcal{C}$ with respect to $\mathbf{y}_{i}$ is given by,

$\begin{split}\frac{\delta\mathcal{C}}{\delta\mathbf{y}_{i}}&=\sum_{j}\frac{\delta\mathcal{C}}{\delta
d_{\mathbb{B}^{n}}(\mathbf{y}_{i},\mathbf{y}_{j})}\frac{\delta
d_{\mathbb{B}^{n}}(\mathbf{y}_{i},\mathbf{y}_{j})}{\delta\mathbf{y}_{i}}\\\
&=2\sum_{j}(p_{ij}-q_{ij})(1+d_{\mathbb{B}^{n}}(\mathbf{y}_{i},\mathbf{y}_{j})^{2})^{-1}\frac{\delta
d_{\mathbb{B}^{n}}(\mathbf{y}_{i},\mathbf{y}_{j})}{\delta\mathbf{y}_{i}}\end{split}$
(12)

The partial gradient of the distance with respect to the low dimensional
embedding $\mathbf{y}_{i}$ is given by

$\small\frac{\delta
d_{\mathbb{B}^{n}}(\mathbf{y}_{i},\mathbf{y}_{j})}{\delta\mathbf{y}_{i}}=\frac{4}{\beta\sqrt{\gamma^{2}-1}}\left(\frac{||\mathbf{y}_{j}||^{2}-2\langle\mathbf{y}_{i},\mathbf{y}_{j}\rangle+1}{\alpha^{2}}\mathbf{y}_{i}-\frac{\mathbf{y}_{j}}{\alpha}\right)$
(13)

where $\alpha=1-||\mathbf{y}_{i}||^{2}$, $\beta=1-||\mathbf{y}_{j}||^{2}$,
$\gamma=1+\frac{2}{\alpha\beta}||\mathbf{y}_{i}-\mathbf{y}_{j}||^{2}$. We use
the Riemannian stochastic gradient descent [3] for the KL-divergence
$\mathcal{C}$. The gradient of the distance loss $\mathcal{H}$ with respect to
$\mathbf{y}_{i}$ is computed as,

$\frac{\delta\mathcal{H}}{\delta\mathbf{y}_{i}}=-4(\lVert\mathbf{x}_{i}\rVert^{2}-\lVert\mathbf{y}_{i}\rVert^{2})\mathbf{y}_{i}$
(14)

We constrain the embeddings to the Poincaré ball after each update as in [23].

Stagewise Training. We found the following training strategy is useful for
producing better visualization in practice. We train the low-dimensional
embeddings only with the local similarity loss for the first 500 iterations.
Then we add the distance loss. The stopping criterion is the same as in t-SNE
[31]. The reason of the strategy is that it is hard to move the points when
they are approaching the boundary of the Poincaré ball.

##  4 Experiments and Results

Baselines. We compare CO-SNE with 4 baselines,

  * •

The standard t-SNE [31]: this is the standard t-SNE which adopts Euclidean
distance for computing the similarities in the high-dimensional space and the
low-dimensional space.

  * •

Principal Component Analysis (PCA) [12]: PCA is a commonly used dimensionality
reduction method which attempts to maintain the maximal variation of the data.
However, as a linear dimensionality method, PCA cannot reduce high-dimensional
data to two dimensions in a meaningful way [2].

  * •

HoroPCA [5]: HoroPCA is a recently proposed extension of PCA on hyperbolic
space. HoroPCA proposed to parameterize geodesic subspaces by ideal points in
the Poincaré ball. HoroPCA also generalized the notion of projection and the
objective function of PCA to hyperbolic space. HoroPCA can be used as a data
whitening method for hyperbolic data and as a visualization method as well.

  * •

UMAP [19]: UMAP is a recently proposed dimensionality reduction and
visualization method based on the ideas from Riemannian geometry and topology.
UMAP is competitive with t-SNE and can better preserve the global structure of
the data.

Tasks. We consider 4 datasets to validate the effectiveness of CO-SNE: 1)
synthetic datasets sampled from a mixture of hyperbolic normal distributions,
2) cellular differentiation data, 3) supervised hyperbolic embeddings and 4)
unsupervised hyperbolic embeddings.

Implementation Details. For PCA and the standard t-SNE, we use the
implementation from [25]. For UMAP, we use the implementation from [20]. For
HoroPCA, we use the implementation from
https://github.com/HazyResearch/HoroPCA which is provided by the original
authors. We implement CO-SNE based on t-SNE by modifying the way of computing
similarities and the optimization procedure.

Hyperparameters. For PCA, UMAP and HoroPCA, we use the default
hyperparameters. For SNE-based methods, we initialize the low-dimensional
embedding using a normal distribution with mean 0.01 and unit variance. The
training follows the standard setups as in [25]. For CO-SNE, the scaling
parameter $\gamma$ is set to 0.1. The hyperparameter $\lambda_{1}$ is usually
set to 10.0 and the hyperparameter $\lambda_{2}$ is usually set to 0.01.

|  |
---|---|---
UMAP | PCA | t-SNE
|  |
HoroPCA | HT-SNE | CO-SNE
Figure 4: The projection of high-dimensional hyperbolic datapoints sampled
from a mixture of hyperbolic normal distributions in a two-dimensional space
with different methods. CO-SNE produces two-dimensional hyperbolic embeddings
which preserve the hierarchical and similarity structure of the high-
dimensional hyperbolic datapoints.

###  4.1 Synthetic Point Clusters

We first use a synthetic dataset to validate the efficacy of CO-SNE. We
randomly generate five point clusters of 20 points each in the 5D hyperbolic
space. Each follows a hyperbolic normal distribution with the unit variance
and the mean located on a different axis. The first and second means are close
to the origin, at [0.1, 0, 0, 0, 0] and [0, -0.2, 0, 0, 0] respectively,
whereas the third and fourth means are far from and equi-distance to the
origin, at [0, 0, 0.9, 0, 0] and [0, 0, 0, -0.9, 0] respectively. The last
mean is right at the origin [0, 0, 0, 0, 0]. Figure 4 shows the 2D
visualization results of these 5D points by different methods.

CO-SNE produces a much better visualization compared with the baselines. With
CO-SNE, the projected two-dimensional hyperbolic embeddings can preserve the
local similarity structure and the global hierarchical structure of high-
dimensional datapoints well. Also, CO-SNE can prevent the high-dimensional
datapoints from being projected too close which usually happens with Euclidean
distance based methods. Note that HT-SNE does not have not enough repulsion to
push close low-dimensional embeddings apart. We provide a detailed analysis on
the drawbacks of each baseline for visualizing high-dimensional hyperbolic
datapoints.

  * •

Standard t-SNE: Euclidean distances are used for computing similarities in the
high-dimensional space. Hyperbolic distances grow much faster than Euclidean
distances. For high-dimensional hyperbolic datapoints which are close to
boundary of the Poincaré ball, the standard t-SNE wrongly underestimates the
distance between them. As a consequence, the standard t-SNE would take
dissimilar high-dimensional data points as neighbors. The resulting low-
dimensional embeddings would collapse into one point which leads to poor
visualization. In summary, t-SNE cannot preserve the global hierarchy of the
hyperbolic data.

  * •

PCA and HoroPCA: as mentioned above, PCA and HoroPCA are linear dimensionality
reduction methods which are not generally suitable for visualization in a two-
dimensional space. Both PCA and HoroPCA cannot preserve local similarity of
the hyperbolic data.

  * •

UMAP: UMAP suffers from the same issue as t-SNE since Euclidean distance is
used for computing high-dimensional similarities.

We mainly compare CO-SNE with HoroPCA since HoroPCA is specifically designed
for hyperbolic data.

---
a) Canonical hematopoetic cell lineage tree [15]
|  |
---|---
b) HoroPCA | c) CO-SNE
Figure 5: CO-SNE produces better visualization of the high-dimensional
biological datapoints than HoroPCA which captures the hierarchical and
similarity structure in the data. a) The hierarchy of the original data from
[15]. b) The two-dimensional embeddings generated by HoroPCA. c) The two-
dimensional embeddings generated by CO-SNE.

###  4.2 Biological Dataset

Biological data can reveal naturally occurring hierarchies, such as in single
cell RNA sequencing data [15]. In [15], they analyze cellular differentiation
data, the transition of immature cells to specialized types. The immature
cells can be viewed as the root of the tree and can branch off into several
different types of cells, creating hierarchical data of cells in different
states of progress in the transition process. One dataset we adapt from [15]
is the mouse myelopoiesis dataset presented by [24], where there are 532 cells
of 9 types. Two of the types, HSPC-1 and HSPC-2, form the root of the
hierarchy, while megakaryocytic (Meg), erythrocytic (Eryth), monocyte-
dendritic cell precursor (MDP), monocytic (Mono), and myelocyte (myelocytes
and metamyelocytes) type cells are states father from the root. Granulocytic
(Gran) cells are a precursor to myelocytes and multi-lineage primed (Multi-
Lin) cells are in an intermediate state.

The data has originally 382 dimensions (noisy) and like [15] we first reduce
it to 20 dimensions via PCA to reduce the noises. We then scale the data to
fit within the Poincaré ball and run CO-SNE and HoroPCA to produce two-
dimensional hyperbolic embeddings. Noted that this dataset is not centered.
The results are shown in Figure 5. The proposed low-embeddings by CO-SNE
capture the hierarchical structure in the original data.

|
---|---
a) HoroPCA | b) CO-SNE
Figure 6: CO-SNE produces better visualization of high-dimensional Poincaré
word embeddings than HoroPCA. a) The two-dimensional Poincaré word embeddings
generated from HoroPCA. b) The two-dimensional Poincaré word embeddings
generated from CO-SNE. In the embeddings generated by CO-SNE, the word
$feline$ and $canine$ are close to $canivore$, which is not the case in
HoroPCA.

###  4.3 Hierarchical Word Embeddings

Hyperbolic space has been used to embed hierarchical representations of
symbolic data. In [23], the authors adopt hyperbolic space for embedding
taxonomies, in particular, the transitive closure of WordNet noun hierarchy
[21]. As shown in [23], higher-dimensional hyperbolic embeddings often lead to
better representations, but they are harder to visualize. Following [23], we
embed the hypernymy relations of the mammals subtree of WordNet in hyperbolic
space. We use the open source implementation provided by [23] to train the
ten-dimensional embeddings. We use HoroPCA and CO-SNE to visualize the learned
embeddings in a two-dimensional hyperbolic space.

Figure 6 shows that compared with HoroPCA, CO-SNE can better preserve the
hierarchical and similarity structure of the high-dimensional datapoints. For
example, the word $feline$ and $canine$ are close to $canivore$ in the CO-SNE
embeddings, which is not the case in HoroPCA. The embeddings produced by CO-
SNE also more resemble the two-dimensional embeddings as shown in Figure 2b of
[23].

|
---|---
a) HoroPCA | b) CO-SNE
Figure 7: CO-SNE produces better visualization of hyperbolic neural networks’
(HNNs) features than HoroPCA. a) The visualization produced by HoroPCA. b) The
visualization produced by CO-SNE. In CO-SNE, the classes are well separated
and have a clear hierarchical structure.

###  4.4 Features of Hyperbolic Neural Networks

We apply CO-SNE to visualize the embeddings produced by hyperbolic neural
networks for supervised image classification. We train a hyperbolic neural
network (HNN) [7] with feature clipping [9] on MNIST. The clipping value is
1.0 and the feature dimension is 64. We use HoroPCA and CO-SNE to reduce the
dimensionality of the features to two. The full test set of MNIST cannot be
used due to the out-of-memory issue of HoroPCA. So for each class, we randomly
sample 100 images.

Figure 7 shows the two-dimensional embeddings generated by HoroPCA and CO-SNE.
The visualization produced by CO-SNE is significantly better the visualization
produced by HoroPCA. In CO-SNE, the classes are well separated and have a
clear hierarchical structure. We further train hyperbolic classifiers [7] on
the frozen two-dimensional features. We use a learning rate of 0.001 and the
number of epoch is 10. The accuracy of the features generated by HoroPCA is
30.2% while the accuracy of the features generated by CO-SNE is 61.2%. This
implies that low-dimensional features generated by CO-SNE are more separated
and respect the structure of original high-dimensional embeddings.

|
---|---
a) HoroPCA | b) CO-SNE
Figure 8: CO-SNE produces better visualization of the high-dimensional latent
space representations generated by the encoder of the Poincaré VAE than
HoroPCA. a) The two-dimensional latent Poincaré embeddings generated from
HoroPCA. b) The two-dimensional latent Poincaré embeddings generated from CO-
SNE. CO-SNE can capture the hierarchical and clustering structures of the
high-dimensional latent representations while HoroPCA cannot.

##  5 Latent Space of Poincaré VAE

Variational autoencoders (VAEs) [14] is a popular unsupervised learning
method. Standard VAEs assume the latent space is Euclidean. [18] extends VAEs
by assuming the latent space is hyperbolic. Different from the standard VAEs,
Poincaré VAEs can embed tree-like structures more efficiently. For using CO-
SNE to visualize the latent space of Poincaré VAEs, we train a Poincaré VAE
with a latent dimension of five on MNIST [16] following from [18]. We further
generate the latent space representations of 1000 randomly sampled images with
the encoder.

Figure 8 shows the visualizations produced by HoroPCA and CO-SNE. Clearly, CO-
SNE produces much better visualization than HoroPCA. In particular, we can
easily observe the hierarchical and clustering structures in the latent space
which are totally distorted in the visualization produced by HoroPCA. Thus,
CO-SNE can be used to understand the latent space of Poincaré VAEs and
facilitate the development of better unsupervised hyperbolic learning methods.

###  5.1 Discussion

CO-SNE can be used to visualize hyperbolic datapoints in a two-dimensional
hyperbolic space while maintaining the local similarities and hierarchical
structure. CO-SNE still suffers from the same weaknesses as t-SNE. In
particular, as a general method for dimensionality reduction, the curse of
intrinsic dimensionality and the non-convexity, please see [31] for more
details. More ablations and results can be found in the Supplementary. In
particular, we investigate the effect of hyperparameters for balancing the KL-
divergence and the distance loss.

##  6 Summary

We propose CO-SNE which can be used to visualize hyperbolic data in a two-
dimensional hyperbolic space. We use the generalized versions of the normal
and Cauchy distributions in hyperbolic space to compute the high-dimensional
and the low-dimensional similarities, respectively. We apply CO-SNE to
visualize synthetic hyperbolic data, biological data and embeddings learned by
supervised and unsupervised representation learning methods. CO-SNE produces
much better visualizations than several popular visualization methods.

Acknowledgements. This work was supported, in part, by Berkeley Deep Drive and
the National Science Foundation (NSF) under Grant No. 2131111. Any opinions,
findings, and conclusions or recommendations expressed in this material are
those of the authors and do not necessarily reflect the views of the NSF.

##  7 Supplementary

###  7.1 Riemannian Geometry

We give a brief overview of Riemannian geometry, for more details please refer
to [4]. A Riemannian manifold $(\mathcal{M},\mathfrak{g})$ is a real smooth
manifold $\mathcal{M}$ with a Riemannian metric $\mathfrak{g}$. The Riemannian
metric $\mathfrak{g}$ is a smoothly varying inner product which is defined on
the tangent space $T_{\mathbf{x}}{\mathcal{M}}$ of $\mathcal{M}$. Given
$\mathbf{x}\in\mathcal{M}$ and two vectors $\mathbf{v},\mathbf{w}\in
T_{\mathbf{x}}{\mathcal{M}}$, we can use the Riemannian metric $\mathfrak{g}$
to compute the inner product
$\langle\mathbf{v},\mathbf{w}\rangle_{\mathbf{x}}$ as
$\mathfrak{g}(\mathbf{v},\mathbf{w})$. The norm of $\mathbf{v}\in
T_{\mathbf{x}}{\mathcal{M}}$ is defined as
$\lVert\mathbf{v}\rVert_{\mathbf{x}}=\sqrt{\langle\mathbf{v},\mathbf{v}\rangle}_{\mathbf{x}}$.
A geodesic generalizes the notion of straight line in the manifold which is
defined as a curve $\gamma:[0,1]\rightarrow\mathcal{M}$ of constant speed that
is everywhere locally a distance minimizer. The exponential map and the
inverse exponential map are defined as follows: given
$\mathbf{x},\mathbf{y}\in\mathcal{M},\mathbf{v}\in
T_{\mathbf{x}}{\mathcal{M}}$, and a geodesic $\gamma$ of length
$\lVert\mathbf{v}\rVert$ such that
$\gamma(0)=\mathbf{x},\gamma(1)=\mathbf{y},\gamma^{\prime}(0)=\mathbf{v}/\lVert\mathbf{v}\rVert$,
the exponential map
$\textnormal{Exp}_{\mathbf{x}}:T_{\mathbf{x}}{\mathcal{M}}\rightarrow\mathcal{M}$
satisfies $\textnormal{Exp}_{\mathbf{x}}(\mathbf{v})=\mathbf{y}$ and the
inverse exponential map
$\textnormal{Exp}^{-1}_{\mathbf{x}}:\mathcal{M}\rightarrow
T_{\mathbf{x}}{\mathcal{M}}$ satisfies
$\textnormal{Exp}^{-1}_{\mathbf{x}}(\mathbf{y})=\mathbf{v}$.

###  7.2 Hyperbolic Student’s t-Distribution

Recall the way to define the the Student’s t-distribution which expresses the
random variable $t$ as,

$t=\frac{u}{\sqrt{v/n}}$ (15)

where $u$ is a random variable sampled from a standard normal distribution and
$v$ is a random variable sampled from a $\chi^{2}$-distribution of $n$ degrees
of freedom. The $\chi^{2}$-distribution can also be derived from normal
distribution. Let $u_{1}$, $u_{2}$, …, $u_{n}$ be independent standard normal
random variables, then the sum of the squares,

$v=\sum_{i=1}^{n}u_{i}^{2}$ (16)

is a $\chi^{2}$-distribution with $n$ degrees of freedom. Thus, the
probability density function of the $\chi^{2}$-distribution can be derived
from the probability density function of the normal distribution which is,

$T_{n}(v)=\frac{1}{2^{n/2}\Gamma(n/2)}v^{(n-2)/2}e^{-v/2}$ (17)

Using Equation 15, we can further derive the probability density function of
the Student’s t-distribution,

$f_{n}(t)=\frac{1}{\sqrt{n}B(1/2,n/2)}(1+\frac{t^{2}}{n})^{-(n+1)/2}$ (18)

where $B$ is the Beta function.

The probability density function of hyperbolic Cauchy distribution can be
derived in a similar way using hyperbolic normal distribution.

|  $\lambda_{2}$ = 0.0 |  $\lambda_{2}$ = 0.01 |  $\lambda_{2}$ = 0.05 |  $\lambda_{2}$ = 0.1 |  $\lambda_{2}$ = 0.2
---|---|---|---|---|---
$\lambda_{1}$ = 0 |  |  |  |  |
$\lambda_{1}$ = 5 |  |  |  |  |
$\lambda_{1}$ = 10 |  |  |  |  |
$\lambda_{1}=15$ |  |  |  |  |
$\lambda_{1}$ = 20 |  |  |  |  |
Figure 9: The effect of $\lambda_{1}$ and $\lambda_{2}$ on the quality of the
visualization. We show the visualization results of CO-SNE on a mixture of
five hyperbolic normal distributions in a five-dimensional hyperbolic space.
We can observe that $\lambda_{1}$ is responsible for preserving the local
similarity structure and $\lambda_{2}$ is responsible for preserving the
global hierarchical structure. _This shows that both the KL-divergence and the
distance loss are important for producing good visualization_. CO-SNE is
robust to wide choices of $\lambda_{1}$ and $\lambda_{2}$. Generally,
$\lambda_{1}$ should be larger than $\lambda_{2}$ since the magnitude of the
gradients of the KL-divergence is smaller.

###  7.3 Hyperbolic Cauchy Distribution

Similar to the Student’s t-Distribution, the probability density function of
Cauchy distribution can derived from the probability density function of the
normal distribution. In particular, let $X$ and $Y$ be independent standard
normal random variables, then $Z=\frac{X}{X+Y}$ is a Cauchy random variable.
The probability density function of Cauchy distribution can be written as,

$f(x;x_{0},\gamma)=\frac{1}{\pi\gamma[1+(\frac{x-x_{0}}{\gamma})^{2}]}$ (19)

The probability density function of hyperbolic Cauchy distribution can be
derived in a similar way using hyperbolic normal distribution.

The repulsion and attraction forces in CO-SNE depend on the term
$p_{ij}-q_{ij}$ in Equation 12 of the main text. $p_{ij}$ is fixed during
training which depends on the distribution of the high-dimensional datapoints.
To create more repulsion forces between two close low-dimensional embeddings
$y_{i}$ and $y_{j}$, we aim at increasing the probability that the point
$y_{i}$ would select the point $y_{j}$ as its neighbor (i.e., $q_{ij}$). By
using a small $\gamma$ in hyperbolic Cauchy distribution, the distance between
$y_{i}$ and $y_{j}$ is scaled up. When the point $y_{i}$ is fixed, the
probability of selecting $y_{j}$ as a neighbor (i.e., $q_{ij}$) is scaled up
relatively to some point $y_{k}$ which is far away from $y_{i}$. As a result,
$p_{ij}-q_{ij}$ can potentially become negative. This creates additional
repulsion forces to push the low-dimensional points apart.

###  7.4 The Effect of $\lambda_{1}$ and $\lambda_{2}$

Recall the objective function of CO-SNE,

$\mathcal{L}=\lambda_{1}\mathcal{C}+\lambda_{2}\mathcal{H}$ (20)

$\lambda_{1}$ and $\lambda_{2}$ are used to balance the KL-divergence
$\mathcal{C}$ and the distance loss $\mathcal{H}$ and can be regarded as the
learning rates. For ablation studies on the effect of $\lambda_{1}$ and
$\lambda_{2}$, we reuse the settings in Section 4.1 of the main text.

We consider different settings of $\lambda_{1}$ and $\lambda_{2}$ to
investigate the effect of the KL-divergence and the distance loss. The results
are shown in Figure 9. We have several observations from the results.

  1. 1.

In the first row, $\lambda_{1}=0.0$. This means that only the distance loss is
presented. We can observe that the low-dimensional embeddings can only
approximate the magnitude of the high-dimensional datapoints but not the
similarity structure.

  2. 2.

In the first column, $\lambda_{2}=0.0$. This means that only the KL-divergence
is presented. We can observe that the low-dimensional embeddings can only
preserve the similarity structure in the high-dimensional datapoints but not
the hierarchical information.

  3. 3.

In other cases, we can observe that a larger $\lambda_{2}$ can better preserve
the hierarchical structure in the high-dimensional datapoints but may distort
the similarity structure. A larger $\lambda_{1}$ may also lead to a bad
visualization of the similarity structure since the KL-divergence might
diverge. Nevertheless, CO-SNE is robust to wide choices of $\lambda_{1}$ and
$\lambda_{2}$. _Both the KL-divergence and the distance loss are important for
producing good visualization_.

## References

  * [1] Gregorio Alanis-Lobato, Pablo Mier, and Miguel A Andrade-Navarro.  Efficient embedding of complex networks to hyperbolic space via their laplacian.  Scientific reports, 6(1):1–10, 2016.
  * [2] Sanjeev Arora, Wei Hu, and Pravesh K Kothari.  An analysis of the t-sne algorithm for data visualization.  In Conference On Learning Theory, pages 1455–1462. PMLR, 2018.
  * [3] Silvere Bonnabel.  Stochastic gradient descent on riemannian manifolds.  IEEE Transactions on Automatic Control, 58(9):2217–2229, 2013.
  * [4] Manfredo Perdigao do Carmo.  Riemannian geometry.  Birkhäuser, 1992.
  * [5] Ines Chami, Albert Gu, Dat P Nguyen, and Christopher Ré.  Horopca: Hyperbolic dimensionality reduction via horospherical projections.  In International Conference on Machine Learning, pages 1419–1429. PMLR, 2021.
  * [6] Hyunghoon Cho, Benjamin DeMeo, Jian Peng, and Bonnie Berger.  Large-margin classification in hyperbolic space.  In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1832–1840. PMLR, 2019.
  * [7] Octavian-Eugen Ganea, Gary Bécigneul, and Thomas Hofmann.  Hyperbolic neural networks.  arXiv preprint arXiv:1805.09112, 2018.
  * [8] Caglar Gulcehre, Misha Denil, Mateusz Malinowski, Ali Razavi, Razvan Pascanu, Karl Moritz Hermann, Peter Battaglia, Victor Bapst, David Raposo, Adam Santoro, et al.  Hyperbolic attention networks.  arXiv preprint arXiv:1805.09786, 2018.
  * [9] Yunhui Guo, Xudong Wang, Yubei Chen, and Stella X Yu.  Free hyperbolic neural networks with limited radii.  arXiv preprint arXiv:2107.11472, 2021.
  * [10] Mangesh Gupte, Pravin Shankar, Jing Li, Shanmugauelayut Muthukrishnan, and Liviu Iftode.  Finding hierarchy in directed online social networks.  In Proceedings of the 20th international conference on World wide web, pages 557–566, 2011.
  * [11] Joy Hsu, Jeffrey Gu, Gong-Her Wu, Wah Chiu, and Serena Yeung.  Learning hyperbolic representations for unsupervised 3d segmentation.  arXiv preprint arXiv:2012.01644, 2020.
  * [12] Ian T Jolliffe and Jorge Cadima.  Principal component analysis: a review and recent developments.  Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 374(2065):20150202, 2016.
  * [13] Valentin Khrulkov, Leyla Mirvakhabova, Evgeniya Ustinova, Ivan Oseledets, and Victor Lempitsky.  Hyperbolic image embeddings.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6418–6428, 2020.
  * [14] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114, 2013.
  * [15] Anna Klimovskaia, David Lopez-Paz, Léon Bottou, and Maximilian Nickel.  Poincaré maps for analyzing complex hierarchies in single-cell data.  Nature communications, 11(1):1–9, 2020.
  * [16] Yann LeCun.  The mnist database of handwritten digits.  http://yann. lecun. com/exdb/mnist/.
  * [17] Qi Liu, Maximilian Nickel, and Douwe Kiela.  Hyperbolic graph neural networks.  arXiv preprint arXiv:1910.12892, 2019.
  * [18] Emile Mathieu, Charline Le Lan, Chris J Maddison, Ryota Tomioka, and Yee Whye Teh.  Continuous hierarchical representations with poincar$\backslash$’e variational auto-encoders.  arXiv preprint arXiv:1901.06033, 2019.
  * [19] Leland McInnes, John Healy, and James Melville.  Umap: Uniform manifold approximation and projection for dimension reduction.  arXiv preprint arXiv:1802.03426, 2018.
  * [20] Leland McInnes, John Healy, Nathaniel Saul, and Lukas Grossberger.  Umap: Uniform manifold approximation and projection.  The Journal of Open Source Software, 3(29):861, 2018.
  * [21] George A Miller.  Wordnet: a lexical database for english.  Communications of the ACM, 38(11):39–41, 1995.
  * [22] Yoshihiro Nagano, Shoichiro Yamaguchi, Yasuhiro Fujita, and Masanori Koyama.  A wrapped normal distribution on hyperbolic space for gradient-based learning.  In International Conference on Machine Learning, pages 4693–4702. PMLR, 2019.
  * [23] Maximilian Nickel and Douwe Kiela.  Poincar$\backslash$’e embeddings for learning hierarchical representations.  arXiv preprint arXiv:1705.08039, 2017.
  * [24] Andre Olsson, Meenakshi Venkatasubramanian, Viren K Chaudhri, Bruce J Aronow, Nathan Salomonis, Harinder 1 Singh, and H. Leighton Grimes.  Single-cell analysis of mixed-lineage states leading to a binary cell fate choice.  Nature, 2016.
  * [25] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.  Scikit-learn: Machine learning in Python.  Journal of Machine Learning Research, 12:2825–2830, 2011.
  * [26] Xavier Pennec.  Intrinsic statistics on riemannian manifolds: Basic tools for geometric measurements.  Journal of Mathematical Imaging and Vision, 25(1):127–154, 2006\.
  * [27] Sam T Roweis and Lawrence K Saul.  Nonlinear dimensionality reduction by locally linear embedding.  science, 290(5500):2323–2326, 2000.
  * [28] Ryohei Shimizu, Yusuke Mukuta, and Tatsuya Harada.  Hyperbolic neural networks++.  arXiv preprint arXiv:2006.08210, 2020.
  * [29] Joshua B Tenenbaum, Vin De Silva, and John C Langford.  A global geometric framework for nonlinear dimensionality reduction.  science, 290(5500):2319–2323, 2000.
  * [30] Abraham Albert Ungar.  A gyrovector space approach to hyperbolic geometry.  Synthesis Lectures on Mathematics and Statistics, 1(1):1–194, 2008\.
  * [31] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  Journal of machine learning research, 9(11), 2008.
  * [32] Melanie Weber, Manzil Zaheer, Ankit Singh Rawat, Aditya Menon, and Sanjiv Kumar.  Robust large-margin learning in hyperbolic space.  arXiv preprint arXiv:2004.05465, 2020.
  * [33] Zhenzhen Weng, Mehmet Giray Ogut, Shai Limonchik, and Serena Yeung.  Unsupervised discovery of the long-tail in instance segmentation using hierarchical self-supervision.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2603–2612, 2021.
  * [34] Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun.  Graph neural networks: A review of methods and applications.  AI Open, 1:57–81, 2020.


11institutetext: Indian Institute of Technology, Gandhinagar, India
11email<EMAIL_ADDRESS>22institutetext: Indian Institute of Technology,
Gandhinagar, India
22email<EMAIL_ADDRESS>
33institutetext: Indian Institute of Technology, Gandhinagar, India
33email<EMAIL_ADDRESS>
#  TabLeX: A Benchmark Dataset for Structure and Content Information
Extraction from Scientific Tables

Harsh Desai 11 Pratik Kayal 22 Mayank Singh 33

###### Abstract

Information Extraction (IE) from the tables present in scientific articles is
challenging due to complicated tabular representations and complex embedded
text. This paper presents TabLeX, a large-scale benchmark dataset comprising
table images generated from scientific articles. TabLeX consists of two
subsets, one for table structure extraction and the other for table content
extraction. Each table image is accompanied by its corresponding LaTeX source
code. To facilitate the development of robust table IE tools, TabLeX contains
images in different aspect ratios and in a variety of fonts. Our analysis
sheds light on the shortcomings of current state-of-the-art table extraction
models and shows that they fail on even simple table images. Towards the end,
we experiment with a transformer-based existing baseline to report performance
scores. In contrast to the static benchmarks, we plan to augment this dataset
with more complex and diverse tables at regular intervals.

###### Keywords:

Information Extraction LaTeXScientific Articles.

##  1 Introduction

Tables are compact and convenient means of representing relational information
present in diverse documents such as scientific papers, newspapers, invoices,
product descriptions, and financial statements [12]. Tables embedded in the
scientific articles provide a natural way to present data in a structured
manner [5]. They occur in numerous variations, especially visually, such as
with or without horizontal and vertical lines, spanning multiple columns or
rows, non-standard spacing, alignment, and text formatting [6]. Besides, we
also witness diverse semantic structures and presentation formats, dense
embedded text, and formatting complexity of the typesetting tools [23]. These
complex representations and formatting options lead to numerous challenges in
automatic tabular information extraction (hereafter, ‘TIE’).

Table detection vs. extraction: In contrast to table detection task
(hereafter, ‘TD’) that refers to identifying tabular region (e.g., finding a
bounding box that encloses the table) in an document, TIE refers to two post
identification tasks: (i) table structure recognition (hereafter, ‘TSR’) and
(ii) table content recognition (hereafter, ‘TCR’). TSR refers to the
extraction of structural information like rows and columns from the table, and
TCR refers to content extraction that is embedded inside the tables. Figure 1
shows an example table image with its corresponding structure and content
information in the TeX language. Note that, in this paper, we only focus on
the two TIE tasks.

Limitations in existing state-of-the-art TIE systems: There are several state-
of-the-art tools (Camelot111https://github.com/camelot-
dev/camelot,Tabula222https://github.com/chezou/tabula-py,
PDFPlumber333https://github.com/jsvine/pdfplumber, and Adobe Acrobat
SDK444https://www.adobe.com/devnet/acrobat/overview.html) for text-based TIE.
On the contrary, Tesseract-based OCR [24] is commercially available tool which
can be used for image-based TIE. However, these tools perform poorly on the
tables embedded in the scientific papers due to the complexity of tables in
terms of spanning cells and presence of mathematical content. The recent
advancements in deep learning architectures (Graph Neural Networks [32] and
Transformers [28]) have played a pivotal role in developing the majority of
the image-based TIE tools. We attribute the limitations in the current image-
based TIE primarily due to the training datasets’ insufficiency. Some of the
critical issues with the training datasets can be (i) the size, (ii) diversity
in the fonts, (iii) image resolution (in dpi), (iv) aspect ratios, and (v)
image quality parameters (blur and contrast).

Our proposed benchmark dataset: In this paper, we introduce TabLeX, a
benchmark dataset for information extraction from tables embedded inside
scientific documents compiled using LaTeX-based typesetting tools. TabLeX is
composed of two subsets—a table structure subset and a table content subset—to
extract the structure and content information. The table structure subset
contains more than three million images, whereas the table content subset
contains over one million images. Each tabular image accompanies its
corresponding ground-truth program in TeX macro language. In contrast to the
existing datasets [3, 30, 31], TabLeX comprises images with 12 different fonts
and multiple aspect ratios.

Main contributions: The main contributions of the paper are:

  1. 1.

Robust preprocessing pipeline to process the scientific documents (created in
TeX language) and extract the tabular spans.

  2. 2.

A large-scale benchmark dataset, TabLeX, comprising more than three million
images for structure recognition task and over one million images for the
content recognition task.

  3. 3.

Inclusion of twelve font types and multiple aspect ratios during the dataset
generation process.

  4. 4.

Evaluation of state-of-the-art computer vision based baseline [7] on TabLeX.

The paper outline : We organize the paper into several sections. Section 2
reviews existing datasets and corresponding extraction methodologies. Section
3 describes the preprocessing pipeline and presents TabLeX dataset statistics.
Section 4 details three evaluation metrics. Section 5 presents the the
baseline and discusses the experimental results and insights. Finally, we
conclude and identify the future scope in Section 6.

(a)

(b)

(c)

Figure 1: Example of table image along with its structure and content
information from the dataset. Tokens in content information are character-
based, and the ‘¦’ token acts as a delimiter to identify words out of a
continuous stream of characters.

##  2 The Current State Of The Research

In recent years, we witness a surge in the digital documents available online
due to the high availability of communication facilities and large-scale
internet infrastructure. In particular, the availability of scientific
research papers that contain complex tabular structures has grown
exponentially. However, we witness fewer research efforts to extract embedded
tabular information automatically. Table 1 lists some of the popular datasets
for TD and TIE tasks. Among the two tasks, there are very few datasets
available for the TIE, specifically from scientific domains containing complex
mathematical formulas and symbols. As the current paper primarily focuses on
the TIE from scientific tables, we discuss some of the popular datasets and
their limitations in this domain.

Table 1: Datasets and methods used for Table Detection (TD), Table Structure
Recognition (TSR) and Table Recognition (TR). * represents scientific paper
datasets.

Datasets | TD | TSR | TR | Format | # Tables | Methods
---|---|---|---|---|---|---
Marmot[26]* | ✓ | $\times$ | $\times$ | PDF | 958 | Pdf2Table[21],TableSeer[15],[10]
PubLayNet[31]* | ✓ | $\times$ | $\times$ | PDF | 113k | F-RCNN[19], M-RCNN[11]
DeepFigures[22]* | ✓ | $\times$ | $\times$ | PDF | 1.4m | Deepfigures[22]
ICDAR2013[9] | ✓ | ✓ | ✓ | PDF | 156 | Heuristics+ML
ICDAR2019[8] | ✓ | ✓ | $\times$ | Images | 3.6k | Heuristics+ML
UNLV[20] | ✓ | ✓ | $\times$ | Images | 558 | T-Recs[13]
TableBank[14]* | ✓ | ✓ | $\times$ | Images | 417k (TD) | F-RCNN[19]
TableBank[14]* | ✓ | ✓ | $\times$ | Images | 145k (TSR) | WYGIWYS[2]
SciTSR[1]* | $\times$ | ✓ | $\checkmark$ | PDF | 15k | GraphTSR[1]
Table2Latex[3]* | $\times$ | ✓ | ✓ | Images | 450k | IM2Tex[4]
Synthetic data [18] | $\times$ | ✓ | ✓ | Images | Unbounded | DCGNN[18]
PubTabNet[30]* | $\times$ | ✓ | ✓ | Images | 568k | EDD[30]
TabLeX (ours)* | $\times$ | ✓ | ✓ | Images | 1m+ | TRT [7]

Scientific tabular datasets: Table2Latex [3] dataset contains 450k scientific
table images and its corresponding ground-truth in LaTeX. It is curated from
the arXiv555http://arxiv.org/ repository. To the best of our knowledge,
Table2Latex is the only dataset that contains ground truth in TeX language but
is not publicly available. TableBank [14] contains 145k table images along
with its corresponding ground truth in the HTML representation. TableBank [14]
contains table images from both Word and LaTeX documents curated from the
internet and arXiv5, respectively. However, it does not contain content
information to perform the TCR task. PubTabNet [30] contains over 568k table
images and corresponding ground truth in the HTML language. PubTabNet is
curated from the PubMed Central repository666https://pubmed.ncbi.nlm.nih.gov/.
However, it is only limited to the biomedical and life sciences domain. SciTSR
[1] contains 15k tables in the PDF format with table cell content and the
coordinate information in the JSON format. SciTSR has been curated from the
arXiv5 repository. However, manual analysis tells that 62 examples out of 1000
randomly sampled examples are incorrect [29].

TIE methodologies: The majority of the TIE methods employ encoder-decoder
architecture [3, 14, 30]. Table2Latex [3] uses IM2Tex [4] model where encoder
consists of convolutional neural network (CNN) followed by bidirectional LSTM
and decoder consists of standard LSTM. TableBank [14] uses WYGIWS [2] model,
an encoder-decoder architecture, where encoder consists of CNN followed by a
recurrent neural network (RNN) and decoder consists of standard RNN. PubTabNet
[30] uses a proposed encoder-dual-decoder (EDD) [30] model which consists of
CNN encoder and two RNN decoders called structure and cell decoder,
respectively. In contrast to the above works, SciTSR [1] proposed a graph
neural network-based extraction methodology. The network takes vertex and edge
features as input and computes their representations using graph attention
blocks, and performs classification over these edges.

TIE metrics: Table2Latex [3] used BLEU [16] score (a text-based metric) and
exact match accuracy (a vision-based metric) for evaluation. TableBank [14]
also conducted BLEU [16] metric for evaluation. Tesseract OCR [25] uses the
Word Error Rate (WER) metric for evaluation of the output. SciTSR [1] uses
micro- and macro-averaged precision, recall, and F1-score to compare the
output against the ground truth, respectively. In contrast to the above
standard evaluation metrics in NLP literature, PubTabNet [30] proposed a new
metric called Tree-Edit-Distance-based Similarity (TEDS) for evaluation of the
output HTML representation.

The challenges: Table 1 shows that there are only two datasets for Image-based
TCR from scientific tables, that is, Table2Latex [3] and PubTabNet [30]. We
address some of the challenges from previous works with TabLeX which includes
(i) large-size for training (ii) Font-agnostic learning (iii) Image-based
scientific TCR (iv) domain independent

##  3 The TabLeX Dataset

This section presents the detailed curation pipeline to create the TabLeX
dataset. Next, we discuss the data acquisition strategy.

###  3.1 Data Acquisition

We curate data from popular preprint repository arXiv5. We downloaded paper
(uploaded between Jan 2019–Sept 2020) source code and corresponding compiled
PDF. These articles belong to eight subject categories. Figure 2 illustrates
category-wise paper distribution. As illustrated, the majority of the papers
belong to three subject categories, physics (33.93%), computer science
(25.79%), and mathematics (23.27%). Overall, we downloaded 347,502 papers and
processed them using the proposed data processing pipeline (described in the
next section).

Figure 2: Total number of papers in arXiv’s subject categories. Here SS
denotes Systems Science.

###  3.2 Data Processing Pipeline

The following steps present a detailed description of the data processing
pipeline.

|
---|---
(a) A real table image | (b) Corresponding LaTeX source code
|
(c) Extracted table snippet | (d) LaTeX code template
|
(e) Code with Charter font package | (f) The final generated table image
Figure 3: Data processing pipeline.

####  3.2.1 LaTeX Code Pre-Processing Steps

  1. 1.

Table Snippets Extraction: A table snippet is a part of LaTeX code that begins
with ‘`\begin{tabular}`’ and ends with ‘`\end{tabular}`’ command. During its
extraction, we removed citation command ‘`\cite{}`’, reference command
‘`\ref{}`’, label command ‘`\label{}`’, and graphics command
‘`\includegraphics[]{}`’, along with $\sim$ symbol (preceding these commands).
Also, we remove command pairs (along with the content between them) like
‘`\begin{figure}`’ and ‘`\end{figure}`’, and ‘`\begin{subfigure}`’ and
‘`\end{subfigure}`’, as they cannot be predicted from the tabular images.
Furthermore, we also remove the nested table environments. Figure 3a and 3b
show an example table and its corresponding LaTeX source code, respectively.

  2. 2.

Comments Removal: Comments are removed by removing the characters between
‘`%`’ token and newline token ‘`\n`’. This step was performed because comments
do not contribute to visual information.

  3. 3.

Column Alignment and Rows Identification: We keep all possible alignment
tokens (‘`l`’, ‘`r`’, and ‘`c`’) specified for column styling. The token ‘`|`’
is also kept to identify vertical lines between the columns, if present. The
rows are identified by keeping the ‘`\\`’ and ‘`\tabularnewline`’ tokens.
These tokens signify the end of each row in the table. For example, Figure 3c
shows extracted table snippet from LaTeX source code (see Figure 3b)
containing the column and rows tokens with comment statements removed.

  4. 4.

Font Variation: In this step, the extracted LaTeX code is augmented with
different font styles. We experiment with a total 12 different LaTeX font
packages777https://www.overleaf.com/learn/latex/font˙typefaces. We use popular
font packages from PostScript family which includes ‘`courier`’, ‘`helvet`’,
‘`palatino`’, ‘`bookman`’, ‘`mathptmx`’, ‘`utopia`’ and also other font
packages such as ‘`tgbonum`’, ‘`tgtermes`’, ‘`tgpagella`’, ‘`tgschola`’,
‘`charter` and ‘`tgcursor`’. For each curated image, we create 12 variations,
one in each of the font style.

  5. 5.

Image Rendering: Each table variant is compiled into a PDF using a LaTeX code
template (described in Figure 3d). Here, ‘table snippet’ represents the
extracted tabular LaTeX code and ‘font package’ represents the LaTeX font
package used to generated the PDF. The corresponding table PDF files are then
converted into JPG images using the Wand
library888https://github.com/emcconville/wand which uses ImageMagick [27] API
to convert PDF into images. Figure 3e shows the embedded code and font
information within the template. Figure 3f shows the final table image with
Charter font. During conversion, we kept the image’s resolution as 300 dpi,
set the background color of the image to white, and removed the transparency
of the alpha channel by replacing it with the background color of the image.
We use two types of aspect ratio variations during the conversion, described
as follows:

    1. (a)

Conserved Aspect Ratio: In this case, the bigger dimension (height or width)
of the image is resized to 400 pixels999We use ‘400’ pixels as an experimental
number.. We then resize the smaller dimension (width or height) by keeping the
original aspect ratio conserved. During resizing, we use a blur factor of 0.8
to keep images sharp.

    2. (b)

Fixed Aspect Ratio: The images are resized to a fixed size of 400$\times$400
pixels using a blur factor of 0.8. Note that this resizing scheme can lead to
extreme levels of image distortions.

####  3.2.2 LaTeX Code Post-Processing Steps for Ground-Truth Preparation

  1. 1.

Filtering Noisy LaTeX tokens: We filter out LaTeX environment tokens (tokens
starting with ‘`\`’) having very less frequency (less than 5000) in the
overall corpus compared to other LaTeX environment tokens and replaced it with
‘`\LATEX_TOKEN`’ as these tokens will have a little contribution. This
frequency-based filtering reduced the corpus’s overall vocabulary size, which
helps in training models on computation or memory-constrained environments.

  2. 2.

Structure Identification: In this post-processing step, we create Table
Structure Dataset (TSD). It consists of structural information corresponding
to the table images. We keep tabular environment parameters as part of
structure information and replace tokens representing table cells’ content
with a placeholder token ‘CELL’. This post-processing step creates ground
truth for table images in TSD. Specifically, the vocabulary comprises digits
(0-9), ‘&’, ‘CELL’, ‘`\\`’, ‘`\hline`’, ‘`\multicolumn`’, ‘`\multirow`’,
‘`\bottomrule`’, ‘`\midrule`’, ‘`\toprule`’, ‘`|`’, ‘l’, ‘r’, ‘c’, ‘{’, and
‘}’. A sample of structure information is shown in Figure 1b, where the ‘CELL’
token represents a cell structure in the table. Structural information can be
used to identify the number of cells, rows, and columns in the table using
‘CELL’, `‘\\’` and alignment tokens (‘c’, ‘l’, and ‘r’), respectively. Based
on output sequence length, we divided this dataset further into two variants
TSD-250 and TSD-500, where the maximum length of the output sequence is 250
and 500 tokens, respectively.

  3. 3.

Content Identification: Similar to TSD, we also create a Table Content Dataset
(TCD). This dataset consists of content information including alphanumeric
characters, mathematical symbols, and other LaTeX environment tokens. Content
tokens are identified by removing tabular environment parameters and keeping
only the tokens that identify table content. Specifically, the vocabulary
includes all the alphabets (a-z and A-Z), digits (0-9), LaTeX environment
tokens (‘`\textbf`’, `\hspace`’, etc.), brackets (‘`(`’, ‘`)`’, ‘`{`’, ‘`}`’,
etc.) and all other possible symbols (‘`$`’, ‘`&`’, etc.). In this dataset,
based on output sequence length, we divided it further into two variants
TCD-250 and TCD-500, where the maximum length of the output sequence is 250
and 500 tokens, respectively. A sample of content information is shown in
Figure 1c.

###  3.3 Dataset Statistics

We further partition each of the four dataset variants TSD-250, TSD-500,
TCD-25, and TCD-500, into training, validation, and test sets in a ratio of
80:10:10. Table 2 shows the summary of the dataset. Note that the number of
samples present in the TSD and the corresponding TCD can differ due to
variation in the output sequence’s length. Also, the average number of tokens
per sample in TCD is significantly higher than the TSD due to more information
in the tables’ content than the corresponding structure. Figure 4 demonstrates
histograms representing the token distribution for the TSD and TCD. In the
case of TSD, the majority of tables contain less than 25 tokens. Overall the
token distribution shows a long-tail behavior. In the case of TCD, we witness
a significant proportion between 100–250 tokens. The dataset is licensed under
Creative Commons Attribution 4.0 International License and available for
download at https://www.tinyurl.com/tablatex.

Table 2: TabLeX Statistics. ML denotes the maximum length of output sequence. Samples denotes the total number of table images. T/S denotes the average number of tokens per sample. VS denotes the vocabulary size. AR and AC denote the average number of rows and columns among all the samples in the four dataset variants, respectively. Dataset | ML | Samples | Train/Val/Test | T/S | VS | AR | AC
---|---|---|---|---|---|---|---
TSD-250 | 250 | $2938392$ |  $2350713$ / $293839$ / $293840$ | $74.09$ | 25 | 6 | 4
TSD-500 | 500 | $3191891$ |  $2553512$ / $319189$ / $319190$ | $95.39$ | 25 | 8 | 5
TCD-250 | 250 | $1105636$ |  $884508$ / $110564$ / $110564$ | 131.96 | 718 | 4 | 4
TCD-500 | 500 | $1937686$ |  $1550148$ / $193769$ / $193769$ | $229.06$ | 737 | 5 | 4

(a) TSD-250

(b) TSD-500

(c) TCD-250

(d) TCD-500

Figure 4: Histograms representing the number of tokens distribution for the
tables present in the dataset.

##  4 Evaluation Metrics

We experiment with three evaluation metrics to compare the predicted sequence
of tokens ($T_{PT}$) against the sequence of tokens in the ground truth
($T_{GT}$). Even though the proposed metrics are heavily used in NLP research,
few TIE works have leveraged them in the past. Next, we discuss these metrics
and illustrate the implementation details using a toy example described in
Figure 5.

(a)

(b)

(c)

(d)

Figure 5: (a) A toy table, (b) corresponding ground truth token sequence for
TSR and TCR tasks, (c) model output sequence for TCR task, and (d) model
output sequence for TSR task.

  1. 1.

Exact Match Accuracy (EMA): EMA outputs the fraction of predictions with exact
string match of $T_{PT}$ against $T_{GT}$. A model having a higher value of
EMA is considered a good model. In Figure 5, for the TSR task, $T_{PT}$ misses
‘`\hline`’ token compared to the corresponding $T_{GT}$, resulting in EMA = 0.
Similarly, the $T_{PT}$ exactly matches the $T_{GT}$ for the TCR task,
resulting in EMA = 1.

  2. 2.

Bilingual Evaluation Understudy Score (BLEU): The BLEU [16] score is primarily
used in Machine Translation literature to compare the quality of the
translated sentence against the expected translation. Recently, several TIE
works [2, 4] have adapted BLEU for evaluation. It counts the matching n-grams
in $T_{PT}$ against the n-grams in $T_{GT}$. The comparison is made regardless
of word order. The higher the BLEU score, the better is the model. We use
SacreBLEU [17] implementation for calculating the BLEU score. We report scores
for the most popular variant, BLEU-4. BLEU-4 refers to the product of brevity
penalty (BP) and a harmonic mean of precisions for unigrams, bigrams, 3-grams,
and 4-grams (for more details see [16]). Figure 5, there is an exact match
between $T_{PT}$ and $T_{GT}$ for TCR task yielding BLEU = 100.00. In the case
of TSR, the missing ‘`\hline`’ token in $T_{PT}$ yields a lower BLEU = 89.66.

  3. 3.

Word Error Rate (WER): WER is defined as a ratio of the Levenshtein distance
between the $T_{PT}$ and $T_{GT}$ to the length of $T_{GT}$. It is a standard
evaluation metric for several OCR tasks. Since it measures the rate of error,
models with lower WER are better. We use
jiwer101010https://github.com/jitsi/jiwer Python library for WER computation.
In Figure 5, for the TCR task, the WER between $T_{PT}$ and $T_{GT}$ is 0.
Whereas in the case of TSR, the Levenshtein distance between $T_{PT}$ and
$T_{GT}$ is one due to the missing ‘`\hline`’ token. Since the length of
$T_{GT}$ is 35, WER comes out to be 0.02.

##  5 Experiments

In this section, we experiment with a deep learning-based model for TSR and
TCR tasks. We adapt an existing model [7] architecture proposed for the scene
text recognition task and train it from scratch on the TabLeX dataset. It uses
partial ResNet-101 along with a fully connected layer as a feature extractor
module for generating feature embeddings with a cascaded Transformer [28]
module for encoding the features and generating the output. Figure 6 describes
the detailed architecture of the model. Note that in contrast to the scene
image as an input in [7], we input a tabular image and predict the LaTeX token
sequence. We term it as the TIE-ResNet-Transformer model (TRT).

Figure 6: TIE-ResNet-Transformer model architecture.

###  5.1 Implementation Details

For both TSR and TCR tasks, we train a similar model. We use the third
intermediate hidden layer of partial ResNet-101 and an FC layer as a feature
extractor module to generate a feature map. The generated feature map (size =
$625\times 1024$) is further embedded into 256 dimensions, resulting in
reduced size of $625\times 256$. We experiment with four encoders and eight
decoders with learnable positional embeddings. The models are trained for ten
epochs with a batch size of 32 using cross-entropy loss function and Adam
Optimizer with an initial learning rate of $0.1$ and 2000 warmup training
steps, decreasing with Noam’s learning rate decay scheme. A dropout rate of
0.1 and $\epsilon_{ls}=0.1$ label smoothing parameter is used for
regularization. For decoding, we use the greedy decoding technique in which
the model generates an output sequence of tokens in an auto-regressive manner,
consuming the previously generated tokens to generate the next token. All the
experiments were performed on 4$\times$NVIDIA V100 graphics card.

###  5.2 Results

Table 3: TRT results on the TabLeX dataset.

Dataset | Aspect Ratio | EMA(%) | BLEU | WER(%)
---|---|---|---|---
TCD 250 | Conserved | 21.19 | 95.18 | 15.56
Fixed | 20.46 | 96.75 | 14.05
TCD 500 | Conserved | 11.01 | 91.13 | 13.78
Fixed | 11.23 | 94.34 | 11.12
TSD 250 | Conserved | 70.54 | 74.75 | 3.81
Fixed | 74.02 | 70.59 | 4.98
TSD 500 | Conserved | 70.91 | 82.72 | 2.78
Fixed | 71.16 | 61.84 | 9.34

Table 3 illustrates that higher sequence length significantly degrades the EMA
score for both the TSR and TCR tasks. For TCD-250 and TCD-500, high BLEU
scores ($>$90) suggest that the model can predict a large chunk of LaTeX
content information correctly. However, the errors are higher for images with
a conserved aspect ratio than with a fixed aspect ratio, which is confirmed by
comparing their BLEU score and WER metrics. Similarly, for TSD-250 and
TSD-500, high EMA and lower WER suggest that the model can correctly identify
structure information for most of the tables. TCD yields a lower EMA score
than TSD. We attribute this to several reasons. One of the reasons is that the
TCR model fails to predict some of the curly braces (‘{’ and ‘}’) and dollar
(‘$’) tokens in the predictions. After removing curly braces and dollar tokens
from the ground truth and predictions, EMA scores for conserved and fixed
aspect ratio images in TCD-250 increased to 68.78% and 75.33%, respectively.
Similarly, for TCD-500, EMA for conserved and fixed aspect ratio images
increases to 49.23% and 59.94%, respectively. In contrast, TSD do not contain
dollar tokens, and curly braces are only present at the beginning of the
column labels, leading to higher EMA scores. In the future, we believe the
results can be improved significantly by proposing better vision-based DL
architectures for TSR and TCR tasks.

##  6 Conclusion

This paper presents a benchmark dataset TabLeX for structure and content
information extraction from scientific tables. It contains tabular images in
12 different fonts with varied visual complexities. We also proposed a novel
preprocessing pipeline for dataset generation. We evaluate the existing state-
of-the-art transformer-based model and show excellent future opportunities in
developing algorithms for IE from scientific tables. In the future, we plan to
continuously augment the dataset size and add more complex tabular structures
for training and prediction.

## Acknowledgment

This work was supported by The Science and Engineering Research Board (SERB),
under sanction number ECR/2018/000087.

## References

  * [1] Chi, Z., Huang, H., Xu, H., Yu, H., Yin, W., Mao, X.: Complicated table structure recognition. CoRR abs/1908.04729 (2019), http://arxiv.org/abs/1908.04729
  * [2] Deng, Y., Kanervisto, A., Rush, A.M.: What you get is what you see: A visual markup decompiler. ArXiv abs/1609.04938 (2016)
  * [3] Deng, Y., Rosenberg, D., Mann, G.: Challenges in end-to-end neural scientific table recognition. In: 2019 International Conference on Document Analysis and Recognition (ICDAR). pp. 894–901 (2019). https://doi.org/10.1109/ICDAR.2019.00148
  * [4] Deng, Y., Kanervisto, A., Ling, J., Rush, A.M.: Image-to-markup generation with coarse-to-fine attention. In: Proceedings of the 34th International Conference on Machine Learning - Volume 70. p. 980–989. ICML’17, JMLR.org (2017)
  * [5] Douglas, S., Hurst, M., Quinn, D., et al.: Using natural language processing for identifying and interpreting tables in plain text. In: Proceedings of the Fourth Annual Symposium on Document Analysis and Information Retrieval. pp. 535–546 (1995)
  * [6] Embley, D.W., Hurst, M., Lopresti, D.P., Nagy, G.: Table-processing paradigms: a research survey. Int. J. Document Anal. Recognit. 8(2-3), 66–86 (2006). https://doi.org/10.1007/s10032-006-0017-x, https://doi.org/10.1007/s10032-006-0017-x
  * [7] Feng, X., Yao, H., Yi, Y., Zhang, J., Zhang, S.: Scene text recognition via transformer. arXiv preprint arXiv:2003.08077 (2020)
  * [8] Gao, L., Huang, Y., Déjean, H., Meunier, J., Yan, Q., Fang, Y., Kleber, F., Lang, E.: Icdar 2019 competition on table detection and recognition (ctdar). In: 2019 International Conference on Document Analysis and Recognition (ICDAR). pp. 1510–1515 (2019). https://doi.org/10.1109/ICDAR.2019.00243
  * [9] Göbel, M., Hassan, T., Oro, E., Orsi, G.: Icdar 2013 table competition. In: 2013 12th International Conference on Document Analysis and Recognition. pp. 1449–1453 (2013). https://doi.org/10.1109/ICDAR.2013.292
  * [10] Hao, L., Gao, L., Yi, X., Tang, Z.: A table detection method for pdf documents based on convolutional neural networks. DAS pp. 287–292 (2016)
  * [11] He, K., Gkioxari, G., Dollár, P., Girshick, R.B.: Mask R-CNN. CoRR abs/1703.06870 (2017), http://arxiv.org/abs/1703.06870
  * [12] Kasar, T., Bhowmik, T.K., Belaïd, A.: Table information extraction and structure recognition using query patterns. In: 2015 13th International Conference on Document Analysis and Recognition (ICDAR). pp. 1086–1090 (2015). https://doi.org/10.1109/ICDAR.2015.7333928
  * [13] Kieninger, T., Dengel, A.: A paper-to-html table converting system. In: Proceedings of document analysis systems (DAS). vol. 98, pp. 356–365 (1998)
  * [14] Li, M., Cui, L., Huang, S., Wei, F., Zhou, M., Li, Z.: Tablebank: Table benchmark for image-based table detection and recognition. CoRR abs/1903.01949 (2019), http://arxiv.org/abs/1903.01949
  * [15] Liu, Y., Bai, K., Mitra, P., Giles, C.L.: Tableseer: Automatic table metadata extraction and searching in digital libraries. In: Proceedings of the 7th ACM/IEEE-CS Joint Conference on Digital Libraries. p. 91–100. JCDL ’07, Association for Computing Machinery, New York, NY, USA (2007). https://doi.org/10.1145/1255175.1255193, https://doi.org/10.1145/1255175.1255193
  * [16] Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. pp. 311–318. Association for Computational Linguistics, Philadelphia, Pennsylvania, USA (Jul 2002). https://doi.org/10.3115/1073083.1073135, https://www.aclweb.org/anthology/P02-1040
  * [17] Post, M.: A call for clarity in reporting BLEU scores. In: Proceedings of the Third Conference on Machine Translation: Research Papers. pp. 186–191. Association for Computational Linguistics, Belgium, Brussels (Oct 2018), https://www.aclweb.org/anthology/W18-6319
  * [18] Qasim, S.R., Mahmood, H., Shafait, F.: Rethinking table parsing using graph neural networks. CoRR abs/1905.13391 (2019), http://arxiv.org/abs/1905.13391
  * [19] Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. In: Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., Garnett, R. (eds.) Advances in Neural Information Processing Systems. vol. 28, pp. 91–99. Curran Associates, Inc. (2015), https://proceedings.neurips.cc/paper/2015/file/14bfa6bb14875e45bba028a21ed38046-Paper.pdf
  * [20] Shahab, A., Shafait, F., Kieninger, T., Dengel, A.: An open approach towards the benchmarking of table structure recognition systems. In: Proceedings of the 9th IAPR International Workshop on Document Analysis Systems. p. 113–120. DAS ’10, Association for Computing Machinery, New York, NY, USA (2010). https://doi.org/10.1145/1815330.1815345, https://doi.org/10.1145/1815330.1815345
  * [21] Shigarov, A., Mikhailov, A., Altaev, A.: Configurable table structure recognition in untagged pdf documents. In: Proceedings of the 2016 ACM Symposium on Document Engineering. p. 119–122. DocEng ’16, Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2960811.2967152, https://doi.org/10.1145/2960811.2967152
  * [22] Siegel, N., Lourie, N., Power, R., Ammar, W.: Extracting scientific figures with distantly supervised neural networks. In: Proceedings of the 18th ACM/IEEE on Joint Conference on Digital Libraries. p. 223–232. JCDL ’18, Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3197026.3197040, https://doi.org/10.1145/3197026.3197040
  * [23] Singh, M., Sarkar, R., Vyas, A., Goyal, P., Mukherjee, A., Chakrabarti, S.: Automated early leaderboard generation from comparative tables. In: European Conference on Information Retrieval. pp. 244–257. Springer (2019)
  * [24] Smith, R.: An overview of the tesseract ocr engine. In: Proc. Ninth Int. Conference on Document Analysis and Recognition (ICDAR). pp. 629–633 (2007)
  * [25] Smith, R.: An overview of the tesseract ocr engine. In: Ninth International Conference on Document Analysis and Recognition (ICDAR 2007). vol. 2, pp. 629–633. IEEE (2007)
  * [26] Tao, X., Liu, Y., Fang, J., Qiu, R., Tang, Z.: Dataset, ground-truth and performance metrics for table detection evaluation. In: Document Analysis Systems, IAPR International Workshop on. pp. 445–449. IEEE Computer Society, Los Alamitos, CA, USA (mar 2012). https://doi.org/10.1109/DAS.2012.29, https://doi.ieeecomputersociety.org/10.1109/DAS.2012.29
  * [27] The ImageMagick Development Team: Imagemagick, https://imagemagick.org
  * [28] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: Advances in neural information processing systems. pp. 5998–6008 (2017)
  * [29] Wu, G., Zhou, J., Xiong, Y., Zhou, C., Li, C.: Tablerobot: an automatic annotation method for heterogeneous tables. Personal and Ubiquitous Computing pp. 1–7 (2021)
  * [30] Zhong, X., ShafieiBavani, E., Jimeno Yepes, A.: Image-based table recognition: Data, model, and evaluation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.M. (eds.) Computer Vision – ECCV 2020. pp. 564–580. Springer International Publishing, Cham (2020)
  * [31] Zhong, X., Tang, J., Jimeno-Yepes, A.: Publaynet: largest dataset ever for document layout analysis. CoRR abs/1908.07836 (2019), http://arxiv.org/abs/1908.07836
  * [32] Zhou, J., Cui, G., Zhang, Z., Yang, C., Liu, Z., Sun, M.: Graph neural networks: A review of methods and applications. CoRR abs/1812.08434 (2018), http://arxiv.org/abs/1812.08434