# Clipped Hyperbolic Classifiers Are Super-Hyperbolic Classifiers

Yunhui Guo1 Xudong Wang1 Yubei Chen2 Stella X. Yu1
1UC Berkeley / ICSI 2Facebook AI Research

###### Abstract

Hyperbolic space can naturally embed hierarchies, unlike Euclidean space.
Hyperbolic Neural Networks (HNNs) exploit such representational power by
lifting Euclidean features into hyperbolic space for classification,
outperforming Euclidean neural networks (ENNs) on datasets with known semantic
hierarchies. However, HNNs underperform ENNs on standard benchmarks without
clear hierarchies, greatly restricting HNNs’ applicability in practice.

Our key insight is that HNNs’ poorer general classification performance
results from vanishing gradients during backpropagation, caused by their
hybrid architecture connecting Euclidean features to a hyperbolic classifier.
We propose an effective solution by simply clipping the Euclidean feature
magnitude while training HNNs.

Our experiments demonstrate that clipped HNNs become super-hyperbolic
classifiers: They are not only consistently better than HNNs which already
outperform ENNs on hierarchical data, but also on-par with ENNs on MNIST,
CIFAR10, CIFAR100 and ImageNet benchmarks, with better adversarial robustness
and out-of-distribution detection.

##  1 Introduction

Many datasets are inherently hierarchical. WordNet [30] has a hierarchical
conceptual structure, users in social networks such as Facebook or twitter
form hierarchies based on different occupations and organizations [11].

Representing such hierarchical data in Euclidean space cannot capture and
reflect their semantic or functional resemblance [1, 34]. Hyperbolic space,
i.e., non-Euclidean space with constant negative curvature, has been leveraged
to embed data with hierarchical structures with low distortion owing to the
nature of exponential growth in volume with respect to its radius [34, 41,
40]. For instance, hyperbolic space has been used for analyzing the
hierarchical structure in single cell data [20], learning hierarchical word
embedding [34], embedding complex networks [1], etc.

Recent algorithms operate directly in hyperbolic space to exploit more
representational power. Examples are Hyperbolic Perceptron [47], Hyperbolic
Support Vector Machine [5], and Hyperbolic Neural Networks (HNNs) [8], an
alternative to standard Euclidean neural networks (ENNs).

---
a) HNNs employ a hybrid architecture.
|
b) Standard benchmarks. | c) Few-shot learning tasks.
Figure 1:  We propose an effective solution for training HNNs by clipping the
Euclidean features. Clipped HNNs become super-hyperbolic classifiers: They are
not only consistently better than HNNs which already outperform ENNs on
hierarchical data, but also on-par with ENNs on standard benchmarks. a) HNNs
employ a hybrid architecture. The Euclidean part converts an input into
Euclidean embedding. Then the Euclidean embedding is projected onto the
Poincaré model of hyperbolic space via exponential map Exp${}_{0}(\cdot)$.
Finally, the hyperbolic embeddings are classified with Poincaré hyperplanes.
Clipped HNNs utilize a reduced region of hyperbolic space. b) Clipped HNNs
outperform baseline HNNs on standard benchmarks. c) Clipped HNNs outperform
both HNNs and ENNs on 1-s (shot) 1-w (way) and 5-s (shot) 5-w (way) few-shot
learning tasks.

HNNs adopt a hybrid architecture [18] (Figure 1): An ENN is first used for
extracting image features in Euclidean space; they are then projected onto
hyperbolic space to be classified by a hyperbolic multiclass logistic
regression [8].

While HNNs outperform ENNs on several datasets with explicit hierarchies [8],
there are several serious limitations. 1) HNNs underperform ENNs on standard
classification benchmarks with flat or non-hierarchical semantic structures.
2) Even for image datasets that possess latent hierarchical structures there
are no experimental evidence that HNNs can capture such structures or provide
on-par performance with ENNs [18]. 3) Existing improvements on HNNs mainly
focus on reducing the number of parameters [42] or incorporating different
types of neural network layers such as attention [10] or convolution [42].
Unfortunately, why HNNs are worse than ENNs on standard benchmarks has not
been investigated or understood.

Our key insight is that HNNs’ poorer general classification performance is
caused by their hybrid architecture connecting Euclidean features to a
hyperbolic classifier. It leads to _vanishing gradients_ during training. In
particular, the training dynamics of HNNs push the hyperbolic embeddings to
the boundary of the Poincaré ball [2] which causes the gradients of Euclidean
parameters to vanish.

We propose a simple yet effective solution to this problem by simply clipping
the Euclidean feature magnitude during training, thereby preventing the
hyperbolic embedding from approaching the boundary during training. Our
experiments demonstrate that clipped HNNs become super-hyperbolic classifiers:
They are not only consistently better than HNNs which already outperform ENNs
on hierarchical data, but also on-par with ENNs on MNIST, CIFAR10, CIFAR100
and ImageNet benchmarks, with better adversarial robustness and out-of-
distribution detection.

Our paper makes the following contributions. 1) Our detailed analysis reveals
the underlying issue of vanishing gradients that makes HNNs worse than ENNs on
standard classification benchmarks. 2) We propose a simple yet effective
feature clipping solution. 3) Our extensive experimentation demonstrates that
clipped HNNs outperform standard HNNs and become on-par with ENNs on standard
benchmarks. They are also more robust to adversarial attacks and exhibit
stronger out-of-distribution detection capability than their Euclidean
counterparts.

##  2 Related Work

Supervised Learning in Hyperbolic Space. Several hyperbolic neural networks
were proposed in the seminal work of HNNs [8], including multinomial logitstic
regression (MLR), fully connected and recurrent neural networks which can
operate directly on hyperbolic embeddings, outperforming Euclidean variants on
text entailment and noisy-prefix prediction tasks. Hyperbolic Neural
Networks++ [42] reduces the number of parameters of HNNs and also introduces
hyperbolic convolutional layers. Hyperbolic attention networks [10] rewrite
the operations in the attention layers using gyrovector operations [45],
delivering gains on neural machine translation, learning on graphs and visual
question answering. Hyperbolic graph neural network [25] extends the
representational geometry of Graph Neural Networks (GNNs) [55] to hyperbolic
space. Hyperbolic graph attention networks [53] further studies GNNs with
attention mechanisms in hyperbolic space. HNNs have been used for few-shot
classification and person re-identification [18].

Unsupervised Learning in Hyperbolic Space. [32] uses a wrapped normal
distribution in hyperbolic space to construct hyperbolic variational
autoencoders (VAEs) [19], whereas [29] uses Gaussian generalizations in
hyperbolic space to construct Poincaré VAEs. [17] applies hyperbolic neural
networks to unsupervised 3D segmentation of complex volumetric data.

Our work differs from all the above-mentioned methods which focus on the
application of HNNs to data with natural tree structures. We extend HNNs to
standard classification benchmarks which may not have hierarchies. By
improving HNNs to the level of ENNs in these scenarios, we greatly enhance the
general applicability of HNNs.

##  3 Super-Hyperbolic Classifiers from Clipping

Our goal is to understand why HNNs underperform ENNs on standard image
classification benchmarks and propose corresponding solutions. First, we
review the basics of Riemannian geometry and HNNs. Then, we analyze the
vanishing gradient problem in training HNNs. Finally, we present the proposed
method and discuss its connections to existing methods.

###  3.1 Preliminaries

Smooth Manifold. An $n$-dimensional topological manifold $\mathcal{M}$ is a
topological space that is locally Euclidean of dimension $n$: Every point
$\mathbf{x}\in\mathcal{M}$ has a neighborhood that is homeomorphic to an open
subset of $\mathbb{R}^{n}$. A smooth manifold is a topological manifold with
additional smooth structure which is a maximal smooth atlas.

Riemannian Manifold. A Riemannian manifold $(\mathcal{M},\mathfrak{g})$ is a
real smooth manifold with a Riemannian metric $\mathfrak{g}$. The Riemannian
metric $\mathfrak{g}$ is defined on the tangent space
$T_{\mathbf{x}}{\mathcal{M}}$ of $\mathcal{M}$ which is a smoothly varying
inner product.

Inner Product and Norm on Riemannian Manifold. For $\mathbf{x}\in\mathcal{M}$
and any two vectors $\mathbf{v},\mathbf{w}\in T_{\mathbf{x}}{\mathcal{M}}$,
the inner product $\langle\mathbf{v},\mathbf{w}\rangle_{\mathbf{x}}$ is
defined as $\mathfrak{g}(\mathbf{v},\mathbf{w})$. With the definition of inner
product, for $\mathbf{v}\in T_{\mathbf{x}}{\mathcal{M}}$, the norm is defined
as
$\lVert\mathbf{v}\rVert_{\mathbf{x}}=\sqrt{\langle\mathbf{v},\mathbf{v}\rangle}_{\mathbf{x}}$.

Geodesics on Riemannian Manifold. A geodesic is a curve
$\gamma:[0,1]\rightarrow\mathcal{M}$ of constant speed that is locally
minimizing the distance between two points on the manifold.

Exponential Map on Riemannian Manifold. Given
$\mathbf{x},\mathbf{y}\in\mathcal{M},\mathbf{v}\in
T_{\mathbf{x}}{\mathcal{M}}$, and a geodesic $\gamma$ of length
$\lVert\mathbf{v}\rVert$ such that
$\gamma(0)=\mathbf{x},\gamma(1)=\mathbf{y},\gamma^{\prime}(0)=\mathbf{v}/\lVert\mathbf{v}\rVert$,
the exponential map
$\textnormal{Exp}_{\mathbf{x}}:T_{\mathbf{x}}{\mathcal{M}}\rightarrow\mathcal{M}$
satisfies $\textnormal{Exp}_{\mathbf{x}}(\mathbf{v})=\mathbf{y}$ and the
inverse exponential map
$\textnormal{Exp}^{-1}_{\mathbf{x}}:\mathcal{M}\rightarrow
T_{\mathbf{x}}{\mathcal{M}}$ satisfies
$\textnormal{Exp}^{-1}_{\mathbf{x}}(\mathbf{y})=\mathbf{v}$. For more details,
please refer to [4, 24]

Poincaré Ball Model for Hyperbolic Space. A hyperbolic space is a Riemannian
manifold with constant negative curvature. There are several isometric models
for hyperbolic space, one of the commonly used models is Poincaré ball model
[34, 8] which can be derived using stereoscopic projection of the hyperboloid
model [2]. The $n$-dimensional Poincaré ball model of constant negative
curvature $-c$ is defined as
$(\mathbb{B}^{n}_{c},\mathfrak{g}_{\mathbf{x}}^{c})$, where
$\mathbb{B}^{n}_{c}$ =
$\\{\mathbf{x}\in\mathbb{R}^{n}:c\lVert\mathbf{x}\rVert<1\\}$ and
$\mathfrak{g}_{\mathbf{x}}^{c}=(\gamma_{\mathbf{x}}^{c})^{2}I_{n}$ is the
Riemannian metric tensor. $I_{n}$ is the Euclidean metric tensor. The
conformal factor is defined as,

$\gamma_{\mathbf{x}}^{c}=\frac{2}{1-c\lVert\mathbf{x}\rVert^{2}}$ (1)

The conformal factor induces the inner product
$\langle\mathbf{u},\mathbf{v}\rangle_{\mathbf{x}}^{c}=(\gamma_{\mathbf{x}}^{c})^{2}\langle\mathbf{u},\mathbf{v}\rangle$
and norm
$\lVert\mathbf{v}\rVert^{c}_{\mathbf{x}}=\gamma_{\mathbf{x}}^{c}\lVert\mathbf{v}\rVert$
for all $\mathbf{u},\mathbf{v}\in T_{\mathbf{x}}\mathbb{B}^{n}_{c}$. The
exponential map of Poincaré ball model can be written analytically with the
operations of gyrovector space.

---
Figure 2: Hyperbolic distance grows exponentially as we move towards the
boundary of the Poincaré ball.
---
Figure 3: Poincaré hyperplane defined by $\mathbf{a}$ and $\mathbf{p}$. Blue
line is the orthogonal projection of $\mathbf{x}$ to the Poincaré hyperplane.

Distance in Poincaré Ball Model. Figure 2 shows that hyperbolic distance grows
much faster than Euclidean distance as we move towards the boundary of the
Poincaré ball. As we will show later, this fundamental fact would lead to an
optimization issue when we construct a neural network consisting of both
Euclidean and hyperbolic layers.

Gyrovector Space. A gyrovector space [45, 46] is an algebraic structure that
provides an analytic way to operate in hyperbolic space. Gyrovector space can
be used to define various operations such as scalar multiplication,
subtraction, exponential map, inverse exponential map in Poincaré ball model.

The basic operation in gyrovector space is called Möbius addition
$\oplus_{c}$. With Möbius addition $\oplus_{c}$, we can define vector addition
of two points in Poincaré ball model as,

$\mathbf{u}\oplus_{c}\mathbf{v}=\frac{(1+2c\langle\mathbf{u},\mathbf{v}\rangle+c\lVert\mathbf{v}\rVert^{2})\mathbf{u}+(1-c\lVert\mathbf{u}\rVert^{2})\mathbf{v}}{1+2c\langle\mathbf{u},\mathbf{v}\rangle+c^{2}\lVert\mathbf{u}\rVert^{2}\lVert\mathbf{v}\rVert^{2}}$
(2)

for all $\mathbf{u},\mathbf{v}\in\mathbb{B}^{n}_{c}$. Particularly,
$\lim_{c\rightarrow 0}\oplus_{c}$ converges to the standard $+$ in the
Euclidean space. For more details, please refer to the Supplementary.

Hyperbolic Neural Networks. Hyperbolic neural networks consist of an Euclidean
sub-network and a hyperbolic classifier (Figure 1). The Euclidean sub-network
$E(\mathbf{x})$ converts an input $\mathbf{x}$ such as an image into a
representation $\mathbf{x}^{E}$ in Euclidean space. $\mathbf{x}^{E}$ is then
projected onto hyperbolic space $\mathbb{B}^{n}_{c}$ via an exponential map
$\textnormal{Exp}^{c}_{\mathbf{0}}(\cdot)$ as
$\mathbf{x}^{H}\in\mathbb{B}^{n}_{c}$. The hyperbolic classifier
$H(\mathbf{x}^{H})$ performs classification based on $\mathbf{x}^{H}$ with the
standard cross-entropy loss $\ell$.

Let the parameters of the Euclidean sub-network be $\mathbf{w}^{E}$ and the
parameters of the hyperbolic classifier be $\mathbf{w}^{H}$. Given the loss
function $\ell$, the optimization problem can be formalized as,

$\min_{\mathbf{w}^{E},\mathbf{w}^{H}}\ell(H(\textnormal{Exp}_{\mathbf{0}}^{c}((E(\mathbf{x};\mathbf{w}^{E}));\mathbf{w}^{H}),y)$
(3)

where the outer and inner functions are
$H:\mathbb{B}^{n}_{c}\rightarrow\mathbb{R}$ and
$E:\mathbb{R}^{m}\rightarrow\mathbb{R}^{n}$. As shown in [8], the exponential
map is defined as,

$\textnormal{Exp}_{\mathbf{0}}^{c}(\mathbf{v})=\tanh(\sqrt{c}\lVert\mathbf{v}\rVert)\frac{\mathbf{v}}{\sqrt{c}\lVert\mathbf{v}\rVert}$
(4)

The construction of hyperbolic classifier relies on the following definition
of Poincaré hyperplanes,

######  Definition 3.1 (Poincaré hyperplanes [8])

For $\mathbf{p}\in\mathbb{B}^{n}_{c}$, $\mathbf{a}\in
T_{\mathbf{p}}\mathbb{B}^{n}_{c}\setminus\\{\mathbf{0}\\}$, the Poincaré
hyperplane is defined as,

$\tilde{H}^{c}_{\mathbf{a},\mathbf{p}}\coloneqq\\{\mathbf{x}\in\mathbb{B}^{n}_{c}:\langle-\mathbf{p}\oplus_{c}\mathbf{x},\mathbf{a}\rangle=0\\}$
(5)

where $\mathbf{a}$ is the normal vector. Figure 3 shows the Poincaré
hyperplane defined by $\mathbf{a}$ and $\mathbf{p}$.

[8] shows that in hyperbolic space the probability that a given
$\mathbf{x}\in\mathbb{B}^{n}_{c}$ is classified as class $k$ is,

$\small
p(y=k|\mathbf{x})\propto\exp(\langle-\mathbf{p}_{k}\oplus_{c}\mathbf{x},\mathbf{a}_{k}\rangle)\sqrt{\mathfrak{g}^{c}_{\mathbf{p_{k}}}(\mathbf{a}_{k},\mathbf{a}_{k})}d_{c}(\mathbf{x},\tilde{H}^{c}_{\mathbf{a}_{k},\mathbf{p}})$
(6)

where $d_{c}(\mathbf{x},\tilde{H}^{c}_{\mathbf{a}_{k},\mathbf{p}})$ is the
distance of the embedding $\mathbf{x}$ to the Poincaré hyperplane of class $k$
as shown in Figure 3. In hyperbolic classifier, the parameters are the vectors
$\\{\mathbf{p}_{k}\\}$ for each class $k$.

###  3.2 Vanishing Gradient Problem in Training Hyperbolic Neural Networks

Training Hyperbolic Neural Networks with Backpropagation. The standard
backpropagation algorithm [38] is used for training HNNs [8, 18]. During
backpropagation, the gradient of the Euclidean parameters $\mathbf{w}^{E}$ can
be computed as,

$\frac{\partial\ell}{\partial\mathbf{w}^{E}}=(\frac{\partial\mathbf{x}^{H}}{\partial\mathbf{w}^{E}})^{T}\frac{\partial\ell}{\partial\mathbf{x}^{H}}$
(7)

where $\mathbf{x}^{H}$ is the hyperbolic embedding of the input $\mathbf{x}$,
$\frac{\partial\mathbf{x}^{H}}{\partial\mathbf{w}^{E}}$ is the Jacobian matrix
and $\frac{\partial\ell}{\partial\mathbf{x}^{H}}$ is the gradient of the loss
function with respect to the hyperbolic embedding $\mathbf{x}^{H}$. It is
noteworthy that since $\mathbf{x}^{H}$ is an embedding in hyperbolic space,
$\frac{\partial\ell}{\partial\mathbf{x}^{H}}\in
T_{\mathbf{x}^{H}}\mathbb{B}^{n}_{c}$ is the Riemannian gradient [3] and

$\frac{\partial\ell}{\partial\mathbf{x}^{H}}=\frac{(1-\lVert\mathbf{x}^{H}\rVert^{2})^{2}}{4}\nabla\ell(\mathbf{x}^{H})$
(8)

where $\nabla\ell(\mathbf{x}^{H})$ is the Euclidean gradient.

Vanishing Gradient Problem. We conduct an experiment to show the vanishing
gradient problem during training hyperbolic neural networks. We train a LeNet-
like convolutional neural network [23] with hyperbolic classifier on the MNIST
data. We use a two-dimensional Poincaré ball for visualization. Figure 4 shows
the trajectories of the hyperbolic embeddings of six randomly sampled inputs
during training. The arrows indicate the movement of each embedding after one
gradient update step. It can be observed that at initialization all the
hyperbolic embeddings are close to the center of the Poincaré ball. During
training, the hyperbolic embeddings gradually move to the boundary of the
ball. The magnitude of the gradient diminishes during training as the training
loss decays. However, at the end of training, while the training loss slightly
increases, the gradient vanishes due to the issue that the hyperbolic
embeddings approach the boundary of the ball.

From Equation 6, we can see that in order to maximize the probability of the
correct prediction, we need to increase the distance of the hyperbolic
embedding to the corresponding Poincaré hyperplane, i.e.,
$d_{c}(\mathbf{x}^{H},\tilde{H}^{c}_{\mathbf{a}_{k},\mathbf{p}})$. The
training dynamics of HNNs thus push the hyperbolic embeddings to the boundary
of the Poincaré ball in which case $\lVert\mathbf{x}^{H}\rVert^{2}$ approaches
one. The inverse of the Riemannian metric tensor becomes zero which causes
$\lVert\frac{\partial\ell}{\partial\mathbf{x}^{H}}\rVert^{2}$ to be small.
From Equation 7, it is easy to see that if
$\lVert\frac{\partial\ell}{\partial\mathbf{x}^{H}}\rVert^{2}$ is small, then
$\lVert\frac{\partial\ell}{\partial\mathbf{w}^{E}}\rVert^{2}$ is small and the
optimization makes no progress on $\mathbf{w}^{E}$.

---
a)
b)
Figure 4: Hyperbolic neural networks suffer from vanishing gradient problem
during training with backpropagation. a) The trajectories of the hyperbolic
embeddings of six randomly sampled inputs during training in a 2-dimensional
Poincaré ball. The arrows indicate the change of location of each embedding
with each gradient update. The embeddings move to the boundary of the ball
during optimization which causes vanishing gradient problem. b) The gradient
vanishes while the training loss goes up at the end of training.

_Vanishing gradient problem_ [15, 37, 36, 12] is one of the difficulties in
training deep neural networks using backpropagation. Vanishing gradient
problem occurs when the magnitude of the gradient is too small for the
optimization to make progress. For Euclidean neural networks, vanishing
gradient problem can be alleviated by architecture designs [16, 14], proper
weight initialization [31] and carefully chosen activation functions [49].
However, the vanishing gradient problem in training HNNs is not exploited in
existing literature.

The Effect of Gradient Update of Euclidean Parameters on the Hyperbolic
Embedding. We derive the effect of a single gradient update of the Euclidean
parameters on the hyperbolic embedding, for more details please refer to the
Supplementary. For the Euclidean sub-network
$E:\mathbb{R}^{m}\rightarrow\mathbb{R}^{n}$, consider the first-order Taylor-
expansion with a single gradient update,

$\begin{split}E(\mathbf{w}^{E}_{t+1})&=E(\mathbf{w}_{t}^{E}+\eta\frac{\partial\ell}{\partial\mathbf{w}^{E}})\\\
&\approx E(\mathbf{w}^{E}_{t})+\eta(\frac{\partial
E(\mathbf{w}_{t}^{E})}{\partial\mathbf{w}_{t}^{E}})^{T}\frac{\partial\ell}{\partial\mathbf{w}^{E}}\\\
\end{split}$ (9)

where $\eta$ is the learning rate. The gradient of the exponential map can be
computed as,

$\small\begin{split}\nabla\textnormal{Exp}_{\mathbf{0}}^{c}(\mathbf{v})&=\frac{\mathbf{v}}{\sqrt{c}\lVert\mathbf{v}\rVert}\nabla\tanh({\sqrt{c}\lVert\mathbf{v}\rVert})+\tanh({\sqrt{c}\lVert\mathbf{v}\rVert})\nabla\frac{\mathbf{v}}{\sqrt{c}\lVert\mathbf{v}\rVert}\\\
&=1-\tanh^{2}({\sqrt{c}\lVert\mathbf{v}\rVert})+\tanh({\sqrt{c}\lVert\mathbf{v}\rVert})\frac{1}{\sqrt{c}}\frac{2}{\lVert\mathbf{v}\rVert}\end{split}$
(10)

Let $\mathbf{x}_{t+1}^{H}$ be the projected point in hyperbolic space, i.e.,

$\mathbf{x}_{t+1}^{H}=\textnormal{Exp}_{\mathbf{0}}^{c}(E(\mathbf{w}^{E}_{t+1}))$
(11)

By applying the first-order Taylor-expansion on the exponential map and
following standard derivations, we can find that,

$\begin{split}\mathbf{x}_{t+1}^{H}&=\mathbf{x}_{t}^{H}+C(E(\mathbf{w}^{E}_{t})^{T})\frac{\partial\ell}{\partial\mathbf{w}^{E}}\\\
\end{split}$ (12)

where
$C(E(\mathbf{w}^{E}_{t}))=\nabla\textnormal{Exp}_{\mathbf{0}}^{c}(E(\mathbf{w}^{E}_{t}))^{T}\eta(\frac{\partial
E(\mathbf{w}_{t}^{E})}{\partial\mathbf{w}_{t}^{E}})^{T}$. Thus once
$\lVert\mathbf{x}_{t}^{H}\rVert$ approaches one, from Equation 7 and Equation
8 we can find that the hyperbolic embedding stagnates no matter how large the
training loss is.

###  3.3 Clipped Hyperbolic Neural Networks

Euclidean Feature Clipping. There are several possible solutions to address
the vanishing gradient problem for training HNNs. One tentative solution is to
replace all the Euclidean layers with hyperbolic layers, however it is not
clear how to directly map the original input images onto hyperbolic space.
Another solution is to use normalized gradient descent [13] for optimizing the
Euclidean parameters to reduce the effect of gradient magnitude. However we
observed that this introduces instability during training and makes it harder
to tune the learning rate for optimizing Euclidean parameters.

We address the vanishing gradient problem by first reformulating the
optimization problem in Equation 3 with a regularization term which controls
the magnitude of hyperbolic embeddings,

$\min_{\mathbf{w}^{E},\mathbf{w}^{H}}\ell(H(\mathbf{x}^{H};\mathbf{w}^{H}),y)+\beta\lVert\mathbf{x}^{H}\rVert^{2}$
(13)

where
$\mathbf{x}^{H}=\textnormal{Exp}_{\mathbf{0}}^{c}((E(\mathbf{x};\mathbf{w}^{E}))$
and $\beta>0$ is a hyperparameter. By minimizing the training loss, the
hyperbolic embeddings tend to move to the boundary of the Poincaré ball which
causes the vanishing gradient problem. The additional regularization term is
used to prevent the hyperbolic embeddings from approaching the boundary.

Figure 5: The relation between the clipping value $r$ and the effective radius
of the Poincaré ball, i.e.,
$\textnormal{Exp}_{\mathbf{0}}^{c}(\textnormal{CLIP}(\mathbf{x}^{E};r))$.
Clipped HNNs utilize a reduced region of Poincaré ball.

While the soft constraint introduced in Equation 13 is effective, it
introduces additional complexity to the optimization process and has worse
performance. We instead employ the following hard constraint which regularizes
the Euclidean embedding before the exponential map whenever its norm exceeds a
given threshold,

$\textnormal{CLIP}(\mathbf{x}^{E};r)=\min\\{1,\frac{r}{\lVert\mathbf{x}^{E}\rVert}\\}\cdot\mathbf{x}^{E}$
(14)

where $\mathbf{x}^{E}=E(\mathbf{x};\mathbf{w}^{E})$ and $r>0$ is a
hyperparameter. The clipped Euclidean embedding is projected via exponential
map as
$\textnormal{CLIP}(\mathbf{x}^{H};r)=\textnormal{Exp}_{\mathbf{0}}^{c}(\textnormal{CLIP}(\mathbf{x}^{E};r))$.
The relation between the clipping value and the effective radius of Poincaré
ball is depicted in Figure 5.

Using the relation between the hyperbolic distance and the Euclidean distance
to the origin,

$d_{c}(0,x)=s\ln(\frac{s+x}{s-x})$ (15)

where $s=1/\sqrt{|c|}$, $c$ is the curvature and $x$ is Euclidean distance to
the origin. $c$ is usually set to -1. The clipping value $r$ can be further
converted into hyperbolic radius $r_{\mathbb{B}^{n}_{c}}$ as below,

$r_{\mathbb{B}^{n}_{c}}=d_{c}(0,\textnormal{CLIP}(\mathbf{x}^{H};r))=2r$ (16)

The hyperbolic radius $r_{\mathbb{B}^{n}_{c}}$ is measured in hyperbolic
distance. The hyperbolic embeddings are within a hyperbolic ball of radius of
$r_{\mathbb{B}^{n}_{c}}$.

Discussion on Feature Clipping. The proposed _Feature Clipping_ imposes a hard
constraint on the maximum norm of the hyperbolic embedding to prevent the
inverse of the Riemannian metric tensor from approaching zero. Therefore there
is always a gradient signal for optimizing the hyperbolic embedding. Although
decreasing the norm of the hyperbolic embedding shrinks the effective radius
of the embedding space, we found that it does no harm to accuracy while
alleviating the vanishing gradient problem.

A radius limited hyperbolic classifier is a super-hyperbolic classifier, not a
nearly Euclidean classifier. In the Supplementary, we show that clipped
hyperbolic space well maintains the hyperbolic property and delivers better
results for learning hierarchical word embeddings.

Discussion on Hyperbolic Embedding Literature. Similar regularization
approaches have been used in the hyperbolic embedding literature to prevent
numerical issues when optimizing hyperbolic embeddings [25, 35]. In contrast,
our work is focused on the hyperbolic neural networks for image classification
and its unique vanishing gradient issue, which is drastically different from
[25, 35] in terms of model architecture and the focused problem. In hyperbolic
neural networks, the gradients are backpropagated through the hyperbolic
layers to the Euclidean layers which causes the gradients to vanish. The
vanishing gradient issue will not occur in [25, 35] since Euclidean layers are
not adopted.

Lorentz model is used recently to overcome the numerical issues of Poincaré
ball model for learning word embeddings [35]. However, it is most effective
only in low dimensions [25]. For image datasets of ImageNet-scale, hyperbolic
neural networks with high-dimensional embeddings are necessary for enough
model capacity.

---
a) Accuracy of each class on MNIST.
b) Baseline HNNs
c) Clipped HNNs
Figure 6: Clipped HNNs greatly outperform baseline HNNs. a) The per class test
accuracy of baseline HNNs and Clipped HNNs. b) The Poincaré decision
hyperplanes and the hyperbolic embeddings of sampled test images of baseline
HNNs. c) The Poincaré decision hyperplanes and the hyperbolic embeddings of
sampled test images of clipped HNNs. Clipped HNNs learn more discriminative
feature in hyperbolic space. The per class accuracy indicates that baseline
HNNs learn biased feature space which hurts the performance on certain
classes.

##  4 Experimental Results

We conduct four types of experiments: standard balanced classification tasks,
few-shot learning tasks, adversarial robustness and out-of-distribution
detection. The results show that clipped HNNs are on par with ENNs on standard
recognition datasets while demonstrating better performance in terms of few-
shot classification, adversarial robustness and out-of-distribution detection.

Datasets. We consider four commonly used image classification datasets: MNIST
[22], CIFAR10 [21], CIFAR100 [21] and ImageNet [7]. See details in the
Supplementary. To our best knowledge, this paper is the first attempt to
extensively evaluate hyperbolic neural networks on the standard image
classification datasets for supervised classification.

Baselines and Networks. We compare the performance of HNNs training
with/without the proposed feature clipping method [8, 18] and their Euclidean
counterparts. For MNIST, we use a LeNet-like convolutional neural network [23]
which has two convolutional layers with max pooling layers in between and
three fully connected layers. For CIFAR10 and CIFAR100, we use WideResNet
[52]. For ImageNet, we use a standard ResNet18 [14].

Training Setups. For training ENNs, we use SGD with momentum. For training
HNNs, the Euclidean parameters of HNNs are trained using SGD, and the
hyperbolic parameters of HNNs are optimized using stochastic Riemann gradient
descent [3], just like the previous method. For training networks on MNIST, we
train the network for 10 epochs with a learning rate of 0.1. The batch size is
64. For training networks on CIFAR10 and CIFAR100, we train the network for
100 epochs with an initial learning rate of 0.1 and use cosine learning rate
scheduler [27]. The batch size is 128. For training networks on ImageNet, we
train the network for 100 epochs with an initial learning rate of 0.1 and the
learning rate decays by 10 every 30 epochs. The batch size is 256. We find the
HNNs are robust to the choice of the hyperparameter $r$, thus we fix $r$ to be
1.0 in all the experiments. For more discussions and results on the effect of
$r$, please see the Supplementary. For baseline HNNs, we use a clipping value
of 15 similar to [25, 35] to address the numerical issue. The experiments on
MNIST, CIFAR10 and CIFAR100 are repeated for 5 times and we report both
average accuracy and standard deviation. All the experiments are done on 8
NVIDIA TITAN RTX GPUs.

Results on Standard Benchmarks. Table 1 shows the results of different
networks on the considered benchmarks. On MNIST, we can observe that the
accuracy of the improved clipped HNNs is about 5% higher than the baseline
HNNs and match the performance of ENNs. On CIFAR10, CIFAR100 and ImageNet, the
improved HNNs achieve 6%, 3% and 3% improvement over baseline HNNs. The
results show that HNNs can perform well even on datasets which lack explicit
hierarchical structure.

Figure 6 shows the Poincaré hyperplanes of all the classes and the hyperbolic
embeddings of 1000 sampled test images extracted by the baseline HNNs and
clipped HNNs. Note that the Poincaré hyperplanes consist of arcs of Euclidean
circles that are orthogonal to the boundary of the ball. We also color the
points in the ball based on the classification results. It can be observed
that by regularizing the magnitude of the hyperbolic embedding, all the
embeddings locate in a restricted region of the whole Poincaré ball and the
network learns more regular and discriminative features in hyperbolic space.

Standard Classification
---
Task |  Euclidean [14] |  Hyperbolic [8] | C-Hyperbolic
MNIST |  99.12$\pm$0.34 |  94.42$\pm$0.29 |  99.08$\pm$0.31
CIFAR10 |  94.81$\pm$0.42 |  88.82$\pm$0.51 |  94.76$\pm$0.44
CIFAR100 |  76.24$\pm$0.35 |  72.26$\pm$0.41 |  75.88$\pm$0.38
ImageNet | 69.82 | 65.74 | 68.45
Few-Shot Classification on CUB Dataset
1-Shot 5-Way |  51.31$\pm$0.91 |  61.18$\pm$0.24 |  64.66$\pm$0.24
5-Shot 5-Way |  70.77$\pm$0.69 |  79.51$\pm$0.16 |  81.76$\pm$0.15
Few-Shot Classification on MiniImageNet Dataset
1-Shot 5-Way |  49.42$\pm$0.78 |  51.88$\pm$0.20 |  53.01$\pm$0.22
5-Shot 5-Way |  51.88$\pm$0.20 |  72.63$\pm$0.16 |  72.66$\pm$0.15
Table 1: Clipped HNN approaches ENN on standard classification benchmarks.
Clipped hyperbolic ProtoNet (C-Hyperbolic) greatly outperforms standard
hyperbolic ProtoNet (Hyperbolic) and Euclidean ProtoNet (Euclidean) on few-
shot learning tasks.

Few-Shot Learning. We show that the proposed feature clipping can also improve
the performance of Hyperbolic ProtoNet [18] for few-shot learning. Different
from the standard ProtoNet [43] which computes the prototype of each class in
Euclidean space, Hyperbolic ProtoNet computes the class prototype in
hyperbolic space using hyperbolic averaging. Hyperbolic features are shown to
be more effective than Euclidean features for few-shot learning [18].

We follow the experimental settings in [18] and conduct experiments on CUB
[48] and miniImageNet dataset [39]. We consider 1-shot 5-way and 5-shot 5-way
tasks as in [18]. The evaluation is repeated for 10000 times and we report the
average performance and the 95% confidence interval. Table 1 shows that the
proposed feature clipping further improves the accuracy of Hyperbolic ProtoNet
for few-shot classification by as much as 3%.

(a) On MNIST

(b) On MNIST

(c) On CIFAR10

Figure 7:  Clipped HNNs are more robust to ENNs against adversarial attacks. We show the results of adversarial robustness of clipped HNNs and ENNs to different attack methods and perturbations. Clipped HNNs are consistently better than ENNs.  Table 2: Clipped HNNs consistently outperform ENNs (shaded in gray) on out-of-distribution (OOD) detection with softmax scores when trained on CIFAR10 and tested on OOD datasets. OOD Dataset | $\mathbf{FPR95}\downarrow$ | $\mathbf{AUROC}\uparrow$ | $\mathbf{AUPR}\uparrow$
---|---|---|---
ISUN | 46.30$\pm$0.78 | 91.50$\pm$0.16 | 98.16$\pm$0.05
45.28$\pm$0.65 | 91.61$\pm$0.21 | 98.09$\pm$0.06
Place365 | 51.09$\pm$0.92 | 87.56$\pm$0.37 | 96.76$\pm$0.15
54.77$\pm$0.76 | 86.82$\pm$0.41 | 96.17$\pm$0.20
Texture | 65.04$\pm$0.91 | 82.80$\pm$0.35 | 94.59$\pm$0.20
47.12$\pm$0.62 | 89.91$\pm$0.20 | 97.39$\pm$0.09
SVHN | 71.66$\pm$0.84 | 86.58$\pm$0.21 | 97.06$\pm$0.06
49.89$\pm$1.03 | 91.34$\pm$0.22 | 98.13$\pm$ 0.06
LSUN-Crop | 22.22$\pm$0.78 | 96.05$\pm$0.10 | 99.16$\pm$0.03
23.87$\pm$0.73 | 95.65$\pm$0.22 | 98.98$\pm$0.07
LSUN-Resize | 41.06$\pm$1.07 | 92.67$\pm$0.16 | 98.42$\pm$0.04
41.49$\pm$1.24 | 92.97$\pm$0.24 | 98.46 $\pm$0.07
Mean | 49.56 | 89.53 | 97.36
43.74 | 91.38 | 97.87
Table 3: Clipped HNNs consistently outperform ENNs (shaded in gray) on out-of-distribution (OOD) detection with softmax scores when trained on CIFAR100 and tested on OOD datasets. On average, they are on par with ENNs by AUPR and far better by PRR95 and AUROC. OOD Dataset | $\mathbf{FPR95}\downarrow$ | $\mathbf{AUROC}\uparrow$ | $\mathbf{AUPR}\uparrow$
---|---|---|---
ISUN | 74.07$\pm$0.87 | 82.51$\pm$0.39 | 95.83$\pm$0.11
68.37$\pm$0.90 | 81.31$\pm$0.43 | 94.96$\pm$0.20
Place365 | 81.01$\pm$1.07 | 76.90$\pm$0.45 | 94.02$\pm$0.15
79.66$\pm$0.69 | 76.94$\pm$0.28 | 93.91$\pm$0.18
Texture | 83.67$\pm$0.68 | 77.52$\pm$0.32 | 94.47$\pm$0.10
64.91$\pm$0.80 | 83.26$\pm$0.25 | 95.77$\pm$0.08
SVHN | 84.56$\pm$0.78 | 84.32$\pm$0.22 | 96.69$\pm$0.07
53.11$\pm$1.04 | 89.53$\pm$0.26 | 97.71$\pm$0.07
LSUN-Crop | 43.46$\pm$0.79 | 93.09$\pm$0.23 | 98.58$\pm$0.05
51.08$\pm$1.17 | 87.21$\pm$0.39 | 96.83$\pm$0.13
LSUN-Resize | 71.50$\pm$0.73 | 82.12$\pm$0.40 | 95.69$\pm$0.13
63.86$\pm$1.10 | 82.36$\pm$0.42 | 95.16$\pm$0.13
Mean | 73.05 | 82.74 | 95.88
63.50 | 83.43 | 95.72

Adversarial Robustness. We show that clipped HNNs are more robust to
adversarial attacks including FGSM [9] and PGD [28] than ENNs. For attacking
networks trained on MNIST using FGSM, we consider the perturbation
$\epsilon=0.05,0.1,0.2,0.3$. For attacking networks trained on MNIST using
PGD, we consider the perturbation $\epsilon=0.05,0.1,0.15,0.2$. The number of
steps is 40. For attacking networks trained on CIFAR10 using PGD, we consider
the perturbation $\epsilon=0.8/255,1.6/255,3.2/255$. The number of steps is 7.

From Figure 7 we can see that across all the cases clipped HNNs show more
robustness than ENNs to adversarial attacks. For more discussions and results
using vanilla HNNs, please see the Supplementary.

Out-of-Distribution Detection. We conduct experiments to show that clipped
HNNs have stronger out-of-distribution detection capability than ENNs. Out-of-
distribution detection aims at determining whether or not a given input is
from the same distribution as the training data. We follow the experimental
settings in [26]. The in-distribution datasets are CIFAR10 and CIFAR100. The
out-of-distribution datasets are ISUN [50], Place365 [54], Texture [6], SVHN
[33], LSUN-Crop [51] and LSUN-Resize [51]. For detecting out-of-distribution
data, we use both softmax score and energy score as described in [26]. For
metrics, we consider FPR95, AUROC and AUPR [26]. Table 2 and 3 show the
results of using softmax score on CIFAR10 and CIFAR100 respectively. We can
see that HNNs and ENNs achieve similar AUPR, however HNNs achieve much better
performance in terms of FPR95 and AUROC. In particular, HNNs reduce FPR95 by
5.82% and 9.55% on CIFAR10 and CIFAR100 respectively. For results using energy
score and vanilla HNNs, please see the Supplementary.

The Effect of Feature Dimension. Figure 8 shows the change of test accuracy as
we vary the feature dimension on CIFAR10 and CIFAR100. Clipped HNNs are much
better than ENNs when the feature dimension is low. One possible reason is
that when the dimension is low in the Euclidean case, the data are hard to be
linearly separated. However in hyperbolic space, since the Poincaré
hyperplanes are “curved”, the data are more likely to be linearly separated
even in two dimensions.

|
---|---

·

Figure 8: Clipped HNNs are better than ENNs when the feature dimension is low.
Left: Test accuracy on CIFAR10. Right: Test accuracy on CIFAR100. We change
the feature dimension from 2 to 128.

##  5 Summary

We propose a simple yet effective solution called _Feature Clipping_ to
address the vanishing gradient problem in training HNNs. We conduct extensive
experiments on commonly used image dataset benchmarks. To the best of our
knowledge, this is the first time that HNNs can be applied to image datasets
of ImageNet-scale. Clipped HNNs show significant improvement over baseline
HNNs and match the performance of ENNs. The proposed feature clipping also
improves the performance of HNNs for few-shot learning. Further experimental
studies reveal that clipped HNNs are more robust to adversarial attacks such
as PGD and FGSM. Clipped HNNs also show stronger out-of-distribution detection
capability than ENNs. When the feature dimension is low, clipped HNNs even
outperform ENNs.

Acknowledgements. This work was supported, in part, by Berkeley Deep Drive and
the National Science Foundation under Grant No. 2131111. Any opinions,
findings, and conclusions or recommendations expressed here are those of the
authors and do not necessarily reflect the views of the National Science
Foundation.

##  6 Organization of the supplementary

###  6.1 Background and datasets

  1. 1.

In Section 7, we introduce the background on Gyrovector space.

  2. 2.

In Section 9, we give the detailed statistics of the datasets.

###  6.2 Effect of the gradient update

In Section 8, we derive the effect of the gradient update of the Euclidean
parameters on the hyperbolic embeddings.

###  6.3 Effect of the clipped value $r$

In Section 10, we show the effect of different choices of $r$. _To conclude,
there is a sweet spot in terms of choosing $r$ which is neither too large
(causing vanishing gradient problem) nor too small (not enough capacity). The
performance of clipped HNNs is also robust to the choice of the hyperparameter
$r$ if it is around the sweet spot._

###  6.4 Adversarial robustness

In Section 11, we show more results on adversarial robustness. _Clipped HNNs
show more robustness to ENNs and greatly improve the robustness of vanilla
HNNs._

###  6.5 More on out-of-distribution detection

In Section 12, we show more results on out-of-distribution (OOD) detection
using ENNs, vanilla HNNs and clipped HNNs. _Clipped HNNs show stronger OOD
detection capability compared with ENNs and greatly improve the OOD detection
capability of vanilla HNNs._

###  6.6 Using softmax with temperature scaling as a workaround

In Section 13 we show that using softmax with temperature scaling as a
workaround for addressing the vanishing gradient problem. _When feature
dimension is high, softmax with temperature scaling severely underperforms the
proposed feature clipping. The results again confirm the effectiveness of the
proposed approach._

###  6.7 Clipped hyperbolic space is still hyperbolic

In Section 14, we show that clipped hyperbolic space is still hyperbolic.
_Using clipped hyperbolic space for learning word embeddings is better than
using the unclipped version._

###  6.8 With norm regularization term

In Section 15, we show the results using norm regularization during training.
_The proposed feature clipping outperforms using the norm regularization
term._

###  6.9 More discussions on Lorentz model

In Section 16, we give more discussions on Lorentz model and why we focus on
hyperbolic neural networks based on Poincaré ball model.

##  7 Gyrovector space

We give more details on gyrovector space, for a more systematic treatment,
please refer to [45, 46, 44].

Gyrovector space provides a way to operate in hyperbolic space with vector
algebra. Gyrovector space to hyperbolic geometry is similar to standard vector
space to Euclidean geometry. The geometric objects in gyrovector space are
called gyroevectors which are equivalent classes of directed gyrosegments.
Similar to the vectors in Euclidean space which are added according to
parallelogram law, gyrovectors are added according to gyroarallelogram law.
Technically, gyrovector spaces are gyrocommutative gyrogroups of gyrovectors
that admit scalar multiplications.

We start from the introduction of gyrogroups which give rise to gyrovector
spaces.

######  Definition 7.1 (Gyrogroups)

A groupoid $(G,\oplus)$ is a gyrogroup if it satisfies the follow axioms,

  1. 1.

There exist one element $0\in G$ satisfies $0\oplus a=a$ for all $a\in G$.

  2. 2.

For each $a\in G$, there exist an element $\ominus a\in G$ which satisfies
$\ominus a\oplus a=0$

  3. 3.

For every $a,b,c\in G$, there exist a unique element gry$[a,b]c\in G$ such
that $\oplus$ satisfies the left gyroassociative law $a\oplus(b\oplus
c)=(a\oplus b)\oplus$gry$[a,b]c$.

  4. 4.

The map gry$[a,b]c$: $G\rightarrow G$ given by $c\mapsto$ gry$[a,b]c$ is an
automorphism of the groupoid $(G,\oplus)$: gyr$[a,b]\in Aut(G,\oplus)$. The
automorphism gyr$[a,b]$ of $G$ is called the gyroautomorphism of $G$ generated
by $a,b\in G$.

  5. 5.

The operation gry: $G\times G\rightarrow Aut(G,\oplus)$ is called gyrator of
$G$. The gyroautomorphism gyr$[a,b]$ generated by any $a,b\in G$ has the left
loop property: gyr$[a,b]$ = gyr$[a\oplus b,b]$.

In particular, Möbius complex disk groupoid $(\mathbb{D},\oplus_{M})$ is a
gyrocommunicative gyrogroup, where $\mathbb{D}=\\{z\in\mathbb{C}:|z|<1\\}$ and
$\oplus_{M}$ is the Möbius addition. The same applies to the $s$-ball
$\mathbb{V}_{s}$ which is defined as,

$\mathbb{V}_{s}=\\{\mathbf{v}\in\mathbb{V}:\lVert\mathbf{v}\rVert<s\\}$ (17)

Gyrocommutative gyrogroups which admit scalar multiplication $\oplus$ become
gyrovector space $(G,\oplus,\otimes)$. Möbius gyrogroups
$(\mathbb{V},\oplus_{M})$ admit scalar multiplication $\oplus_{M}$ become
Möbius gyrovector space $(\mathbb{V},\oplus_{M},\otimes_{M})$.

######  Definition 7.2 (Möbius Scalar Multiplication)

Let $(\mathbb{V}_{s},\oplus_{M})$ be a Möbius gyrogroup, the Möbius scalar
multiplication $\otimes_{M}$ is defined as,

$r\otimes_{M}\mathbf{v}=s\frac{(1+\frac{\lVert\mathbf{v}\rVert}{s})^{r}-(1-\frac{\lVert\mathbf{v}\rVert}{s})^{r}}{(1+\frac{\lVert\mathbf{v}\rVert}{s})^{r}+(1-\frac{\lVert\mathbf{v}\rVert}{s})^{r}}\frac{\mathbf{v}}{\lVert\mathbf{v}\rVert}$
(18)

where $r\in\mathbb{R}$ and $\mathbf{v}\in\mathbb{V}_{s}$,
$\mathbf{v}\neq\mathbf{0}$.

######  Definition 7.3 (Gyrolines)

Let $\mathbf{a},\mathbf{b}$ be two distinct points in the gyrovector space
$(G,\oplus,\otimes)$. The gyroline in $G$ which passes through
$\mathbf{a},\mathbf{b}$ is the set of points:

$L=\mathbf{a}\oplus(\ominus\mathbf{a}\oplus\mathbf{b})\otimes t$ (19)

where $t\in\mathbb{R}$.

|  |
---|---|---
Figure 9: We show the change of the test accuracy as we vary the
hyperparameter $r$. A large $r$ leads to _vanishing gradient problem_ and a
small $r$ causes insufficient capacity. Both lead to a drop in test accuracy.

It can be proven that gyrolines in a Möbius gyrovector space coincide with the
geodesics of the Poincaré ball model of hyperbolic geometry.

With the aid of operations in gyrovector spaces, we can define important
properties of the Poincaré ball model in closed-form expressions.

######  Definition 7.4 (Exponential Map and Logarithmic Map)

As shown in [8], the exponential map
$\exp^{c}_{\mathbf{x}}:T_{\mathbf{x}}\mathbb{B}^{n}_{c}\rightarrow\mathbb{B}^{n}_{c}$
is defined as,

$\scriptsize\exp^{c}_{\mathbf{x}}(\mathbf{v})=\mathbf{x}\oplus_{c}(\tanh(\frac{\sqrt{c}\lambda_{\mathbf{x}}^{c}\lVert\mathbf{v}\rVert}{2})\frac{\mathbf{v}}{\sqrt{c}\lVert\mathbf{v}\rVert}),\quad\quad\forall\mathbf{x}\in\mathbb{B}_{c}^{n},\mathbf{v}\in
T_{\mathbf{x}}\mathbb{B}^{n}_{c}$ (20)

The logarithmic map $\log_{\mathbf{x}}^{c}:\mathbb{B}^{n}_{c}\rightarrow
T_{\mathbf{x}}\mathbb{B}^{n}_{c}$ is defined as,

$\scriptsize\log_{\mathbf{x}}^{c}=\frac{2}{\sqrt{c}\lambda^{c}_{\mathbf{x}}}\tanh^{-1}(\sqrt{c}\lVert\ominus_{c}\mathbf{x}\oplus_{c}\mathbf{y}\rVert)\frac{\ominus_{c}\mathbf{x}\oplus_{c}\mathbf{y}}{\lVert\ominus_{c}\mathbf{x}\oplus_{c}\mathbf{y}\rVert},\quad\quad\mathbf{x},\mathbf{y}\in\mathbb{B}^{n}_{c}$
(21)

The distance between two points in the Poincaré ball can be defined as,

###### Definition 7.5

(Poincaré Distance between Two Points)

$d_{c}(\mathbf{x},\mathbf{y})=\frac{2}{\sqrt{c}}\tanh^{-1}(\sqrt{c}\lVert\ominus_{c}\mathbf{x}\oplus_{c}\mathbf{y}\rVert)$
(22)

Table 4: The results of out-of-distribution detection on CIFAR10 with softmax
score. The results of ENNs are shaded in dark gray. The results of vanilla
HNNs are shaded in light gray. Clipped HNNs achieve higher average performance
across all the datasets and greatly improve the OOD detection capability of
vanilla HNNs.

OOD Dataset | $\mathbf{FPR95}\downarrow$ | $\mathbf{AUROC}\uparrow$ | $\mathbf{AUPR}\uparrow$
---|---|---|---
ISUN | 46.30 $\pm$ 0.78 | 91.50 $\pm$ 0.16 | 98.16 $\pm$ 0.05
98.37 $\pm$ 0.20 | 29.72 $\pm$ 0.58 | 74.54 $\pm$ 0.19
45.28 $\pm$ 0.65 | 91.61 $\pm$ 0.21 | 98.09 $\pm$ 0.06
Place365 | 51.09 $\pm$ 0.92 | 87.56 $\pm$ 0.37 | 96.76 $\pm$ 0.15
96.10 $\pm$ 0.32 | 44.82 $\pm$ 0.65 | 80.67 $\pm$ 0.34
54.77 $\pm$ 0.76 | 86.82 $\pm$ 0.41 | 96.17 $\pm$ 0.20
Texture | 65.04 $\pm$ 0.91 | 82.80 $\pm$ 0.35 | 94.59 $\pm$0.20
97.62 $\pm$ 0.15 | 33.87 $\pm$ 0.40 | 74.52 $\pm$ 0.17
47.12 $\pm$ 0.62 | 89.91 $\pm$ 0.20 | 97.39 $\pm$ 0.09
SVHN | 71.66 $\pm$ 0.84 | 86.58 $\pm$ 0.21 | 97.06 $\pm$ 0.06
91.03 $\pm$ 0.53 | 61.33 $\pm$ 0.53 | 86.39 $\pm$ 0.27
49.89 $\pm$ 1.03 | 91.34 $\pm$ 0.22 | 98.13 $\pm$ 0.06
LSUN-Crop | 22.22 $\pm$ 0.78 | 96.05 $\pm$ 0.10 | 99.16 $\pm$ 0.03
96.18 $\pm$ 0.36 | 37.29 $\pm$ 0.63 | 75.94 $\pm$ 0.27
23.87 $\pm$ 0.73 | 95.65 $\pm$ 0.22 | 98.98 $\pm$ 0.07
LSUN-Resize | 41.06 $\pm$1.07 | 92.67 $\pm$ 0.16 | 98.42 $\pm$ 0.04
99.62 $\pm$ 0.10 | 22.05 $\pm$ 0.32 | 71.88 $\pm$ 0.15
41.49 $\pm$ 1.24 | 92.97 $\pm$ 0.24 | 98.46 $\pm$ 0.07
Mean | 49.56 | 89.53 | 97.36
96.49 | 38.18 | 77.32
43.74 | 91.38 | 97.87

Table 5: The results of out-of-distribution detection on CIFAR100 with softmax
score. The results of ENNs are shaded in dark gray. The results of vanilla
HNNs are shaded in light gray. Clipped HNNs achieve comparable performance to
ENNs in terms of AUPR and higher average performance in terms of PRR95 and
AUROC.

OOD Dataset | $\mathbf{FPR95}\downarrow$ | $\mathbf{AUROC}\uparrow$ | $\mathbf{AUPR}\uparrow$
---|---|---|---
ISUN | 74.07 $\pm$ 0.87 | 82.51 $\pm$ 0.39 | 95.83 $\pm$ 0.11
80.97 $\pm$ 0.65 | 69.24 $\pm$ 0.52 | 89.98 $\pm$ 0.22
68.37 $\pm$ 0.90 | 81.31 $\pm$ 0.43 | 94.96 $\pm$ 0.20
Place365 | 81.01 $\pm$ 1.07 | 76.90 $\pm$ 0.45 | 94.02 $\pm$ 0.15
82.75 $\pm$ 0.66 | 71.97 $\pm$ 0.50 | 92.27 $\pm$ 0.22
79.66 $\pm$ 0.69 | 76.94 $\pm$ 0.28 | 93.91 $\pm$ 0.18
Texture | 83.67 $\pm$ 0.68 | 77.52 $\pm$ 0.32 | 94.47 $\pm$ 0.10
75.33 $\pm$ 0.92 | 75.14 $\pm$ 0.49 | 92.39 $\pm$ 0.20
64.91 $\pm$ 0.80 | 83.26 $\pm$ 0.25 | 95.77 $\pm$ 0.08
SVHN | 84.56 $\pm$ 0.78 | 84.32 $\pm$ 0.22 | 96.69 $\pm$ 0.07
62.83 $\pm$ 0.70 | 84.29 $\pm$ 0.23 | 96.31 $\pm$ 0.06
53.11 $\pm$ 1.04 | 89.53 $\pm$ 0.26 | 97.71 $\pm$ 0.07
LSUN-Crop | 43.46 $\pm$ 0.79 | 93.09 $\pm$ 0.23 | 98.58 $\pm$ 0.05
56.66 $\pm$ 0.67 | 89.30 $\pm$ 0.17 | 97.71 $\pm$ 0.04
51.08 $\pm$ 1.17 | 87.21 $\pm$ 0.39 | 96.83 $\pm$ 0.13
LSUN-Resize | 71.50 $\pm$ 0.73 | 82.12 $\pm$ 0.40 | 95.69 $\pm$ 0.13
75.50 $\pm$ 0.81 | 73.40 $\pm$ 0.68 | 91.41 $\pm$ 0.28
63.86 $\pm$ 1.10 | 82.36 $\pm$ 0.42 | 95.16 $\pm$ 0.13
Mean | 73.05 | 82.74 | 95.88
72.34 | 77.22 | 93.35
63.50 | 83.43 | 95.72

Table 6: The results of out-of-distribution detection on CIFAR10 with energy
score. The results of ENNs are shaded in dark gray. The results of vanilla
HNNs are shaded in light gray.

OOD Dataset | $\mathbf{FPR95}\downarrow$ | $\mathbf{AUROC}\uparrow$ | $\mathbf{AUPR}\uparrow$
---|---|---|---
ISUN | 34.19 $\pm$ 0.97 | 93.07 $\pm$ 0.24 | 98.42 $\pm$ 0.07
99.31 $\pm$ 0.15 | 28.69 $\pm$ 0.35 | 74.14 $\pm$ 0.10
25.39 $\pm$ 0.32 | 95.48 $\pm$ 0.09 | 99.01 $\pm$ 0.04
Place365 | 43.34 $\pm$ 1.22 | 88.50 $\pm$ 0.48 | 96.76 $\pm$ 0.17
97.57 $\pm$ 0.37 | 43.96 $\pm$ 0.87 | 80.36 $\pm$ 0.46
45.17 $\pm$ 1.19 | 89.61 $\pm$ 0.28 | 97.20 $\pm$ 0.14
Texture | 58.51 $\pm$ 0.77 | 82.98 $\pm$ 0.20 | 94.55 $\pm$ 0.14
95.93 $\pm$ 0.29 | 35.02 $\pm$ 0.41 | 74.87 $\pm$ 0.14
49.70 $\pm$ 0.94 | 90.66 $\pm$ 0.20 | 97.98 $\pm$ 0.04
SVHN | 49.04 $\pm$ 1.05 | 91.57 $\pm$ 0.13 | 98.12 $\pm$ 0.05
96.71 $\pm$ 0.37 | 59.65 $\pm$ 0.56 | 86.16 $\pm$ 0.25
57.33 $\pm$ 1.34 | 88.45 $\pm$ 0.20 | 97.44 $\pm$ 0.06
LSUN-Crop | 9.48 $\pm$ 0.60 | 98.21 $\pm$ 0.07 | 99.63 $\pm$ 0.02
98.18 $\pm$ 0.27 | 36.34 $\pm$ 0.63 | 75.64 $\pm$ 0.26
24.78 $\pm$ 0.73 | 95.06 $\pm$ 0.15 | 98.92 $\pm$ 0.05
LSUN-Resize | 28.28 $\pm$ 0.66 | 94.31 $\pm$ 0.14 | 98.72 $\pm$ 0.04
99.91 $\pm$ 0.06 | 21.34 $\pm$ 0.48 | 71.60 $\pm$ 0.21
22.52 $\pm$ 0.67 | 96.15 $\pm$ 0.09 | 99.18 $\pm$ 0.02
Mean | 37.14 | 91.44 | 97.70
97.94 | 37.50 | 77.13
37.48 | 92.57 | 98.29

Table 7: The results of out-of-distribution detection on CIFAR100 with energy
score. The results of ENNs are shaded in dark gray. The results of vanilla
HNNs are shaded in light gray.

OOD Dataset | $\mathbf{FPR95}\downarrow$ | $\mathbf{AUROC}\uparrow$ | $\mathbf{AUPR}\uparrow$
---|---|---|---
ISUN | 74.49 $\pm$ 0.60 | 82.45 $\pm$ 0.33 | 95.84 $\pm$ 0.12
81.73 $\pm$ 0.54 | 70.38 $\pm$ 0.28 | 90.76 $\pm$ 0.20
68.75 $\pm$ 0.93 | 81.33 $\pm$ 0.31 | 94.93 $\pm$ 0.16
Place365 | 81.20 $\pm$ 0.86 | 77.02 $\pm$ 0.34 | 94.13 $\pm$ 0.13
82.73 $\pm$ 0.98 | 74.04 $\pm$ 0.55 | 93.20 $\pm$ 0.24
79.51 $\pm$ 0.69 | 77.23 $\pm$ 0.37 | 93.97 $\pm$ 0.17
Texture | 83.19 $\pm$ 0.31 | 77.74 $\pm$ 0.35 | 94.54 $\pm$ 0.11
72.77 $\pm$ 0.52 | 77.38 $\pm$ 0.39 | 93.38 $\pm$ 0.21
65.03 $\pm$ 0.52 | 83.38 $\pm$ 0.29 | 95.85 $\pm$ 0.10
SVHN | 84.12 $\pm$ 0.59 | 84.41 $\pm$ 0.16 | 96.72 $\pm$ 0.04
53.37 $\pm$ 0.67 | 86.37 $\pm$ 0.30 | 96.78 $\pm$ 0.08
55.44 $\pm$ 1.00 | 89.43 $\pm$ 0.25 | 97.69 $\pm$ 0.06
LSUN-Crop | 43.80 $\pm$ 1.29 | 93.04 $\pm$ 0.22 | 98.56 $\pm$ 0.05
87.32 $\pm$ 0.36 | 83.09 $\pm$ 0.20 | 96.40 $\pm$ 0.05
74.89 $\pm$ 0.73 | 84.98 $\pm$ 0.18 | 96.46 $\pm$ 0.08
LSUN-Resize | 71.86 $\pm$ 0.69 | 81.86 $\pm$ 0.27 | 95.60 $\pm$ 0.09
81.81 $\pm$ 0.71 | 72.96 $\pm$ 0.59 | 91.64 $\pm$ 0.23
64.35 $\pm$ 0.62 | 82.64 $\pm$ 0.36 | 95.27 $\pm$ 0.14
Mean | 73.11 | 82.75 | 95.90
76.62 | 77.37 | 93.69
67.99 | 83.17 | 95.70

##  8 The Effect of gradient update of Euclidean parameters on the hyperbolic
embedding

We derive the effect of the a single gradient update of the Euclidean
parameters on the hyperbolic embedding. For the Euclidean sub-network
$E:\mathbb{R}^{m}\rightarrow\mathbb{R}^{n}$. Consider the first-order Taylor-
expansion of the Euclidean network with a single gradient update,

$\begin{split}E(\mathbf{w}^{E}_{t+1})&=E(\mathbf{w}_{t}^{E}+\eta\frac{\partial\ell}{\partial\mathbf{w}^{E}})\\\
&\approx E(\mathbf{w}^{E}_{t})+\eta(\frac{\partial
E(\mathbf{w}_{t}^{E})}{\partial\mathbf{w}_{t}^{E}})^{T}\frac{\partial\ell}{\partial\mathbf{w}^{E}}\\\
\end{split}$ (23)

Meanwhile, the exponentional map of the Poincaré ball is,

$\textnormal{Exp}_{\mathbf{0}}^{c}(\mathbf{v})=\tanh(\sqrt{c}\lVert\mathbf{v}\rVert)\frac{\mathbf{v}}{\sqrt{c}\lVert\mathbf{v}\rVert}$
(24)

The gradient of the exponential map can be computed as,

$\scriptsize\begin{split}\nabla\textnormal{Exp}_{\mathbf{0}}^{c}(\mathbf{v})&=\frac{\mathbf{v}}{\sqrt{c}\lVert\mathbf{v}\rVert}\nabla\tanh({\sqrt{c}\lVert\mathbf{v}\rVert})+\tanh({\sqrt{c}\lVert\mathbf{v}\rVert})\nabla\frac{\mathbf{v}}{\sqrt{c}\lVert\mathbf{v}\rVert}\\\
&=1-\tanh^{2}({\sqrt{c}\lVert\mathbf{v}\rVert})+\tanh({\sqrt{c}\lVert\mathbf{v}\rVert})\frac{1}{\sqrt{c}}\frac{2}{\lVert\mathbf{v}\rVert}\end{split}$
(25)

Let $\mathbf{x}_{t+1}^{H}$ be the projected point in hyperbolic space, i.e,

$\mathbf{x}_{t+1}^{H}=\textnormal{Exp}_{\mathbf{0}}^{c}(E(\mathbf{w}^{E}_{t+1}))$
(26)

Again we can apply the first-order Taylor-expansion on the exponential map,

$\begin{split}\mathbf{x}_{t+1}^{H}&=\textnormal{Exp}_{\mathbf{0}}^{c}(E(\mathbf{w}^{E}_{t+1}))\\\
&\approx\textnormal{Exp}_{\mathbf{0}}^{c}(E(\mathbf{w}^{E}_{t})+\eta(\frac{\partial
E(\mathbf{w}_{t}^{E})}{\partial\mathbf{w}_{t}^{E}})^{T}\frac{\partial\ell}{\partial\mathbf{w}^{E}})\\\
\end{split}$ (27)

Denote $\eta(\frac{\partial
E(\mathbf{w}_{t}^{E})}{\partial\mathbf{w}_{t}^{E}})^{T}\frac{\partial\ell}{\partial\mathbf{w}^{E}}$
by $J_{\mathbf{w}^{E}_{t}}$, we have

$\begin{split}\mathbf{x}_{t+1}^{H}&=\textnormal{Exp}_{\mathbf{0}}^{c}(E(\mathbf{w}^{E}_{t+1}))\\\
&\approx\textnormal{Exp}_{\mathbf{0}}^{c}(E(\mathbf{w}^{E}_{t})+J_{\mathbf{w}^{E}_{t}})\\\
&\approx\textnormal{Exp}_{\mathbf{0}}^{c}(E(\mathbf{w}^{E}_{t}))+(\frac{\partial\textnormal{Exp}_{\mathbf{0}}^{c}(E(\mathbf{w}^{E}_{t}))}{\partial
E(\mathbf{w}^{E}_{t})})^{T}J_{\mathbf{w}^{E}_{t}}\\\
&=\mathbf{x}_{t}^{H}+(\frac{\partial\textnormal{Exp}_{\mathbf{0}}^{c}(E(\mathbf{w}^{E}_{t}))}{\partial
E(\mathbf{w}^{E}_{t})})^{T}J_{\mathbf{w}^{E}_{t}}\end{split}$ (28)

Denote
$(\frac{\partial\textnormal{Exp}_{\mathbf{0}}^{c}(E(\mathbf{w}^{E}_{t}))}{\partial
E(\mathbf{w}^{E}_{t})})^{T}\eta(\frac{\partial
E(\mathbf{w}_{t}^{E})}{\partial\mathbf{w}_{t}^{E}})^{T}$ by
$C(E(\mathbf{w}^{E}_{t}))$,

$\begin{split}\mathbf{x}_{t+1}^{H}&=\mathbf{x}_{t}^{H}+C(E(\mathbf{w}^{E}_{t}))^{T}\frac{\partial\ell}{\partial\mathbf{w}^{E}}\\\
\end{split}$ (29)

##  9 Datasets

The statistics of the datasets are shown in Table 8.

| MNIST | CIFAR10 | CIFAR100 | ImageNet
---|---|---|---|---
# of Training Examples | 60,000 | 50,000 | 50,000 | 1,281,167
# of Test Examples | 10,000 | 10,000 | 10,000 | 50,000
Table 8: The statistics of the datasets.

##  10 The effect of hyperparameter r

We conduct ablation studies to show the effect of the hyperparameter $r$ which
is the maximum norm of the Euclidean embedding. In Figure 9 we show the change
of test accuracy as we vary the hyperparameter $r$ on MNIST, CIFAR10 and
CIFAR100. We repeat the experiments for each choice of $r$ five times and
report both average accuracy and standard deviation. On the one hand, it can
be observed that a larger $r$ leads to a drop in test accuracy. As we point
out, this is caused by the vanishing gradient problem in training hyperbolic
neural networks. On the other hand, a small $r$ can also lead to a drop in
test accuracy especially on more complex tasks such as CIFAR10 and CIFAR100.
The plausible reason is that a small $r$ reduces the capacity of the embedding
space which is detrimental for learning discriminative features.

To conclude, there is a sweet spot in terms of choosing $r$ which is neither
too large (causing vanishing gradient problem) nor too small (not enough
capacity). The performance of clipped HNNs is also robust to the choice of the
hyperparameter $r$ if it is around the sweet spot.

Figure 10: Clipped HNNs show more adversarial robustness compared with ENNs.
We show the clean image and the corresponding adversarial image and the
predictions of the network of 10 randomly sampled images. In several cases,
clipped HNNs make correct predictions on the adversarial images while
Euclidean neural networks make wrong predictions.

##  11 More results on adversarial robustness

Although we observe that with adversarial training, hyperbolic neural networks
achieve similar robust accuracy to Euclidean neural networks, in a further
study, we consider training models using a small $\epsilon$ but attacking with
a larger $\epsilon$ with FGSM on MNIST. In Table 9 we show the results of
training the networks using $\epsilon=0.05$ and attacking with $\epsilon$ =
0.1, 0.2 and 0.3. We can observe that for attacking with larger $\epsilon$
such as 0.2 and 0.3, clipped HNNs show more robustness to ENNs. Moreover,
clipped HNNs greatly improve the robustness of vanilla HNNs. The possible
explanation is that the proposed feature clipping reduces the adversarial
noises in the forward pass and also improve the performance of vanilla HNNs.
One of the future directions is to systematically understand and analyze the
reason behind the robustness of clipped HNNs. In Figure 10, we show the clean
and adversarial images generated by FGSM with clipped HNNs and ENNs
respectively. The predictions of the networks are shown above the image. It
can be observed that clipped HNNs show more adversarial robustness compared
with ENNs.

Network $\epsilon$ | 0.1 | 0.2 | 0.3
---|---|---|---
ENNs | 94.51 $\pm$ 0.32 % | 67.85 $\pm$ 2.12 % | 42.18 $\pm$ 1.32 %
Vanilla HNNs |  81.08 $\pm$ 4.03 % |  46.57 $\pm$ 2.09 % |  17.21 $\pm$ 3.27%
Clipped HNNs |  93.34 $\pm$ 0.16 % |  74.97 $\pm$ 1.02 % |  46.27 $\pm$ 1.88 %

Table 9: Adversarial training with FGSM ($\epsilon$ = 0.05) on MNIST. For
attacking with larger $\epsilon$ such as 0.1, 0.2 and 0.3, clipped HNNs
greatly improve the robustness of vanilla HNNs and show more robustness to
ENNs when the attacking with large perturbations ($\epsilon$ = 0.2 and
$\epsilon$ = 0.3).

##  12 More results on out-of-distribution detection (OOD)

###  12.1 Results with energy score

In Table 6 and 7 we show the results of using energy score [26] on CIFAR10 and
CIFAR100 for out-of-distribution detection. We can observe that on CIFAR10,
clipped HNNs achieve comparable performance in terms of FPR95 and perform much
better in terms other AUROC and AUPR compared with ENNs. On CIFAR100, clipped
HNNs achieve comparable performance in terms of AUPT and perform much better
in terms other FPR95 and AUROC compared with ENNs. The results are consistent
with the case of using softmax score.

###  12.2 Clipped HNNs greatly improve the OOD detection capability of vanilla
HNNs.

In Table 4 \- Table 7, we also show the results of using vanilla HNNs for out-
of-distribution detection. Across all the datasets and scores, we can see that
clipped HNNs greatly improve the OOD detection capability of vanilla HNNs.
This shows that vanilla HNNs have poor OOD detection ability which can greatly
limit their practical applicability.

|
---|---
Figure 11: We show the change of the test accuracy as we vary the temperature
parameter $T$. The red horizontal line is the result of the hyperbolic neural
networks with the proposed feature clipping. Softmax with temperature scaling
with a carefully tuned temperature can approach the performance of the
proposed feature clipping. However, it is sensitive to the feature dimension
and the temperature parameter. Left: the embedding dimension is 2. Right: the
embedding dimension is 64.

##  13 Softmax with temperature scaling

We consider softmax with temperature scaling as an alternative for addressing
the vanishing gradient problem in training hyperbolic neural networks. Softmax
with temperature scaling introduces an additional temperature parameter $T$ to
adjust the logits before applying the softmax function. Softmax with
temperature scaling can be formulated as,

$\scriptsize\textnormal{Softmax}(\bm{Z}/T)_{i}=\frac{e^{Z_{i}/T}}{\sum_{j=1}^{K}e^{Z_{j}/T}}\quad\textnormal{for}\quad
i=1,...,K,\quad\bm{Z}=(Z_{1},...,Z_{K})$ (30)

In hyperbolic neural networks, $\bm{Z}$ is the output of the hyperbolic fully-
connected layer and $K$ is the number of classes. If the additional
temperature parameter $T$ is smaller than 1, the magnitude (in the Euclidean
sense) of the hyperbolic embedding will be scaled up which prevents it from
approaching the boundary of the ball.

In Figure 11, we show the performance of training hyperbolic neural networks
with temperature scaling compared with the proposed feature clipping. We
consider feature dimensions of 2 and 64 respectively. Different temperature
parameters are considered and the experiments are repeated for 10 times with
different random seeds. We show both the average accuracy and the standard
deviation. We can observe that softmax with temperature scaling and a
carefully tuned temperature parameter can approach the performance of the
proposed feature clipping when the feature dimension is 2. However, the
feature dimension is 64, softmax with temperature scaling severely
underperforms the proposed feature clipping. The results again confirm the
effectiveness of the proposed approach.

|
---|---
Figure 12: A magnitude-clipped hyperbolic space is still hyperbolic and
behaves drastically different from Euclidean space. Left: the comparison of
hyperbolic line segment with Euclidean line segment. Right: the comparison of
hyperbolic triangle with Euclidean triangle.

##  14 A magnitude-clipped hyperbolic space is still hyperbolic

The metric in the hyperbolic space with the clipping strategy is still
drastically different from that in the Euclidean space, even with magnitude
clipping. For the first example, consider two points: $a$ = [0.5, 0.55], $b$ =
[0.3, -0.6], the magnitude of both points is smaller than 0.76. The hyperbolic
distance between the two points is 3.1822 while the Euclidean distance is
1.1673. This is a two-dimensional example, with a larger embedding dimension,
the difference will be much more significant. In Figure 12: left, we compare
the hyperbolic line segment with Euclidean line segment given the point $a$
and the point $b$.

A magnitude-clipped hyperbolic space is still hyperbolic, as the hyperbolic
geometry still holds: unlike Euclidean triangles, where the angles always add
up to $\pi$ radians (180∘, a straight angle), in hyperbolic geometry the sum
of the angles of a hyperbolic triangle is always strictly less than $\pi$
radians (180∘, a straight angle). The difference is referred to as the defect.
For a second example, consider three points: A = [0.5, 0.55], B = [0.3, -0.6],
C = [-0.1, 0.1]. Their magnitude are all smaller than 0.76. For the triangle
ABC, the defect is 58.21∘ in hyperbolic space and 0∘ in Euclidean space. This
again shows that the clipped hyperbolic space still well maintains the
hyperbolic property. In Figure 12: right, we compare the hyperbolic triangle
with Euclidean triangle given the point A, the point B and the point C.

We apply the proposed clipping strategy to learn word embedding as in [34]. We
perform the reconstruction task on the transitive closure of the WordNet noun
hierarchy. We compare the embedding quality of the Euclidean space, the
hyperbolic space, clipped hyperbolic space using mean average precision (mAP).
The embedding dimension is 10. The results are summarized in Table 10.

We have two conclusions here. First, learning word embeddings with hyperbolic
space provides better results than learning in Euclidean space. Second, using
hyperbolic space with clipping is slightly better than using hyperbolic space
without clipping.

Method | mAP
---|---
Euclidean space | 0.059
Clipped hyperbolic space | 0.860
Hyperbolic space | 0.851
Table 10: Learning with word embeddings with clipped hyperbolic space
outperforms both with Euclidean space and vanilla hyperbolic space.

##  15 Additional regularization to minimize the norm of the Euclidean
embedding during training

The results of using the regularization term are shown in Table 11.

Method | On CIFR10 | On CIFAR100
---|---|---
vanilla HNN | 88.82 | 72.26
w/ regularization | 92.71 | 73.34
w/ clipping | 94.76 | 75.88
Table 11: The proposed feature clipping outperforms vanilla HNNs and HNNs with
regularization.

We can see that the clipping strategy outperforms the regularization approach.
The reason is that with regularization, the loss function consists of two
terms: one is the cross-entropy loss and the other is the regularization loss.
It is difficult to balance the two terms . During training, if the cross-
entropy loss becomes small, the optimization focuses on minimizing the
embedding norm, however a small embedding norm is also detrimental to the
performance.

##  16 More discussions on Lorentz model

Lorentz model is another commonly used model for hyperbolic space. The Lorentz
model of $n$-dimensional hyperbolic space is defined as,

$\mathcal{H}^{n}=\\{\mathbf{x}\in\mathbb{R}^{n+1}:\langle\mathbf{x},\mathbf{x}\rangle_{\mathcal{L}}=-1,x_{0}>0\\}$
(31)

where $\langle\mathbf{x},\mathbf{x}\rangle_{\mathcal{L}}$ is the _Lorentzian
scalar product_ which is defined as,

$\langle\mathbf{x},\mathbf{y}\rangle_{\mathcal{L}}=-x_{0}y_{0}+\sum_{i=1}^{n}x_{i}y_{i}$
(32)

The distance function of Lorentz model is given as,

$d_{\mathcal{H}}(\mathbf{x},\mathbf{y})=\operatorname{arccosh}(-\langle\mathbf{x},\mathbf{y}\rangle_{\mathcal{L}})$
(33)

Lorentz model is used recently to overcome the numerical issues of the
distance function in Poincaré ball model for learning word embeddings [35].
However, it is most effective only in low dimensions [25]. For image datasets
of ImageNet-scale, hyperbolic neural networks with high-dimensional embeddings
are necessary for enough model capacity. Moreover, current hyperbolic neural
network layers are only designed for Poinaré ball model. To extend hyperbolic
neural networks layers for Lorentz model can be an interesting future work.

## References

  * [1] Gregorio Alanis-Lobato, Pablo Mier, and Miguel A Andrade-Navarro.  Efficient embedding of complex networks to hyperbolic space via their laplacian.  Scientific reports, 6(1):1–10, 2016.
  * [2] James W Anderson.  Hyperbolic geometry.  Springer Science & Business Media, 2006.
  * [3] Silvere Bonnabel.  Stochastic gradient descent on riemannian manifolds.  IEEE Transactions on Automatic Control, 58(9):2217–2229, 2013.
  * [4] Manfredo Perdigao do Carmo.  Riemannian geometry.  Birkhäuser, 1992.
  * [5] Hyunghoon Cho, Benjamin DeMeo, Jian Peng, and Bonnie Berger.  Large-margin classification in hyperbolic space.  In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1832–1840. PMLR, 2019.
  * [6] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi.  Describing textures in the wild.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3606–3613, 2014.
  * [7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [8] Octavian-Eugen Ganea, Gary Bécigneul, and Thomas Hofmann.  Hyperbolic neural networks.  arXiv preprint arXiv:1805.09112, 2018.
  * [9] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  arXiv preprint arXiv:1412.6572, 2014.
  * [10] Caglar Gulcehre, Misha Denil, Mateusz Malinowski, Ali Razavi, Razvan Pascanu, Karl Moritz Hermann, Peter Battaglia, Victor Bapst, David Raposo, Adam Santoro, et al.  Hyperbolic attention networks.  arXiv preprint arXiv:1805.09786, 2018.
  * [11] Mangesh Gupte, Pravin Shankar, Jing Li, Shanmugauelayut Muthukrishnan, and Liviu Iftode.  Finding hierarchy in directed online social networks.  In Proceedings of the 20th international conference on World wide web, pages 557–566, 2011.
  * [12] Boris Hanin.  Which neural net architectures give rise to exploding and vanishing gradients?  arXiv preprint arXiv:1801.03744, 2018.
  * [13] Elad Hazan, Kfir Y Levy, and Shai Shalev-Shwartz.  Beyond convexity: Stochastic quasi-convex optimization.  arXiv preprint arXiv:1507.02030, 2015.
  * [14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [15] Sepp Hochreiter.  The vanishing gradient problem during learning recurrent neural nets and problem solutions.  International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 6(02):107–116, 1998.
  * [16] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  Neural computation, 9(8):1735–1780, 1997.
  * [17] Joy Hsu, Jeffrey Gu, Gong-Her Wu, Wah Chiu, and Serena Yeung.  Learning hyperbolic representations for unsupervised 3d segmentation.  arXiv preprint arXiv:2012.01644, 2020.
  * [18] Valentin Khrulkov, Leyla Mirvakhabova, Evgeniya Ustinova, Ivan Oseledets, and Victor Lempitsky.  Hyperbolic image embeddings.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6418–6428, 2020.
  * [19] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114, 2013.
  * [20] Anna Klimovskaia, David Lopez-Paz, Léon Bottou, and Maximilian Nickel.  Poincaré maps for analyzing complex hierarchies in single-cell data.  Nature communications, 11(1):1–9, 2020.
  * [21] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [22] Yann LeCun.  The mnist database of handwritten digits.  http://yann. lecun. com/exdb/mnist/, 1998.
  * [23] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324, 1998.
  * [24] John M Lee.  Introduction to Riemannian manifolds.  Springer, 2018.
  * [25] Qi Liu, Maximilian Nickel, and Douwe Kiela.  Hyperbolic graph neural networks.  arXiv preprint arXiv:1910.12892, 2019.
  * [26] Weitang Liu, Xiaoyun Wang, John D Owens, and Yixuan Li.  Energy-based out-of-distribution detection.  arXiv preprint arXiv:2010.03759, 2020.
  * [27] Ilya Loshchilov and Frank Hutter.  Sgdr: Stochastic gradient descent with warm restarts.  arXiv preprint arXiv:1608.03983, 2016.
  * [28] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.  Towards deep learning models resistant to adversarial attacks.  arXiv preprint arXiv:1706.06083, 2017.
  * [29] Emile Mathieu, Charline Le Lan, Chris J Maddison, Ryota Tomioka, and Yee Whye Teh.  Continuous hierarchical representations with poincar$\backslash$’e variational auto-encoders.  arXiv preprint arXiv:1901.06033, 2019.
  * [30] George A Miller.  Wordnet: a lexical database for english.  Communications of the ACM, 38(11):39–41, 1995.
  * [31] Dmytro Mishkin and Jiri Matas.  All you need is a good init.  arXiv preprint arXiv:1511.06422, 2015.
  * [32] Yoshihiro Nagano, Shoichiro Yamaguchi, Yasuhiro Fujita, and Masanori Koyama.  A wrapped normal distribution on hyperbolic space for gradient-based learning.  In International Conference on Machine Learning, pages 4693–4702. PMLR, 2019.
  * [33] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng.  Reading digits in natural images with unsupervised feature learning.  2011\.
  * [34] Maximilian Nickel and Douwe Kiela.  Poincar$\backslash$’e embeddings for learning hierarchical representations.  arXiv preprint arXiv:1705.08039, 2017.
  * [35] Maximillian Nickel and Douwe Kiela.  Learning continuous hierarchies in the lorentz model of hyperbolic geometry.  In International Conference on Machine Learning, pages 3779–3788. PMLR, 2018.
  * [36] Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli.  The emergence of spectral universality in deep networks.  In International Conference on Artificial Intelligence and Statistics, pages 1924–1932. PMLR, 2018.
  * [37] Jeffrey Pennington, Samuel S Schoenholz, and Surya Ganguli.  Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice.  arXiv preprint arXiv:1711.04735, 2017.
  * [38] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams.  Learning representations by back-propagating errors.  nature, 323(6088):533–536, 1986.
  * [39] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  International journal of computer vision, 115(3):211–252, 2015\.
  * [40] Frederic Sala, Chris De Sa, Albert Gu, and Christopher Ré.  Representation tradeoffs for hyperbolic embeddings.  In International conference on machine learning, pages 4460–4469. PMLR, 2018.
  * [41] Rik Sarkar.  Low distortion delaunay embedding of trees in hyperbolic plane.  In International Symposium on Graph Drawing, pages 355–366. Springer, 2011.
  * [42] Ryohei Shimizu, Yusuke Mukuta, and Tatsuya Harada.  Hyperbolic neural networks++.  arXiv preprint arXiv:2006.08210, 2020.
  * [43] Jake Snell, Kevin Swersky, and Richard S Zemel.  Prototypical networks for few-shot learning.  arXiv preprint arXiv:1703.05175, 2017.
  * [44] Abraham A Ungar.  Hyperbolic trigonometry and its application in the poincaré ball model of hyperbolic geometry.  Computers & Mathematics with Applications, 41(1-2):135–147, 2001\.
  * [45] Abraham A Ungar.  Analytic hyperbolic geometry: Mathematical foundations and applications.  World Scientific, 2005.
  * [46] Abraham Albert Ungar.  A gyrovector space approach to hyperbolic geometry.  Synthesis Lectures on Mathematics and Statistics, 1(1):1–194, 2008\.
  * [47] Melanie Weber, Manzil Zaheer, Ankit Singh Rawat, Aditya Menon, and Sanjiv Kumar.  Robust large-margin learning in hyperbolic space.  arXiv preprint arXiv:2004.05465, 2020.
  * [48] P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona.  Caltech-UCSD Birds 200.  Technical Report CNS-TR-2010-001, California Institute of Technology, 2010\.
  * [49] Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li.  Empirical evaluation of rectified activations in convolutional network.  arXiv preprint arXiv:1505.00853, 2015.
  * [50] Pingmei Xu, Krista A Ehinger, Yinda Zhang, Adam Finkelstein, Sanjeev R Kulkarni, and Jianxiong Xiao.  Turkergaze: Crowdsourcing saliency with webcam based eye tracking.  arXiv preprint arXiv:1504.06755, 2015.
  * [51] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao.  Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop.  arXiv preprint arXiv:1506.03365, 2015.
  * [52] Sergey Zagoruyko and Nikos Komodakis.  Wide residual networks.  arXiv preprint arXiv:1605.07146, 2016.
  * [53] Yiding Zhang, Xiao Wang, Xunqiang Jiang, Chuan Shi, and Yanfang Ye.  Hyperbolic graph attention network.  arXiv preprint arXiv:1912.03046, 2019.
  * [54] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba.  Places: A 10 million image database for scene recognition.  IEEE transactions on pattern analysis and machine intelligence, 40(6):1452–1464, 2017.
  * [55] Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun.  Graph neural networks: A review of methods and applications.  AI Open, 1:57–81, 2020.