Inspecting class hierarchies in classification-based metric learning models

Hyeongji Kim1,2*, Pekka Parviainen2, Terje Berge1, Ketil Malde1,2

1 Institute of Marine Research, Bergen, Norway

2 Department of Informatics, University of Bergen, Norway

*<EMAIL_ADDRESS>
## Abstract

Most classification models treat all misclassifications equally. However,
different classes may be related, and these hierarchical relationships must be
considered in some classification problems. These problems can be addressed by
using hierarchical information during training. Unfortunately, this
information is not available for all datasets. Many classification-based
metric learning methods use class representatives in embedding space to
represent different classes. The relationships among the learned class
representatives can then be used to estimate class hierarchical structures. If
we have a predefined class hierarchy, the learned class representatives can be
assessed to determine whether the metric learning model learned semantic
distances that match our prior knowledge. In this work, we train a softmax
classifier and three metric learning models with several training options on
benchmark and real-world datasets. In addition to the standard classification
accuracy, we evaluate the hierarchical inference performance by inspecting
learned class representatives and the hierarchy-informed performance, i.e.,
the classification performance, and the metric learning performance by
considering predefined hierarchical structures. Furthermore, we investigate
how the considered measures are affected by various models and training
options. When our proposed ProxyDR model is trained without using predefined
hierarchical structures, the hierarchical inference performance is
significantly better than that of the popular NormFace model. Additionally,
our model enhances some hierarchy-informed performance measures under the same
training options. We also found that convolutional neural networks (CNNs) with
random weights correspond to the predefined hierarchies better than random
chance.

## Introduction

Neural network-based classifiers have shown impressive classification
accuracy. For instance, a convolutional neural network (CNN) classifier [1]
surpassed human-level top-5 classification accuracy (94.9%) [2] on the
1000-class classification challenge on the ImageNet dataset [3]. Most training
loss functions in neural network classifiers treat all misclassifications
equally. However, in practice, the severity of various misclassifications may
differ considerably. For instance, in an autonomous vehicle system, mistaking
a person as a tree can result in a more catastrophic consequence than
mistaking a streetlight as a tree [4]. In addition, some classification tasks
include a large number of classes, such as the 1000-class ImageNet
classification challenge, and hierarchical relationships may exist among these
classes. When several classification models achieved similar accuracy, one
would prefer to choose models in which the wrongly predicted classes are
“hierarchically” close to the ground-truth classes. To address the severity of
misclassifications and relationships among classes, hierarchical information
can be used. For instance, predefined hierarchical structures can be
incorporated into training by replacing standard labels with soft labels based
on this hierarchical information [4]. Some metric learning approaches also use
hierarchical information [5, 6, 7].
In general, metric learning methods learn embedding functions in which similar
data points are close and dissimilar data points are far apart according to
the distance metric in the learned embedding space. For class-labeled
datasets, data points in the same class are regarded as similar, while data
points in different classes are regarded as dissimilar. Metric learning can be
applied in image retrieval tasks [8] to identify relevant images or few-shot
classification tasks [9, 10], which are classification tasks with only a few
examples per class. Usually, metric learning methods assume that there are no
special relations among classes, i.e., a flat hierarchy is assumed. However, a
predefined class hierarchy can be incorporated into the training process of
metric learning models to improve the hierarchy-informed performance [5, 6,
7].
Class hierarchical structures can be defined in several ways. Hierarchical
structures can be defined by domain experts [6] or extracted from WordNet
[11], which is a database that contains semantic relations among English
words. For instance, ImageNet classes [3] are organized according to WordNet.
However, hierarchy determination from WordNet requires that a class name or
its higher class is in the WordNet database. When these approaches are not
applicable, hierarchy can be inferred by estimating the class distance matrix
based on learned classifiers. For instance, the confusion matrix of a
classifier can be used to estimate relations among class pairs [12]. Each row
in the confusion matrix can be treated as a vector, and the distance between
these vectors can be calculated to estimate the class hierarchical structure.
However, this approach can be cumbersome for hierarchy-informed classification
tasks, as they require separate training and evaluation (validation) processes
to determine the hierarchical structure. Moreover, this approach becomes
challenging when some classes contain a very small number of data points, as
some elements in the confusion matrix may be uninformative. One type of metric
learning model uses a unique position in embedding space (class
representative) to represent each class [13, 14, 15, 16]. Class
representatives can also be used to infer hierarchical structures by
considering their distances [6, 17]. This approach does not require a separate
training process, as class representatives can be learned automatically.
On the other hand, when we have a predefined hierarchy, the learned class
representatives can improve our understanding of the trained metric learning
model. For example, we can determine if the semantic distance learned by a
metric learning model matches our prior knowledge (that is, the predefined
hierarchy). For instance, when a model has been trained to classify species,
we can determine if the model regards a dog as closer to a cat than to a rose.
Furthermore, these inspections can be used to evaluate the trustworthiness of
a model. However, previous works have paid little attention to the
relationships among class representatives. In this work, we assess several
metric learning methods and training options by focusing on their learned
class representatives and hierarchy-informed performance. Moreover, we attempt
to determine conditions that improve the hierarchy inference performance and
evaluate whether such models enhance the hierarchy-informed performance. We
also investigate different training options with predefined hierarchies for
comparison.

### Problem settings

The predefined hierarchical distance, i.e., the distance between two classes,
often needs to be considered when training models with hierarchical
information and evaluating model performance. Barz and Denzler [5] and
Bertinetto et al. [4] used bounded ($\left[0,1\right]$) dissimilarity based on
the height of the lowest common ancestor (LCA) between two classes. In this
work, similar to Garnot and Landrieu [6], we define the hierarchical distance
as the shortest path distance between two classes. For instance, in the
predefined hierarchical tree shown in Fig. 1, the hierarchical distance
between the classes “tiger” and “woman” is $4$ ($=2+2$), as we need to move
two steps upward and two steps downward to move from one class to the other.
Similarly, the hierarchical distance between the classes “tiger” and “shark”
is $6$. Hierarchical structures can be expressed as trees or directed acyclic
graph (DAG) structures [18]. In this work, we consider tree-structured
hierarchies. In other words, each node cannot have more than one parent node.

Fig 1: Pruned CIFAR100 tree structure for visualization. The whole
hierarchical structure is shown in a table in S2 Appendix.

Let $\mathcal{X}\subseteq\mathbb{R}^{d_{I}}$ be an input space and
$\mathcal{Y}$ be a set of classes. The data points $x\in\mathcal{X}$ and
corresponding classes $c\in\mathcal{Y}$ are sampled from the joint
distribution $\mathcal{D}$. The feature mapping
$f:\mathcal{X}\rightarrow\mathcal{Z}$ extracts feature vectors according to
the inputs, where $\mathcal{Z}=\mathbb{R}^{d_{F}}$ is a raw feature space. In
this work, we assume that this mapping is modeled by a neural network.

### Softmax classification

The softmax classifier is commonly used in neural network-based classification
tasks. The softmax classifier estimates the class probability $p(c|x)$, which
is the probability that data point $x$ belongs to class $c$, as follows:

$\displaystyle
p(c|x)=\frac{\exp(l_{c}(x))}{\sum\limits_{y\in\mathcal{Y}}{\exp(l_{y}(x))}},$
(1)

where logit $l_{y}(x)$ is the neural network output according to input
$x\in\mathcal{X}$ and class $y\in\mathcal{Y}$. Usually, logit $l_{y}(x)$ is
calculated as:

$\displaystyle l_{y}(x)=W_{y}^{T}f(x)+b_{y},$ (2)

where the feature vector $f(x)$ is an output of a penultimate layer, $W_{y}$
is a weight vector, and $b_{y}$ is a bias term for class $y$. The probability
$p(c|x)$ estimated by the model is often called the confidence value (score).
Based on the estimated class probabilities, we can use the cross-entropy loss
to train the model. This loss measures the difference between the predicted
and target probability distributions. The cross-entropy (CE) loss of a mini-
batch $B\subseteq\mathcal{D}$ is defined as:

$\displaystyle L_{CE}=-\frac{1}{|B|}\sum\limits_{(x_{i},c_{i})\in
B}{\log(p(c_{i}|x_{i}))}.$ (3)

### Metric learning

While softmax classifiers are commonly used in deep learning classification,
metric learning approaches can be beneficial, as they better control data
points in embedding space. Hence, metric learning can provide information on
the similarity between different data points. Metric learning approaches can
be divided into two categories: (direct) embedding-based methods and
classification-based methods. Embedding-based methods [19, 20] directly
compare data points in embedding space to train an embedding function (feature
map) $f(\cdot)$. Because embedding-based methods compare data points directly,
they must be trained with pairs (using the contrastive loss) or triplets
(using the triplet loss) of data points. Thus, embedding-based methods have
high training complexity and require special mining algorithms [13] to prevent
slow convergence speeds [14]. On the other hand, classification-based methods
[13, 15, 16, 21, 22] use class representatives to represent classes. According
to the classification loss (cross-entropy loss), class representatives guide
data points to converge to class-specific positions. Classification-based
methods converge faster than embedding-based methods due to their reduced
sampling complexity (training with single data points). In this paper, we
focus on classification-based metric learning methods.

#### NormFace

NormFace [13], which is also known as normalized softmax [23], modifies Eq. 2
in the softmax classifier. NormFace was motivated by the normalization of
features during feature comparisons to improve face verification during the
testing phase. To apply normalization during both the testing and training
phases, NormFace normalizes the feature (embedding) vectors and weight vectors
and uses a zero bias term. Previous experiments on metric learning models [8]
have shown that NormFace and its variants [24, 15, 16] achieved competitive
performance on metric learning tasks.
We denote $\tilde{v}=\frac{v}{\left\|v\right\|}$ for any nonzero vector $v$
and the angle between vectors $\tilde{W}_{y}$ and $\tilde{f}(x)$ as
$\theta_{y}$. Then, as
$\left\|\tilde{W}_{y}\right\|=1=\left\|\tilde{f}(x)\right\|$, we obtain the
following equation:

$\displaystyle\tilde{W}_{y}^{T}\tilde{f}(x)=\left\|\tilde{W}_{y}\right\|\left\|\tilde{f}(x)\right\|\cos{\theta_{y}}=\cos{\theta_{y}}.$
(4)

According to Eq. 4, the class probability $p(c|x)$ estimated by NormFace can
be expressed as:

$\displaystyle
p(c|x)=\frac{\exp(s\tilde{W}_{c}^{T}\tilde{f}(x))}{\sum\limits_{y\in\mathcal{Y}}{\exp(s\tilde{W}_{y}^{T}\tilde{f}(x))}}=\frac{\exp(s\cos{\theta_{c}})}{\sum\limits_{y\in\mathcal{Y}}{\exp(s\cos{\theta_{y}})}},$
(5)

where $s>0$ is a scaling factor. NormFace learns embeddings according to this
estimation and the cross-entropy loss in Eq. 3.
We next investigate the geometrical meaning of the NormFace classification
results. If a data point $x$ is classified as belonging to class $c$ by
NormFace, according to Eq. 5, we obtain
$\exp(s\cos{\theta_{c}})\geq\exp(s\cos{\theta_{y}})$, i.e.,
$\cos{\theta_{c}}\geq\cos{\theta_{y}}$. As the cosine function is a
monotonically decreasing function in the interval $\left(0,\pi\right)$, we
obtain $\theta_{c}\leq\theta_{y}$. In terms of angles, the normalized
embedding vector $\tilde{f}(x)$ is closer (or equally close) to
$\tilde{W}_{c}$ than $\tilde{W}_{y}$. As we can classify data points using
normalized weight vectors according to this geometrical interpretation, we can
consider $\tilde{W}_{y}$ as the class representative for a class
$y\in\mathcal{Y}$. Fig. 2 visualizes the above explanation.

Fig 2: Visualization of NormFace classification results for two classes with
$\tilde{W}_{1}$=(0,1) and $\tilde{W}_{2}$=(1,0). As $\tilde{f}(x)$ is closer
to $\tilde{W}_{1}$ than $\tilde{W}_{2}$, data point $x$ is classified as
belonging to class 1.

#### Proxies and prototypes

Although NormFace [13] uses a learnable weight vector as a class
representative for each class, the average point for a class can also be used
as a class representative. To prevent confusion, we define two kinds of class
representatives for each class: a proxy representative and a prototype
representative. We refer to the learnable weight vectors as proxy
representatives. In NormFace, the weight vectors $\tilde{W}_{y}$ are proxies.
In this work, the proxy representative are used to define the training loss
and directly update the network. Note that we can predefine the proxy
representatives, and the proxies can be fixed. On the other hand, we refer to
the (normalized) average embedding for each class as a prototype
representative. We denote the average embedding for class $y$ as vector
$\mu_{y}$. In normalized space, this vector is defined as:

$\displaystyle\mu_{y}=\frac{1}{|X_{y}|}\sum_{x\in X_{y}}\tilde{f}(x),$

where $X_{y}\subseteq\mathcal{X}$ is a set of data points in class $y$. Then,
the prototype representatives are defined as
$\tilde{\mu}_{y}=\frac{\mu_{y}}{\left\|\mu_{y}\right\|}$. A metric learning
method known as the prototypical network [9] was devised for few-shot learning
tasks. During the training process, this network uses local prototypes based
on special mini-batches known as episodes. In their work, the authors used
unnormalized average embedding as prototypes because they considered Euclidean
space. In this paper, we do not use prototypes for training, and (global)
prototypes using training data are used only for evaluation.
Note that the proxy and prototype representatives are not necessarily the
same. For instance, when we train the model with fixed proxies, the positions
of the prototypes change during training, while the positions of the proxies
remain fixed. Moreover, the confidence values are not necessarily maximized at
the proxies [25]. In this case, the data points may not converge to their
proxies. This concept is explained further in the next subsubsection.

#### SD softmax and DR formulation

In our previous work [25], we analyzed a softmax formulation for metric
learning in which negative squared distances were used as logit values. We
called this formulation the “softmax-based formulation” in our previous paper.
In this work, we refer to this formulation as “squared distance (SD) softmax”
to prevent confusion with the standard softmax formulation shown in Eq. 1.

We denote the class representative for class $y$ as $R_{y}$ and the distance
to class $y$ as $d_{x,y}:=d(f(x),R_{y})$ according to the distance function
$d(\cdot,\cdot)$. For the normalized embeddings, we define
$d_{x,y}:=d(\tilde{f}(x),R_{y})$ by assuming that point $R_{y}$ is normalized.
Then, the class probability $p(c|x)$ estimated using the SD softmax
formulation is:

$\displaystyle
p(c|x)=\frac{\exp(-d_{x,c}^{2})}{\sum\limits_{y\in\mathcal{Y}}{\exp(-d_{x,y}^{2})}}.$
(6)

The SD softmax formulation uses the difference in the squared distance for
training.
Consider the following equation:

$\displaystyle\left\|\tilde{W}_{y}-\tilde{f}(x)\right\|^{2}=\left\|\tilde{W}_{y}\right\|^{2}+\left\|\tilde{f}(x)\right\|^{2}-2\tilde{W}_{y}^{T}\tilde{f}(x)=1+1-2\cos\theta_{y}.$

Then, we obtain:

$\displaystyle
s\cos\theta_{y}-s=-\frac{s}{2}\left\|\tilde{W}_{y}-\tilde{f}(x)\right\|^{2}.$

Thus, we obtain the equation:

$\displaystyle\frac{\exp(s\cos{\theta_{c}})}{\sum\limits_{y\in\mathcal{Y}}{\exp(s\cos{\theta_{y}})}}=\frac{\exp(s\cos{\theta_{c}}-s)}{\sum\limits_{y\in\mathcal{Y}}{\exp(s\cos{\theta_{y}}-s)}}=\frac{\exp(-\frac{s}{2}\left\|\tilde{W}_{c}-\tilde{f}(x)\right\|^{2})}{\sum\limits_{y\in\mathcal{Y}}{\exp(-\frac{s}{2}\left\|\tilde{W}_{y}-\tilde{f}(x)\right\|^{2})}},$
(7)

(This equation is taken from [13]). The above equation shows that the NormFace
formulation in Eq. 5 can be considered an SD softmax formulation 6 that uses
the Euclidean distance on a hypersphere with radius $\sqrt{\frac{s}{2}}$.
In [25], we found that the SD softmax formulation had two main limitations.
First, the estimated probability and corresponding loss may be affected by
scaling changes. However, while this limitation must be considered when a
Euclidean space is used, this limitation can be addressed by using normalized
embeddings, as in the NormFace model [13]. The second limitation is that the
estimated class probabilities are not optimized at the class representatives.
For instance, the maximum estimated class probability $p(c|x)$ is not found at
the class representative of class $c$. The NormFace model also encounters this
issue. In the example shown in Fig. 2 with scaling factor $s=2$, point
$(-\frac{1}{\sqrt{2}},\frac{1}{\sqrt{2}})$ is the point that maximizes the
confidence value of class $1$.
To address the above limitations, we proposed the distance ratio (DR)-based
formulation [25] for metric learning models. Mathematically, the DR
formulation estimates the class probability $p(c|x)$ as:

$\displaystyle
p(c|x)=\frac{\frac{1}{d_{x,c}^{s}}}{\sum\limits_{y\in\mathcal{Y}}{\frac{1}{d_{x,y}^{s}}}}=\frac{d_{x,c}^{-s}}{\sum\limits_{y\in\mathcal{Y}}{d_{x,y}^{-s}}}.$
(8)

The DR formulation uses ratios of distances for training.
Moreover, the DR formulation [25] resolves the above two limitations of the SD
softmax formulation. In the example shown in Fig. 2, for any scaling factor
$s$, point $(0,1)=\tilde{W}_{1}$ is the point that maximizes the confidence
value of class $1$. Thus, our experiments showed that the DR formulation has
faster or comparable training speed in Euclidean (unnormalized) embedding
spaces.

#### Exponential moving average (EMA) approach

Zhe et al. [21] theoretically showed that, while training NormFace [13], the
commonly used gradient descent update method for the proxies based on the
training loss cannot guarantee that the updated proxies approach the
corresponding prototypes. To address this issue, they proposed using the
normalized exponential moving average (EMA) to update the proxies.
Mathematically, when updating a proxy for a data point $x$ in class $c$, the
proxy $\tilde{W}_{c}$ is updated as:

$\displaystyle\tilde{W}_{c}:=\frac{\alpha\tilde{f}(x)+(1-\alpha)\tilde{W}_{c}}{\left\|\alpha\tilde{f}(x)+(1-\alpha)\tilde{W}_{c}\right\|},$
(9)

where $0<\alpha<1$ is a parameter that controls the speed and stability of the
updates. Their experimental results showed that the EMA approach achieved
better performance than the standard NormFace model on multiple datasets.

#### Adaptive scaling factor approach

Based on previous observations that the convergence and performance of
NormFace models [13] depend on the scale parameter $s$, Zhang et al. [22]
proposed AdaCos, which is a NormFace model trained by using the adaptive scale
factor $s$ in Eq. (5). Moreover, they suggested to use parameter $s$, which
significantly changes the probability $p(c|x)$ estimated by Eq. (5), where $c$
is the class of data point $x$. In other words, they attempted to find a
parameter $s$ that maximizes $\left\|\frac{\partial
p(c|x)(\theta_{c})}{\partial\theta_{c}}\right\|$ by approximating an $s$ value
that satisfies the equation:

$\displaystyle\left\|\frac{\partial^{2}p(c|x)(\theta_{c}^{\prime})}{\partial\theta_{c}^{\prime
2}}\right\|=0,$ (10)

where $\theta_{c}^{\prime}:=\text{clip}(\theta_{c},0,\frac{\pi}{2})$ and
$\text{clip}(\cdot,\cdot,\cdot)$ is a function that limits a value within a
specified range. They proposed two AdaCos models: static (fixed) and dynamic
versions. The static model determines a good scale parameter $s$ before
training the NormFace model based on observations of the angles between the
data points and proxies. In the static model, the scale parameter $s$ is not
updated. The dynamic model updates the scale parameter $s$ during each
iteration based on the current angles between the data points and proxies.

#### CORR loss

In contrast to previous metric learning approaches that ignored hierarchical
relationships among classes, Barz and Denzler [5] used normalized embeddings
to achieve hierarchy-informed classification. First, they predefined the
positions of the proxies using a given hierarchical structure. These proxies
are fixed, i.e., they are not updated during training. Then, they used the
predefined proxies to train the models according to the CORR loss. For a data
point $x$ in class $c$, the CORR loss ensures that the embedding vector
$\tilde{f}(x)$ is close to the corresponding proxy $\tilde{W}_{c}$. The CORR
loss of a mini-batch $B\subseteq\mathcal{D}$ is defined as:

$\displaystyle L_{CORR}=\frac{1}{|B|}\sum\limits_{(x_{i},c_{i})\in
B}{\left(1-\tilde{W}_{c_{i}}^{T}\tilde{f}(x_{i})\right)}=\frac{1}{|B|}\sum\limits_{(x_{i},c_{i})\in
B}{\left(1-\cos{\theta_{c_{i}}}\right)},$ (11)

where $\theta_{c_{i}}$ is the angle between $\tilde{W}_{c_{i}}$ and
$\tilde{f}(x_{i})$.

## Methods

We investigated several methods and training options. The details of the
settings are described in the following sections. The codes for our
experiments will be available in
https://github.com/hjk92g/Inspecting_Hierarchies_ML.

##### Dataset

We conducted experiments using three plankton datasets (small microplankton,
large microplankton, and mesozooplankton) and two benchmark datasets (CIFAR100
[26] and NABirds [27]). Table 1 summarizes the number of classes and images in
each dataset. The three plankton datasets were obtained from the Institute of
Marine Research (IMR), where flow imaging microscopy is used during routine
monitoring. The plankton samples were imaged using three FlowCams, ©2022
Yokogawa Fluid Imaging Technologies, Inc., with different magnification
settings. The three plankton datasets contain nonliving classes of artifacts
and debris. Moreover, these datasets contain class names that are not in the
WordNet database [11]. In addition, the three plankton datasets have severe
class imbalances. For example, each class in the small microplankton (MicroS)
dataset contains 1 to 456 images. Further details on the plankton datasets can
be found in S2 Appendix. We randomly divided each plankton dataset into 70$\%$
training data, 10$\%$ validation data, and 20$\%$ test data. For the plankton
datasets, we use images without any augmentation. As the input images have
diverse image sizes, we resized the input images to $128\times 128$. The
CIFAR100 dataset contains 100 classes and 600 images per class. The CIFAR100
contains 50000 training and 10000 test images, and we randomly divided the
original training data into 45000 training and 5000 validation data.
Furthermore, we applied random transformations (15 degree range rotations, 10%
range translations, 10% range scaling, 10 degree range shearing, and
horizontal flips) on the CIFAR100 training data. We did not resize images from
their original size of $32\times 32$. The NABirds dataset contains 23929
training and 24633 test images. Similar to the plankton datasets, we randomly
divided this dataset into 70$\%$ training data, 10$\%$ validation data, and
20$\%$ test data. We resized the images to $128\times 128$ pixels. We also
applied the same augmentations applied to the CIFAR100 dataset on the NABirds
dataset. The predefined hierarchical structures are available in S1 Appendix.

Table 1:  Summary of the studied datasets. Dataset |  Target particle (plankton datasets) |  Images per class |  Images |  Classes
---|---|---|---|---
Small microplankton (MicroS) |  $5$ to $50\,\mu m$ |  1 to 456 |  6738 |  109
Large microplankton (MicroL) |  $35$ to $500\,\mu m$ |  2 to 613 |  8348 |  102
Mesozooplankton (MesoZ) |  $180$ to $2000\,\mu m$ |  3 to 486 |  6738 |  52
CIFAR100 [26] |  — |  600 |  60000 |  100
NABirds [27] |  — |  13 to 120 |  48562 |  555

##### Models

We consider four types of models: the softmax classifier, NormFace, ProxyDR
(explained below), and a CORR loss-based model. We focus on metric learning
models with normalized embeddings, as normalization is commonly used in metric
learning models to improve performance [13, 28, 5]. ProxyDR is a model that
uses proxies for classification and DR formulations (8) to estimate class
probabilities $p(c|x)$. Similar to the NormFace model, ProxyDR uses the
Euclidean distance on a hypersphere. The difference between the two models is
that ProxyDR uses DR formulations, while NormFace uses SD softmax formulations
(as shown in Eq. 7). For the plankton datasets, we used a pretrained Inception
version 3 architecture [29] as the backbone. For the CIFAR100 and NABirds
datasets, we used a pretrained ResNet50 [30] as the backbone. In addition to
the backbones, we applied a learnable linear transformation to obtain
$128$-dimensional embeddings $f(x)$. For the plankton datasets, we incorporate
the size information. Specifically, when a data point $x$ has size
$v_{size}=\left[width,height\right]^{T}$, we take the elementwise logarithm
$v^{\prime}_{size}=\left[\log(width),\log(height)\right]^{T}$. By applying a
linear transformation, we obtain an embedding vector $f_{size}(x)$ according
to the size information, i.e.,
$f_{size}(x)=W_{size}^{T}v^{\prime}_{size}+b_{size}$, where $W_{size}$ is a
learnable matrix with shape $2\times 128$ and $b_{size}\in\mathbb{R}^{2}$ is a
learnable vector. Then, we add this vector to the original embedding vector,
namely, $f(x):=f(x)+f_{size}(x)$.

##### Training settings

We trained the models according to the backbone weights and other weights
(linear transformations, proxies). We used the Adam optimizer [31] with a
learning rate of $10^{-4}$. Except in the case of a dynamic approach
(explained below), we use $10.0$ as a scaling factor in both NormFace and
ProxyDR. We set the training batch size in all experiments to $32$. For the
plankton datasets, we trained the models for $50$ epochs. For the CIFAR100 and
NABirds datasets, we trained the models for $100$ epochs. During each epoch,
we assessed the model accuracy. We chose the model with the highest validation
accuracy for testing. For each setting, we trained the models five times with
different seeds for the random split.

##### Training options

The different training options are described as follows. The dynamic approach
affects the scale factors in Eqs. 5 and 8. The EMA and MDS approaches both
affect the proxy calculations.

  * •

Standard: standard training with a fixed scale factor $s=10$, with proxies
updated using the cross-entropy loss (3).

  * •

EMA (exponential moving average): training using the normalized exponential
moving average [21], as shown in Eq. 9, to update the proxies. When multiple
data points have the same class $c$ in a mini-batch, we apply a modified
expression. Specifically, instead of applying the EMA using a single data
point, as in Eq. 9, we use the normalized average embedding (local prototype)
of the mini-batch. Mathematically, when updating a proxy for $m$ data points
$x_{B;1},\cdots,x_{B;m}$ with class $c$ in mini-batch $B$, the normalized
average embedding is defined as:

$\displaystyle\tilde{\mu}_{B;c}=\frac{\sum\limits_{i=1}^{m}{\tilde{f}(x_{B;i})}}{\left\|\sum\limits_{i=1}^{m}{\tilde{f}(x_{B;i})}\right\|}.$

Then, proxy $\tilde{W}_{c}$ is updated as:

$\displaystyle\tilde{W}_{c}:=\frac{\alpha\tilde{\mu}_{B;c}+(1-\alpha)\tilde{W}_{c}}{\left\|\alpha\tilde{\mu}_{B;c}+(1-\alpha)\tilde{W}_{c}\right\|},$
(12)

where $\alpha$ is the same parameter as in Eq. 9. We set the parameter
$\alpha$ to $0.001$.

  * •

Dynamic: training with a dynamic scale factor, similar to AdaCos [22]. In
contrast to the original paper, which chooses a scale factor using an
approximate expression, we use the Adam [31] optimizer to determine a scale
factor that satisfies Eq. 10. More details are included in S1 Appendix.

  * •

MDS (multidimensional scaling): training according to predefined hierarchical
information. We use the hierarchical information to set (fixed) proxies.
First, as in the distance calculation method in the problem setting
subsection, we use a predefined hierarchy to generate a distance matrix. We
denote the (hierarchical) distance between the $i$th and $j$th classes as
$d_{H}(i,j)$. For each hierarchical distance $d_{H}$, we apply a
transformation $d_{T}=\frac{\sqrt{2}d_{H}}{\beta+d_{H}}$ for a scalar
$\beta>0$ to address the limited Euclidean distance ($\leq\sqrt{2}$) on unit
spherical spaces. We set $\beta=1.0$ in all of our experiments. (Note that if
$d$ is a metric, the transformed distance $\frac{\sqrt{2}d}{\beta+d}$ is also
a metric.) Then, according to the transformed distance matrix $D_{T}$, we use
multidimensional scaling (MDS) to set the proxies. Mathematically, we minimize
a value known as the normalized stress, which can be expressed as:

$\displaystyle\text{Stress}(D_{\tilde{W}}):=\frac{\left\|D_{\tilde{W}}-D_{T}\right\|_{F}}{\left\|D_{T}\right\|_{F}},$
(13)

where $D_{\tilde{W}}$ is a pairwise Euclidean distance matrix according to
proxies $\tilde{W}_{y}$ and $\left\|\cdot\right\|_{F}$ is the Frobenius norm.
We use the Adam optimizer with a learning rate of $10^{-3}$ for 1000
iterations to obtain the proxies. While we used stochastic gradient descent
for the MDS option, different methods, such as those applied by Barz and
Denzler [5], can also be used for MDS. During training, we fix the obtained
proxies and update only the embedding function $f(\cdot)$.

##### Performance measures

In our experiments, we consider three types of performance measures: standard
classification measures, hierarchical inference performance measures, and
hierarchy-informed performance measures. The hierarchical inference
performance measures are used to estimate how well the learned class
representatives match the predefined hierarchies. The hierarchy-informed
performance measures are used to estimate how well a model performs on
classification or similarity measures according to the predefined hierarchies.
We used the top-k accuracy as a standard classification measure, the mean
correlation as the hierarchical inference performance measure, and the average
hierarchical distance (AHD), hierarchical precision at $k$ (HP@k),
hierarchical similarity at $k$ (HS@k), and average hierarchical similarity at
$k$ (AHS@k) as hierarchy-informed performance measures. The utilized measures
are defined as follows.

  * •

Top-$k$ accuracy: The classification accuracy was calculated, with correct
classification defined as whether the labeled class is in the top-$k$
predictions (most likely classes), i. e., the $k$ classes with the highest
confidence values. We report results for $k=1,5$.

  * •

Mean correlations: We introduce this measure to evaluate how well the learned
class representatives match the predefined hierarchical structure. We obtain
the class representatives (either proxies or prototypes) from the learned
model using training data points. Then, we obtain the pairwise distance matrix
$D_{L}$ based on the class representatives. We compare matrix $D_{L}$ with the
distance matrix $D_{H}$ based on the predefined hierarchical structure.
Specifically, for each class (row), we calculate Spearman’s rank correlation
coefficient according to the two matrices. Then, we determine the mean
correlation using Fisher transformations. More precisely, we apply a Fisher
transformation $\operatorname{arctanh}(\cdot)$ on each correlation
coefficient, take the average of the transformed values, and apply
$\tanh(\cdot)$ on the average value.
For the plankton datasets, the predefined hierarchical structures of the
living classes are based on biological taxonomies. However, these datasets
also contain some nonliving classes, such as “large bubbles” and “dark
debris”. Considering that the predefined hierarchical structures of the living
classes are scientifically defined, for the plankton datasets, we report mean
correlations among whole classes or only among living classes.

  * •

AHD: The average hierarchical distance of the top-$k$ predictions [4] was
calculated as the average hierarchical distance $d_{H}$ between the labeled
classes and each of the top-$k$ most likely classes. In contrast to Bertinetto
et al. [4], who considered only misclassified cases, we consider all cases.
Hence, in our calculations, the final denominators differ. When $k=1$, the AHD
measure is the same as the average hierarchical cost (AHC) measure defined by
Garnot and Landrieu [6]. We report results for $k=1,5$.

  * •

HP@$k$: The hierarchical precision at $k$ [32] was taken as a performance
measure. Specifically, let us denote a (hierarchical) neighborhood set of a
class $c$ with the distance threshold $\epsilon$ as $N(c,\epsilon)$, i.e.,
$n\in N(c,\epsilon)\Longleftrightarrow d_{H}(c,n)\leq\epsilon$. We define
$\text{hCorrectSet}(c,k)$ as the neighborhood set $N(c,\epsilon)$ with the
smallest $\epsilon$ such that $|\text{hCorrectSet}(c,k)|\geq k$. Then, the
hierarchical precision at $k$ is calculated as the fraction of the top-$k$
predictions in $\text{hCorrectSet}(c,k)$. We report the results for $k=5$.

  * •

HS@$k$: The hierarchical similarity at $k$ is a measure that was introduced by
Barz and Denzler [5] with the name “hierarchical precision at $k$”, although
this metric does not evaluate precision and instead assesses similarity. Here,
we use a different measure with the same name that was defined by Frome et al.
[32]. Hence, we renamed the measure “hierarchical similarity at $k$”. When $c$
is a label of a query data point $x$, let
$R=\left(\left(x_{1},c_{1}\right),\cdots,\left(x_{m},c_{m}\right)\right)$ be
the ordered list of image-label pairs based on the distance (sorted by
ascending distance) to point $x$ in the normalized embedding space.
Considering $\cos(\theta)=1-\frac{\left\|u_{1}-u_{2}\right\|^{2}}{2}$, where
$u_{1}$ and $u_{2}$ are unit vectors and $\theta$ is the angle between $u_{1}$
and $u_{2}$, we defined the similarity between the $i$th and $j$th classes
$s_{H}(i,j)$ as:

$\displaystyle s_{H}(i,j)=1-\frac{d_{T}(i,j)^{2}}{2},$ (14)

where $i$ and $j$ are the indices for classes ($1\leq i,j\leq|\mathcal{Y}|$).
The hierarchical similarity at $k$ is then defined as:

$\displaystyle
HS@k:=\frac{\sum\limits_{i=1}^{k}{s_{H}(I(c),I(c_{i}))}}{\max_{\pi}\sum\limits_{i=1}^{k}{s_{H}(I(c),I(c_{\pi_{i}}))}},$
(15)

where $I(\cdot)$ is an index function that outputs the corresponding index
(between $1$ and $|\mathcal{Y}|$) for a class and $\pi$ is an index
permutation that ranges from $1$ to $m$. We report results for $k=50,250$.

  * •

AHS@$K$: The average hierarchical similarity at $K$ was introduced by Barz and
Denzler [5] as the “average hierarchical precision at $K$”. Due to similar
reasons as for HS@$k$, we renamed the measure. The average hierarchical
similarity at $K$ is defined as the area under the curve of HS@$k$ from $k=1$
to $k=K$. We report results for $K=250$.

## Results

### Main results

The performance measure evaluation results are shown in bar plots with 95%
confidence intervals. Dashed lines separate models trained with or without
predefined hierarchical knowledge. We show only the results on the CIFAR100
and NABirds datasets in the main text. All results are included in S3
Appendix. To reduce spurious findings, we focus on consistent trends across
the five datasets.
Figs. 3 and 4 and the figures in S3 Appendix show the top-$k$ accuracy for
various training settings. The NormFace and ProxyDR models achieved comparable
top-$k$ accuracy for both $k$ values on most datasets. The softmax loss model
obtained low top-$k$ accuracy. While the result was not significant, using the
dynamic option achieved higher top-1 accuracy than standard training. When the
EMA option was added to the standard and dynamic options, the top-5 accuracy
decreased, except for standard NormFace on the NABirds dataset. While the CORR
loss achieved top-1 accuracy that was comparable to that achieved by other
training options with predefined hierarchical information, this model obtained
low top-5 accuracy on all datasets. The top-5 accuracy with the CORR loss was
even lower than that with the softmax loss, except on the NABirds dataset.
Although the results were better than those of the CORR loss model, the use of
predefined hierarchical information during training for NormFace and ProxyDR
also reduced the top-5 accuracy. These results show the opposite trend to the
changes in the top-1 accuracy, which showed comparable or enhanced
performance. Moreover, the dynamic MDS approach obtained lower top-5 accuracy
than the MDS approach without the dynamic option.

Fig 3: Top-$k$ accuracy results (A: $k=1$, B: $k=5$) on the CIFAR100 dataset.
Fig 4: Top-$k$ accuracy results (A: $k=1$, B: $k=5$) on the NABirds dataset.

Figs. 5 and 6 and the figures in S3 Appendix show the mean correlation values
for various training options. When we consider training options that do not
use predefined hierarchical information, ProxyDR obtains higher mean
correlations than NormFace, except for the EMA approaches. The ProxyDR model
with the dynamic option obtained higher mean correlations than the standard
ProxyDR model. As expected, the use of predefined hierarchical information
greatly increased the mean correlations based on prototypes in both the
NormFace and ProxyDR models, except the ProxyDR model on the NABirds dataset.
When we consider training options that utilize predefined hierarchical
information, the CORR loss model achieved the highest mean correlations based
on prototypes in most cases. The ProxyDR model typically achieved the second-
best mean correlations.

Fig 5: Correlation measures on the CIFAR100 dataset. (A) Values using proxies.
(B) Values using prototypes. The mean correlation value based on proxies with
the MDS option was $0.8580$.  Fig 6: Correlation measures on the NABirds
dataset. (A) Values using proxies. (B) Values using prototypes. The mean
correlation value based on proxies with the MDS option was $0.4476$ (this
value is small because the dataset contains 555 classes and the embedding
dimension is 128.).

Figs. 7 and 8 and the figures in S3 Appendix show the hierarchical performance
measures obtained with various training options. The softmax loss option
achieved the worst hierarchical performance, except for AHS@$50$ on the
NABirds dataset. Although these measures are used to assess the hierarchy-
informed performance, some of the measures are not substantially affected by
the use of predefined hierarchical information during training. For instance,
the use of predefined hierarchical information in the CIFAR100 dataset (Fig.
7) did not show noticeable improvements in terms of the AHD (k=1), HS@$50$,
and AHS@$250$ measures. Moreover, while the use of predefined hierarchical
information significantly improved the HS@$250$ results on most datasets, only
marginal improvements were observed for the CIFAR100 dataset. On the other
hand, the use of predefined hierarchical information significantly improved
the AHD (k=5) and HP@$5$ results on all datasets. The CORR loss achieved the
best results on these two measures, except on the NABirds dataset. Adding the
dynamic option to the standard and MDS options improved performance on these
two measures, except for the ProxyDR model on the CIFAR100 dataset. When we
consider training options that do not use predefined hierarchical information,
the ProxyDR model shows better performance than NormFace in terms of these two
measures, except for the EMA and dynamic options on the CIFAR100 dataset.
Under the same settings, ProxyDR performed better than NormFace in terms of
the HS@$250$ measure, except for the EMA option on the CIFAR100 dataset.
Moreover, under the same settings, the ProxyDR model with the dynamic option
achieved the highest HS@$250$ and AHS@$250$ values among the compared models.

Fig 7: Hierarchical performance measures on the CIFAR100 dataset. The symbol
$\downarrow$ denotes that lower values indicate better performance. The symbol
$\uparrow$ denotes that higher values indicate better performance. (A) AHD
(k=1): $\downarrow$. (B) AHD (k=5): $\downarrow$. (C) HP@5: $\uparrow$. (D)
HS@50: $\uparrow$. (E) HS@250: $\uparrow$. (F) AHS@250: $\uparrow$.  Fig 8:
Hierarchical performance measures on the NABirds dataset. The symbol
$\downarrow$ denotes that lower values indicate better performance. The symbol
$\uparrow$ denotes that higher values indicate better performance. (A) AHD
(k=1): $\downarrow$. (B) AHD (k=5): $\downarrow$. (C) HP@5: $\uparrow$. (D)
HS@50: $\uparrow$. (E) HS@250: $\uparrow$. (F) AHS@250: $\uparrow$.

### Additional mean correlation results

To investigate the changes in the class representatives, we evaluated the mean
correlations at the end of each training epoch. We report only the results on
the CIFAR100 and NABirds datasets. Moreover, we report results for ProxyDR
models with the standard and dynamic options. More results are included in S3
Appendix. Furthermore, we report mean correlations based on random networks,
i.e., networks with random weights, pretrained networks, and unnormalized and
normalized input spaces.
Figs. 9 and 10 visualize the changes in the mean correlation values during
ProxyDR model training (averaged values from five different seeds).
Surprisingly, we found that the prototypes of the approximately untrained
networks (pretrained on ImageNet [3] and trained on the target dataset, e.g.,
CIFAR100 or NABirds, for only one epoch) already have relatively high
correlations (approximately $0.4$), with predefined hierarchical structures.
While the accuracy curves show no noticeable differences, using the dynamic
option modified the transition in the mean correlations. In particular,
prototype-based mean correlations increased after 20 to 40 training epochs,
and the maximum values were obtained near the end of training (approximately
100 epochs). The proxy-based mean correlations started at low values, and the
difference with the prototype-based mean correlations was reduced. Moreover,
the training epoch during which the validation accuracy is maximized often
differs from the training epoch during which the correlation measures are
maximized.

Fig 9: Changes in accuracy and mean correlations for the CIFAR100 dataset
(ProxyDR). (A) Accuracy curve with standard training. (B) Mean correlation
curve with standard training. (C) Accuracy curve with the dynamic option. (D)
Mean correlation curve with the dynamic option.  Fig 10: Changes in accuracy
and mean correlations for the NABirds dataset (ProxyDR). (A) Accuracy curve
with standard training. (B) Mean correlation curve with standard training. (C)
Accuracy curve with the dynamic option. (D) Mean correlation curve with the
dynamic option.

Table 2 shows prototype-based mean correlation values and their 95 percent
confidence intervals based on five different seeds. The mean correlations with
the random networks show that the prototypes and ground-truth hierarchy are
correlated. While these values are smaller than the other cases shown in the
table, the results show that random networks have some degree of semantic
understanding.

Table 2:  Mean correlations based on prototypes.

Space |  Random weights |  Pretrained |  Input space |  Normalized input space
---|---|---|---|---
CIFAR100 |  $0.2841\pm 0.0303$ |  $0.3816\pm 0.0116$ |  $0.3414\pm 0.0353$ |  $0.2609\pm 0.0416$
NABirds |  $0.1088\pm 0.0839$ |  $0.2648\pm 0.0349$ |  $0.1740\pm 0.0056$ |  $0.2369\pm 0.0039$

“Random weights” and “pretrained” mean embedding space based on the ResNet50
[30] backbone.

## Discussion

In this work, we investigate classification and hierarchical performance under
different models and training options. Our experiments reveal several
important findings. Under the training options that do not consider predefined
hierarchical information, the ProxyDR model achieved better hierarchical
inference performance than NormFace in most cases. Furthermore, under the same
training options, ProxyDR achieved better hierarchy-informed performance in
terms of the AHD (k=5) and HP@$5$ measures. Moreover, we observed that the use
of a dynamic scaling factor improved the hierarchical inference performance.
The changes in the mean correlation values (Figs. 9 and 10) verified the
effect of the dynamic training option. These results reveal the importance of
dynamic training approach. We also found that some hierarchy-informed
performance measures are not significantly improved by the use of known
hierarchical structures. This finding indicates that multiple hierarchy-
informed performance measures should be considered to compare the hierarchy-
informed performance of different models. We also observed a trade-off between
the hierarchy-informed performance and top-5 accuracy. While the CORR loss
model typically achieved the best hierarchical performance, this model
obtained the lowest top-5 accuracy among the experimental models. Similarly,
the use of predefined hierarchical information in NormFace and ProxyDR
significantly improved the AHD (k=5) and HP@$5$ performance but reduced the
top-5 accuracy. In contrast to previous works that observed a trade-off
between hierarchy-informed performance and top-1 accuracy [4], we did not
observe this trade-off with top-1 accuracy.
Surprisingly, we found that prototypes based on CNNs with random weights
showed correspondence with predefined hierarchies that was higher than random
chance. Because CNNs combine convolutional and pooling layers, most CNN
architectures have translation invariance properties. Because of such priors,
random networks may know weak perceptual similarity [33]. Another possible
reason for this result is that prototypes of input spaces show higher
correspondence with predefined hierarchies than random chance. The Johnson-
Lindenstrauss lemma [34] shows that linear projections using random matrices
approximately preserve distances. If this property holds for nonlinear
projections based on random neural networks, network-based prototypes may also
correspond to predefined hierarchies. This phenomenon suggests that when we
use metric learning models with proxies, the proxies can be assigned based on
prototypes instead of starting at random positions. This may improve training
during the initial epochs, as we start from proxies that are more semantically
reasonable than random positions.

Although we observed that the DR formulation improves the hierarchical
inference performance and hierarchy-informed performance when training models
without predefined hierarchies, we did not study the reasons underlying these
phenomena. We suggest one possible hypothesis. While NormFace prevents sudden
changes in the scaling factor by using normalized embeddings, its loss
function is based on the squared difference of the distance (as it uses the SD
softmax formulation). As this loss can increase the squared difference of the
distance among different proxies, the absolute distance between any pairs of
proxies may be increased. This tendency can result in larger distances, even
among semantically similar classes, and proxy positions that are less
organized in terms of visual similarity. On the other hand, the DR
formulation-based loss is based on distance ratios. Thus, there is no tendency
to increase absolute distances among proxies, and proxies can be structured
according to visual similarity. However, further investigations are needed to
verify this hypothesis and reveal the underlying cause.

## Conclusion

The hierarchy-informed performance must be improved to more broadly adopt
classification models. We explored this concept with classification-based
metric learning models in situations in which hierarchical information is and
is not available during training. Our results show that when the class
hierarchical relations are unknown, the ProxyDR model achieves the best
hierarchical inference and hierarchy-informed performance. In contrast, with
hierarchy-informed training, the CORR loss model achieves the best hierarchy-
informed performance but the lowest top-5 accuracy on most datasets. Since
some hierarchy-informed measures may not be improved by the use of
hierarchical information during training, multiple hierarchy-informed
performance measures should be used to obtain appropriate comparisons.
Additionally, our experiments reveal that during classification-based metric
learning, initializing proxies based on prototypes may be beneficial.

## Acknowledgments

Hyeongji Kim, Terje Berge, and Ketil Malde acknowledge the Ministry of Trade,
Industry and Fisheries for financial support. The authors thank Gayantonia
Franze, Hege Lyngvær Mathisen, Magnus Reeve, and Mona Ring Kleiven, from the
Plankton Research Group at Institute of Marine Research (IMR), for their
contributions to the plankton datasets.

## References

  * 1. He K, Zhang X, Ren S, Sun J.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.  In: Proceedings of the IEEE international conference on computer vision; 2015. p. 1026–1034.
  * 2. Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, et al.  Imagenet large scale visual recognition challenge.  International journal of computer vision. 2015;115(3):211–252.
  * 3. Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L.  Imagenet: A large-scale hierarchical image database.  In: 2009 IEEE conference on computer vision and pattern recognition. Ieee; 2009. p. 248–255.
  * 4. Bertinetto L, Mueller R, Tertikas K, Samangooei S, Lord NA.  Making better mistakes: Leveraging class hierarchies with deep networks.  In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2020. p. 12506–12515.
  * 5. Barz B, Denzler J.  Hierarchy-based image embeddings for semantic image retrieval.  In: 2019 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE; 2019. p. 638–647.
  * 6. Garnot VSF, Landrieu L.  Leveraging Class Hierarchies with Metric-Guided Prototype Learning.  In: British Machine Vision Conference (BMVC); 2021.
  * 7. Jayathilaka M, Mu T, Sattler U.  Ontology-based n-ball concept embeddings informing few-shot image classification.  arXiv preprint arXiv:210909063. 2021;.
  * 8. Musgrave K, Belongie S, Lim SN.  A metric learning reality check.  In: European Conference on Computer Vision. Springer; 2020. p. 681–699.
  * 9. Snell J, Swersky K, Zemel R.  Prototypical Networks for Few-shot Learning.  Advances in Neural Information Processing Systems. 2017;30:4077–4087.
  * 10. Chen WY, Liu YC, Kira Z, Wang YCF, Huang JB.  A closer look at few-shot classification.  arXiv preprint arXiv:190404232. 2019;.
  * 11. Miller GA.  WordNet: An electronic lexical database.  MIT press; 1998.
  * 12. Godbole S.  Exploiting confusion matrices for automatic generation of topic hierarchies and scaling up multi-way classifiers.  Annual Progress Report, Indian Institute of Technology–Bombay, India. 2002;.
  * 13. Wang F, Xiang X, Cheng J, Yuille AL.  Normface: L2 hypersphere embedding for face verification.  In: Proceedings of the 25th ACM international conference on Multimedia; 2017. p. 1041–1049.
  * 14. Kim S, Kim D, Cho M, Kwak S.  Proxy anchor loss for deep metric learning.  In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2020. p. 3238–3247.
  * 15. Wang H, Wang Y, Zhou Z, Ji X, Gong D, Zhou J, et al.  Cosface: Large margin cosine loss for deep face recognition.  In: Proceedings of the IEEE conference on computer vision and pattern recognition; 2018. p. 5265–5274.
  * 16. Deng J, Guo J, Xue N, Zafeiriou S.  Arcface: Additive angular margin loss for deep face recognition.  In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2019. p. 4690–4699.
  * 17. Wan A, Dunlap L, Ho D, Yin J, Lee S, Petryk S, et al.  {NBDT}: Neural-Backed Decision Tree.  In: International Conference on Learning Representations; 2021.Available from: https://openreview.net/forum?id=mCLVeEpplNE.
  * 18. Silla CN, Freitas AA.  A survey of hierarchical classification across different application domains.  Data Mining and Knowledge Discovery. 2011;22(1):31–72.
  * 19. Weinberger KQ, Blitzer J, Saul L.  Distance metric learning for large margin nearest neighbor classification.  Advances in neural information processing systems. 2005;18.
  * 20. Hadsell R, Chopra S, LeCun Y.  Dimensionality reduction by learning an invariant mapping.  In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06). vol. 2. IEEE; 2006. p. 1735–1742.
  * 21. Zhe X, Ou-Yang L, Yan H.  Improve L2-normalized Softmax with Exponential Moving Average.  In: 2019 International Joint Conference on Neural Networks (IJCNN). IEEE; 2019. p. 1–7.
  * 22. Zhang X, Zhao R, Qiao Y, Wang X, Li H.  Adacos: Adaptively scaling cosine logits for effectively learning deep face representations.  In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2019. p. 10823–10832.
  * 23. Zhai A, Wu HY.  Classification is a strong baseline for deep metric learning.  British Machine Vision Conference (BMVC). 2019;.
  * 24. Liu W, Wen Y, Yu Z, Li M, Raj B, Song L.  Sphereface: Deep hypersphere embedding for face recognition.  In: Proceedings of the IEEE conference on computer vision and pattern recognition; 2017. p. 212–220.
  * 25. Kim H, Parviainen P, Malde K.  Distance-Ratio-Based Formulation for Metric Learning.  arXiv preprint arXiv:220108676. 2022;.
  * 26. Krizhevsky A.  Learning multiple layers of features from tiny images; 2009.
  * 27. Van Horn G, Branson S, Farrell R, Haber S, Barry J, Ipeirotis P, et al.  Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2015. p. 595–604.
  * 28. Movshovitz-Attias Y, Toshev A, Leung TK, Ioffe S, Singh S.  No fuss distance metric learning using proxies.  In: Proceedings of the IEEE International Conference on Computer Vision; 2017. p. 360–368.
  * 29. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z.  Rethinking the inception architecture for computer vision.  In: Proceedings of the IEEE conference on computer vision and pattern recognition; 2016. p. 2818–2826.
  * 30. He K, Zhang X, Ren S, Sun J.  Deep residual learning for image recognition.  In: Proceedings of the IEEE conference on computer vision and pattern recognition; 2016. p. 770–778.
  * 31. Kingma DP, Ba J.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:14126980. 2014;.
  * 32. Frome A, Corrado GS, Shlens J, Bengio S, Dean J, Ranzato M, et al.  Devise: A deep visual-semantic embedding model.  Advances in neural information processing systems. 2013;26.
  * 33. Zhang R, Isola P, Efros AA, Shechtman E, Wang O.  The unreasonable effectiveness of deep features as a perceptual metric.  In: Proceedings of the IEEE conference on computer vision and pattern recognition; 2018. p. 586–595.
  * 34. Johnson W, Lindenstrauss J.  Extensions of Lipschitz mappings into a Hilbert space.  Contemporary Mathematics. 1984;26:189–206.

## Supporting information

## S1 Appendix

### Detailed explanation of the dynamic (adaptive) scaling factors in the
NormFace and ProxyDR models.

Zhang et al. [22] rewrote Eq. 5 as follows:

$\displaystyle
p(c|x)=\frac{\exp(s\cos{\theta_{c}})}{\exp(s\cos{\theta_{c}})+B_{x}},$

where $c$ is the corresponding class of point $x$ and
$B_{x}=\sum\limits_{y\neq c,y\in\mathcal{Y}}{\exp(s\cos{\theta_{y}})}$. They
found that $\theta_{y}$ was close to $\frac{\pi}{2}$ during the training
process for $y\neq c$, i.e., for different classes. Thus,
$B_{x}\approx\sum\limits_{y\neq
c,y\in\mathcal{Y}}{\exp(s\cos{\frac{\pi}{2}})}=\sum\limits_{y\neq
c,y\in\mathcal{Y}}{\exp(0)}=\left|\mathcal{Y}\right|-1$.
In Eq. 10, $\frac{\partial^{2}p(c|x)(\theta_{c})}{\partial\theta_{c}^{2}}$ can
be written as:

$\displaystyle\frac{\partial^{2}p(c|x)(\theta_{c})}{\partial\theta_{c}^{2}}=\frac{-sB_{x}\exp({s\cos{\theta_{c}}})\psi_{NormFace}(s,\theta_{c})}{\left(\exp({s\cos{\theta_{c}}})+B_{x}\right)^{3}},$

where
$\psi_{NormFace}(s,\theta_{c})=\cos{\theta_{c}}\left(\exp({s\cos{\theta_{c}}})+B_{x}\right)+s\sin^{2}{\theta_{c}}\left(\exp({s\cos{\theta_{c}}})-B_{x}\right)$.
Moreover, they used $s=\frac{\log{B_{x}}}{\cos{\theta_{c}}}$ to approximate
the solution for Eq. 10. For the static version, they used
$\left|\mathcal{Y}\right|-1$ to estimate $B_{x}$ and $\frac{\pi}{4}$ to
estimate $\theta_{c}$. For the dynamic version, they used $B_{x;avg}$ to
estimate $B_{x}$ and $\theta_{c;med}$ to estimate $\theta_{c}$, where
$B_{x;avg}$ is the average of $B_{x}$ in a mini-batch and $\theta_{c;med}$ is
the median of the $\theta_{c}$ values in a mini-batch. They clipped the
$\theta_{c;med}$ value to be in the range $\left[0,\frac{\pi}{4}\right]$.
Instead of using $s=\frac{\log{B_{x}}}{\cos{\theta_{c}}}$, in our
implementation, we use the Adam optimizer [31] to update a scale factor $s$
that minimizes $\psi_{NormFace}^{2}(s,\theta_{c})$, i.e.,
$\psi_{NormFace}(s,\theta_{c})\approx 0$. For $\psi_{NormFace}(s,\theta_{c})$,
we use $\left|\mathcal{Y}\right|-1$ to estimate $B_{x}$ and $\frac{\pi}{4}$ to
estimate $\theta_{c}$ to initialize the value of $s$. During model training,
we use $B_{x;avg}$ to estimate $B_{x}$ and $\theta_{c;med}$ to estimate
$\theta_{c}$.
We can also apply the dynamic scaling factor to the ProxyDR model. First, we
define $d_{x,y}:=\left\|\tilde{f}(x)-\tilde{W}_{y}\right\|$. We rewrite Eq. 8
as:

$\displaystyle p(c|x)=\frac{d_{x,c}^{-s}}{d_{x,c}^{-s}+B_{x}},$

where $B_{x}=\sum\limits_{y\neq c,y\in\mathcal{Y}}{d_{x,y}^{-s}}$. Assuming
$\theta_{y}\approx\frac{\pi}{2}$ for $y\neq c$, we obtain
$B_{x}\approx\sum\limits_{y\neq
c,y\in\mathcal{Y}}{\left(\frac{\pi}{2}\right)^{-s}}=\left(\left|\mathcal{Y}\right|-1\right)\left(\frac{\pi}{2}\right)^{-s}$.
The expression $\frac{\partial^{2}p(c|x)(\theta_{c})}{\partial\theta_{c}^{2}}$
can be written as:

$\displaystyle\frac{\partial^{2}p(c|x)(\theta_{c})}{\partial\theta_{c}^{2}}=\frac{B_{x}s\theta_{c}^{(s-2)}\psi_{ProxyDR}(s,\theta_{c})}{\left(B_{x}\theta_{c}^{s}+1\right)^{3}}$

where $\psi_{ProxyDR}(s,\theta_{c})=B_{x}(s+1)\theta_{c}^{s}-s+1$. We then use
the Adam optimizer to update a scale factor $s$ that minimizes
$\psi_{ProxyDR}^{2}(s,\theta_{c})$.

## S2 Appendix

### Dataset details.

All plankton images were obtained using FlowCam (Yokogawa Fluid Imaging
Technologies). FlowCam is a flow imaging microscope that captures particles
flowing through glass flowcells with well-defined volumes. The three plankton
datasets were obtained according to different types of samples (live and Lugol
fixed whole seawater or $180\,\mu m$ WP2 plankton net samples) using three
different FlowCams (FlowCam 8400, FlowCam VS, and FlowCam Macro) with various
magnifications. Thus, the datasets include particles ranging from $5$ to
$2000\,\mu m$ in size, thus representing nano-, micro-, and mesozooplankton.
The plankton samples were obtained from three coastal monitoring stations
(Institute of Marine Research) along the Norwegian coast, including Holmfjord
in the north, Austevoll in the west and Torungen in the south. In addition,
for the nano- and microplankton, seawater samples were obtained from a tidal
zone at a depth of 1 meter at the research station at Flødevigen in southern
Norway, which is approximately 2 nautical miles from the southern monitoring
station at Torungen. The sampling period for the three datasets covered all
seasons over a period of approximately 2.5 years.

##### Small microplankton (MicroS)

This dataset contains images of fixed and live seawater samples acquired at a
depth of $5\,m$ at the three monitoring stations and a depth of $1\,m$ in the
tidal zone (see above). The seawater samples were carefully filtered through a
$80\,\mu m$ mesh to ensure that $100\,\mu m$ flowcell was not clogged and
imaged using a $10\times$ objective. This FlowCam configuration results in a
total magnification of $100\times$ and images particles ranging from $5$ to
$50\,\mu m$. Before resizing, one pixel in an image represented $0.7330\,\mu
m$.

##### Large microplankton (MicroL)

This dataset contains images of fixed and live seawater samples acquired at a
depth of $5\,m$ at the three monitoring stations and a depth of $1\,m$ in the
tidal zone (see above). The seawater samples were not filtered and were imaged
using a $2\times$ objective, targeting $35$ to $500\,\mu m$ particles. Before
resizing, one pixel in an image represented $2.9730\,\mu m$. Due to instrument
repair and adjustments to improve image quality, the camera settings were
modified during the 3 years of imaging to acquire this dataset. Therefore, the
image appearance and quality are slightly variable.

##### Mesozooplankton (MesoZ)

This dataset contains images of mesozooplankton samples acquired at the three
coastal monitoring stations (see above) and a transect in the Norwegian Sea
(Svinøysnittet). The samples were obtained using an IMR (Institute of Marine
Research) standard plankton net (WP2) or a multinet mammoth (both $180\,\mu m$
mesh) and fixed with $4\%$ formaldehyde. The images were acquired by two
FlowCam instruments (one in Bergen and one in Flødevigen), and the image
appearance differs slightly between the two instruments. The FlowCam macro was
equipped with a $0.5\times$ objective, resulting in a total magnification of
12.5 and imaging organisms ranging from $180$ to $2000\,\mu m$. Before
resizing, one pixel in an image represented $9.05\,\mu m$. All images in the
mesozooplankton dataset are in grayscale.

##### NABirds

Data provided by the Cornell Lab of Ornithology, with thanks to photographers
and contributors of crowdsourced data at AllAboutBirds.org/Labs. This material
is based upon work supported by the National Science Foundation under Grant
No. 1010818.

##### Hierarchical structures of the datasets

Figs. S1, S2, and S3 and Table S1 show the hierarchical structures of the
datasets. As the NABirds dataset contains too many (555) classes to visualize,
we do not show the hierarchical structures of the NABirds dataset [27]. We
used the hierarchy provided by the Cornell Lab of Ornithology.

Fig S1: The hierarchical structure of the MicroS dataset used in our experiment. Best viewed by zooming in.  Fig S2: The hierarchical structure of the MicroL dataset used in our experiment. Best viewed by zooming in.  Fig S3: The hierarchical structure of the MesoZ dataset used in our experiment. Best viewed by zooming in.  Table S1: The hierarchical structure of the CIFAR100 dataset used in our experiment. Level 0 |  Level 1 |  Level 2 |  Level 3 (class level)
---|---|---|---
Animals |  Invertebrates |  insects |  bee, beetle, butterfly, caterpillar, cockroach
|  |  non-insect invertebrates |  crab, lobster, snail, spider, worm
|  Mammals |  aquatic mammals |  beaver, dolphin, otter, seal, whale
|  |  large carnivores |  bear, leopard, lion, tiger, wolf
|  |  large omnivores and herbivores |  camel, cattle, chimpanzee, elephant, kangaroo
|  |  medium-sized mammals |  fox, porcupine, possum, raccoon, skunk
|  |  people |  baby, boy, girl, man, woman
|  |  small mammals |  hamster, mouse, rabbit, shrew, squirrel
|  Non-mammal vertebrates |  fish |  aquarium fish, flatfish, ray, shark, trout
|  |  reptiles |  crocodile, dinosaur, lizard, snake, turtle
Artificial |  Artificial (indoor) |  food containers |  bottles, bowls, cans, cups, plates
|  |  household electrical devices |  clock, computer keyboard, lamp, telephone, television
|  |  household furniture |  bed, chair, couch, table, wardrobe
|  Artificial (outdoor) |  large man-made outdoor things |  bridge, castle, house, road, skyscraper
|  |  vehicles 1 |  bicycle, bus, motorcycle, pickup truck, train
|  |  vehicles 2 |  lawn-mower, rocket, streetcar, tank, tractor
Nature (non-animal) |  Nature (non-specific organism) |  large natural outdoor scenes |  cloud, forest, mountain, plain, sea
|  Plants |  flowers |  orchids, poppies, roses, sunflowers, tulips
|  |  fruit and vegetables |  apples, mushrooms, oranges, pears, sweet peppers
|  |  trees |  maple, oak, palm, pine, willow

## S3 Appendix

Results for all five datasets.
Figs. S4, S5, S6, S7, and S8 show the top-$k$ accuracies on the different
datasets.

Fig S4: Top-$k$ accuracy results (A: $k=1$, B: $k=5$) on the MicroS dataset.
Fig S5: Top-$k$ accuracy results (A: $k=1$, B: $k=5$) on the MicroL dataset.
Fig S6: Top-$k$ accuracy results (A: $k=1$, B: $k=5$) on the MesoZ dataset.
Fig S7: Top-$k$ accuracy results (A: $k=1$, B: $k=5$) on the CIFAR100 dataset.
Fig S8: Top-$k$ accuracy results (A: $k=1$, B: $k=5$) on the NABirds dataset.

Figs. S9, S10, S11, S12, and S13 show the mean correlation values on the
different datasets.

Fig S9: Correlation measures on the MicroS dataset. (Top) Values using proxies
(A: whole classes, B: living classes). (Bottom) Values using prototypes (C:
whole classes, D: living classes). ‘Living’ indicates that only biological
classes were used (no nonliving classes). The mean correlation values based on
proxies with the MDS option were $0.9306$ (whole) and $0.9011$ (living).  Fig
S10: Correlation measures on the MicroL dataset. (Top) Values using proxies
(A: whole classes, B: living classes). (Bottom) Values using prototypes (C:
whole classes, D: living classes). The mean correlation values based on
proxies with the MDS option were $0.9543$ (whole) and $0.9426$ (living). Fig
S11: Correlation measures on the MesoZ dataset. (Top) Values using proxies (A:
whole classes, B: living classes). (Bottom) Values using prototypes (C: whole
classes, D: living classes). The mean correlation values based on proxies with
the MDS option were $0.9783$ (whole) and $0.9602$ (living).  Fig S12:
Correlation measures on the CIFAR100 dataset. (A) Values using proxies. (B)
Values using prototypes. The mean correlation value based on proxies with the
MDS option was $0.8580$.  Fig S13: Correlation measures on the NABirds
dataset. (A) Values using proxies. (B) Values using prototypes. The mean
correlation value based on proxies with the MDS option was $0.4476$ (this
value is small as the dataset contains 555 classes and the embedding dimension
is 128.).

Figs. S14, S15, S16, S17, and S18 show the hierarchy-informed performance
measures on the different datasets.

Fig S14: Hierarchical performance measures on the MicroS dataset. The symbol
$\downarrow$ denotes that lower values indicate better performance. The symbol
$\uparrow$ denotes that higher values indicate better performance. (A) AHD
(k=1): $\downarrow$. (B) AHD (k=5): $\downarrow$. (C) HP@5: $\uparrow$. (D)
HS@50: $\uparrow$. (E) HS@250: $\uparrow$. (F) AHS@250: $\uparrow$.  Fig S15:
Hierarchical performance measures on the MicroL dataset. The symbol
$\downarrow$ denotes that lower values indicate better performance. The symbol
$\uparrow$ denotes that higher values indicate better performance. (A) AHD
(k=1): $\downarrow$. (B) AHD (k=5): $\downarrow$. (C) HP@5: $\uparrow$. (D)
HS@50: $\uparrow$. (E) HS@250: $\uparrow$. (F) AHS@250: $\uparrow$.  Fig S16:
Hierarchical performance measures on the MesoZ dataset. The symbol
$\downarrow$ denotes that lower values indicate better performance. The symbol
$\uparrow$ denotes that higher values indicate better performance. (A) AHD
(k=1): $\downarrow$. (B) AHD (k=5): $\downarrow$. (C) HP@5: $\uparrow$. (D)
HS@50: $\uparrow$. (E) HS@250: $\uparrow$. (F) AHS@250: $\uparrow$.  Fig S17:
Hierarchical performance measures on the CIFAR100 dataset. The symbol
$\downarrow$ denotes that lower values indicate better performance. The symbol
$\uparrow$ denotes that higher values indicate better performance. (A) AHD
(k=1): $\downarrow$. (B) AHD (k=5): $\downarrow$. (C) HP@5: $\uparrow$. (D)
HS@50: $\uparrow$. (E) HS@250: $\uparrow$. (F) AHS@250: $\uparrow$.  Fig S18:
Hierarchical performance measures on the NABirds dataset. The symbol
$\downarrow$ denotes that lower values indicate better performance. The symbol
$\uparrow$ denotes that higher values indicate better performance. (A) AHD
(k=1): $\downarrow$. (B) AHD (k=5): $\downarrow$. (C) HP@5: $\uparrow$. (D)
HS@50: $\uparrow$. (E) HS@250: $\uparrow$. (F) AHS@250: $\uparrow$.

Additional results on mean correlations
Figs. S19, S20, S21, S22, S23 and S24 visualize the changes in the mean
correlation values with different models and training options, showing the
averaged values for five different seeds. Figs. S19 and S22 show that the
accuracy of the NormFace models decreased after a certain number of epochs
when the models were trained with the dynamic option. Interestingly, the mean
correlation values increased even when the accuracy decreased. When using
predefined hierarchical information and dynamic options (if applicable), the
prototype-baesd mean correlation values approached the fixed proxy-based mean
correlation values, except the NormFace model on the NABirds dataset.

Fig S19: Changes in accuracy and mean correlations for the CIFAR100 dataset
(NormFace). (A) Accuracy curve with standard training. (B) Mean correlation
curve with standard training. (C) Accuracy curve with the dynamic option. (D)
Mean correlation curve with the dynamic option. (E) Accuracy curve with the
MDS option. (F) Mean correlation curve with the MDS option. (G) Accuracy curve
with the MDS & dynamic options. (H) Mean correlation curve with the MDS &
dynamic options.  Fig S20: Changes in accuracy and mean correlations for the
CIFAR100 dataset (ProxyDR). (A) Accuracy curve with standard training. (B)
Mean correlation curve with standard training. (C) Accuracy curve with the
dynamic option. (D) Mean correlation curve with the dynamic option. (E)
Accuracy curve with the MDS option. (F) Mean correlation curve with the MDS
option. (G) Accuracy curve with the MDS & dynamic options. (H) Mean
correlation curve with the MDS & dynamic options.  Fig S21: Changes in
accuracy and mean correlations for the CIFAR100 dataset (CORR). (A) Accuracy
curve with CORR loss training. (B) Mean correlation curve with CORR loss
training.  Fig S22: Changes in accuracy and mean correlations for the NABirds
dataset (NormFace). (A) Accuracy curve with standard training. (B) Mean
correlation curve with standard training. (C) Accuracy curve with the dynamic
option. (D) Mean correlation curve with the dynamic option. (E) Accuracy curve
with the MDS option. (F) Mean correlation curve with the MDS option. (G)
Accuracy curve with the MDS & dynamic options. (H) Mean correlation curve with
the MDS & dynamic options.  Fig S23: Changes in accuracy and mean correlations
for the NABirds dataset (ProxyDR). (A) Accuracy curve with standard training.
(B) Mean correlation curve with standard training. (C) Accuracy curve with the
dynamic option. (D) Mean correlation curve with the dynamic option. (E)
Accuracy curve with the MDS option. (F) Mean correlation curve with the MDS
option. (G) Accuracy curve with the MDS & dynamic options. (H) Mean
correlation curve with the MDS & dynamic options.  Fig S24: Changes in
accuracy and mean correlations for the NABirds dataset (CORR). (A) Accuracy
curve with CORR loss training. (B) Mean correlation curve with CORR loss
training.

##### Rank correlations between performance measures

We investigate whether different performance measures are correlated. While we
cannot derive causal relationships, this investigation is useful for
determining if there are trade-offs between various performance measures. We
used rank correlation coefficients between the mean correlation values and
other measures. We ignored the softmax loss results, as this model is not
based on metric learning.
In Table S2, we investigate whether mean correlation values based on proxies
(Figs. S9, S10, S11, S12, and S13) are correlated with other performance
measures (Figs. S4, S5, S6, S7, S8, S14, S15, S16, S17, and S18). The table
shows that the AHD (k=5), HP@5, HS@250, and AHS@250 measures are correlated
with the proxy-based mean correlation values. Moreover, there does not appear
to be any trade-off between the top-1 accuracies and proxy-based mean
correlations. However, a trade-off between the top-5 accuracy and proxy-based
mean correlations is observed on some datasets.

Table S2:  Rank (Spearman) correlation coefficients of different measures with
mean correlations (based on proxies)

Measure |  MicroS |  MicroL |  MesoZ |  CIFAR100 |  NABirds
---|---|---|---|---|---
Top-1 |  0.2039 |  0.2260 |  0.2171 |  0.1675 |  0.3112
|  $(1.0336\times 10^{-1})$ |  $(7.0284\times 10^{-2})$ |  $(8.2377\times 10^{-2})$ |  $(1.8239\times 10^{-1})$ |  $(1.1627\times 10^{-2})$
Top-5 |  -0.1350 |  -0.3716 |  -0.2120 |  -0.3364 |  -0.5669
|  $(2.8361\times 10^{-1})$ |  $(\bm{2.3043\times 10^{-3}})$ |  $(9.0045\times 10^{-2})$ |  $(\bm{6.1476\times 10^{-3}})$ |  $(\bm{8.4964\times 10^{-7}})$
AHD (k=1) |  -0.3423 |  -0.3855 |  -0.2959 |  -0.1821 |  -0.4267
|  $(\bm{5.2583\times 10^{-3}})$ |  $(\bm{1.5179\times 10^{-3}})$ |  $(1.6684\times 10^{-2})$ |  $(1.4658\times 10^{-1})$ |  $(\bm{3.9270\times 10^{-4}})$
AHD (k=5) |  -0.7407 |  -0.8258 |  -0.9026 |  -0.8856 |  -0.7023
|  $(\bm{1.7522\times 10^{-12}})$ |  $(\bm{2.5540\times 10^{-17}})$ |  $(\bm{9.6705\times 10^{-25}})$ |  $(\bm{1.1648\times 10^{-22}})$ |  $(\bm{7.0541\times 10^{-11}})$
HP@5 |  0.7392 |  0.8350 |  0.9001 |  0.8955 |  0.7070
|  $(\bm{2.0489\times 10^{-12}})$ |  $(\bm{5.4193\times 10^{-18}})$ |  $(\bm{2.0612\times 10^{-24}})$ |  $(\bm{7.8000\times 10^{-24}})$ |  $(\bm{4.6412\times 10^{-11}})$
HS@50 |  0.7671 |  0.7962 |  0.5634 |  0.1878 |  0.5276
|  $(\bm{9.2047\times 10^{-14}})$ |  $(\bm{2.2284\times 10^{-15}})$ |  $(\bm{1.0275\times 10^{-6}})$ |  $(1.3410\times 10^{-1})$ |  $(\bm{6.3074\times 10^{-6}})$
HS@250 |  0.7911 |  0.8167 |  0.8860 |  0.5514 |  0.7210
|  $(\bm{4.4738\times 10^{-15}})$ |  $(\bm{1.1029\times 10^{-16}})$ |  $(\bm{1.0514\times 10^{-22}})$ |  $(\bm{1.9333\times 10^{-6}})$ |  $(\bm{1.2570\times 10^{-11}})$
AHS@250 |  0.7950 |  0.8186 |  0.8034 |  0.3335 |  0.6862
|  $(\bm{2.6376\times 10^{-15}})$ |  $(\bm{8.1611\times 10^{-17}})$ |  $(\bm{8.1089\times 10^{-16}})$ |  $(\bm{6.6355\times 10^{-3}})$ |  $(\bm{2.8150\times 10^{-10}})$

$p$ values are written in parentheses. Significant results ($p$ value $<0.01$)
are written in bold text.

In Table S3, we investigate whether the top-1 accuracy (Figs. S4, S5, S6, S7,
and S8) is correlated with other performance measures (Figs. S4, S5, S6, S7,
S8, S14, S15, S16, S17, and S18). The table shows that the AHD (k=1), HS@50,
and AHS@250 measures are correlated with the top-1 accuracy. Note that these
are the measures that did not show noticeable changes on some datasets when
predefined hierarchical information was used during training.

Table S3:  Rank (Spearman) correlation coefficients of different measures with
the top-1 accuracy

Measure |  MicroS |  MicroL |  MesoZ |  CIFAR100 |  NABirds
---|---|---|---|---|---
Top-5 |  0.2199 |  0.0365 |  0.1033 |  0.4398 |  -0.1043
|  $(7.8357\times 10^{-2})$ |  $(7.7302\times 10^{-1})$ |  $(4.1302\times 10^{-1})$ |  $(\bm{2.4659\times 10^{-4}})$ |  $(4.0815\times 10^{-1})$
AHD (k=1) |  -0.8005 |  -0.8302 |  -0.9309 |  -0.9507 |  -0.9509
|  $(\bm{1.2168\times 10^{-15}})$ |  $(\bm{1.2369\times 10^{-17}})$ |  $(\bm{2.9921\times 10^{-29}})$ |  $(\bm{9.4257\times 10^{-34}})$ |  $(\bm{8.6632\times 10^{-34}})$
AHD (k=5) |  -0.1906 |  -0.3259 |  -0.2908 |  -0.2877 |  -0.6136
|  $(1.2829\times 10^{-1})$ |  $(\bm{8.0758\times 10^{-3}})$ |  $(1.8752\times 10^{-2})$ |  $(2.0115\times 10^{-2})$ |  $(\bm{5.4992\times 10^{-8}})$
HP@5 |  0.2263 |  0.2956 |  0.2886 |  0.2439 |  0.6029
|  $(6.9923\times 10^{-2})$ |  $(1.6811\times 10^{-2})$ |  $(1.9712\times 10^{-2})$ |  $(5.0222\times 10^{-2})$ |  $(\bm{1.0692\times 10^{-7}})$
HS@50 |  0.4325 |  0.5085 |  0.6979 |  0.8967 |  0.6832
|  $(\bm{3.2041\times 10^{-4}})$ |  $(\bm{1.5301\times 10^{-5}})$ |  $(\bm{1.0420\times 10^{-10}})$ |  $(\bm{5.5302\times 10^{-24}})$ |  $(\bm{3.6174\times 10^{-10}})$
HS@250 |  0.1369 |  0.3057 |  0.2945 |  0.5535 |  0.4915
|  $(2.7674\times 10^{-1})$ |  $(1.3279\times 10^{-2})$ |  $(1.7269\times 10^{-2})$ |  $(\bm{1.7312\times 10^{-6}})$ |  $(\bm{3.2203\times 10^{-5}})$
AHS@250 |  0.3394 |  0.4233 |  0.5067 |  0.8517 |  0.5733
|  $(\bm{5.6730\times 10^{-3}})$ |  $(\bm{4.4195\times 10^{-4}})$ |  $(\bm{1.6583\times 10^{-5}})$ |  $(\bm{2.4413\times 10^{-19}})$ |  $(\bm{6.0057\times 10^{-7}})$
Mean correlation (proxy) |  0.2039 |  0.2260 |  0.2171 |  0.1675 |  0.3112
|  $(1.0336\times 10^{-1})$ |  $(7.0284\times 10^{-2})$ |  $(8.2377\times 10^{-2})$ |  $(1.8239\times 10^{-1})$ |  $(1.1627\times 10^{-2})$
Mean correlation (prototype) |  0.2150 |  0.2055 |  0.1738 |  0.1978 |  0.2664
|  $(8.5503\times 10^{-2})$ |  $(1.0051\times 10^{-1})$ |  $(1.6616\times 10^{-1})$ |  $(1.1423\times 10^{-1})$ |  $(3.1936\times 10^{-2})$

$p$ values are written in parentheses. Significant results ($p$ value $<0.01$)
are written in bold text.

In Table S4, we investigate whether the top-5 accuracy (Figs. S4, S5, S6, S7,
and S8) is correlated with other performance measures (Figs. S4, S5, S6, S7,
S8, S14, S15, S16, S17, and S18). The table shows that no measures are
correlated with the top-5 accuracy on all datasets.

Table S4:  Rank (Spearman) correlation coefficients of different measures with
the top-5 accuracy

Measure |  MicroS |  MicroL |  MesoZ |  CIFAR100 |  NABirds
---|---|---|---|---|---
Top-1 |  0.2199 |  0.0365 |  0.1033 |  0.4398 |  -0.1043
|  $(7.8357\times 10^{-2})$ |  $(7.7302\times 10^{-1})$ |  $(4.1302\times 10^{-1})$ |  $(\bm{2.4659\times 10^{-4}})$ |  $(4.0815\times 10^{-1})$
AHD (k=1) |  -0.0716 |  0.0033 |  -0.0406 |  -0.4245 |  0.1196
|  $(5.7106\times 10^{-1})$ |  $(9.7911\times 10^{-1})$ |  $(7.4809\times 10^{-1})$ |  $(\bm{4.2428\times 10^{-4}})$ |  $(3.4268\times 10^{-1})$
AHD (k=5) |  0.2417 |  0.4842 |  0.2732 |  0.4561 |  0.4949
|  $(5.2436\times 10^{-2})$ |  $(\bm{4.3828\times 10^{-5}})$ |  $(2.7654\times 10^{-2})$ |  $(\bm{1.3422\times 10^{-4}})$ |  $(\bm{2.7791\times 10^{-5}})$
HP@5 |  -0.2765 |  -0.5318 |  -0.3113 |  -0.4902 |  -0.5733
|  $(2.5757\times 10^{-2})$ |  $(\bm{5.1600\times 10^{-6}})$ |  $(1.1590\times 10^{-2})$ |  $(\bm{3.3964\times 10^{-5}})$ |  $(\bm{5.9953\times 10^{-7}})$
HS@50 |  0.0115 |  -0.3164 |  -0.2331 |  0.2515 |  -0.0071
|  $(9.2784\times 10^{-1})$ |  $(1.0231\times 10^{-2})$ |  $(6.1677\times 10^{-2})$ |  $(4.3295\times 10^{-2})$ |  $(9.5522\times 10^{-1})$
HS@250 |  -0.0677 |  -0.4343 |  -0.1965 |  0.0955 |  -0.3192
|  $(5.9231\times 10^{-1})$ |  $(\bm{3.0034\times 10^{-4}})$ |  $(1.1677\times 10^{-1})$ |  $(4.4900\times 10^{-1})$ |  $(\bm{9.5513\times 10^{-3}})$
AHS@250 |  0.0249 |  -0.3603 |  -0.1446 |  0.3157 |  -0.2203
|  $(8.4393\times 10^{-1})$ |  $(\bm{3.1949\times 10^{-3}})$ |  $(2.5059\times 10^{-1})$ |  $(1.0404\times 10^{-2})$ |  $(7.7867\times 10^{-2})$
Mean correlation (proxy) |  -0.1350 |  -0.3716 |  -0.2120 |  -0.3364 |  -0.5669
|  $(2.8361\times 10^{-1})$ |  $(\bm{2.3043\times 10^{-3}})$ |  $(9.0045\times 10^{-2})$ |  $(\bm{6.1476\times 10^{-3}})$ |  $(\bm{8.4964\times 10^{-7}})$
Mean correlation (prototype) |  -0.1040 |  -0.4784 |  -0.2910 |  -0.3675 |  -0.5457
|  $(4.0985\times 10^{-1})$ |  $(\bm{5.5679\times 10^{-5}})$ |  $(1.8677\times 10^{-2})$ |  $(\bm{2.5967\times 10^{-3}})$ |  $(\bm{2.5927\times 10^{-6}})$

$p$ values are written in parentheses. Significant results ($p$ value $<0.01$)
are written in bold text.