# Abnormal Event Detection in Urban Surveillance Videos Using GAN and Transfer
Learning

Ali Atghaei, Soroush Ziaeinejad, and Mohammad Rahmati
Department of Computer Engineering and Information Technology
Amirkabir University of Technology (Tehran Polytechnic), Tehran, Iran
Email: {atghaei, ziaeinejad<EMAIL_ADDRESS>
###### Abstract

Abnormal event detection (AED) in urban surveillance videos has multiple
challenges. Unlike other computer vision problems, the AED is not solely
dependent on the content of frames. It also depends on the appearance of the
objects and their movements in the scene. Various methods have been proposed
to address the AED problem. Among those, deep learning–based methods show the
best results. This paper is based on deep learning methods and provides an
effective way to detect and locate abnormal events in videos by handling
spatio-temporal data. This paper uses generative adversarial networks (GANs)
and performs transfer learning algorithms on pre-trained convolutional neural
network (CNN) which result in an accurate and efficient model. The efficiency
of the model is further improved by processing the optical-flow information of
the video. This paper runs experiments on two benchmark datasets for AED
problem (UCSD Peds1 and UCSD Peds2) and compares the results with other
previous methods. The comparisons are based on various criteria such as area
under curve (AUC) and true positive rate (TPR). Experimental results show that
the proposed method can effectively detect and locate abnormal events in crowd
scenes.

###### Index Terms:

Deep learning, event detection, generative adversarial network, machine
learning, neural networks, transfer learning.

##  I Introduction

Using smart surveillance cameras has recently become very popular. Compared to
human monitoring systems, these smart cameras have more consistent behavior
and can provide higher accuracy and quicker response. The higher efficiency of
these systems has attracted researchers and developers active in developing
automated surveillance systems [1]. By considering the widespread use of
automated surveillance systems in different applications, it is expected that
computer vision–based systems will be able to automatically process a large
amount of visual information. Abnormal Event Detection (AED) in videos is one
of the most popular computer vision topics. Because of the nature and
subjective definitions of ‘abnormality’ (or ‘anomaly’), AED is a very
challenging and content-dependent problem [2]. with its major problems
addressed, AED can be used for a wide variety of applications such as crowd
analysis, subway stations and urban pathways surveillance, summarization of
surveillance videos, and smart home monitoring. In the case of using AED for
crowd analysis, this system is expected to understand the crowd behavior in a
public place for a period of time and inform human agents if it observes an
unusual event in the video.

An event is considered abnormal if it is unlikely or unexpected to occur. In
statistics, ‘anomaly’ is defined as an unusual behavior in a distribution or
an outlier data point in a data space. Anomaly detection systems are trained
with a normal dataset and construe an outlier as abnormal behavior. These
systems usually try to jointly detect and locate abnormal behavior. Locating
means detecting the location of an abnormal event by showing the abnormal
pixels of each frame. For AED to have high accuracy and quick response, an
effective data representation method is required. Data space in AED is spatio-
temporal with both appearance and movement involved in the process.
Researchers use various methods to locate abnormal parts of a video file. The
most popular method is gridding, which splits a sequence of frames to smaller
fixed-size 3D patches by applying a fixed grid on the frames [3].

Crowd scene analysis has multiple challenges such as occlusion, shadowing, and
overlapping of moving objects. Different algorithms have been proposed to
overcome these challenge but all of them have their advantages and
disadvantages. There are several methods to understand the movement
information of scene elements. These methods are able to precisely model the
direction and speed of each individual object. However, these methods are
usually very time consuming. Besides, the perspective distortion of urban
surveillance videos adds to the complexity of the problem as it causes
different scale and movement patterns based on object locations and camera
position. Because of different lighting situations and subtle difference
between normal and abnormal cases, an accurate discriminative model is needed
to detect abnormal patches and frames. Various machine learning models such as
deep neural networks require a large amount of labeled data. However, AED is
an unsupervised learning problem for which gathering a large amount of labeled
data is a time consuming and arduous task [2].

Several methods are proposed for AED problem. Trajectory-based methods give a
highly accurate model by using the movement of scene entities. Recently, some
methods such as Histogram of Gradients (HoG) or Histogram of Optical Flows
(HOF) are used to model spatio-temporal properties of the videos. In these
methods, an entity is considered abnormal if the model has never observed its
movement pattern before. However, these methods are not efficient for crowd
scenes because of their high time complexity and also the problem of moving
object occlusion [4], [5]. Researchers have published several papers to
address the AED problem. A considerable portion of these published research is
evaluated with $\text{UCSD Peds}1$ and UCSD Peds2 datasets [6].

Researchers had used traditional machine learning methods before creating deep
learning models. Although these methods are still in use, they have become
limited to specific applications. An HMM model using combined dynamic texture
as the feature set [6], social force method using spatio-temporal data
filtering [7], sparse representation method [8], optical flow clustering
method [9], bag-of-visual-word model to represent images [10], and a GMM model
using 3D gradient images [11] are examples of these traditional models.

Another class of machine learning models is deep learning networks. Recent
research activities commonly use these models to solve complex problems.
Researchers also use deep models to address the AED problem. A one-class SVM
model using optical flow features which extracted by an auto-encoder network
is proposed in [3], and a one-class SVM model using feature set which is
extracted by a pre-trained deep model is proposed in [2].

This paper uses a new method to address the AED problem based on two essential
steps: Spatio-temporal features processing and motion analysis using optical
flow images. Fig. 1 shows the block diagram of the learning phase of the
proposed method which has three main parts: pre-processing, training the GAN,
and analyzing the optical flow images. The testing phase includes a pre-
processing step which is then followed by a spatio-temporal representation.
The representation is then calculated and fed to the trained discriminator
network as an input. In the final step, abnormal patches of all the frames are
determined and an output image is created to show the appearance-motion
abnormalities of each frame.

The rest of this paper is organized as follows: Section 2 describes our
feature extraction methods and different types of analys that we use. This
section also shows the block diagram of the training and testing phases.
Section 3 explains the use of transfer learning by importing a part of VGG16
network and Section 4 reports the experimental results and comparisons with
state-of-the-art methods. Finally, Section 5 concludes the paper.

##  II Appearance-Motion and Motion-Only Analyses

To detect abnormalities in videos, researchers often analyze either appearance
features or motion features. However, processing both appearance and motion
features is usually necessary because this combination may include additional
important information.

In this work, we take advantages of both appearance and motion features. In
the proposed model, the input which is a sequence of original frames moves in
two different paths. One path (straight path shown in Fig. 1) is appearance-
motion analysis which extracts 3D spatio-temporal features. Another path
(downward path shown in Fig. 1) is motion-only analysis which extracts
abnormal directions or speeds using optical flow images.

Figure 1: Training phase of the proposed method.

###  II-A Pre-processing of Input Images

Pre-processing of the data is typically the first step to be taken prior to
using any learning model. In this step, certain image processing tasks are
applied to the frames to resolve their appearance challenges and prepare them
to enter the learning model. Examples of the tasks that can be done in the
pre-processing step are histogram equalization, foreground extraction, edge
detection of foreground objects, obtaining spatio-temporal representations,
and patch extraction. A 3D structure is formed to jointly consider the motion
features and the appearance features. Fig. 2 shows the spatio-temporal
representation of three frames chosen by picking every other frame in a set of
consecutive frames.111Using every other frame instead of using three
consecutive frames better shows the movement of the objects. The spatio-
temporal representation of frame number $t$ is

$D_{t}=<I_{t},I_{\text{t-2}},I_{\text{t-4}}>,$ (1)

where $I\textsubscript{t}$ is the edge image of frame number $t$.

Figure 2: Spatio-temporal representation of three frames.

###  II-B Patch Extraction From Appearance-Motion Representation

Frame patches are determined by adding a grid overlay to the appearance-motion
representation of the frames. Among all patches, those that have at lease a
minimum amount of foreground pixels are chosen to avoid noises and useless
data. These patches are gathered from video frames and inputted to the network
for learning and testing processes. Fig. 3 shows the output of the patch
selection unit.

Figure 3: Patch selection from spatio-temporal representation.

###  II-C Objects Movement Analysis

In this work we focus on the hue and intensity of the output of Gunner-
Farneback optical flow algorithm [12]. The hue shows the path direction and
the intensity shows the speed of moving objects in a video sequence.

Figure 4: Testing phase of the proposed method.

Fig 4 shows the block diagram of the testing phase of the proposed method.
Sorted histograms of direction and speed intensities are calculated in all
video frames. Motion abnormalities are then detected based on the sorted
histograms. The first $5\%$ of hue/intensity values with the lowest
frequencies in the sorted histogram are considered as abnormal. As a result,
motion abnormalities are obtained from this step. The result of this analysis,
together with that of the appearance-motion analysis can effectively detect
and locate abnormal patches in the video.

##  III Transfer Learning to Benefit From VGG16

Data acquisition is one of the main challenges of every machine learning
problem. Also, most of the machine learning methods- especially deep
learning–based methods- are very time consuming. One of the potential
solutions is to transfer a pre-trained model with a specific data domain to a
related but different domain without the need for re-learning or providing new
datasets. For instance, a model that learned to detect cars in a video
sequence can detect unseen trucks without re-learning procedure. This concept
is referred to “transfer learning” that introduced in [13].

In this work we use GAN that introduced in [14] to learn the normal data
distribution using the normal appearance-motion patches. In each step, the
generator outputs patches and gives them to a discriminator that uses a pre-
trained VGG16 network. The discriminator then guesses the originality of its
input patches. The final error of the discriminator is calculated based on the
accuracy of its detections and returned to both the generator and the
discriminator. Thus, the generator improves its fake generation accuracy and
the discriminator improves its ability to distinguish between real and fake
images.

In this work, the first six layers of a pre-trained VGG16 network are used to
obviate the need for training more than $10$ million learnable parameters and
gathering a large amount of various data. Fig. 5 shows the results of this
network after $16000$ iterations. At first, the generator generates random
noises. However, after $16000$ iterations the generated patches become very
similar to pedestrians in the real patches. This means that the generator
outputs are improving.

Figure 5: The generator outputs after $16000$ iterations.

##  IV Experimental results

The efficiency of the proposed method is evaluated with AUC and EER criteria.
The comparison between the proposed method and the state of the art shows that
the proposed method is very effective in detecting and locating abnormalities
in the videos.

Python programming language and Keras module are used to implement the GAN
subnetworks. The computer used in this experiment has NVIDIA GeForce $1050$
GPU, $32$ GB of RAM, and an Intel Core i$7$ CPU running at $3.1$ GHz. To
evaluate the model, roc_curve function of Python sklearn module is used to
calculate the area under the ROC curve. The generator has four convolutional
layers and the discriminator has two parts: The first part is a portion of
VGG16 from the input to the pooling layer of the fourth block. This part is
constant and unlearnable. The second part consists of two fully-connected
learnable layers which are concatenated to the first part.

###  IV-A The utilized datasets: UCSD Peds1 and UCSD Peds2

UCSD Peds dataset includes two subsets: UCSD Peds1 and UCSD Peds2. Both show a
crowded pedestrian zone where bikers, skaters, and carts are considered
abnormal entities. This dataset has 50 training and 48 testing video samples.
The difference between the two subsets is their frame size and the angle of
the camera. In UCSD Peds1, the camera is placed on a high altitude. UCSD Peds1
has perspective distortion and a resolution of 238*158 pixels. In UCSD Peds2
the camera is placed on a lower altitude. It has a resolution of 360*240
pixels and NO perspective distortion.

###  IV-B Model Evaluation Criteria

There are two types of evaluation for AED models and approaches: Frame-level
evaluation and pixel-level evaluation. In frame-level evaluation, the model
accuracy is calculated based on the abnormal frame detection without
localizing the abnormality. On the contrary, pixel-level evaluation localizes
abnormalities and checks the model output with the pixels of ground-truth
images. If a model can correctly detect at least 40% of abnormal pixels, its
detection is considered true.

The area under the ROC curve which is drawn based on true positive rate (TPR)
and false positive rate (FPR) is a common criterion to evaluate and compare
the performance and accuracy of different models. TPR is calculated as

$\text{TPR}=\frac{TP}{P}=\frac{TP}{TP+FN},$ (2)

where $TP$ is the number of frames that are correctly detected as abnormal
frames and $FN$ is the number of frames that are incorrectly considered as
normal frames. Equation (2) can also be used in the pixel-level mode but
instead of the number of frames, the number of pixels should be counted. FPR
is calculated as

$\text{FPR}=\frac{FP}{N}=\frac{FP}{FP+TN},$ (3)

where $FP$ is the number of frames that are incorrectly detected as abnormal
frames and $TN$ is the number of frames that are correctly considered as
normal frames.

Fig. 6 shows the Equal error rate (EER) which is a point in the ROC at the
intersection of the curve and a line that traverses from (0,1) and (1,0).
False negative rate (FNR) is the proportion of positives which yield negative
test outcomes that calculated as:

$\text{FNR}=\frac{FN}{P}=\frac{FN}{TP+FN},$ (4)

At EER, FPR and FNR are equal. A lower EER shows that the algorithm is more
accurate with less error. Another important criterion is time complexity. An
algorithm is more attractive to use in different applications if its overall
execution time is sufficiently short.

Figure 6: EER and ROC.

###  IV-C Results with UCSD Peds1 and UCSD Peds2

After training the model with the training data, the model is tested with
testing data and the system is evaluated with the above-mentioned criteria by
using the ground-truth of the dataset. Fig. 7 shows an instant of testing the
system when it is detecting a biker as an abnormality.

Figure 7: Detected abnormality with UCSD Peds1.

Figs. 8(a) and 9(a) show that compared to the state-of-the-art AED methods,
the proposed method is more effective in both frame-level and pixel-level
evaluations as it has a smaller EER and a larger AUC.222ROC curves of the
previous methods are derived from [15].

Figure 8: Frame-level ROC comparison of the proposed method with state of the
art. (a) when UCSD Peds1 is used and (b) when UCSD Peds2 is used. Figure 9:
Frame-level ROC comparison of the proposed method with state of the art. (a)
when UCSD Peds1 is used and (b) when UCSD Peds2 is used.

Fig. 9(a) shows that the overall accuracy and AUC of all methods in the pixel-
level evaluation are less than those in the frame-level evaluation. This is
because the pixel-level evaluation is a more stringent evaluation: In addition
to evaluating the ability of an AED method in detecting the abnormalities of
the frames, it also evaluates the ability of the AED method in locating the
abnormalities. This results in larger EER and smaller AUC values as compared
to frame-level evaluation. Table I lists the values of AUC and EER of each
method and further validates the effectiveness of the proposed method.

The main reason for the superiority of the proposed method is that is uses
both appearance and motion analyses. With UCSD Peds1, detection of motion
abnormalities is important: Skaters, carts, and bikers usually move faster
than pedestrians. Also with more challenging abnormalities such as low-speed
bikers, appearance analysis of the proposed method is useful in achieving
effective results. The proposed method also has remarkable results with UCSD
Peds2 dataset. Table II shows the superiority of the proposed method over the
state-of-the-art methods using EER comparison.

TABLE I: EER and AUC comparisons with UCSD Peds1. Method |  Frame-level |  Frame-level |  Pixel-level |  Pixel-level
---|---|---|---|---
EER | AUC | EER | AUC
MDT [7] | 25% | 81% | 58% | 44%
Social force [8] | 31% | 68% | 79% | 20%
AMDN [3] | 16% | 92% | 40% | 67%
Proposed | 14% | 93% | 36% | 73%
TABLE II: Frame-level and pixel-level EER comparisons with UCSD Peds2. Method |  Frame-level EER |  Pixel-level EER
---|---|---
MDT [7] | 24% | 54%
Social force [8] | 42% | 80%
AMDN [3] | 16% | 42%
Proposed | 15% | 17%

###  IV-D Time Analysis

For GAN to be sufficiently accurate, its learning process requires a large
number of repetitions. The time complexity for both training and testing
phases is one of the major problems in most deep neural networks including
GAN. However, we decrease the time complexity of the GAN by using transfer
learning. In this work, VGG16 network is used to extract the appearance
features. VGG16 is a well-known pre-trained network which is trained with
ImageNet dataset. ImageNet dataset contains more than 14 million images from
more than 21800 categories. VGG16 is trained by a large number of various
data. Thus, its feature extractor is general enough and we can use its first
layers as the feature extractor of our method. To show the effectiveness of
the proposed method, we noted the execution times of different parts of our
method. Major execution times that should be analyzed in the learning process
of the proposed method are pre-processing time, optical flow extraction time,
3D representation time, and classification time. Table III shows these
execution times for the proposed method.

TABLE III:  Execution time of each task for one frame. Pre- | Optical | 3D | Classification | Total
---|---|---|---|---
processing | flow | representation |  |
| extraction |  |  |
0.001 s | 0.020 s | 0.001 s | 0.290 s | 0.312 s

##  V Conclusion

In this paper, a new GAN-based method is proposed to address AED problem. This
method uses both appearance-motion and motion-only representations of the
input data. Therefore, it effectively detects various abnormalities in shape,
skeleton, speed, and direction. The proposed method transfers the knowledge of
a pre-trained CNN (VGG16) to its discriminator CNN to solve this unsupervised
problem. This knowledge transfer makes the training phase of the proposed
method highly efficient. Experimental case studies are carried out to compare
the proposed method with the state of the art. The experiments are based on
UCSD Peds1 and UCSD Peds2 datasets.

The results show the effectiveness of the proposed method as it has a lower
EER and higher AUC as compared to other methods. In addition to the
performance studies, time complexity study is also carried out which shows
that the proposed method is sufficiently time effective. With the utilized
personal computer, the proposed method can detect and locate abnormalities of
a frame in a short time. By using GPU oriented codes and with the advancements
of the computers, the proposed method will also be applicable for real-time
AED with higher video frame rates (e.g., 30+ fps).

## References

  * [1] R. Chalapathy and S. Chawla, “Deep learning for anomaly detection: A survey,” _arXiv preprint arXiv:1901.03407_ , 2019.
  * [2] M. Sabokrou, M. Fayyaz, M. Fathy, Z. Moayed, and R. Klette, “Deep-anomaly: Fully convolutional neural network for fast anomaly detection in crowded scenes,” _Computer Vision and Image Understanding_ , vol. 172, pp. 88–97, 2018.
  * [3] D. Xu, Y. Yan, E. Ricci, and N. Sebe, “Detecting anomalous events in videos by learning deep representations of appearance and motion,” _Computer Vision and Image Understanding_ , vol. 156, pp. 117–127, 2017.
  * [4] F. Jiang, J. Yuan, S. A. Tsaftaris, and A. K. Katsaggelos, “Anomalous video event detection using spatiotemporal context,” _Computer Vision and Image Understanding_ , vol. 115, no. 3, pp. 323–333, 2011.
  * [5] S. Wu, B. E. Moore, and M. Shah, “Chaotic invariants of lagrangian particle trajectories for anomaly detection in crowded scenes,” _IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ , pp. 2054–2060, 2010.
  * [6] V. Mahadevan, W. Li, V. Bhalodia, and N. Vasconcelos, “Anomaly detection in crowded scenes,” _IEEE Computer Society Conference on Computer Vision and Pattern Recognition_ , pp. 1975–1981, 2010.
  * [7] R. Mehran, A. Oyama, and M. Shah, “Abnormal crowd behavior detection using social force model,” _IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 935–942, 2009.
  * [8] Y. Cong, J. Yuan, and J. Liu, “Abnormal event detection in crowded scenes using sparse representation,” _Pattern Recognition_ , vol. 46, no. 7, pp. 1851–1864, 2013.
  * [9] V. Saligrama and Z. Chen, “Video anomaly detection based on local statistical aggregates,” _IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 2112–2119, 2012.
  * [10] M. Javan Roshtkhari and M. D. Levine, “Online dominant and anomalous behavior detection in videos,” _IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 2611–2618, 2013.
  * [11] Y. Feng, Y. Yuan, and X. Lu, “Learning deep event models for crowd anomaly detection,” _Neurocomputing_ , vol. 219, pp. 548–556, 2017.
  * [12] G. Farneback, “Two-frame motion estimation based on polynomial expansion,” _Scandinavian conference on Image analysis_ , pp. 363–370, 2003.
  * [13] L. Y. Pratt, “Discriminability-based transfer between neural networks,” pp. 204–211, 1993.
  * [14] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” _Advances in neural information processing systems_ , pp. 2672–2680, 2014.
  * [15] M. Ravanbakhsh, M. Nabi, E. Sangineto, L. Marcenaro, C. Regazzoni, and N. Sebe, “Abnormal event detection in videos using generative adversarial nets,” _2017 IEEE International Conference on Image Processing (ICIP)_ , pp. 1577–1581, 2017.


# Test-Time Amendment with a Coarse Classifier for Fine-Grained Classification

Kanishk Jain1, Shyamgopal Karthik2, Vineet Gandhi1
1IIIT Hyderabad 2University of Tübingen

###### Abstract

We investigate the problem of reducing mistake severity for fine-grained
classification. Fine-grained classification can be challenging, mainly due to
the requirement of domain expertise for accurate annotation. However, humans
are particularly adept at performing coarse classification as it requires
relatively low levels of expertise. To this end, we present a novel approach
for Post-Hoc Correction called Hierarchical Ensembles (HiE) that utilizes
label hierarchy to improve the performance of fine-grained classification at
test-time using the coarse-grained predictions. By only requiring the parents
of leaf nodes, our method significantly reduces _avg. mistake severity_ while
improving _top-1_ accuracy on the iNaturalist-19 and tieredImageNet-H
datasets, achieving a new state-of-the-art on both benchmarks. We also
investigate the efficacy of our approach in the semi-supervised setting. Our
approach brings notable gains in _top-1_ accuracy while significantly
decreasing the severity of mistakes as training data decreases for the fine-
grained classes. The simplicity and post-hoc nature of HiE renders it
practical to be used with any off-the-shelf trained model to improve its
predictions further. The code is available at:
https://github.com/kanji95/Hierarchical-Ensembles

##  1 Introduction

Over the past decade, large-scale datasets have been instrumental in driving
the rapid progress of computer vision/image recognition. However, in settings
that require experts to annotate samples, collecting large amounts of labeled
data can be prohibitively expensive. _Fine-Grained Visual Classification
(FGVC)_ is one such example, where one would need a domain expert to be able
to identify the category for a particular sample. However, it can be cheaper
to obtain coarse labels for the same samples in these settings. For instance,
in Figure 1 while it is evident to spot and identify a butterfly or a bird,
one would need a _Lepidopterist_ or an _Ornithologists_ to categorize them as
_Junonia Genoveva_ or _Water Ouzel_.

Therefore, one popular research direction in recent times has been to utilize
the availability of a larger set of images with coarse labels to improve the
performance (i.e _top-1_ accuracies) of the neural network models on various
fine-grained classification benchmarks Su et al. (2021); Su and Maji (2021).
Using coarse labels in a classification setting introduces the concept of a
label hierarchy, where all the labels in a dataset are connected through some
taxonomy. This taxonomy could either be defined from the biological taxonomy,
as with many species recognition datasets or could be derived from language-
based ontologies such as WordNet Miller (1995). Another mainstream usage of
label hierarchies in recent times is to use them to reduce the severity of
mistakes committed by various classification models Deng et al. (2010); Barz
and Denzler (2019); Bertinetto et al. (2020); Karthik et al. (2021); Garg et
al. (2022). For instance, mistaking a car for a bus is a better mistake than
mistaking a car for a lamppost. This originated from the extensive work on
cost-sensitive classification Domingos (1999); Zadrozny and Elkan (2001).
However, defining pairwise costs can be a tedious process and scales
quadratically with the number of classes. Therefore, the recent works have all
focused on using the label hierarchy and defining costs automatically based on
graph distances (e.g. using the height of the least common ancestor between
the predicted and the ground truth class as a proxy for cost). The research in
this direction aims to invent methods that at least retain the accuracies of
the backbone baseline models while reducing the severity of mistakes committed
by it.

Figure 1: While it is straightforward to identify the images based on their
coarse labels, it is challenging to distinguish them based on fine-grained
labels without domain expertise.

While these two research directions might seem entirely different, they work
with the same inputs (i.e., images, labels, and the label hierarchy).
Therefore, one would ideally want a method that can use additional coarsely
labelled samples to improve the model’s accuracy and utilise the label
hierarchy to reduce the severity of the mistakes. However, these two goals
have been tackled independently. Post-hoc correction methods Deng et al.
(2010); Karthik et al. (2021), learning hierarchy-aware features Bertinetto et
al. (2020); Garg et al. (2022); Chang et al. (2021) as well as structured
embedding spaces Barz and Denzler (2019); Frome et al. (2013) have been
proposed to reduce the severity of mistakes. Coarse labels have been used for
self-training/pseudo-labelling Phoo and Hariharan (2021) and with an
additional objective to regularise/enhance the training Su and Maji (2021) in
the semi-supervised learning frameworks.

In this work, we propose a simple method that is able to bring significant
improvements to both of these problems. Our fundamental insight is that models
trained for different granularities focus on different aspects. This is also
corroborated by recent work Chang et al. (2021); Garg et al. (2022), which
shows that when training a model jointly, coarse-level label prediction
exacerbates fine-grained feature learning. Therefore, we train two separate
models for the coarse and fine-grained labels. At inference, we employ a
modified decision rule, which computes the $\mathrm{argmax}$ over the
normalized score of coarse and fine grained predictions. We refer to the
proposed strategy as a Hierarchical Ensemble (HiE). We find that employing HiE
not only reduces the severity of mistakes, but is able to improve _top-1_
accuracies, which are far more pronounced in the semi-supervised setting. More
formally, we make the following contributions:

  1. 1.

We introduce a simple post-hoc mechanism called Hierarchical Ensembles for
incorporating cues from multiple models trained for different granularities of
a hierarchy tree.

  2. 2.

We demonstrate that HiE brings considerable reductions in mistake severity
while improving the _top-1 accuracy_. HiE significantly outperforms two
competent baselines and six other methods from the prior art and achieves
state-of-the-art performance on both iNaturalist-19 and tieredImageNet-H
benchmarks.

  3. 3.

We illustrate that HiE brings consistent gains when plugged with existing
semi-supervised methods exploiting additional coarse labeled data to improve
the fine-grained classification. On iNaturalist-19 dataset, HiE combined with
Moco He et al. (2020) and hierarchical loss Su and Maji (2021), recovers 86%
of the underlying performance while using only 0.5% of the fine-grained
annotations.

##  2 Related Work

Learning from Taxonomic Labels: There has been a long history of utilizing
label hierarchies for better classification performance. Sila and Freitas
Silla and Freitas (2011) perform a comprehensive survey of hierarchical
classification methods applied to various tasks in different application
domains. Broadly, the methods that train hierarchy-aware features can be
categorized into three categories a) _Label-Embedding methods_ , b)
_Hierarchical architectures_ , and c) _Hierarchical loss functions_.

The label-embedding methods project the categories into a semantic embedding
space Frome et al. (2013); Xian et al. (2016) instead of viewing them as one-
hot encodings and are widely used, especially in zero-shot learning Xian et
al. (2018). The semantic embedding space can be derived either from side-
information such as class attributes Xian et al. (2016), taxonomies Bertinetto
et al. (2020), or from natural language Frome et al. (2013). These can be
further enhanced by learning these embeddings on a hypersphere Barz and
Denzler (2019) or hyperbolic spaces Nickel and Kiela (2017); Khrulkov et al.
(2020); Liu et al. (2020).

On the other hand, hierarchical loss functions try to modify the training
objective to incorporate the label hierarchy Wu et al. (2016); Bertinetto et
al. (2020); Garnot and Landrieu (2021); Garg et al. (2022); Valmadre (2022).
The most common approach here is to introduce a loss at every level in the
hierarchy Wu et al. (2016); Bertinetto et al. (2020); Garg et al. (2022). The
challenge in this direction has been to ensure that the coarse-grained task
does not hamper the accuracy of the fine-grained classifier Chang et al.
(2021); Garg et al. (2022).

There have also been attempts to modify the architecture based on the label
hierarchy. For instance, popular ideas revolve around having branches at
different layers of the model Bilal et al. (2017), predicting the conditional
probabilities at each node in the hierarchy Redmon and Farhadi (2017); Ridnik
et al. (2021), or partitioning the feature space using the levels in the
hierarchies Chang et al. (2021).

Most closely related to our work are the works that attempt to incorporate the
label hierarchy information post-hoc during inference. Here, traditional
Conditional Risk Minimization (CRM) Duda et al. (1973) was applied in this
setting by Deng et al. (2010) and recently revisited by Karthik et al. (2021).
Our proposed approach is also applied during inference; however, it is
orthogonal to CRM, and we demonstrate that CRM can be used to improve our
results further. Concurrent to our work, Novack et al. (2023) proposed a
similar approach to utilize subclasses to improve superclass recognition
performance in a post-hoc manner with vision-language models Radford et al.
(2021).

Fine-Grained Visual Classification: Contrary to the popular image recognition
benchmarks, fine-grained visual classification focuses on recognizing the
differences between similar-looking categories Van Horn et al. (2018a); Wah et
al. (2011); Maji et al. (2013). Here, methods often focus on learning
discriminative features that use local information since most categories share
a similar global structure Angelova and Zhu (2013); Lin et al. (2015); Zhang
et al. (2014). However, the most related to our work are the ones that use
class taxonomies to improve fine-grained classification performance. This is
most frequently done in the semi-supervised setting Su et al. (2021), where
one has access to an additional set of weakly labeled samples. Here,
approaches have explored incorporating a hierarchical loss on the coarse
labels Su and Maji (2021) or using the coarse labels to filter pseudo-labels
Phoo and Hariharan (2021) in addition to the standard semi-supervised learning
techniques of consistency regularization Sohn et al. (2020), pseudo-labeling
Lee et al. (2013); Rizve et al. (2021), and contrastive learning He et al.
(2020). Our proposed approach is complementary to these works and can be
applied to all of these works to bring additional improvements.

##  3 Methodology

###  3.1 Problem Formulation

We consider the fine-grained classification task with a label hierarchy
$\mathcal{H}$ of $L$ levels defined over the class labels. The leaf nodes of
$\mathcal{H}$ correspond to fine-grained classes at level $L$, and the class
labels at level $l<L$ are considered coarse-grained labels. The goal is to use
the information from coarse annotations to improve the performance on fine
labels. Formally, given a dataset $\mathcal{X}=\\{(x_{i},{y}_{i}^{l})\mid
i=1,2,\dots,N\\}$ consisting of $N$ training images and their respective
ground truth labels at level $l$, where label
${y}_{i}^{l}\in\mathcal{Y}^{l}=\\{1,2,...,N_{l}\\}$, and $N_{l}$ corresponds
to the number of classes at level $l$, the task is to train a classifier
$f^{L}_{\theta}:\mathcal{X}\rightarrow p(\mathcal{Y}^{L})$, on fine-grained
class labels by making use of the information available through coarse-grained
labels $\mathcal{Y}^{l<L}$.

###  3.2 Proposed Method

Figure 2: Consider a four class hierarchy tree and corresponding fine-grained
predictions ($q_{i}$) and coarse level predictions ($r_{i}$) obtained using
independent models. Our work aims to improve the fine grained classification
by leveraging the predictions at the coarse level. In the given example, the
class prediction changes from ‘rose’ to ‘bus’ after the post-hoc correction.

Suppose we have two classifiers, $f_{\theta}^{L}$ and $f_{\phi}^{L-1}$ trained
on hierarchy levels $L,L-1$ with $\theta$ and $\phi$ being their parameters,
respectively, then for a sample $x\sim D$ , we have,
$\hat{y}_{L}=f_{\theta}^{L}(x)$ and $\hat{y}_{L-1}=f_{\phi}^{L-1}(x)$ as
logits of two classifiers for the input $x$. We further define:

$Q=[q_{1},q_{2},...,q_{N_{L}}]=\mathrm{softmax}(\hat{y}_{L})$
$R=[r_{1},r_{2},...,r_{N_{L-1}}]=\mathrm{softmax}(\hat{y}_{L-1})$

We reweigh the fine-grained predictions $\mathcal{Q}$, given the softmax
probabilities $\mathcal{R}$ obtained from the coarse-grained classifier:

$P(i|x,\theta,\phi,\mathcal{H})=\frac{P(i|x;\theta)P(i_{parent}|x;\phi)}{\sum_{j=1}^{N_{L}}P(j|x;\theta)P(j_{parent}|x;\phi)},\quad\text{where}~{}i_{parent}=\mathcal{H}(i)$
(1)

The denominator is the normalization term, ensuring
$\sum_{i=1}^{N_{L}}P(i|x,\mathcal{H})=1$. In the studied setting,
$P(i|x,\mathcal{H})$ simplifies to:

$P(i|x,\mathcal{H})=s_{i}=\frac{q_{i}\mbox{ .
}r_{i_{parent}}}{\sum_{j=1}^{N_{L}}q_{j}\mbox{ . }r_{j_{parent}}}$ (2)

The main modification we make in the classification strategy at the fine
grained level is in the decision rule, which now selects the class that
maximizes the normalized score:

$\underset{i}{\mathrm{argmax}}\mbox{ }P(i|x,\mathcal{H})$ (3)

In Figure 2, we illustrate a four-class example, comparing standard cross-
entropy predictions with those from our proposed method. We would like to
point out that the proposed technique can be interpreted as a Hierarchical
Ensemble. Existing literature in neural networks-based ensemble learning has
often focused on classifiers trained at a single granularity Wang et al.
(2020) and has largely overlooked ensembles with hierarchical context. In this
work, we highlight the practical benefits of hierarchical ensembles for
improving the fine-grained classification task by choosing the simplest
possible method of multiplying the predictions at different levels of
hierarchy.

We show that if we make a correct prediction at the coarse level, the proposed
Hierarchical Ensemble (HiE) is guaranteed to improve the downstream
predictions at the fine-grained classification task.

######  Theorem 3.1.

Assuming, $Q=[q_{1},q_{2},...,q_{N_{L}}]$ and
$R=[r_{1},r_{2},...,r_{N_{L-1}}]$ are the predictions obtained at the fine and
coarse grained labels for a given input $x$, such that
$\sum_{i=1}^{N_{L}}q_{i}=1$ and $\sum_{i=1}^{N_{L-1}}r_{i}=1$. For the ground
truth labels $g$ and $g_{parent}$ at the fine grained and the coarse grained
levels respectively:

Now assuming that the coarse label is correctly predicted by the coarse
prediction network i.e. $\mathrm{argmax}(R)=g_{parent}$, we wish to prove
that:

$\frac{q_{g}\mbox{ . }r_{g_{parent}}}{\sum_{j=1}^{N_{L}}q_{j}\mbox{ .
}r_{j_{parent}}}\geq q_{g}.$

###### Proof.

The denominator iterates over the fine grained predictions and multiplies them
with their parent’s prediction scores. This is equivalent to iterating over
the coarse label predictions and multiplying with the sum of prediction scores
for all its children. By rewriting the denominator, we obtain:

$\sum_{j=1}^{N_{L}}q_{j}\mbox{ .
}r_{j_{parent}}=\sum_{j=1}^{N_{L-1}}r_{j}\sum_{i\in j_{child}}q_{i}$

Assuming, $\sum_{i\in j_{child}}q_{i}=z_{j}$

$\sum_{j=1}^{N_{L-1}}r_{j}\sum_{i\in
j_{child}}q_{i}=\sum_{j=1}^{N_{L-1}}r_{j}\mbox{ . }z_{j}=R^{T}Z$

Invoking Holder’s inequality, using $a=\infty$ and $b=1$, ($1/a+1/b=1$), we
obtain:

$\begin{split}R^{T}Z&\leq{\|R\|}_{\infty}{\|Z\|}_{1}\\\ \end{split}$

Since, $\|R\|_{\infty}=\mathrm{max}(R)=r_{g_{parent}}$ and
$\|Z\|_{1}=\sum_{i=1}^{N_{L}}q_{i}=1$. We can say:

$R^{T}Z\leq{\|R\|}_{\infty}{\|Z\|}_{1}\\\ \leq r_{g_{parent}}$

Given the above equation, we can conclude that:

$\frac{r_{g_{parent}}}{\sum_{j=1}^{N_{L}}q_{j}\mbox{ . }r_{j_{parent}}}\geq
1.$

∎

When correct predictions are made at both the coarse and fine-grained levels,
the method will help increase the confidence of the prediction. A more
interesting case occurs when the coarse-level prediction is correct; however,
the fine-grained classifier makes a wrong prediction. The hierarchical
ensemble can shift the prediction to the correct sub-tree, reducing the
mistake severity and potentially improving the _top-1_ accuracy.

##  4 Experiments & Results

Like prior works Bertinetto et al. (2020); Garg et al. (2022); Karthik et al.
(2021), we evaluate our approach on the iNaturalist-19 Van Horn et al. (2018b)
and tieredImageNet-H Ren et al. (2018) datasets. The iNaturalist-19 dataset is
a species classification dataset containing plants, animals, fungi, etc., and
has an 8-level hierarchy. The tieredImageNet-H dataset contains classes from a
wide range of categories, including musical instruments, animal breeds, tools,
and plant species. It has a 13-level hierarchy derived from the WordNet
hierarchy. However, rather than using the full hierarchy of categories, for
our method, we only use the leaf classes and their parent categories during
training and inference. The iNaturalist-19 dataset includes 1010 fine-grained
“species” classes and 72 coarse-grained classes for the “genus” taxonomy. The
tieredImageNet-H dataset has 608 leaf classes and 201 parent classes. We train
classifiers for each level independently, using the same hyperparameter
settings.

In addition, we also evaluate the performance of our approach in a semi-
supervised setting. Specifically, we want to test the performance on fine-
grained classification tasks using a large number of coarsely annotated
examples and a limited number of finely annotated examples. We use the
iNaturalist-19 dataset for this setting; the coarse labels for all the images
are used, while for fine-grained labels, we sample a limited number of images
per class label during training. We used the “species” and “genus” taxonomies
for the fine and coarse level labels and performed experiments for cases when
the number of images per fine label is 100, 50, 25, and 10. Similar to the
supervised setting, we train separate classifiers on coarse and fine labels.

To align with prior efforts Karthik et al. (2021); Garg et al. (2022), all of
our experiments are performed using the ResNet-18 backbone. We randomly crop a
portion of images and resize them to $224\times 224$ resolution for both
datasets. For the iNaturalist-19 dataset, we initialize the ResNet-18 weights
with a pre-trained ImageNet model and train the classifiers using a customized
SGD optimizer for 100 epochs, with different learning rates for the backbone
network and the fully connected layer (0.01 and 0.1, respectively). For the
tieredImageNet-H dataset, we train the ResNet-18 from scratch (since it is
derived from the ImageNet dataset) and use the AMSGrad variant of the Adam
optimizer with a learning rate of $1e^{-5}$ for 120 epochs. We conduct five
runs for each experiment in tables 1 and 2 and report the mean and standard
deviations.

Evaluation Metrics:  Like previous works Bertinetto et al. (2020); Karthik et
al. (2021), we validate our approach using the _top-1_ accuracy, _avg. mistake
severity_ and _hierarchical distance@k_ metrics. _Avg. mistake severity_
metric computes the average height of the lowest common ancestor (LCA) between
the predicted class and the ground truth class for the incorrectly classified
samples. Since this metric considers only the incorrect predictions, it is
possible to reduce the mistake severity by making more low cost mistakes
Karthik et al. (2021), as a consequence of which the _top-1_ accuracy
decreases. To overcome this issue, _hierarchical distance@k_ is used, which
measures the average height of _top-k_ predictions with the ground truth class
for all samples. For $k>1$, this metric captures the quality of ranking
imposed by the network in the label space.

###  4.1 Making better mistakes

In this section, we demonstrate that the proposed hierarchical ensembles
significantly reduce the severity of mistakes compared to the current state-
of-the-art methods. We compare our method with Barz and Denzler (2019),
YOLO-v2 Redmon and Farhadi (2017); both approaches proposed by Bertinetto et
al. (2020) with varying parameters - soft-labels and HXE. We also compare
against the method proposed by Chang et al. (2021) for multi-granular
classification and HAF Garg et al. (2022) imposing consistency on prediction
heads across label hierarchy. We contrast the performance of the methods
mentioned above to the cross-entropy baseline.

Method |  Top-$1$ Error($\downarrow$) |  Mistakes severity($\downarrow$) |  Hier dist@1($\downarrow$) |  Hier dist@5($\downarrow$) |  Hier dist@20($\downarrow$)
---|---|---|---|---|---
Without CRM
Cross-Entropy |  36.44 $\pm$ 0.061 |  2.39 $\pm$ 0.007 |  0.87 $\pm$ 0.004 |  1.97 $\pm$ 0.002 |  3.25 $\pm$ 0.002
Cross-Entropy-H |  37.63 $\pm$ 0.054 |  2.37 $\pm$ 0.005 |  0.89 $\pm$ 0.009 |  1.86 $\pm$ 0.001 |  2.96 $\pm$ 0.001
HiE-Self |  36.36 $\pm$ 0.047 |  2.36 $\pm$ 0.014 |  0.86 $\pm$ 0.005 |  1.44 $\pm$ 0.003 |  2.31 $\pm$ 0.004
Barz & Denzler Barz and Denzler (2019) |  62.63 $\pm$ 0.278 | 1.99 $\pm$ 0.008 |  1.24 $\pm$ 0.005 | 1.49 $\pm$ 0.005 | 1.97 $\pm$ 0.005
YOLO-v2 Redmon and Farhadi (2017) |  44.37 $\pm$ 0.106 |  2.42 $\pm$ 0.003 |  1.08 $\pm$ 0.004 |  1.90 $\pm$ 0.003 |  2.87 $\pm$ 0.010
HXE $\alpha$=0.1 Bertinetto et al. (2020) |  41.48 $\pm$ 0.204 |  2.41 $\pm$ 0.009 |  1.00 $\pm$ 0.006 |  1.77 $\pm$ 0.011 |  2.69 $\pm$ 0.021
HXE $\alpha$=0.6 Bertinetto et al. (2020) |  45.45 $\pm$ 0.014 |  2.24 $\pm$ 0.006 |  1.02 $\pm$ 0.003 |  1.70 $\pm$ 0.005 |  2.55 $\pm$ 0.005
Soft-labels $\beta=30$ Bertinetto et al. (2020) |  41.67 $\pm$ 0.134 |  2.32 $\pm$ 0.010 |  0.97 $\pm$ 0.006 |  1.50 $\pm$ 0.006 |  2.23 $\pm$ 0.005
Soft-labels $\beta=4$ Bertinetto et al. (2020) |  74.70 $\pm$ 0.212 | 1.82 $\pm$ 0.005 |  1.36 $\pm$ 0.004 | 1.49 $\pm$ 0.003 | 1.96 $\pm$ 0.004
Chang et al. Chang et al. (2021) |  37.23 $\pm$ 0.175 |  2.28 $\pm$ 0.006 |  0.85 $\pm$ 0.004 |  1.75 $\pm$ 0.005 |  3.02 $\pm$ 0.008
HAF Garg et al. (2022) | 36.40 $\pm$ 0.092 |  2.28 $\pm$ 0.012 | 0.83 $\pm$ 0.002 |  1.62 $\pm$ 0.002 |  2.55 $\pm$ 0.003
HiE (Ours) | 35.33 $\pm$ 0.037 |  2.15 $\pm$ 0.003 | 0.76 $\pm$ 0.001 | 1.33 $\pm$ 0.001 |  2.19 $\pm$ 0.003
| With CRM
Cross-Entropy Karthik et al. (2021) |  36.51 $\pm$ 0.083 |  2.33 $\pm$ 0.001 |  0.85 $\pm$ 0.002 |  1.32 $\pm$ 0.001 |  1.86 $\pm$ 0.002
Cross-Entropy-H |  37.70 $\pm$ 0.038 |  2.33 $\pm$ 0.011 |  0.88 $\pm$ 0.003 |  1.33 $\pm$ 0.002 |  1.87 $\pm$ 0.004
HiE-Self | 36.45 $\pm$ 0.057 |  2.34 $\pm$ 0.019 |  0.85 $\pm$ 0.003 |  1.32 $\pm$ 0.003 |  1.86 $\pm$ 0.008
Barz & Denzler |  75.61 $\pm$ 0.098 |  4.62 $\pm$ 0.032 |  3.49 $\pm$ 0.025 |  3.66 $\pm$ 0.024 |  3.84 $\pm$ 0.018
YOLO-v2 |  45.17 $\pm$ 0.046 |  2.43 $\pm$ 0.001 |  1.10 $\pm$ 0.001 |  1.50 $\pm$ 0.001 |  1.99 $\pm$ 0.002
HXE $\alpha$=0.1 |  41.47 $\pm$ 0.220 |  2.38 $\pm$ 0.011 |  0.99 $\pm$ 0.008 |  1.41 $\pm$ 0.006 |  1.93 $\pm$ 0.005
HXE $\alpha$=0.6 |  45.60 $\pm$ 0.017 |  2.21 $\pm$ 0.008 |  1.01 $\pm$ 0.003 |  1.40 $\pm$ 0.004 | 1.40 $\pm$ 0.004
Soft-labels $\beta=30$ |  41.99 $\pm$ 0.126 |  2.31 $\pm$ 0.009 |  0.97 $\pm$ 0.007 |  1.40 $\pm$ 0.005 |  1.91 $\pm$ 0.005
Soft-labels $\beta=4$ |  77.34 $\pm$ 0.262 | 2.06 $\pm$ 0.012 |  1.60 $\pm$ 0.007 |  1.72 $\pm$ 0.008 |  2.14 $\pm$ 0.007
Chang et al. Chang et al. (2021) |  37.31 $\pm$ 0.145 |  2.24 $\pm$ 0.008 |  0.84 $\pm$ 0.002 |  1.30 $\pm$ 0.002 |  1.84 $\pm$ 0.002
HAF Garg et al. (2022) |  36.48 $\pm$ 0.095 |  2.25 $\pm$ 0.012 | 0.82 $\pm$ 0.003 | 1.29 $\pm$ 0.004 |  1.84 $\pm$ 0.002
HiE (Ours) | 35.42 $\pm$ 0.025 | 2.14 $\pm$ 0.015 | 0.75 $\pm$ 0.005 | 1.23 $\pm$ 0.005 | 1.79 $\pm$ 0.005
Table 1: Results comparing _top-1_ error(%) and hierarchical metrics for
iNaturalist-19. Results in the Top block are reported without using CRM
Karthik et al. (2021) technique and Bottom block are reported using CRM.
Methods highlighted with aqua green are the best performing methods in top-$1$
error (%). We use dark green for the top performers and light green for the
runners-up in each metric.

We also propose two custom baselines called Cross-Entropy-H and HiE-Self.
Cross-Entropy-H obtains coarse class predictions by marginalizing the
prediction of its fine-grained children classes. It then employs an additional
hierarchical loss term Su and Maji (2021) over the obtained coarse
predictions. HiE-Self applies a hierarchical ensemble using the coarse
predictions obtained post marginalization over fine-grained classes (instead
of training a separate network for the coarse level).

We also experiment with the CRM technique proposed by Karthik et al. (2021).
Since CRM is a post-hoc approach that reweighs the probability distribution of
samples obtained from any trained model, it can be applied to pre-trained
models of all the approaches. Therefore, in each of the Tables 1-2, we group
the results to report evaluation metrics with and without using CRM at test
time.

Method |  Top-$1$ error($\downarrow$) |  Mistakes severity($\downarrow$) |  Hier dist@1($\downarrow$) |  Hier dist@5($\downarrow$) |  Hier dist@20($\downarrow$)
---|---|---|---|---|---
Without CRM
Cross-Entropy |  30.64 $\pm$ 0.030 |  7.07 $\pm$ 0.010 |  2.17 $\pm$ 0.006 |  5.70 $\pm$ 0.003 |  7.25 $\pm$ 0.003
Cross-Entropy-H |  32.87 $\pm$ 0.042 |  7.13 $\pm$ 0.031 |  2.35 $\pm$ 0.003 |  5.70 $\pm$ 0.009 |  7.14 $\pm$ 0.012
HiE-Self |  30.78 $\pm$ 0.054 |  7.05 $\pm$ 0.028 |  2.19 $\pm$ 0.004 |  5.35 $\pm$ 0.009 |  6.92 $\pm$ 0.007
Barz & Denzler Barz and Denzler (2019) |  39.73 $\pm$ 0.240 |  6.80 $\pm$ 0.019 |  2.70 $\pm$ 0.022 |  5.48 $\pm$ 0.271 | 6.21 $\pm$ 0.005
YOLO-v2 Redmon and Farhadi (2017) |  33.37 $\pm$ 0.082 |  7.02 $\pm$ 0.004 |  2.34 $\pm$ 0.016 |  5.85 $\pm$ 0.011 |  7.43 $\pm$ 0.016
HXE $\alpha$=0.1 Bertinetto et al. (2020) |  30.72 $\pm$ 0.036 |  7.00 $\pm$ 0.019 |  2.15 $\pm$ 0.005 |  5.62 $\pm$ 0.008 |  7.08 $\pm$ 0.015
HXE $\alpha$=0.6 Bertinetto et al. (2020) |  34.50 $\pm$ 0.007 | 6.73 $\pm$ 0.014 |  2.32 $\pm$ 0.003 |  5.48 $\pm$ 0.001 | 6.78 $\pm$ 0.003
Soft-labels $\beta=30$ Bertinetto et al. (2020) |  30.53 $\pm$ 0.194 |  7.05 $\pm$ 0.009 |  2.15 $\pm$ 0.013 |  5.66 $\pm$ 0.002 |  7.14 $\pm$ 0.008
Soft-labels $\beta=4$ Bertinetto et al. (2020) |  38.99 $\pm$ 0.105 | 6.60 $\pm$ 0.024 |  2.57 $\pm$ 0.004 | 5.13 $\pm$ 0.002 | 6.21 $\pm$ 0.001
Chang et al. Chang et al. (2021) |  33.46 $\pm$ 0.026 |  6.99 $\pm$ 0.010 |  2.34 $\pm$ 0.006 |  5.75 $\pm$ 0.005 |  7.34 $\pm$ 0.010
HAF Garg et al. (2022) | 30.50 $\pm$ 0.010 |  7.03 $\pm$ 0.024 | 2.14 $\pm$ 0.008 |  5.62 $\pm$ 0.011 |  6.99 $\pm$ 0.009
HiE (Ours) | 29.81 $\pm$ 0.081 |  6.95 $\pm$ 0.013 | 2.07 $\pm$ 0.014 | 5.30 $\pm$ 0.001 |  6.86 $\pm$ 0.001
| With CRM
Cross-Entropy Karthik et al. (2021) | 30.56 $\pm$ 0.020 | 7.01 $\pm$ 0.007 | 2.14 $\pm$ 0.006 | 4.93 $\pm$ 0.002 | 6.11 $\pm$ 0.001
Cross-Entropy-H |  32.93 $\pm$ 0.029 |  7.05 $\pm$ 0.006 |  2.32 $\pm$ 0.004 |  5.01 $\pm$ 0.008 |  6.14 $\pm$ 0.003
HiE-Self |  30.79 $\pm$ 0.031 |  7.04 $\pm$ 0.008 |  2.17 $\pm$ 0.004 |  4.98 $\pm$ 0.005 |  6.14 $\pm$ 0.002
Barz & Denzler |  83.55 $\pm$ 0.000 |  11.94 $\pm$ 0.000 |  11.92 $\pm$ 0.000 |  11.91 $\pm$ 0.000 |  11.91 $\pm$ 0.000
YOLO-v2 |  33.98 $\pm$ 0.099 |  6.99 $\pm$ 0.011 |  2.38 $\pm$ 0.012 |  5.05 $\pm$ 0.001 |  6.17 $\pm$ 0.001
HXE $\alpha$=0.1 |  30.80 $\pm$ 0.079 |  6.95 $\pm$ 0.021 |  2.14 $\pm$ 0.005 | 4.94 $\pm$ 0.003 | 6.11 $\pm$ 0.002
HXE $\alpha$=0.6 |  34.68 $\pm$ 0.003 |  6.69 $\pm$ 0.007 |  2.32 $\pm$ 0.001 |  4.99 $\pm$ 0.005 | 6.13 $\pm$ 0.003
Soft-labels $\beta=30$ |  30.69 $\pm$ 0.125 |  6.99 $\pm$ 0.007 |  2.15 $\pm$ 0.008 |  4.95 $\pm$ 0.001 | 6.11 $\pm$ 0.001
Soft-labels $\beta=4$ |  82.72 $\pm$ 0.079 |  7.54 $\pm$ 0.001 |  6.24 $\pm$ 0.005 |  6.94 $\pm$ 0.005 |  7.25 $\pm$ 0.002
Chang et al. Chang et al. (2021) |  33.73 $\pm$ 0.033 | 6.93 $\pm$ 0.015 |  2.34 $\pm$ 0.002 |  5.02 $\pm$ 0.007 |  6.15 $\pm$ 0.001
HAF Garg et al. (2022) |  30.63 $\pm$ 0.007 |  6.97 $\pm$ 0.024 | 2.14 $\pm$ 0.008 |  4.95 $\pm$ 0.004 | 6.11 $\pm$ 0.001
HiE (Ours) | 29.89 $\pm$ 0.082 | 6.93 $\pm$ 0.013 | 2.07 $\pm$ 0.013 | 4.93 $\pm$ 0.001 | 6.11 $\pm$ 0.001
Table 2: Results comparing top-$1$ error(%) and hierarchical metrics on the
test set of tieredImageNet-H. The Top block reports results without using CRM
Karthik et al. (2021) and the Bottom block are reported using CRM. Methods
highlighted with aqua green are the best performing methods in top-$1$ error
(%). We use dark green for the top performers and light green for the runners-
up in each metric.

Apart from HAF, all the other methods from the prior art trade-off the _top-1_
accuracy with the mistake severity (Tables 1&2). The results contrast with the
problem’s main objective, i.e., improve the hierarchical metrics by
maintaining or improving the _top-1_ error. In some of the instances (e.g. HXE
$\alpha=0.6$, Soft-labels $\beta=30$), the _avg. mistake severity_ is reduced
by making additional low-severity mistakes (indicated by an increase in
_top-1_ error and _Hier dist@1_). The multi-head approach by Chang et al.
(2021) brings minor gains, and HAF further improves upon it by introducing
consistency loss among different head predictions.

The proposed minimalist approach of Hierarchical Ensembles (HiE) significantly
outperforms all the methods; on both datasets, with and without CRM. On the
three metrics of _top-1_ error, _avg. mistake severity_ , and _Hier dist@1_ ,
the performance achieved by HiE without CRM betters all other methods post-
CRM. The gains achieved by HiE on _Hier dist@5_ and _Hier dist@20_ metrics are
more pronounced, for instance, improving over HAF without CRM on
iNaturalist-19 dataset by 17% and 14% respectively. These two ranking metrics
compare the ordering of the classes provided by each of these classifiers. The
results illustrate that HiE is able to reliably align the predictions with the
hierarchy. The results are interesting given that HiE only employs the bottom
two levels of the hierarchy, in contrast to other methods which utilize the
entire hierarchy tree.

Compared with the baselines, on the iNaturalist-19 dataset, the Cross-
Entropy-H brings minor gains on _Hier dist@5_ and _Hier dist@20_ metrics;
however, the _top-1_ error increases by over a percent. On tieredImageNet-H,
Cross-Entropy-H fails to give any improvements. In contrast, the HiE-Self
baseline is able to retain the original _top-1_ performance while bringing
noticeable gains on _Hier dist@k_ metrics. It illustrates that hierarchical
ensembles are helpful even when applied on a single network. However, training
separate networks for coarse and fine-grained levels allows us to learn
complementary features which are then combined using our HiE approach.

###  4.2 Semi Supervised Learning

We additionally validate our approach on fine-grained classification in the
semi-supervised setting. We assume that coarse annotations are available for
all the samples; however, fine-grained annotations are available only for a
few samples in each class. The studied setting differs from cases Phoo and
Hariharan (2021), which divide the fine-grained classes into two categories:
base classes with abundant annotated samples and novel classes where only a
few annotated samples are available. This framework Phoo and Hariharan (2021)
also limits the evaluation only to the novel classes. Our setting imposes firm
limits on difficult-to-obtain fine-grained labels. In the extreme setting, we
assume only ten annotated training samples are available for each fine-grained
class. We also evaluate the performance across all the fine-grained classes
together.

Methods | #Imgs/Label | without HiE | with HiE
---|---|---|---
Top-1 Acc | Avg. Mistake Severity | Hierarchical Distance@1 | Top-1 Acc | Avg. Mistake Severity | Hierarchical Distance@1
w/o CRM | w CRM | w/o CRM | w CRM | w/o CRM | w CRM | w/o CRM | w CRM | w/o CRM | w CRM | w/o CRM | w CRM
CrossEntropy | 100 | 53.90 | 53.93 | 2.55 | 2.48 | 1.17 | 1.14 | 56.95 | 56.82 | 2.07 | 2.06 | 0.89 | 0.89
Pseudo-Label | 51.25 | 51.09 | 2.52 | 2.46 | 1.23 | 1.20 | 54.78 | 54.61 | 2.04 | 2.03 | 0.92 | 0.92
MoCo | 54.72 | 54.51 | 2.41 | 2.36 | 1.09 | 1.07 | 57.07 | 56.91 | 2.05 | 2.04 | 0.88 | 0.88
MocoST | 55.77 | 55.59 | 2.42 | 2.37 | 1.07 | 1.05 | 57.99 | 57.78 | 2.07 | 2.06 | 0.87 | 0.87
CrossEntropy-H | 55.76 | 55.71 | 2.48 | 2.42 | 1.10 | 1.07 | 58.50 | 58.32 | 2.08 | 2.07 | 0.86 | 0.86
PseudoLabel-H | 56.18 | 55.98 | 2.47 | 2.41 | 1.08 | 1.06 | 58.80 | 58.79 | 2.09 | 2.08 | 0.86 | 0.86
ST-H | 54.82 | 54.71 | 2.47 | 2.41 | 1.12 | 1.09 | 57.51 | 57.45 | 2.07 | 2.07 | 0.88 | 0.88
Moco-H | 56.05 | 55.85 | 2.47 | 2.42 | 1.09 | 1.07 | 58.56 | 58.49 | 2.09 | 2.09 | 0.87 | 0.87
MocoST-H | 55.74 | 55.49 | 2.47 | 2.41 | 1.10 | 1.07 | 58.32 | 58.17 | 2.08 | 2.08 | 0.87 | 0.87
CrossEntropy | 50 | 45.47 | 45.60 | 2.70 | 2.61 | 1.47 | 1.42 | 50.39 | 50.29 | 2.00 | 1.99 | 0.99 | 0.99
Pseudo-Label | 45.97 | 45.84 | 2.57 | 2.51 | 1.39 | 1.36 | 50.59 | 50.43 | 1.99 | 1.98 | 0.98 | 0.98
MoCo | 47.94 | 47.75 | 2.55 | 2.50 | 1.33 | 1.31 | 51.72 | 51.61 | 2.01 | 2.00 | 0.97 | 0.97
MocoST | 49.31 | 49.20 | 2.54 | 2.48 | 1.29 | 1.26 | 52.87 | 52.73 | 2.02 | 2.00 | 0.95 | 0.95
CrossEntropy-H | 54.87 | 54.83 | 2.50 | 2.43 | 1.13 | 1.10 | 57.69 | 57.50 | 2.08 | 2.07 | 0.88 | 0.88
PseudoLabel-H | 55.17 | 54.97 | 2.48 | 2.40 | 1.11 | 1.08 | 57.97 | 57.84 | 2.07 | 2.07 | 0.87 | 0.87
ST-H | 52.42 | 52.27 | 2.50 | 2.43 | 1.19 | 1.16 | 55.74 | 55.65 | 2.04 | 2.05 | 0.90 | 0.91
Moco-H | 54.76 | 54.58 | 2.48 | 2.42 | 1.12 | 1.10 | 57.65 | 57.45 | 2.07 | 2.08 | 0.88 | 0.88
MocoST-H | 54.53 | 54.26 | 2.48 | 2.42 | 1.13 | 1.11 | 57.37 | 57.21 | 2.07 | 2.08 | 0.88 | 0.89
CrossEntropy | 25 | 36.47 | 36.63 | 2.87 | 2.77 | 1.82 | 1.76 | 43.28 | 43.22 | 1.93 | 1.91 | 1.10 | 1.09
Pseudo-Label | 39.13 | 39.20 | 2.72 | 2.66 | 1.66 | 1.61 | 44.96 | 44.91 | 1.95 | 1.93 | 1.07 | 1.06
MoCo | 39.48 | 39.55 | 2.74 | 2.68 | 1.66 | 1.62 | 45.06 | 45.00 | 1.94 | 1.92 | 1.06 | 1.06
MocoST | 41.22 | 41.12 | 2.72 | 2.63 | 1.60 | 1.55 | 46.37 | 46.30 | 1.96 | 1.95 | 1.05 | 1.05
CrossEntropy-H | 53.26 | 53.18 | 2.51 | 2.44 | 1.17 | 1.14 | 56.43 | 56.21 | 2.06 | 2.05 | 0.90 | 0.90
PseudoLabel-H | 53.30 | 53.22 | 2.51 | 2.43 | 1.17 | 1.14 | 56.57 | 56.44 | 2.06 | 2.05 | 0.89 | 0.89
ST-H | 48.29 | 48.19 | 2.59 | 2.52 | 1.34 | 1.30 | 52.69 | 52.52 | 2.01 | 2.01 | 0.95 | 0.96
Moco-H | 53.01 | 52.97 | 2.50 | 2.43 | 1.17 | 1.14 | 56.29 | 56.25 | 2.06 | 2.05 | 0.90 | 0.90
MocoST-H | 53.21 | 53.02 | 2.49 | 2.44 | 1.17 | 1.14 | 56.34 | 56.12 | 2.05 | 2.06 | 0.90 | 0.90
CrossEntropy | 10 | 24.79 | 25.17 | 3.12 | 2.98 | 2.34 | 2.23 | 33.33 | 33.32 | 1.82 | 1.81 | 1.22 | 1.20
Pseudo-Label | 27.36 | 27.67 | 3.01 | 2.91 | 2.19 | 2.11 | 35.08 | 35.12 | 1.87 | 1.86 | 1.22 | 1.21
MoCo | 27.11 | 26.90 | 3.01 | 2.95 | 2.19 | 2.16 | 34.95 | 34.93 | 1.83 | 1.82 | 1.19 | 1.18
MocoST | 28.59 | 28.56 | 2.95 | 2.88 | 2.11 | 2.05 | 36.03 | 36.00 | 1.84 | 1.82 | 1.17 | 1.16
CrossEntropy-H | 51.36 | 51.31 | 2.54 | 2.47 | 1.23 | 1.20 | 54.92 | 54.80 | 2.05 | 2.04 | 0.92 | 0.92
PseudoLabel-H | 51.35 | 51.28 | 2.56 | 2.49 | 1.24 | 1.21 | 55.02 | 55.02 | 2.04 | 2.04 | 0.92 | 0.92
ST-H | 40.61 | 40.65 | 2.77 | 2.66 | 1.64 | 1.58 | 46.67 | 46.60 | 1.96 | 1.94 | 1.04 | 1.03
Moco-H | 51.23 | 51.27 | 2.57 | 2.50 | 1.25 | 1.22 | 55.13 | 55.03 | 2.06 | 2.06 | 0.92 | 0.93
MocoST-H | 50.33 | 50.26 | 2.56 | 2.50 | 1.27 | 1.24 | 54.11 | 54.04 | 2.04 | 2.05 | 0.94 | 0.94

Table 3: Results of the proposed HiE approach on the semi-supervised setting
on the iNaturalist-19 dataset. Best results on each metric are emphasized in
bold.

We train a cross-entropy baseline exclusively using the annotated fine-grained
samples. We compare its performance with representative semi-supervised
methods, including Pseudo-Label Lee et al. (2013), Self-Training (ST) with
distillation Hinton et al. (2015), Self-Supervised learning (MoCo) with
distillation He et al. (2020) and a combination of Self-Supervised learning
and Self Training (MoCo+ST). We also include a variation of all these methods
(Pseudo-Label-H, MoCo-H, ST-H, MoCoST-H) with hierarchical supervised loss as
described in Su and Maji (2021) with “genus” supervision. The hierarchical
supervised loss is applied to the coarse predictions obtained by marginalizing
the fine-grained class predictions. The proposed hierarchical ensemble is a
complementary approach, and we compare the performance of the above methods
with and without applying the HiE.

The results are illustrated in Table 3. As expected, the performance of the
cross-entropy baseline steeply drops with the reduction in training data. The
performance of semi-supervised methods (Pseudo-Label, ST, MoCo, MoCo-ST)
improves over the cross-entropy baseline; however, they fail to retain the
overall performance. The semi-supervised methods become more effective with
the reduced number of labeled examples (providing better proportional gains
over the cross-entropy baseline). Incorporating hierarchical supervised loss
brings remarkable gains in _top-1_ accuracy across all experiments. For
instance, in the lowest data regime (using only ten fine-grained annotated
samples for training), hierarchical supervised loss brings twofold improvement
over the cross-entropy baseline. The performance gains showcase the efficacy
of utilizing coarse taxonomic labels in the studied setting.

Employing the proposed Hierarchical Ensemble brings consistent gains across
all the methods on the three studied evaluation metrics. The combined approach
of Moco-H/PseudoLabel-H with HiE achieves a new state of the art in all
settings. Using ten annotated examples for each fine-grained class, Moco-H
with HiE achieves a _top-1_ accuracy of 55.13, which is higher than the
performance of cross entropy baselines using a hundred annotated samples per
fine-grained class. The performance gains of HiE are more evident on the _avg.
mistake severity_ and _Hier dist_ @1 metrics, bringing profound gains over
just using the hierarchical supervised loss. For instance, in 100 sample
experiments, the _Hier dist_ @1 drops from 1.17 to 1.10 (6% improvement) by
applying hierarchical supervised loss. In contrast, applying HiE brings 24%
improvement, reducing _Hier dist_ @1 to 0.89.

Finally, we observe that in Table 1, a cross-entropy baseline with Resnet-18
backbone trained on the entirety of the iNaturalist-19 dataset, i.e., 187385
examples with fine-grained labels achieves a _top-1_ accuracy of 63.56%. In
the semi-supervised setting using coarse-grained labels for entire data and
just using 10 examples with fine-grained annotations for each class (10100
training samples), Moco-H with HiE is able to recover the significant portion
of the underlying performance (achieving _top-1_ accuracy of 55.13), reducing
the requirement of expert annotations to only 0.5% of the total number of
samples.

###  4.3 Performance across different architectures

We also showcase the generalizability of the proposed HiE approach across
different vision architectures in Table 4. We utilize commonly used vision
architectures like MobileNet V3 Howard et al. (2017), Resnet variants He et
al. (2016), EfficientNet Tan and Le (2019), DenseNet121 Huang et al. (2017),
and tiny variants of DeiT Touvron et al. (2021), ViT Dosovitskiy et al. (2021)
and SwinT Liu et al. (2021). All networks are trained with the same hyper-
parameter setting. We compare the proposed HiE against the vanilla and CRM-
based post-hoc approach. HiE outperforms other approaches by significant
margins across different architectures and on all metrics. Furthermore,
similar performance gains are observed irrespective of model capacity, avg.
gain of 0.9% in top-1 accuracy across resnet variants.

Model | Vanilla | CRM | HiE
---|---|---|---
Top 1 Acc. | Mistake Severity | Hier. Dist@1 | Top 1 Acc. | Mistake Severity | Hier. Dist@1 | Top 1 Acc. | Mistake Severity | Hier. Dist@1
Mobilenet_V3 | 45.86 | 2.89 | 1.56 | 45.83 | 2.79 | 1.51 | 46.66 | 2.69 | 1.44
ResNet18 | 63.63 | 2.39 | 0.87 | 63.65 | 2.31 | 0.84 | 64.65 | 2.15 | 0.76
ResNet50 | 69.48 | 2.23 | 0.68 | 69.49 | 2.19 | 0.67 | 70.34 | 2.03 | 0.60
ResNet101 | 70.86 | 2.14 | 0.62 | 70.87 | 2.11 | 0.61 | 71.66 | 1.96 | 0.56
EfficientNet-B0 | 67.69 | 2.25 | 0.73 | 67.68 | 2.21 | 0.71 | 68.75 | 2.01 | 0.63
DenseNet121 | 67.86 | 2.27 | 0.73 | 67.89 | 2.23 | 0.72 | 68.97 | 2.04 | 0.63
DeiT | 64.71 | 2.38 | 0.84 | 64.74 | 2.32 | 0.82 | 65.73 | 2.14 | 0.73
ViT | 66.19 | 2.36 | 0.80 | 66.27 | 2.29 | 0.77 | 67.56 | 2.06 | 0.67
SwinT | 75.08 | 2.10 | 0.52 | 75.09 | 2.06 | 0.51 | 75.74 | 1.88 | 0.46

Table 4: Results of the proposed HiE approach with different architectures on
species taxonomy of the iNaturalist-19 dataset.

Hier. Levels | Top-1 Acc. | Mistake Severity | Hier. Dist @ 1 | Hier. Dist @ 5 | Hier. Dist @ 20
---|---|---|---|---|---
{7} | 63.63 | 2.39 | 0.87 | 1.97 | 3.25
{6, 7} | 64.65 | 2.15 | 0.76 | 1.33 | 2.19
{5 - 7} | 64.72 | 2.09 | 0.74 | 1.26 | 2.00
{4 - 7} | 64.48 | 2.12 | 0.75 | 1.26 | 1.91
{3 - 7} | 64.47 | 2.12 | 0.75 | 1.26 | 1.89
{2 - 7} | 64.46 | 2.13 | 0.76 | 1.27 | 1.89
{1 - 7} | 64.44 | 2.14 | 0.76 | 1.27 | 1.89

Table 5: Results of the proposed HiE approach on fine-grained level (species)
of iNaturalist-19 dataset when incorporating multiple hierarchical levels.
Best results are highlighted in bold.

###  4.4 Incorporating multiple hierarchical levels

For all the above experiments, we only use two levels of hierarchies. As an
additional ablation study, we validate the effectiveness of HiE in the
presence of multiple hierarchical levels in Table 5. Specifically, we train a
separate classifier for each hierarchical level and cascade the predictions
top-down. We observe that incorporating the hierarchical information from two
levels (levels 5, 6) above the leaf level gives the best performance; the
gains taper off gradually as we incorporate information from higher levels.
The reason for this dip in performance is that the higher levels do not have
much additional information as they cover too many classes (e.g., on
iNaturalist: kingdom (level 1), phylum (level 2), and class (level 3) have 3,
4 and 9 classes respectively) which does not help much in improving the fine-
grained predictions further.

##  5 Limitation

The main limitation of HiE is the requirement of training an additional
network for the coarse level. However, existing approaches employ a full
hierarchy with a separate classification head for each hierarchical level and
additional losses to enforce the hierarchical structure, making their training
process complex and difficult to converge. Instead, our approach trained with
vanilla cross-entropy loss works with partial hierarchy and trade-offs the
extra compute for adaptability, reproducibility and simplicity in training. A
future avenue for exploration would be to learn disentangled coarse and fine-
grained features imitating HiE in a unified architecture. Another limitation
is the assumption regarding the availability of underlying label hierarchy. To
overcome this limitation, we can exploit large-language models Brown et al.
(2020) to obtain label hierarchy using raw class labels.

##  6 Conclusion

We proposed using Hierarchical Ensembles (HiE) of independently trained
networks over coarse and fine-grained levels of the label hierarchy. In terms
of mistake severity, our proposed post-hoc correction consistently outperforms
state-of-the-art methods in deep hierarchy-aware image classification by large
margins in terms of decrease in _avg. mistake severity_ and _hierarchical
distance@k_ , while simultaneously improving the _top-1_ accuracy. In the
semi-supervised paradigm, we show that HiE delivers consistent performance
gains when used in conjunction with off-the-shelf semi-supervised learning
algorithms. We show that, comparable performance to a fully supervised
baseline can be attained, even using merely 10 annotations for each fine-
grained class on a large fine-grained image classification dataset
encompassing 1010 classes.

## References

  * Angelova and Zhu (2013) Anelia Angelova and Shenghuo Zhu.  Efficient object detection and segmentation for fine-grained recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 811–818, 2013.
  * Barz and Denzler (2019) Björn Barz and Joachim Denzler.  Hierarchy-based image embeddings for semantic image retrieval.  In _WACV_ , 2019.
  * Bertinetto et al. (2020) Luca Bertinetto, Romain Mueller, Konstantinos Tertikas, Sina Samangooei, and Nicholas A Lord.  Making better mistakes: Leveraging class hierarchies with deep networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12506–12515, 2020.
  * Bilal et al. (2017) Alsallakh Bilal, Amin Jourabloo, Mao Ye, Xiaoming Liu, and Liu Ren.  Do convolutional neural networks learn class hierarchy?  _IEEE transactions on visualization and computer graphics_ , 24(1):152–162, 2017.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _NeurIPS_ , 2020.
  * Chang et al. (2021) Dongliang Chang, Kaiyue Pang, Yixiao Zheng, Zhanyu Ma, Yi-Zhe Song, and Jun Guo.  Your" flamingo" is my" bird": Fine-grained, or not.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11476–11485, 2021.
  * Deng et al. (2010) Jia Deng, Alexander C Berg, Kai Li, and Li Fei-Fei.  What does classifying more than 10,000 image categories tell us?  In _European conference on computer vision_ , pages 71–84. Springer, 2010.
  * Domingos (1999) Pedro Domingos.  Metacost: A general method for making classifiers cost-sensitive.  In _KDD_ , 1999.
  * Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  _ICLR_ , 2021.
  * Duda et al. (1973) Richard O Duda, Peter E Hart, et al.  _Pattern classification and scene analysis_ , volume 3.  Wiley New York, 1973.
  * Frome et al. (2013) Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc’Aurelio Ranzato, and Tomas Mikolov.  Devise: A deep visual-semantic embedding model.  _Advances in neural information processing systems_ , 26, 2013.
  * Garg et al. (2022) Ashima Garg, Depanshu Sani, and Saket Anand.  Learning hierarchy aware features for reducing mistake severity.  In _European Conference on Computer Vision_ , pages 252–267. Springer, 2022.
  * Garnot and Landrieu (2021) Vivien Sainte Fare Garnot and Loic Landrieu.  Leveraging class hierarchies with metric-guided prototype learning.  In _BMVC_ , 2021.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * He et al. (2020) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9729–9738, 2020.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_ , 2(7), 2015.
  * Howard et al. (2017) Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  _arXiv preprint arXiv:1704.04861_ , 2017.
  * Huang et al. (2017) Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4700–4708, 2017.
  * Karthik et al. (2021) Shyamgopal Karthik, Ameya Prabhu, Puneet K Dokania, and Vineet Gandhi.  No cost likelihood manipulation at test time for making better mistakes in deep networks.  _ICLR_ , 2021.
  * Khrulkov et al. (2020) Valentin Khrulkov, Leyla Mirvakhabova, Evgeniya Ustinova, Ivan Oseledets, and Victor Lempitsky.  Hyperbolic image embeddings.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6418–6428, 2020.
  * Lee et al. (2013) Dong-Hyun Lee et al.  Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks.  In _Workshop on challenges in representation learning, ICML_ , 2013\.
  * Lin et al. (2015) Di Lin, Xiaoyong Shen, Cewu Lu, and Jiaya Jia.  Deep lac: Deep localization, alignment and classification for fine-grained recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1666–1674, 2015.
  * Liu et al. (2020) Shaoteng Liu, Jingjing Chen, Liangming Pan, Chong-Wah Ngo, Tat-Seng Chua, and Yu-Gang Jiang.  Hyperbolic visual embedding learning for zero-shot recognition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9273–9281, 2020.
  * Liu et al. (2021) Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021.
  * Maji et al. (2013) Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi.  Fine-grained visual classification of aircraft.  _arXiv preprint arXiv:1306.5151_ , 2013.
  * Miller (1995) George A Miller.  Wordnet: a lexical database for english.  _Communications of the ACM_ , 38(11):39–41, 1995\.
  * Nickel and Kiela (2017) Maximillian Nickel and Douwe Kiela.  Poincaré embeddings for learning hierarchical representations.  In _Advances in neural information processing systems_ , pages 6338–6347, 2017.
  * Novack et al. (2023) Zachary Novack, Julian McAuley, Zachary Chase Lipton, and Saurabh Garg.  Chils: Zero-shot image classification with hierarchical label sets.  In _ICML_ , 2023.
  * Phoo and Hariharan (2021) Cheng Perng Phoo and Bharath Hariharan.  Coarsely-labeled data for better few-shot transfer.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 9052–9061, 2021.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  pages 8748–8763, 2021.
  * Redmon and Farhadi (2017) Joseph Redmon and Ali Farhadi.  Yolo9000: better, faster, stronger.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 7263–7271, 2017.
  * Ren et al. (2018) Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenbaum, Hugo Larochelle, and Richard S Zemel.  Meta-learning for semi-supervised few-shot classification.  _arXiv preprint arXiv:1803.00676_ , 2018.
  * Ridnik et al. (2021) Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor.  Imagenet-21k pretraining for the masses.  _arXiv preprint arXiv:2104.10972_ , 2021.
  * Rizve et al. (2021) Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah.  In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning.  _arXiv preprint arXiv:2101.06329_ , 2021.
  * Silla and Freitas (2011) Carlos N Silla and Alex A Freitas.  A survey of hierarchical classification across different application domains.  _Data Mining and Knowledge Discovery_ , 22(1):31–72, 2011.
  * Sohn et al. (2020) Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  _Advances in neural information processing systems_ , 33:596–608, 2020.
  * Su and Maji (2021) Jong-Chyi Su and Subhransu Maji.  Semi-supervised learning with taxonomic labels.  _arXiv preprint arXiv:2111.11595_ , 2021.
  * Su et al. (2021) Jong-Chyi Su, Zezhou Cheng, and Subhransu Maji.  A realistic evaluation of semi-supervised learning for fine-grained classification.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12966–12975, 2021.
  * Tan and Le (2019) Mingxing Tan and Quoc Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In _International conference on machine learning_ , pages 6105–6114. PMLR, 2019.
  * Touvron et al. (2021) Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In _International conference on machine learning_ , pages 10347–10357. PMLR, 2021.
  * Valmadre (2022) Jack Valmadre.  Hierarchical classification at multiple operating points.  In _NeurIPS_ , 2022.
  * Van Horn et al. (2018a) Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie.  The inaturalist species classification and detection dataset.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 8769–8778, 2018a.
  * Van Horn et al. (2018b) Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie.  The inaturalist species classification and detection dataset.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 8769–8778, 2018b.
  * Wah et al. (2011) Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie.  The caltech-ucsd birds-200-2011 dataset.  2011\.
  * Wang et al. (2020) Xiaofang Wang, Dan Kondratyuk, Eric Christiansen, Kris M Kitani, Yair Alon, and Elad Eban.  Wisdom of committees: An overlooked approach to faster and more accurate models.  _arXiv preprint arXiv:2012.01988_ , 2020.
  * Wu et al. (2016) Hui Wu, Michele Merler, Rosario Uceda-Sosa, and John R Smith.  Learning to make better mistakes: Semantics-aware visual food recognition.  In _Proceedings of the 24th ACM international conference on Multimedia_ , pages 172–176, 2016.
  * Xian et al. (2016) Yongqin Xian, Zeynep Akata, Gaurav Sharma, Quynh Nguyen, Matthias Hein, and Bernt Schiele.  Latent embeddings for zero-shot classification.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 69–77, 2016.
  * Xian et al. (2018) Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata.  Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly.  _IEEE transactions on pattern analysis and machine intelligence_ , 41(9):2251–2265, 2018.
  * Zadrozny and Elkan (2001) Bianca Zadrozny and Charles Elkan.  Learning and making decisions when costs and probabilities are both unknown.  In _KDD_ , 2001.
  * Zhang et al. (2014) Ning Zhang, Jeff Donahue, Ross Girshick, and Trevor Darrell.  Part-based r-cnns for fine-grained category detection.  In _European conference on computer vision_ , pages 834–849. Springer, 2014.