[1]Licheng Jiao

[1]School of Artificial Intelligence, Xidian University, Xi’an, 710071, China

2]International Research Center for Intelligent Perception and Computation,
Xi’an 710071, China

3]Key Laboratory of Intelligent Perception and Image Understanding of the
Ministry of Education

# Geometric Prior Guided Feature Representation Learning
for Long-Tailed Classification

Yanbiao Ma<EMAIL_ADDRESS><EMAIL_ADDRESS>Fang Liu
<EMAIL_ADDRESS>Shuyuan Yang<EMAIL_ADDRESS>Xu Liu<EMAIL_ADDRESS>Puhua Chen<EMAIL_ADDRESS>*  [  [

###### Abstract

Real-world data are long-tailed, the lack of tail samples leads to a
significant limitation in the generalization ability of the model. Although
numerous approaches of class re-balancing perform well for moderate class
imbalance problems, additional knowledge needs to be introduced to help the
tail class recover the underlying true distribution when the observed
distribution from a few tail samples does not represent its true distribution
properly, thus allowing the model to learn valuable information outside the
observed domain. In this work, we propose to leverage the geometric
information of the feature distribution of the well-represented head class to
guide the model to learn the underlying distribution of the tail class.
Specifically, we first systematically define the geometry of the feature
distribution and the similarity measures between the geometries, and discover
four phenomena regarding the relationship between the geometries of different
feature distributions. Then, based on four phenomena, feature uncertainty
representation is proposed to perturb the tail features by utilizing the
geometry of the head class feature distribution. It aims to make the perturbed
features cover the underlying distribution of the tail class as much as
possible, thus improving the model’s generalization performance in the test
domain. Finally, we design a three-stage training scheme enabling feature
uncertainty modeling to be successfully applied. Experiments on
CIFAR-10/100-LT, ImageNet-LT, and iNaturalist2018 show that our proposed
approach outperforms other similar methods on most metrics. In addition, the
experimental phenomena we discovered are able to provide new perspectives and
theoretical foundations for subsequent studies. The code will be available at
https://github.com/mayanbiao1234/Geometric-metrics-for-perceptual-manifolds-
in-deep-neural-networks/

###### keywords:

Long-Tailed Classification, Representational learning, Geometric prior
knowledge

##  1 Introduction

Deep learning has made significant progress in image classification, image
segmentation, and other fields benefiting from artificially annotated large-
scale datasets. However, real-world data tends to follow a long-tailed
distribution [38], and unbalanced classes introduce bias into machine learning
models. Numerous approaches have been proposed to mitigate the model bias,
such as class re-balancing [6, 13, 56, 24, 31, 32], information augmentation
[4, 21, 50, 55, 19] and network structure design [21, 51, 56]. However, the
above approach does not work effectively in all cases, and the generalization
ability of the model will be greatly limited when the samples of the tail
class do not accurately represent its true distribution. We discuss two cases
of the relationship between the observed and true distributions of the tail
classes [5].

Figure 1: (a) When the samples uniformly cover the true data distribution, the
model can learn the correct decision boundaries and can correctly classify
unfamiliar samples to be tested. (b) When the samples cover only a portion of
the true distribution, unfamiliar samples to be tested are highly likely to be
misclassified due to the error in the decision boundary. (c) The direction in
which the arrow points is the best direction to expand the sample.

  * []

  * •

Case 1: The observed samples cover the true data distribution uniformly (As
shown in Figure 1a).

  * •

Case 2: The observed samples cover only a small region of the true data
distribution (As shown in Figure 2b).

In case 1, although the sample size of the tail classes is small, these
samples represent the true data distribution. The main reason for the
degradation of the model performance is that at each sampling, samples from
the tail class are used with a small probability to calculate loss and update
parameters, resulting in inadequate learning of the tail class by the model.
Faced with this situation, existing data augmentation methods [4, 47],
undersampling [41], oversampling [42, 50], and rebalancing loss [20, 27, 31]
can reasonably improve performance. Combining decoupled training with the
above approach can improve the performance of the model even further [1, 51].
However, rebalancing strategies, including numerous meta-learning methods [31,
32, 11], do not increase the information outside the observed training domain.
Some meta-learning methods [19] used for information augmentation also
struggle to provide valuable information due to weak guidance from the meta-
data [48]. Additionally, CReST [44], a self-training-based approach, attempts
to label new samples for the tail classes, but the introduced new samples are
already well-recognized by the long-tailed model, resulting in very limited
new information being introduced. In summary, when certain classes are
severely underrepresented (i.e., case 2), these methods have difficulty
finding the right direction for adjusting the decision boundaries, so
improvements sometimes worsen [5, 52]. Therefore, we pursue to mine knowledge
from the well-represented head classes to help recover the true distribution
of the tail classes.

In case 2, if the underlying true distribution of the tail classes cannot be
recovered, then the model always fails to learn the correct decision
boundaries. Even if the model achieves high recognition accuracy in the
training set, it still fails to have satisfactory generalization performance
when faced with test samples outside the training domain. Therefore, if the
direction for recovering the true distribution of tail classes, such as the
direction indicated by the three arrows in Figure 1c, can be found, the
generalization ability of the model on the tail classes will be significantly
improved. It is necessary to explore additional knowledge to guide the tail
class to recover the true distribution. Transfer learning for long-tail
classification aims to introduce new information to facilitate the model’s
learning on tail classes, which can be classified into two categories [52]:
head-to-tail knowledge transfer and model pretraining. Due to the natural
scarcity of tail class data, model pretraining is not effectively applicable
in long-tail scenarios [48, 52]. Therefore, most of the current research
focuses on head-to-tail knowledge transfer. However, the motivations for
previous head-to-tail knowledge transfer were limited to qualitative analysis
or speculation [1, 14, 30, 39]. They assumed that all background information
from head classes exists in tail classes, which is an unsupported assumption
lacking evidence. Moreover, these methods neglected the potential diversity of
foreground information in tail classes, leading to the possibility of
inadequate recovery of the true distribution of tail classes. Our experimental
comparisons also demonstrate that knowledge transfer supported by evidence
outperforms the aforementioned methods. In conclusion, recovering the
underlying true distribution with limited samples remains a meaningful
challenge.

It has been shown that the model bias is caused by the classifier and the
long-tailed data does not unbalance the feature representation learning [13,
56]. Also considering that the dimensionality of the feature space is smaller
than that of the sample space, we focus on recovering the true distribution of
tail classes in the feature space. In this work, our main contributions are
summarized as follows.

  * []

  * •

We systematically define the geometry of the feature distribution and the
similarity measure between the geometries (Section 3.1 and 3.2). Based on
this, four surprising experimental phenomena are found which can be used to
guide and recover the true distribution of the tail classes (Section 3.3). The
most important phenomenon is that similar feature distributions have similar
geometries and the similarity between the geometries of the feature
distributions decreases as the interclass similarity decreases. We introduce a
geometric perspective to recover underrepresented class distributions,
providing a theoretical and experimental basis for subsequent studies of class
imbalance.

  * •

Based on four experimental phenomena, we propose to model the uncertainty
representation of the tail features with geometric information from the
feature distribution of the head class (Section 4.1). Specifically, instead of
treating samples in the feature space as deterministic points, we perturb them
to make the model learn information outside the observed domain by taking into
account the geometry of the class to which the samples belong. Our proposed
feature uncertainty modeling can effectively alleviate the model bias
introduced by underrepresented classes and can be easily integrated into
existing networks.

  * •

We propose a three-stage training scheme to apply feature uncertainty
representation (Section 4.2). The results of the ablation experiments show
that compared to decoupled training, the three-stage training scheme improves
the tail class performance while reducing the degradation of the head class
performance, resulting in more overall performance improvement of the model.

  * •

Experiments on large-scale long-tailed datasets (Section 5) show that our
proposed method significantly improves the performance of tail classes and
exhibits state-of-the-art results compared to other similar methods.

##  2 Related Work

###  2.1 Class Rebalancing

The extreme imbalance in the number of samples in the long-tail data prevents
the classification model from learning the distribution of the tail classes
adequately, which leads to poor performance of the model on the tail classes.
Therefore, methods to rebalance the number of samples and the losses incurred
per class (i.e., resampling and reweighting) are proposed. Resampling methods
are divided into oversampling and undersampling [3, 8, 9, 13, 40, 53]. The
idea of oversampling is to randomly sample the tail classes to equalize the
number of samples and thus optimize the classification boundaries. The
undersampling methods balance the number of samples by randomly removing
samples from the head classes. For example, [41] finds that training with a
balanced subset of a long-tailed dataset is instead better than using the full
dataset. In addition, [13, 56] fine-tune the classifier via a resampling
strategy in the second phase of decoupled training. [42] continuously adjusts
the distribution of resampled samples and the weights of the two-loss terms
during training to make the model perform better. [50] employs the model
classification loss from an additional balanced validation set to adjust the
sampling rate of different classes.

The purpose of reweighting loss is intuitive, and it is proposed to balance
the losses incurred by all classes, usually by applying a larger penalty to
the tail classes on the objective function (or loss function) [7, 10, 35, 36,
43, 54, 26, 24]. [31] proposes to adjust the loss with the label frequencies
to alleviate class bias. [20] not only assigns weights to the loss of each
class but also assigns higher weights to hard samples. Recent studies have
shown that the effect of reweighting losses by the inverse of the number of
samples is modest [27, 28]. Some methods that produce more “smooth" weights
for reweighting perform better [24], such as taking the square root of the
number of samples as the weight [27]. [6] attributes the better performance of
this smoother method to the existence of marginal effects. In addition, [1]
proposes to learn the classifier with class-balanced loss by adjusting the
weight decay and MaxNorm in the second stage. DSB [24] and DCR [26], for the
first time, examined the factors influencing model bias from a geometric
perspective and proposed a rebalancing approach.

Although class rebalancing methods are simple to implement, their limitations
have been increasingly recognized in recent research [52, 29]. Class
rebalancing methods merely increase the weight of the tail class loss without
introducing additional knowledge to assist the tail classes, which often leads
to overfitting of the tail classes and significantly compromises the model’s
generalization performance [29]. Another limitation is that class rebalancing
methods often improve tail class performance at the expense of sacrificing
head class performance, making it challenging to handle data scarcity issues
[52, 48]. As a result, more and more research is focusing on information
augmentation.

###  2.2 Stage-wise training

Decoupling [13] first proposes to decouple the learning process on long-tail
data into feature learning and classifier learning, and it finds that re-
learning the balanced classifier can significantly improve the model
performance. Further, BBN [56] combines the two-stage learning into a two-
branch model. The two branches of the model share parameters, with one branch
learning using the original data and the other learning using the resampled
data. [5] decomposes the features into class-generic features and class-
specific features, and it expands the tail class data by combining class-
generic features of the head class with class-specific features of the tail
class. [55] finds that augmenting data with Mixup in the first stage benefits
feature learning and does negligible damage to classifiers trained using
decoupling. [51] also observes that long-tailed data does not affect feature
learning, and it proposes an adaptive calibration function for improving the
cross-entropy loss. [12] considers the effect of noisy samples on the tail
class and adaptively assigns weights to the tail class samples by meta-
learning in the second stage.

The two-stage training pushes the decision boundary away from the augmented
tail class distribution, thereby improving the performance of the tail
classes. However, this may lead to excessive bias in the decision boundary and
affect the head classes [49]. Therefore, we propose a three-stage training
strategy. The first two stages are indistinguishable from decoupled training,
while in the third stage, we fix the classifier parameters and fine-tune the
feature extractor to adapt it to the improved classification boundary.

###  2.3 Head-to-tail knowledge transfer

Head-to-tail knowledge transfer is more relevant to our work than other
methods. [49] and [22] were first proposed in the face recognition field to
transfer variance between classes to augment classes with fewer samples. [49,
25] assumes that the feature distributions of each class are multivariate
Gaussian, and the feature distributions of the common and under-represented
classes have the same variance, the variance of the head class is used to
estimate the distribution of the tail class. [22] assumes that the intra-class
angle distribution follows a Gaussian distribution, transfers the intra-class
angle distribution of features to the tail class, and constructs a “feature
cloud" for each feature to extend the distribution of the tail class. Both
[49] and [22] assume that certain statistical properties of head and tail
classes are the same (i.e., variance and intra-class angular distribution).
However, these assumptions lack supporting evidence, which limits their
performance in long-tail recognition.

Similar to the adversarial attack, [14] proposes to transform some of the head
samples into tail samples through perturbation-based optimization to achieve
tail class augmentation. However, this method exploits the vulnerabilities of
deep neural networks to generate samples that mislead the model, and these
samples do not exist as tail-class samples in reality. As a result, it cannot
effectively help the tail classes recover their underlying distribution. [5]
decomposes the features of each class into class-generic features and class-
specific features. During training, the class-specific features of the tail
class are fused with the generic features of the head class to generate new
features to expand the tail class. This idea is similar to data augmentation
in image space, such as Cutmix. [39] dynamically estimates a set of centers
for each class, and then calculates the displacement between the head class
feature and the corresponding nearest intra-class center. This displacement is
used to combine with the tail class centers to generate new features, thereby
increasing the feature diversity of the tail class. [21] proposes to transfer
the geometric information of the feature distribution boundaries of the head
class to the tail class by enhancing the weights of the tail class classifier.
The recently proposed CMO [30] considers that the image of the head class has
a rich background, so the image of the tail class can be pasted directly onto
the background image of the head class to increase the richness of the tail
class. Overall, the above-mentioned research assumes that all background
information of head classes exists in tail classes, which is an unsupported
assumption. It also overlooks the potential diversity of tail class foreground
information, which may lead to ineffective recovery of the true distribution
of tail classes.

Distinguishing from the above studies, we pioneered a geometric perspective of
head-tail knowledge transfer. We systematically define the geometry of the
distribution and its similarity measure and find direct evidence that the
geometry of the head class distribution can help the tail class.

##  3 Motivation

We first define a measure of the geometry of the feature distribution, and
then propose a similarity measure between the geometries. Finally, across
several benchmark data sets, we discovered four experimental phenomena
regarding the relationship between geometric information of feature
distributions. Inspired by the experimental phenomenon, we propose to utilize
the feature distribution of the head class to help the tail class to recover
the underlying distribution.

###  3.1 The Geometry of Data Distribution

In the $P$-dimensional space, given data
$X=[x_{1},x_{2},\dots,x_{n}]\in\mathbb{R}^{P\times n}$ that belongs to the
same class, the sample covariance matrix of $X$ can be estimated as

$\begin{split}\Sigma_{X}=\mathbb{E}[\frac{1}{n}\sum_{i=1}^{n}x_{i}x_{i}^{T}]=\frac{1}{n}XX^{T}\in\mathbb{R}^{P\times
P}.\end{split}$

If $\Sigma_{X}=I_{P}$ and $I_{P}$ denotes a unit matrix of order $P$, the
distribution of $X$ is said to be isotropic, while the opposite is said to be
anisotropic. In practice, the data distribution is usually anisotropic.
Considering the two-dimensional case, we can find two vectors $\xi_{1}$ and
$\xi_{2}$, where $\xi_{1}$ points to the direction with the largest sample
variance, and $\xi_{2}$ points to the direction with the largest variance
among the directions orthogonal to $\xi_{1}$. $\xi_{1}$ and $\xi_{2}$ can be
used to anchor the geometry of the two-dimensional distribution. In the high-
dimensional case, since $\Sigma_{X}$ is a real symmetric matrix, any two of
its eigenvectors are orthogonal to each other, and $\xi_{i}$ points to the
direction with the $i$-th largest variance. Analogously to the two-dimensional
case, we can use all the eigenvectors of $\Sigma_{X}$ to anchor the geometry
of the distribution.

######  Definition 1 (The geometry of data distribution).

Given a $P$-dimensional sample set $X$ and the corresponding covariance matrix
$\Sigma_{X}$. The eigendecomposition of $\Sigma_{X}$ yields $P$ eigenvalues
$\\{\lambda_{1},\lambda_{2},\dots,\lambda_{P}$ and the corresponding
$P$-dimensional eigenvectors
$[\xi_{1},\xi_{2},\dots,\xi_{P}]\in\mathbb{R}^{P\times P}$. All eigenvectors
of $\Sigma_{X}$ are considered as bones to anchor the geometry of the
distribution of $X$, denoted as

$\begin{split}GD_{X}(\xi_{1},\xi_{2},\dots,\xi_{P}),\end{split}$

where $\lambda_{1}\geq\lambda_{2}\geq\dots\geq\lambda_{P}\geq 0$,
$\|\xi_{i}\|_{2}=1$, $i=1,2,\dots,P$.

###  3.2 Similarity Measure of Geometry

In the $P$-dimensional space, given two types of data
$X_{1}=[x_{1},\dots,x_{n}]\in\mathbb{R}^{P\times n}$ and
$X_{2}=[x_{1},\dots,x_{n}]\in\mathbb{R}^{P\times m}$, their sample covariance
matrices are estimated as
$\Sigma_{X_{1}}=\frac{1}{n}X_{1}X_{1}^{T}\in\mathbb{R}^{P\times P}$ and
$\Sigma_{X_{2}}=\frac{1}{m}X_{2}X_{2}^{T}\in\mathbb{R}^{P\times P}$,
respectively. Performing the eigendecomposition on $\Sigma_{X_{1}}$ and
$\Sigma_{X_{2}}$, the geometry of the distributions $X_{1}$ and $X_{2}$ are
denoted as $GD_{X_{1}}(\xi_{X_{1}}^{1},\dots,\xi_{X_{1}}^{P})$ and
$GD_{X_{2}}(\xi_{X_{2}}^{1},\dots,\xi_{X_{2}}^{P})$, respectively, where
$\xi_{X_{1}}^{i}$ and $\xi_{X_{2}}^{j}(i,j=1,2,\dots,P)$ are the eigenvectors
of $\Sigma_{X_{1}}$ and $\Sigma_{X_{2}}$, respectively.

######  Definition 2 (Similarity metric between geometry).

Given the geometry of two distributions
$GD_{X_{1}}(\xi_{X_{1}}^{1},\dots,\xi_{X_{1}}^{P})$ and
$GD_{X_{2}}(\xi_{X_{2}}^{1},\dots,\xi_{X_{2}}^{P})$, their similarity is
defined as

$\begin{split}S(GD_{X_{1}},GD_{X_{2}})=\sum_{i=1}^{P}\langle\xi_{X_{1}}^{i},\xi_{X_{2}}^{i}\rangle=\sum_{i=1}^{P}{\xi_{X_{1}}^{i}}^{T}\xi_{X_{2}}^{i}.\end{split}$

The larger $S(GD_{X_{1}},GD_{X_{2}})$, the more similar the geometry of the
distributions $X_{1}$ and $X_{2}$. The upper and lower bounds of
$S(GD_{X_{1}},GD_{X_{2}})$ are

$\begin{split}0\leq S(GD_{X_{1}},GD_{X_{2}})\geq P.\end{split}$

When any pair of eigenvectors $\xi_{X_{1}}^{i}$ and $\xi_{X_{2}}^{i}$ are co-
linear, $S(GD_{X_{1}},GD_{X_{2}})$ reaches the upper bound $P$. When any pair
of eigenvectors $\xi_{X_{1}}^{i}$ and $\xi_{X_{2}}^{i}$ are orthogonal,
$S(GD_{X_{1}},GD_{X_{2}})$ takes the lower bound value $0$. Taking the two-
dimensional distribution as an example, since

$\begin{split}0\leq{\phi_{R1}}^{T}\phi_{B1}+{\phi_{R2}}^{T}\phi_{B2}\leq{\xi_{R1}}^{T}{\xi_{B1}}+{\xi_{R2}}^{T}{\xi_{B2}}\leq
2,\end{split}$

it is clear that the geometry of the two distributions in Figure 8 is more
similar compared to the two distributions shown in Figure 9. The details are
described in Appendix A.

###  3.3 Four Discoveries about the Geometry of the Feature Distribution

First define the class similarity measure. Then introduce the four phenomena
we found and their experimental setup.

######  Definition 3.

Given a sample set $D_{c}=\\{\dots,(x_{i},y_{c}),\dots\\}$ of class $c$, the
average prediction score $\frac{1}{\mid
D_{c}\mid}{\textstyle\sum_{i}}p(y_{c}\mid x_{i},\theta)$ of all samples
belonging to class $c$ is calculated using a deep neural network with trained
parameters $\theta$, where $\mid\\!\\!D_{c}\\!\\!\mid$ denotes the sample
number of class $c$. Define the class

$\begin{split}h:=argmax_{k\neq c}(\frac{1}{\mid
D_{c}\mid}{\textstyle\sum_{i}}p(y_{c}\mid x_{i},\theta))_{k}\end{split}$

that is most similar to class $c$, i.e., the class with the largest logit
other than class $c$. Further, the similarity ranking can be done based on
logit.

We investigated the relationship between class similarity and the geometry
similarity of class distributions on two benchmark datasets: Fashion MNIST
[45] and CIFAR-10 [15]. ResNet-18 [46] was adopted as the backbone network and
various training schemes were applied to make the performance of ResNet-18 on
the two datasets comparable to the state-of-the-art results (See Appendix B
for details). First, the similarity between all classes on the two datasets is
calculated and ranked. Then, we extracted $64$-dimensional features of all
samples from both datasets using ResNet18 and calculated the geometry of all
class feature distributions. Based on this, we summarize further experiments
and findings as follows.

####  3.3.1 Phenomenon 1

As shown in Figure 2, features were extracted using trained ResNet-18 on MNIST
[16], Fashion MNIST and CIFAR-10. We find the sum of the eigenvalues
corresponding to the first five eigenvectors that are used to represent the
geometry of the distribution can reach more than $80\%$ of the sum of all
eigenvalues, which means that most of the information of the data distribution
can be recovered along the first five eigenvectors.

Figure 2: The ratio of the sum of the top five eigenvalues to the sum of all
eigenvalues after eigendecomposition for the feature embeddings of all classes
in the three datasets. The horizontal coordinates are the indexes of the
classes, and the specific class names are in Appendix D.

####  3.3.2 Phenomenon 2

Based on the above observations, we set $P$ in $S(GD_{X_{1}},GD_{X_{2}})$ to
$5$ and calculate the similarity between geometry of all class feature
distributions in Fashion MNIST and CIFAR-10 and plot them in Figure 3a and
Figure 3b. We find that if two classes have high similarity, then the geometry
of their feature distributions also exhibit high similarity. And as the
similarity between classes decreases, the similarity between the geometry of
the class feature distributions shows a decreasing trend. Take  _dog_  in
CIFAR-10 as an example, its most and least similar classes are _cat_ and
_automobile_ , respectively, and the geometry of the three classes are
represented by $GD_{airplane}(\xi_{1},\dots,\xi_{64})$,
$GD_{bird}(\eta_{1},\dots,\eta_{64})$ and
$GD_{horse}(\zeta_{1},\dots,\zeta_{64})$. Calculate the matrices $M1$ and
$M2$ and plot them in Figure 3c and Figure 3d, where
$M1_{i,j}\\!\\!=\\!\\!\langle\xi_{i},\eta_{j}\rangle$ and
$M2_{i,j}\\!\\!=\\!\\!\langle\xi_{i},\zeta_{j}\rangle(i,j=1,\dots,64)$. It can
be observed that $M1$ is closer to a diagonal matrix compared to $M2$,
which corresponds to a more similar geometry of  _dog_  and  _cat_.
Furthermore, we validate our findings using ResNet-50 and VGG-16 as backbone
networks on CIFAR-10. The experimental results are shown in the third row of
Figure 3, and it can be observed that the phenomenon we discovered still holds
under different backbone networks.

Figure 3: (a) The horizontal coordinates are the indexes of the classes, and 1
to 9 indicate the classes that are most similar to the class represented by
the vertical coordinates to the least similar, respectively. Each element
represents the similarity of the geometry between classes. See Appendix D for
detailed class names. (b) Same as (a). (c) The inner product between all
eigenvectors of _dog_ and all eigenvectors of _cat_ in CIFAR-10. The sum of
the first five diagonal elements of $M1$ is equal to the value of the element
in the first column of the first row in (b). (d) The inner product between all
eigenvectors of _dog_ and _automobile_ in CIFAR-10. The third row represents
the results of the experiments using ResNet-50 and VGG-16 as backbone
networks. The axes as well as the meanings of the values are consistent with
(a) and (b).

To prove that the above phenomenon does not occur by chance, we give the
probability that the experimental results in Figure 3c occur randomly. Given
two random vectors in a $P$-dimensional space, let their inner product be
$\delta\in[0,1]$. The probability density function of $\delta$ is represented
as

$\begin{split}f_{P}(\delta)=\frac{\Gamma(\frac{P}{2})}{\Gamma(\frac{P-1}{2})\sqrt{\pi}}(1-\delta^{2})^{\frac{P-3}{2}}.\end{split}$
(1)

The detailed derivation and proof process of the above equation is shown in
the Appendix C. Setting $P$ in $f_{P}(\delta)$ to $64$, when $\delta$ is taken
as the first five diagonal elements of $M1$ respectively, the calculation
result of $f_{P}(\cdot)$ is shown in Figure 4a. Considering only the first
five diagonal elements of $M1$, the probability of the situation shown in
Figure 3c occurring is almost $0$. Not only that, we observed numerous such
phenomena (see Appendix D), thus implying that the phenomena we found could
hardly have occurred by chance.

Figure 4: (a) The function curve of Equation (1). It can be observed that as
the dimensionality increases, any two random vectors tend to be orthogonal to
each other. (b) When two different models are used to extract features of
_dog_ separately, the geometry of the two feature distributions is not
similar. (c) Cosine similarity between feature centers of classes on CIFAR-10.
(d) Cosine similarity between feature centers of classes on CIFAR-10-LT.

####  3.3.3 Phenomenon 3

The phenomenon that features distributions of a similar class has similar
geometry only occurs when all features are extracted using the same model.
Figure 4b shows that there is a low similarity between the geometry of _dog_
computed by two different ResNet18 trained with random initialization. More
examples in Appendix E.

####  3.3.4 Phenomenon 4

We conducted further experiments on CIFAR-10 as well as its long-tailed
version CIFAR-10-LT. In CIFAR-10-LT, _airplane_ , _automobile_ , _bird_ , and
_cat_ are considered head classes and the remaining classes are tail classes.
As shown in Figure 4d, we confirmed that the most similar class to the tail
class usually belongs to the head class [5] and found that if a tail class and
a head class show high similarity in CIFAR-10-LT, they also show high
similarity on CIFAR-10.

####  3.3.5 Summary and inspiration

Combining the above four phenomena, we propose the following idea: the most
similar head class is selected for each tail class in the training process,
and the geometry of the head class feature distribution is taken as a priori
knowledge to guide and recover the underlying true distribution of the tail
class.

##  4 Methodology

We first introduce how to leverage the geometric information of the head class
feature distribution to model the uncertainty representation of the tail class
features, allowing the model to learn the underlying true distribution of the
tail class. Then a three-stage training scheme is proposed to apply the
feature uncertainty representation.

###  4.1 Feature Uncertainty Representation

Given a tail class $t$, the head class that is most similar to class $t$ is
assumed to be $h$. The $P$-dimensional feature embedding belonging to tail
class $t$ is $z_{t}=[z_{t}^{1},\dots,z_{t}^{N_{t}}]^{T}\in\mathbb{R}^{P\times
N_{t}}$ and the feature embedding belonging to head class $h$ is
$z_{h}=[z_{h}^{1},\dots,z_{h}^{N_{h}}]^{T}\in\mathbb{R}^{P\times N_{h}}$,
where $N_{t}$ and $N_{h}$ denote the sample numbers of class $t$ and class
$h$, respectively. The $i$-th feature embedding of $z_{t}$ is denoted by
$z_{t}^{i}$. For the model to learn the underlying distribution of the tail
class $t$, we want to utilize the existing feature embeddings to generate
feature embeddings that can cover the underlying distribution of the tail
class $t$. We therefore propose to model the uncertainty representation of
$z_{t}^{i}$ with the geometry of the feature distribution of class $h$, i.e.,
$z_{t}^{i}$ is no longer considered a deterministic point.

The sample covariance matrix of class $h$ is estimated as
$\Sigma_{h}=\frac{1}{N_{h}}z_{h}z_{h}^{T}\in\mathbb{R}^{P\times P}$. The
eigenvalues of the matrix $\Sigma_{h}$ are denoted as
$[\lambda_{h}^{1},\dots,\lambda_{h}^{P}]\in\mathbb{R}^{P}$, where
$\lambda_{h}^{1}\geq\cdots\geq\lambda_{h}^{P}$. The eigenvector
$[\xi_{h}^{1},\dots,\xi_{h}^{P}]\in\mathbb{R}^{P\times P}$, which corresponds
one-to-one with the eigenvalues, anchors the geometry of the class $h$ feature
distribution, where $\|\xi_{h}^{i}\|_{2}=1,i=1,\dots,P$. Since the
distributions of similar class have similar geometry, we propose to represent
the uncertainty of $z_{t}^{i}$ by centering a single feature embedding
$z_{t}^{i}$ of the tail class $t$ and performing a random translation to
$z_{t}^{i}$ along a random linear combination of
$\xi_{h}^{1},\dots,\xi_{h}^{P}$. Considering that the “scope" of the
distribution is larger in the direction with larger eigenvalues [57], an
additional weight $\lambda_{h}^{i}$ is assigned to $\xi_{h}^{i}(i=1,\dots,P)$
when the eigenvectors are randomly combined, which means that $z_{t}^{i}$ is
translated farther with higher probability in the direction with larger
eigenvalues. In summary, the final form of the proposed method can be
represented as

$\begin{split}FUR(z_{t}^{i})&=\\!\\!\\!\\!\\!\\!\\!\\!\\!\overbrace{z_{t}^{i}+\sum_{j=1}^{P}\epsilon_{j}\lambda_{h}^{j}\xi_{h}^{j}\in\mathbb{R}^{P}}^{Uncertainty\hskip
2.27621ptrepresentation\hskip 2.27621ptof\hskip 2.27621ptz_{t}^{i}}\\\
\epsilon_{j}&\sim N(0,1),j=1,\dots,P.\end{split}$ (2)

$\epsilon_{1},\dots,\epsilon_{P}$ all follow the standard Gaussian
distribution and are independent of each other, and sampling them randomly
multiple times can produce new feature embeddings with different translation
directions and distances. In particular, when
$\epsilon_{1}=1,\epsilon_{2},\dots,\epsilon_{P}=0$, $z_{t}^{i}$ is translated
along $\xi_{h}^{1}$ by a distance $\lambda_{h}^{1}$. And so on, the maximum
translation distances of $z_{t}^{i}$ in the direction represented by each
feature vector individually are $\lambda_{h}^{1},\dots,\lambda_{h}^{P}$,
respectively.

Algorithm 1 Feature Uncertainty Representation

1:A long-tailed dataset $D$ containing $S$ samples. A CNN network
$M=\\{f(x,\theta_{1}),g(z,\theta_{2})\\}$, where $\theta_{1}$ and $\theta_{2}$
denote the parameters of the feature sub-network and classifier, respectively,
and $x$ and $z$ denote the input and feature embedding of the model,
respectively.

2:for $epoch=1$ to $m1$ do

3: Training model $M$ on dataset $D$ without using any class rebalancing
method.

4:end for

5:Using $M$, the head classes that are most similar to each tail class are
calculated, and then the sample covariance matrix of these head classes is
calculated.

6:for $epoch=m1$ to $m2$ do

7: Freeze the parameters $\theta_{1}$ of the feature sub-network.

8: for $iteration=0$ to $\frac{S}{batch\hskip 2.27621ptsize}$ do

9: A mini-batch $\\{(x_{i},y_{i})\\}_{i=1}^{batch\hskip 2.27621ptsize}$ is
sampled from $D$, where the sample numbers from the tail class are $N_{T}$ and
the sample numbers from the head class are $N_{T}(1+N_{A})$.

10: Compute the feature embedding:
$z_{i}=f(x_{i},\theta_{1}),i=1,\dots,(2N_{T}+N_{T}N_{A})$.

11: Uncertainty representation of all features from tail classes:
$FUR(z_{t}^{i})=z_{t}^{i}+{\textstyle\sum_{j=1}^{P}}\epsilon_{j}\lambda_{h}^{j}\xi_{h}^{j}\in\mathbb{R}^{P}$,
$t\in tail\hskip 2.27621ptclass,i\in int[0,N_{t}]$. $h$ denotes the head class
most similar to $t$.

12: $\epsilon_{j}\sim N(0,1),j=1,\dots,P$. $N_{A}$ augmented features are
generated for the true features of each tail class by randomly sampling
$N_{A}$ times of $\epsilon_{j}(1,\dots,P)$.

13: A mini-batch with a balanced distribution containing $2N_{T}(1+N_{A})$
samples is obtained.

14: Compute the cross-entropy loss $L(g(z_{i},\theta_{2}),y_{i})$ and update
the parameters of the classifier:
$\theta_{2}=\theta_{2}-\alpha\nabla_{\theta_{2}}L(g(z_{i},\theta_{2}),y_{i})$.

15: end for

16:end for

17:for $epoch=m2$ to $m3$ do

18: Freeze the parameter $\theta_{2}$ of $g(z,\theta_{2})$.

19: Fine-tuning the parameters of the feature sub-network using the long-
tailed dataset $D$.

20:end for

Our proposed method can be integrated as a flexible module after the feature
sub-network. It generates augmented samples of tail classes in the feature
space to cover the underlying distribution, giving the model better
generalization ability on long-tailed data. Note that this module is only
applied during model training and can be discarded during testing without
affecting the inference speed.

###  4.2 Training Scheme

We propose a three-stage training scheme to apply feature uncertainty
representation so that the model learns information outside the observed
domain. Decoupled training is adopted for the first two phases. In Phase 1,
the long-tailed dataset is used to learn the feature sub-network and
classifier. In Phase 2, the uncertainty representation of the tail feature is
applied to generate new samples for reshaping the decision boundaries. Unlike
decoupled training, we additionally add Phase 3 to fine-tune the feature sub-
network to adapt it to the new decision boundaries.

  * []

  * •

Phase-1: Initialization training. Represent an end-to-end deep neural network
as a combination of a feature sub-network and a classifier:
$M=\\{f(x,\theta_{1}),g(z,\theta_{2})\\}$, where $\theta_{1}$ and $\theta_{2}$
are the parameters of the network. We utilize all images from the dataset to
learn the feature sub-network $f(x,\theta_{1})$ as well as the classifier
$g(z,\theta_{2})$. After training is completed, the head classes that are most
similar to each tail class are selected based on the average prediction score
of the model (see Definition 3), and the geometry of the feature distribution
of these head classes is represented by the eigenvectors of the covariance
matrix, which will be applied to guide the recovery of the tail class
distribution.

  * •

Phase-2: Reshaping decision boundaries. Freeze the parameters of
$f(x,\theta_{1})$ and employ feature uncertainty representation in feature
space for the tail class to fine-tune the classifier to improve the
performance of the tail class. Specifically, in each iteration, we randomly
sample $N_{T}$ images from the tail class, and then generate $N_{A}$ augmented
samples for each true sample by feature uncertainty representation. Meanwhile,
to balance the sample distribution, we directly sample $N_{T}(1+N_{A})$
samples randomly from the head class. The tail class samples and the head
class samples together form a mini-batch containing $2N_{T}(1+N_{A})$ samples
for fine-tuning the classifier. The $N_{A}$ and $N_{T}$ settings are related
to the batch size, and they are described in detail in Section 5.2.

  * •

Phase-3: Fine-tuned feature sub-network. Fine-tuning the decision boundary can
improve the performance of the tail class while compromising the performance
of the head class [49]. This is because the feature sub-network is not well
adapted to the new decision boundary. Therefore, we propose to freeze the
parameters of $g(z,\theta_{2})$ at Phase-3 and fine-tune $f(x,\theta_{1})$
with the original long-tailed data.

The above three-phase training process is summarized in Algorithm 1.

##  5 Experiments

###  5.1 Datasets and Evaluation Metrics

We evaluate the effectiveness and generalizability of our approach at
CIFAR-10/100-LT [2, 15], ImageNet-LT [23], iNaturalist 2018 [37] and OIA-ODIR
[18]. For a fair comparison, the training and test images of all datasets are
officially split [48, 52, 24], and the Top-1 accuracy on the test set is
utilized as a performance metric.

  * []

  * •

Both CIFAR-10 and CIFAR-100 [15] contain $60,000$ images, of which $50,000$
are used for training and $10,000$ for validation, and they contain $10$ and
$100$ classes, respectively. For a fair comparison, we use the long-tailed
version of the CIFAR dataset. The imbalance factor (IF) is defined as the
value of the number of the most frequent class training samples divided by the
number of the least frequent class training samples. The imbalance factors we
employ in our experiments are $10$, $50$, $100$, and $200$.

  * •

ImageNet-LT is an artificially produced unbalanced dataset utilizing its
balanced version (ImageNet-LT-2012 [33]). It with an imbalance factor of
$256$, contains $1000$ classes totaling $115.8k$ images, with a maximum of
$1280$ images and a minimum of $5$ images per class.

  * •

The iNaturalist 2018 dataset is a large-scale real-world dataset that exhibits
a long tail. It contains $437,513$ training samples from $8,142$ classes with
an imbalance factor of $500$ and three validation samples per class.

  * •

The OIA-ODIR dataset was made public in $2019$, and it contains a total of
$10,000$ fundus images in $8$ classes. See Section 5.6 for a more detailed
description.

Table 1: Comparison on CIFAR-10-LT and CIFAR-100-LT. The accuracy (%) of Top-1 is reported. The best and second-best results are shown in underlined bold and bold, respectively. FUR-Decoupled indicates a FUR with decoupled training, and FUR Default indicates a FUR with three-stage training scheme. Dataset | Pub. | CIFAR-10-LT | CIFAR-100-LT
---|---|---|---
Backbone Net | - | ResNet-32
Imbalance factor | - | 200 | 100 | 50 | 10 | 200 | 100 | 50 | 10
Cross Entropy | - | 65.6 | 70.3 | 74.8 | 86.3 | 34.8 | 38.2 | 43.8 | 55.7
BBN [56] | CVPR 2020 | - | 79.8 | 82.1 | 88.3 | - | 42.5 | 47.0 | 59.1
UniMix [47] | NeurIPS 2021 | 78.5 | 82.8 | 84.3 | 89.7 | 42.1 | 45.5 | 51.1 | 61.3
MetaSAug [19] | CVPR 2021 | 76.8 | 80.5 | 84.0 | 89.4 | 39.9 | 46.8 | 51.9 | 61.7
MiSLAS [55] | CVPR 2021 | - | 82.1 | 85.7 | 90.0 | - | 47.0 | 52.3 | 63.2
CDB-W-CE [34] | IJCV 2022 | - | - | - | - | - | 42.6 | - | 58.7
GCL [17] | CVPR 2022 | 79.0 | 82.7 | 85.5 | - | 44.9 | 48.7 | 53.6 | -
RIDE + CR [26] | CVPR 2023 | - | - | - | - | - | 50.7 | 54.3 | 61.4
OFA [5] | ECCV 2020 | 75.5 | 82.0 | 84.4 | 91.2 | 41.4 | 48.5 | 52.1 | 65.3
M2m [14] | CVPR 2020 | - | 78.3 | - | 87.9 | - | 42.9 | - | 58.2
RSG [39] | CVPR 2021 | - | 79.6 | 82.8 | - | - | 44.6 | 48.5 | -
CMO [30] | CVPR 2022 | - | - | - | - | - | 50.0 | 53.0 | 60.2
FDC [25] | TMM 2023 | 79.7 | 83.4 | 86.5 | 90.6 | 45.8 | 50.6 | 54.1 | 61.3
FUR-Decoupled | - | 79.6 | 83.4 | 86.1 | 90.7 | 45.8 | 50.7 | 53.9 | 61.4
FUR | - | 79.8 | 83.7 | 86.2 | 90.9 | 46.2 | 50.9 | 54.1 | 61.8
Table 2: Details of the experimental setup. The $100+50+50$ in Epoch indicates the first phase, the second phase, and the third phase are trained for $100$, $50$, and $50$ epochs, respectively. Dataset | CIFAR-10/100-LT | ImageNet-LT | iNaturalist 2018
---|---|---|---
Backbone | ResNet-32 | ResNeXt-50 | ResNet-50
Epoch | 100+50+50 | 100+50+50 | 100+50+50
Optimizer: SGD
Mm | 0.9
LR | Phase1 | 0.05 | 0.1 | 0.1
Phase2 | 0.001 | 0.001 | 0.001
Phase3 | 0.001 | 0.001 | 0.001
LR decay | Cosine | Linear | Linear
Batch size | 128 | 256 | 512
Warm-up | ✓ | ✗ | ✗
Table 3: Top-1 accuracy (%) of ResNext-50 [21] on ImageNet-LT and Top-1 accuracy (%) of ResNet-50 [22] on iNaturalist 2018 for classification. The best and the second-best results are shown in underline bold and bold, respectively. FUR-Decoupled indicates a FUR with decoupled training, and FUR Default indicates a FUR with three-stage training scheme. Methods | Pub. | ImageNet-LT | iNaturalist 2018
---|---|---|---
ResNext-50 | ResNet-50
Head | Middle | Tail | Overall | Head | Middle | Tail | Overall
BBN [56] | CVPR 2020 | 43.3 | 45.9 | 43.7 | 44.7 | 49.4 | 70.8 | 65.3 | 66.3
DisAlign [51] | CVPR 2021 | 59.9 | 49.9 | 31.8 | 52.9 | 68.0 | 71.3 | 69.4 | 70.2
UniMix [47] | NeurIPS 2021 | - | - | - | 48.4 | - | - | - | 69.2
MetaSAug [19] | CVPR 2021 | - | - | - | 47.3 | - | - | - | 68.7
MiSLAS [55] | CVPR 2021 | 65.3 | 50.6 | 33.0 | 53.4 | 73.2 | 72.4 | 70.4 | 71.6
CDB-W-CE [34] | IJCV 2022 | - | - | - | 38.5 | - | - | - | -
GCL [17] | CVPR 2022 | - | - | - | 54.9 | - | - | - | 72.0
DSB-LADE [24] | ICLR 2023 | 62.6 | 50.4 | 33.6 | 53.2 | 72.3 | 70.7 | 65.8 | 70.5
LDAM + CR [26] | CVPR 2023 | 60.8 | 50.3 | 33.6 | 52.4 | 69.3 | 66.7 | 61.9 | 65.7
OFA [5] | ECCV 2020 | 47.3 | 31.6 | 14.7 | 35.2 | - | - | - | 65.9
RSG [39] | CVPR 2021 | 63.2 | 48.2 | 32.3 | 51.8 | - | - | - | 70.2
GistNet [21] | ICCV 2021 | 52.8 | 39.8 | 21.7 | 42.2 | - | - | - | 70.8
BS + CMO [30] | CVPR 2022 | 62.0 | 49.1 | 36.7 | 52.3 | 68.8 | 70.0 | 72.3 | 70.9
FUR-Decoupled | - | 65.1 | 51.6 | 38.3 | 55.2 | 73.4 | 72.5 | 73.7 | 72.4
FUR | - | 65.4 | 52.2 | 37.8 | 55.5 | 73.6 | 72.9 | 73.1 | 72.6

###  5.2 Implementation Details

Following the accepted settings [6, 51, 55], the batch sizes on ImageNet-LT
and iNaturalist 2018 were taken to be $256$ and $512$, respectively. For a
fair comparison, we are consistent with OFA [5] and take $N_{A}$ to be $3$, so
$N_{T}$ is $32$ and $64$ on ImageNet-LT and iNaturalist 2018, respectively.
More details of the experimental setup are listed in Table 2. The experimental
setup on OIA-ODIR will be presented separately in Section 5.6. We trained
models on CIFAR-10-LT, CIFAR-100-LT and OIA-ODIR using a single NVIDIA 2080Ti
GPU and on ImageNet-LT and iNaturalist 2018 using $4$ NVIDIA 2080Ti GPUs.

###  5.3 Comparative Methods

We train the proposed Feature Uncertainty Representation (FUR) employing
decoupled training and three-stage training schemes, respectively. The FUR is
then compared with classical and state-of-the-art long-tailed knowledge
transfer methods, non-transfer data augmentation methods, and other state-of-
the-art long-tailed recognition methods. The specific methods are classified
as follows.

  * []

  * •

Classical and latest long-tailed knowledge transfer methods, include OFA [5],
M2m [14], RSG [39], GistNet [21], CMO [30] and FDC [25].

  * •

Other state-of-the-art methods. They include the two-stage MiSLAS [55],
DisAlign [51], BBN [56] with two branches, and the non-transfer augmentation
methods UniMix [47], MetaSAug [19], CDB [34], GCL [17] DSB [24] and CR [26].

###  5.4 Results on CIFAR-10-LT and CIFAR-100-LT

The results on CIFAR-10-LT and CIFAR-100-LT are summarized in Table 1, where
our proposed method achieves optimal performance on six long-tailed CIFAR
datasets and second-best results on the remaining two datasets. Our proposed
FUR outperforms GCL by 1% and 2.2% on CIFAR-10-LT and CIFAR-100-LT with IF =
$100$, respectively. On the CIFAR-100-LT with IF = $10$, FUR-Decoupled
outperforms the combined CMO by 1.6%. FUR with a three stage training scheme
outperforms FUR-Decoupled on all datasets, which we will analyze in detail in
the next part of the experiment.

Compared to CMO, which randomly pastes the image foreground of the tail class
onto the background of the head class image, our proposed FUR relies on the
observed prior knowledge to recover the underlying distribution of the tail
class. GCL constructs the same “feature cloud" for each feature of the tail
class to adjust the model logit, without taking into account the differences
in domain characteristics between classes. As a result, FUR outperforms
similar methods on multiple datasets.

###  5.5 Results on ImageNet-LT and iNaturalist 2018

We report in Table 3 not only the overall performance of FUR and FUR-Decoupled
on ImageNet-LT and iNaturalist 2018 but also additionally add the performance
on three subsets of these two datasets, Head (more than $100$ images), Middle
($20\sim 100$ images), and Tail (less than $20$ images). Compared to other
methods, FUR shows the state-of-the-art overall performance on both ImageNet-
LT and iNaturalist 2018.

We argue that although the bias of the classifier is mitigated after decoupled
training, it is ignored whether the feature sub-network can adapt to the new
decision boundaries, which leads to a trade-off in the performance of the head
classes. Therefore we add a third stage to fine-tune the feature extractor to
adapt it to the latest decision boundaries. FUR-Decoupled outperforms the
transfer-based CMO by 3.4% and 4.8%, respectively, on the Head subset of the
two large-scale long-tailed datasets, benefiting from the fact that FUR relies
on prior knowledge rather than randomly recovering the tail class
distribution. Although there is a slight degradation in tail class performance
after the third stage, the overall performance of the model and the
performance on the head subset are better than the decoupled trained model.
Thus both the feature sub-network and the classifier need to be fine-tuned to
rebalance the preferences of the model. In addition, the extraordinary
performance of FUR-Decoupled on tail classes suggests that our method can
recover the underlying distribution of tail classes more efficiently.

###  5.6 Results on OIA-ODIR

The OIA-ODIR dataset contains a total of $10,000$ fundus images in $8$
classes. The eight classes are Normal (N), hypertensive retinopathy (D),
glaucoma (G), cataract (C), agerelated macular degeneration (A), hypertension
complication (H), pathologic myopia (M), other disease/abnormality (O).
Considering that O usually appears together with other diseases, to reduce
ambiguity, we adopt the data splitting scheme of [24], using only the data of
the first $7$ classes, and the number of training samples and test samples for
each class is shown in Figure 5.

Figure 5: The left figure shows the number of training and testing samples for
each class in OIA-ODIR dataset. The right figure presents a comparison of
various methods.

We used ResNet-50 as the backbone network. An adam optimizer with a learning
rate of $0.1$ (linear decay), a momentum of $0.9$, and a weight decay factor
of $0.005$ was adopted to train all networks. In keeping with [24], average
precision (AP) was used as the performance metric of the model. We implemented
BBN, OFA, and CMO on the OIA-ODIR dataset and compared them with our method.
The experimental results (Figure 5) indicate that FUR outperforms the other
three methods.

###  5.7 Visualization Analysis

To clearly demonstrate that FUR can excel in the recovery of the underlying
distribution of tail classes, we visualized the tail features of CIFAR-10-LT
via t SNE. As shown in Figure 6, the training distribution after augmentation
with FUR can cover the test distribution well. The above results further show
that our proposed method efficiently recovers the distribution of tail
classes. This result further indicates that our proposed method accurately
recovers the underlying distribution of the tail classes, allowing the model
to perform better on the test set outside the training domain.

Figure 6: Visualization of tail class feature embedding from CIFAR-10-LT with
an imbalance factor of $200$.

###  5.8 More Analysis and Discussion

After the training in the third stage, the shape of the distribution of the
head class may change. In this case, if we match the most similar head class
for the tail class, it may cause the most similar head class to change. Then
is there a need for further feature augmentation for the tail class? We do not
think it is needed for the following reasons.

In the second stage, we enhanced the features of the tail category to make the
model learn as much information as possible about the true distribution of the
tail category. In the third stage, we used a smaller learning rate and trained
with long-tailed data, which made the model more focused on learning about the
head category. Therefore, after fine-tuning in the third stage, even if there
is a slight change in the representation of the head category, the
representation of the tail category by the model is almost unchanged, and
knowledge enhancement for the tail category has been completed in the second
stage. That is to say, even after training in the third stage, the shift of
the tail class distribution is still small enough that features extracted by a
model trained in three stages can still be well covered by augmented
distributions generated by FUR in stage two. Figure 6 demonstrates that the
data distribution generated by FUR in the second stage can effectively cover
the testing distribution. Therefore, we believe that the model has already
achieved the goal of learning more about the underlying distribution of the
tail classes, and the superior performance on the tail subsets of ImageNet-LT
and iNaturalist 2018 further confirms the superiority of our method.
Additionally, adding extra training to continue adjusting the feature
augmentation for the tail classes would increase the time cost of the
training.

Figure 7: FUR+ denotes the addition of a fourth training phase to retune the
feature augmentation of the tail class, and FUR denotes the method proposed in
this paper.

We added training stage to adjust the feature enhancement of the tail class on
CIFAR-10-LT and CIFAR-100-LT and compared it with the method proposed in this
paper. The parameters of the fourth stage are consistent with those of the
third stage. As shown in the Figure 7, the performance of the two methods is
almost the same, and the confidence intervals of the five groups of results
completely overlap. For the above reasons, we finally proposed a three-stage
training strategy.

##  6 Conclusion

In this work, We discovered four fundamental phenomena regarding the
relationship between the geometry of feature distributions, which provide the
theoretical and experimental basis for subsequent studies of class imbalance.
Inspired by the four phenomena, we propose feature uncertainty representation
(FUR) with geometric information for recovering the true distribution of tail
classes. After three stages of training, the experimental results show that
our proposed method greatly improves the performance of the tail class
compared to other methods and ensures the superior performance of the head
class at the same time.

##  7 Data Availability Statements

All datasets used in this study are open-access and have been cited in the
paper.

## Appendix A

To facilitate the analysis and understanding of the geometry of the feature
distributions and the similarity between the geometry, four two-dimensional
distributions were generated and plotted in Figure 8 and Figure 9. The
geometry of the feature distribution is first introduced. As shown in Figure
8, the direction $\xi_{R1}$ with the largest variance and the direction
$\xi_{R2}$ with the largest variance in the direction orthogonal to $\xi_{R1}$
are selected. It can be seen that the geometry and location of the
distribution can be anchored by $\xi_{R1}$, $\xi_{R2}$ and the center of the
distribution. It is important to note that in this work, we only focus on the
shape of the distribution and ignore the location of the distribution.
Moreover, if the projection is done along these two directions, the
information of the distribution is preserved to the maximum extent.

Figure 8: Two distributions with similar geometry. Figure 9: Two distributions
with low geometry similarity.

Observing Figure 8, we can notice that the geometry of the red and blue
distributions are more similar because their covariances are similar, i.e.,
the pattern of variation of the vertical axis with the horizontal axis is
similar. The direction of the maximum variance mainly determines the shape of
the distribution, and the direction of the second largest variance also plays
a role in the shape of the distribution. Obviously, the geometry of the two
distributions in Figure 8 is more similar than in Figure 9.

## Appendix B

The CIFAR-10 dataset consists of $60,000$ images with size $32\times 32$ from
$10$ classes, each class contains $6,000$ images, of which $5,000$ images are
used for training and $1,000$ images are used for testing. Fashion MNIST has
ten classes, each containing $6000$ training images and $1000$ test images,
each with a size of $28\times 28$.

Figure 10: Class accuracy of ResNet-18 on Fashion MNIST and CIFAR-10. The
class indexes in Figure 2 correspond to the ten numbers of MNIST. For Fashion
MNIST and CIFAR-10, the class indexes (i.e., $1$ to $10$) correspond to the
classes from left to right in the above figure, respectively.

To improve the generalization ability of the model and prevent the model from
overfitting on the training set, we perform three data augmentation operations
on the training set: random flip, random crop, and Cutout. Cutout keeps the
model from being overly dependent on certain areas of the image by randomly
masking out parts of the image. Considering the size of the image is small,
the size $7\times 7$ convolutional kernel of ResNet-18 and the pooling
operation tend to lose spatial information, so we remove the maximum pooling
layer and adopt the size $3\times 3$ convolutional kernel instead of the size
$7\times 7$ convolutional kernel.

We adopt SGD to optimize the model, set the batch size to $128$, and the
initial learning rate to $0.1$. If the loss does not decrease after $10$
consecutive epochs, the learning rate becomes $0.5$ times of the original, and
we train a total of $250$ epochs. ResNet-18 achieved an accuracy of $93.46\%$
on CIFAR-10 and $94.82\%$ on Fashion MNIST. The accuracy rates for each class
are plotted in Figure 10.

## Appendix C

In the following, we derive the probability density function of the inner
product of two random vectors. Without loss of generality, we set $x$ to be a
$P$-dimensional random unit vector and fix $y$ to be a unit vector, i.e.

$\begin{split}x=(x_{1},x_{2},\dots,x_{P}),y=(1,0,\dots,0).\end{split}$

The above equation satisfies $x_{1}^{2}+x_{2}^{2}+\cdots+x_{P}^{2}=1$. Using
spherical transformations

$\begin{split}\left\\{\begin{array}[]{ll}x_{1}=r\cos\varphi_{1},\\\
x_{2}=r\sin\varphi_{1}\cos\varphi_{2},\\\
x_{3}=r\sin\varphi_{1}\sin\varphi_{2}\cos\varphi_{3},\\\ \cdots\\\
x_{n-1}=r\sin\varphi_{1}\sin\varphi_{2}\cdots\sin\varphi_{n-2}\cos\varphi_{n-1},\\\
x_{n}=r\sin\varphi_{1}\sin\varphi_{2}\cdots\sin\varphi_{n-2}\sin\varphi_{n-1},\end{array}\right.\end{split}$

where

$\begin{split}\left\\{\begin{array}[]{ll}0\leq r\leq+\infty,\\\
0\leq\varphi_{1}\leq\pi,\\\ \cdots\\\ 0\leq\varphi_{n-2}\leq\pi,\\\
0\leq\varphi_{n-1}\leq 2\pi.\end{array}\right.\end{split}$

The Jacobi determinant of the above transformation is

$\begin{split}J&=\frac{\partial(x_{1},x_{2},\dots,x_{n})}{\partial(r,\varphi_{1},\dots,\varphi_{n-1})}\\\
&=r^{n-1}\sin^{n-2}\varphi_{1}\sin^{n-3}\varphi_{2}\cdots\sin\varphi_{n-2}.\end{split}$

Since $x$ is a unit vector, $r=1$. Notice that $\langle
x,y\rangle=x_{1}=\cos\varphi_{1}$, so $\cos\langle
x,y\rangle=\cos\varphi_{1}$. According to the geometric probability,

$\begin{split}&P_{n}(\varphi_{1}\leq\theta)=\\\
&\frac{\int_{0}^{\theta}\sin^{n-2}\varphi_{1}d\varphi_{1}\cdots\int_{0}^{\pi}\sin\varphi_{n-2}d\varphi_{n-2}\int_{0}^{2\pi}d\varphi_{n-1}}{\int_{0}^{\pi}\sin^{n-2}\varphi_{1}d\varphi_{1}\cdots\int_{0}^{\pi}\sin^{2}\varphi_{n-2}d\varphi_{n-2}\int_{0}^{2\pi}d\varphi_{n-1}}\\\
&=\frac{S_{n-1}\int_{0}^{\theta}\sin^{n-2}\varphi_{1}d\varphi_{1}}{S_{n}}\end{split}$

Where $S_{n}$ denotes the surface area of the $n$-dimensional unit sphere.
When $k$ is a positive integer,

$\begin{split}\int_{0}^{\pi}\sin^{k-1}\varphi
d\varphi=2\int_{0}^{\frac{\pi}{2}}\sin^{k-1}\varphi d\varphi,\end{split}$

and because

$\begin{split}\int_{0}^{\frac{\pi}{2}}\sin^{n}\varphi
d\varphi=\left\\{\begin{array}[]{ll}\frac{(2m-1)!!}{(2m)!!}\cdot\frac{\pi}{2},n=2m\\\
\frac{(2m)!!}{(2m+1)!!},n=2m+1\end{array}\right.,\end{split}$

the expression of $S_{n}$ is obtained. For convenience, we can use the
$\Gamma$ function to unify the two, then

$\begin{split}S_{n}=\frac{2\pi^{\frac{n}{2}}}{\Gamma(\frac{n}{2})}.\end{split}$

We can obtain

$\begin{split}P_{n}(\varphi_{1}\leq\theta)=\frac{\Gamma(\frac{n}{2})}{\Gamma(\frac{n-1}{2}\sqrt{\pi})}\int_{0}^{\theta}\sin^{n-2}\varphi_{1}d\varphi_{1}.\end{split}$

Further, the probability density function of $\theta$ is calculated as

$\begin{split}f_{n}(\theta)&=\frac{d}{d\theta}P_{n}(\varphi_{1}\leq\theta)\\\
&=\frac{\Gamma(\frac{n}{2})}{\Gamma(\frac{n-1}{2}\sqrt{\pi})}\sin^{n-2}\theta.\end{split}$

We plot the curve of the function $f_{n}(\theta)$ in Figure 11. It can be seen
that the angle between the two random vectors tends to $\pi/2$ as the
dimensionality increases.

Let $\delta=\cos\theta$. Then the probability density function of $\delta$ is

$\begin{split}f_{n}(\delta)=\frac{\Gamma(\frac{n}{2})}{\Gamma(\frac{n-1}{2}\sqrt{\pi})}(1-\delta^{2})^{\frac{n-3}{2}}.\end{split}$

In the main text, the dimensionality of the feature vector is denoted by $P$.
We plot the curve of the function $f_{n}(\delta)$ in Figure 12. It can be seen
that the inner product between two high-dimensional random vectors tends to
$0$ as the dimensionality increases, which means that the two random vectors
tend to be orthogonal. The above results prove that our findings did not
happen by chance and that the experimental phenomena we summarized are
reliable.

Figure 11: The probability density function of the angle between two high-
dimensional random vectors. Figure 12: The probability density function of the
inner product between two high-dimensional random vectors.

## Appendix D

Table 4: Details of all classes in Figure 3a and Figure 3b.

| Shirt | Pullover | Dress | Bag | Trouser | Sneaker | Ankle | Sandal | Coat
---|---|---|---|---|---|---|---|---|---
T-shirt | 3.63 | 2.56 | 1.27 | 0.69 | 1.15 | 0.93 | 1.28 | 1.16 | 1.02
| Dress | Coat | Bag | Pullover | Shirt | T-shirt | Ankle | Sneaker | Sandal
Trouser | 2.02 | 1.52 | 1.56 | 1.52 | 1.57 | 1.15 | 0.83 | 0.86 | 0.64
| Shirt | Coat | T-shirt | Dress | Trouser | Bag | Sandal | Ankle | Sneaker
Pullover | 3.42 | 2.62 | 2.56 | 1.20 | 1.52 | 1.52 | 1.48 | 0.71 | 0.36
| Coat | Trouser | Shirt | T-shirt | Pullover | Bag | Ankle | Sandal | Sneaker
Dress | 2.58 | 2.02 | 1.97 | 1.27 | 1.20 | 1.32 | 1.24 | 0.58 | 0.48
| Pullover | Dress | Shirt | Bag | Trouser | Ankle | Sneaker | T-shirt | Sandal
Coat | 2.62 | 2.58 | 1.96 | 0.99 | 1.52 | 1.03 | 1.22 | 1.02 | 0.72
| Sneaker | Ankle | Bag | Dress | Shirt | Trouser | T-shirt | Pullover | Coat
Sandal | 3.29 | 1.13 | 1.67 | 0.58 | 0.69 | 0.64 | 1.16 | 1.48 | 0.72
| T-shirt | Pullover | Coat | Dress | Bag | Trouser | Ankle | Sandal | Sneaker
Shirt | 3.63 | 3.42 | 1.96 | 1.97 | 1.36 | 1.57 | 1.41 | 0.69 | 0.41
| Sandal | Ankle | Coat | T-shirt | Trouser | Bag | Dress | Shirt | Pullover
Sneaker | 3.29 | 2.65 | 1.22 | 1.07 | 0.86 | 1.24 | 0.48 | 0.41 | 0.36
| Sandal | Dress | Shirt | Coat | Pullover | T-shirt | Ankle | Trouser | Sneaker
Bag | 1.67 | 1.32 | 1.36 | 0.99 | 1.52 | 0.69 | 1.06 | 1.56 | 1.24
| Sneaker | Sandal | Bag | Coat | Dress | Shirt | T-shirt | Trouser | Pullover
Ankle | 2.65 | 1.13 | 1.06 | 1.03 | 1.24 | 1.41 | 1.28 | 0.83 | 0.71

| bird | trunk | ship | cat | horse | deer | automobile | frog | dog
---|---|---|---|---|---|---|---|---|---
airplane | 3.43 | 3.05 | 1.70 | 1.20 | 0.81 | 1.94 | 0.81 | 1.64 | 1.43
| trunk | ship | airplane | forg | cat | horse | bird | dog | deer
automobile | 2.52 | 1.32 | 0.81 | 0.54 | 0.65 | 0.84 | 1.00 | 0.49 | 1.14
| airplane | deer | cat | dog | frog | horse | ship | automobile | trunk
bird | 3.43 | 2.83 | 2.28 | 1.47 | 1.39 | 1.45 | 1.18 | 1.00 | 0.56
| dog | bird | deer | frog | horse | airplane | ship | trunk | automobile
cat | 3.04 | 2.28 | 2.17 | 1.93 | 0.96 | 1.20 | 0.86 | 0.90 | 0.65
| bird | cat | horse | dog | frog | airplane | ship | automobile | truck
deer | 2.83 | 2.17 | 1.54 | 1.46 | 1.14 | 1.94 | 1.13 | 1.14 | 0.46
| cat | horse | bird | deer | frog | airplane | truck | ship | automobile
dog | 3.04 | 2.44 | 1.47 | 1.46 | 0.74 | 1.43 | 1.16 | 1.03 | 0.49
| cat | bird | deer | dog | airplane | ship | horse | automobile | truck
frog | 1.93 | 1.39 | 1.14 | 0.74 | 1.64 | 0.90 | 1.22 | 0.54 | 0.83
| dog | deer | cat | bird | airplane | truck | frog | ship | automobile
horse | 2.44 | 1.54 | 0.96 | 1.45 | 0.81 | 0.85 | 1.22 | 1.02 | 0.84
| airplane | automobile | truck | cat | bird | frog | deer | horse | dog
ship | 1.70 | 1.32 | 1.18 | 0.86 | 1.18 | 0.90 | 1.13 | 1.02 | 1.03
| airplane | automobile | ship | horse | cat | dog | bird | frog | deer
truck | 3.05 | 2.52 | 1.18 | 0.85 | 0.90 | 1.16 | 0.56 | 0.83 | 0.46

Figure 3 shows the similarity of each class to the other classes on Fashion
MNIST and CIFAR-10, and is sorted in descending order of similarity. In Table
4, we list in detail the name of each class in Figure 3a and Figure 3b.

In addition to the geometry similarity between the feature distributions of
_dog_ and _cat_ shown in Figure 3c, we also plot the geometry similarity
between other classes with high similarity in Figure 13. This evidence
strongly suggests that our findings are not accidental.

Figure 13: Other examples with high geometric similarity in CIFAR-10 and
geometry similarity between the same class of feature distributions extracted
by different models.

## Appendix E

In this section, we provide additional experimental results for phenomenon 3.
Features of all classes in CIFAR-10 were extracted using two ResNet-18 trained
with different initialization parameters, and then the geometry similarity
between the feature distributions of the same class extracted by different
models was calculated. All additional experimental results are plotted in
Figure 13, where it can be observed that the same class of features extracted
by the different models does not match phenomenon 2 at all.

## References

  * [1] Shaden Alshammari, Yu-Xiong Wang, Deva Ramanan, and Shu Kong.  Long-tailed recognition via weight balancing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6897–6907, 2022.
  * [2] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma.  Learning imbalanced datasets with label-distribution-aware margin loss.  Advances in neural information processing systems, 32, 2019.
  * [3] Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer.  Smote: synthetic minority over-sampling technique.  Journal of artificial intelligence research, 16:321–357, 2002.
  * [4] Hsin-Ping Chou, Shih-Chieh Chang, Jia-Yu Pan, Wei Wei, and Da-Cheng Juan.  Remix: rebalanced mixup.  In Computer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, Part VI 16, pages 95–110. Springer, 2020.
  * [5] Peng Chu, Xiao Bian, Shaopeng Liu, and Haibin Ling.  Feature space augmentation for long-tailed data.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIX 16, pages 694–710. Springer, 2020.
  * [6] Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie.  Class-balanced loss based on effective number of samples.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9268–9277, 2019.
  * [7] Charles Elkan.  The foundations of cost-sensitive learning.  In International joint conference on artificial intelligence, volume 17, pages 973–978. Lawrence Erlbaum Associates Ltd, 2001.
  * [8] Andrew Estabrooks, Taeho Jo, and Nathalie Japkowicz.  A multiple resampling method for learning from imbalanced data sets.  Computational intelligence, 20(1):18–36, 2004.
  * [9] Hui Han, Wen-Yuan Wang, and Bing-Huan Mao.  Borderline-smote: a new over-sampling method in imbalanced data sets learning.  In Advances in Intelligent Computing: International Conference on Intelligent Computing, ICIC 2005, Hefei, China, August 23-26, 2005, Proceedings, Part I 1, pages 878–887. Springer, 2005.
  * [10] Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang.  Learning deep representation for imbalanced classification.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5375–5384, 2016.
  * [11] Muhammad Abdullah Jamal, Matthew Brown, Ming-Hsuan Yang, Liqiang Wang, and Boqing Gong.  Rethinking class-balanced methods for long-tailed visual recognition from a domain adaptation perspective.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7610–7619, 2020.
  * [12] Shenwang Jiang, Jianan Li, Ying Wang, Bo Huang, Zhang Zhang, and Tingfa Xu.  Delving into sample loss curve to embrace noisy and imbalanced data.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 7024–7032, 2022.
  * [13] Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis.  Decoupling representation and classifier for long-tailed recognition.  In Eighth International Conference on Learning Representations (ICLR), 2020.
  * [14] Jaehyung Kim, Jongheon Jeong, and Jinwoo Shin.  M2m: Imbalanced classification via major-to-minor translation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13896–13905, 2020.
  * [15] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [16] Yann LeCun.  The mnist database of handwritten digits.  http://yann. lecun. com/exdb/mnist/, 1998.
  * [17] Mengke Li, Yiu-ming Cheung, and Yang Lu.  Long-tailed visual recognition via gaussian clouded logit adjustment.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6929–6938, 2022.
  * [18] Ning Li, Tao Li, Chunyu Hu, Kai Wang, and Hong Kang.  A benchmark of ocular disease intelligent recognition: One shot for multi-disease detection.  In Benchmarking, Measuring, and Optimizing: Third BenchCouncil International Symposium, Bench 2020, Virtual Event, November 15–16, 2020, Revised Selected Papers 3, pages 177–193. Springer, 2021.
  * [19] Shuang Li, Kaixiong Gong, Chi Harold Liu, Yulin Wang, Feng Qiao, and Xinjing Cheng.  Metasaug: Meta semantic augmentation for long-tailed visual recognition.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5212–5221, 2021.
  * [20] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017.
  * [21] Bo Liu, Haoxiang Li, Hao Kang, Gang Hua, and Nuno Vasconcelos.  Gistnet: a geometric structure transfer network for long-tailed recognition.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8209–8218, 2021.
  * [22] Jialun Liu, Yifan Sun, Chuchu Han, Zhaopeng Dou, and Wenhui Li.  Deep representation learning on long-tailed data: A learnable embedding augmentation perspective.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2970–2979, 2020.
  * [23] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu.  Large-scale long-tailed recognition in an open world.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2537–2546, 2019.
  * [24] Yanbiao Ma, Licheng Jiao, Fang Liu, Yuxin Li, Shuyuan Yang, and Xu Liu.  Delving into semantic scale imbalance.  In The Eleventh International Conference on Learning Representations, 2023.
  * [25] Yanbiao Ma, Licheng Jiao, Fang Liu, Shuyuan Yang, Xu Liu, and Puhua Chen.  Feature distribution representation learning based on knowledge transfer for long-tailed classification.  IEEE Transactions on Multimedia, 2023.
  * [26] Yanbiao Ma, Licheng Jiao, Fang Liu, Shuyuan Yang, Xu Liu, and Lingling Li.  Curvature-balanced feature manifold learning for long-tailed classification.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15824–15835, 2023.
  * [27] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten.  Exploring the limits of weakly supervised pretraining.  In Proceedings of the European conference on computer vision (ECCV), pages 181–196, 2018.
  * [28] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.  Distributed representations of words and phrases and their compositionality.  Advances in neural information processing systems, 26, 2013.
  * [29] R Channing Moore, Daniel PW Ellis, Eduardo Fonseca, Shawn Hershey, Aren Jansen, and Manoj Plakal.  Dataset balancing can hurt model performance.  In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023.
  * [30] Seulki Park, Youngkyu Hong, Byeongho Heo, Sangdoo Yun, and Jin Young Choi.  The majority can help the minority: Context-rich minority oversampling for long-tailed classification.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6887–6896, 2022.
  * [31] Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, et al.  Balanced meta-softmax for long-tailed visual recognition.  Advances in neural information processing systems, 33:4175–4186, 2020.
  * [32] Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun.  Learning to reweight examples for robust deep learning.  In International conference on machine learning, pages 4334–4343. PMLR, 2018.
  * [33] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  International journal of computer vision, 115:211–252, 2015.
  * [34] Saptarshi Sinha, Hiroki Ohashi, and Katsuyuki Nakamura.  Class-difficulty based methods for long-tailed visual recognition.  International Journal of Computer Vision, 130(10):2517–2531, 2022\.
  * [35] Jingru Tan, Xin Lu, Gang Zhang, Changqing Yin, and Quanquan Li.  Equalization loss v2: A new gradient balance approach for long-tailed object detection.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1685–1694, 2021.
  * [36] Jingru Tan, Changbao Wang, Buyu Li, Quanquan Li, Wanli Ouyang, Changqing Yin, and Junjie Yan.  Equalization loss for long-tailed object recognition.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11662–11671, 2020.
  * [37] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie.  The inaturalist species classification and detection dataset.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8769–8778, 2018.
  * [38] Grant Van Horn and Pietro Perona.  The devil is in the tails: Fine-grained classification in the wild.  arXiv preprint arXiv:1709.01450, 2017.
  * [39] Jianfeng Wang, Thomas Lukasiewicz, Xiaolin Hu, Jianfei Cai, and Zhenghua Xu.  Rsg: A simple but effective module for learning imbalanced datasets.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3784–3793, 2021.
  * [40] Tao Wang, Yu Li, Bingyi Kang, Junnan Li, Junhao Liew, Sheng Tang, Steven Hoi, and Jiashi Feng.  The devil is in classification: A simple framework for long-tail instance segmentation.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIV 16, pages 728–744. Springer, 2020.
  * [41] Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella X Yu.  Long-tailed recognition by routing diverse distribution-aware experts.  arXiv preprint arXiv:2010.01809, 2020.
  * [42] Yiru Wang, Weihao Gan, Jie Yang, Wei Wu, and Junjie Yan.  Dynamic curriculum learning for imbalanced data classification.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 5017–5026, 2019.
  * [43] Yu-Xiong Wang, Deva Ramanan, and Martial Hebert.  Learning to model the tail.  Advances in neural information processing systems, 30, 2017.
  * [44] Chen Wei, Kihyuk Sohn, Clayton Mellina, Alan Yuille, and Fan Yang.  Crest: A class-rebalancing self-training framework for imbalanced semi-supervised learning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10857–10866, 2021.
  * [45] Han Xiao, Kashif Rasul, and Roland Vollgraf.  Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.  arXiv preprint arXiv:1708.07747, 2017.
  * [46] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.  Aggregated residual transformations for deep neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492–1500, 2017.
  * [47] Zhengzhuo Xu, Zenghao Chai, and Chun Yuan.  Towards calibrated model for long-tailed visual recognition from prior perspective.  Advances in Neural Information Processing Systems, 34:7139–7152, 2021.
  * [48] Lu Yang, He Jiang, Qing Song, and Jun Guo.  A survey on long-tailed visual recognition.  International Journal of Computer Vision, 130(7):1837–1872, 2022\.
  * [49] Xi Yin, Xiang Yu, Kihyuk Sohn, Xiaoming Liu, and Manmohan Chandraker.  Feature transfer learning for face recognition with under-represented data.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5704–5713, 2019.
  * [50] Yuhang Zang, Chen Huang, and Chen Change Loy.  Fasa: Feature augmentation and sampling adaptation for long-tailed instance segmentation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3457–3466, 2021.
  * [51] Songyang Zhang, Zeming Li, Shipeng Yan, Xuming He, and Jian Sun.  Distribution alignment: A unified framework for long-tail visual recognition.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2361–2370, 2021.
  * [52] Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng.  Deep long-tailed learning: A survey.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023\.
  * [53] Zizhao Zhang and Tomas Pfister.  Learning fast sample re-weighting without reward data.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 725–734, 2021.
  * [54] Peilin Zhao, Yifan Zhang, Min Wu, Steven CH Hoi, Mingkui Tan, and Junzhou Huang.  Adaptive cost-sensitive online classification.  IEEE Transactions on Knowledge and Data Engineering, 31(2):214–228, 2018.
  * [55] Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia.  Improving calibration for long-tailed recognition.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16489–16498, 2021.
  * [56] Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen.  Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9719–9728, 2020.
  * [57] Yuke Zhu, Yan Bai, and Yichen Wei.  Spherical feature transform for deep metric learning.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIX 16, pages 420–436. Springer, 2020.