# Long-Tailed Classification Based on Coarse-Grained Leading Forest and Multi-
Center Loss

Jinye Yang1, Ji Xu1∗ 1 State Key Laboratory of Public Big Data, Guizhou
University, Guiyang, China<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Long-tailed(LT) classification is an unavoidable and challenging problem in
the real world. Most of the existing long-tailed classification methods focus
only on solving the inter-class imbalance in which there are more samples in
the head class than in the tail class, while ignoring the intra-lass imbalance
in which the number of samples of the head attribute within the same class is
much larger than the number of samples of the tail attribute. The deviation in
the model is caused by both of these factors, and due to the fact that
attributes are implicit in most datasets and the combination of attributes is
very complex, the intra-class imbalance is more difficult to handle. For this
purpose, we proposed a long-tailed classification framework, known as
Cognisance, which is founded on Coarse-Grained Leading Forest (CLF) and Multi-
Center Loss (MCL), aiming to build a multi-granularity joint solution model by
means of invariant feature learning. In this method, we designed an
unsupervised learning method, i.e., CLF, to better characterize the
distribution of attributes within a class. Depending on the distribution of
attributes, we can flexibly construct sampling strategies suitable for
different environments. In addition, we introduce a new metric learning loss,
i.e., MCL, which aims to gradually eliminate confusing attributes during the
feature learning process. More importantly, this approach does not depend on a
specific model structure and can be integrated with existing LT methods as an
independent component. We have conducted extensive experiments and our
approach has state-of-the-art performance in both existing benchmarks
ImageNet-GLT and MSCOCO-GLT, and can improve the performance of existing LT
methods. Our codes are available on GitHub:
https://github.com/jinyery/cognisance

###### Index Terms:

Imbalanced learning, long-tailed learning, invariant feature learning.

##  I Introduction

In real-world applications, training samples typically exhibit a long-tailed
distribution, especially for large-scale datasets [1, 2]. Long-tailed
distribution means that a small number of head categories contain a large
number of samples, while the majority of tail categories have a relatively
limited number of samples. This imbalance can lead to traditional
classification algorithms preferring to handle head categories and performing
poorly when dealing with tail categories. Solving the problem of long-tailed
classification is crucial as tail categories may contain important information
such as rare diseases, critical events, or characteristics of minority groups
[3, 4, 5, 6]. In order to effectively address this challenge, researchers have
proposed various methods. However, what this article wants to emphasize is
that the long-tailed distribution problem that truly troubles the industry is
not all the inter-class long-tailed problem which is currently the most
studied. What truly hinders the further implementation of machine learning
methods in the industry is the intra-class long-tailed problem, such as when
it comes to the long-tailed distribution in unmanned vehicle training data,
the weather distribution, day and night distribution [7], etc., are not the
targets predicted by the model. Therefore, the long-tailed here is not among
classes, but among attributes, where the attribute represents all the factors
that cause intra-class changes, including object-level attributes (such as the
specific vehicle model, brand, color, etc.) and image-level attributes (such
as lighting, climate conditions, etc.), this emerging task is named
Generalized Long-Tailed Classification (GLT) [8].

Figure 1: Inter-class long-tailed distribution and intra-class long-tailed
distribution.

In Fig. 1, it is evident that there is a significant difference in the number
of samples among different categories, especially for the head class such as
“Coast”, which has a much larger number of samples than the tail class such as
“Escalator”. However, even within a category, there is a significant
difference in the number of samples corresponding to different attributes. For
example, in the category “Coast”, the number of samples during the day is much
greater than that at night, and the number of samples on sunny days is also
much greater than that on cloudy days. The imbalance in sample size among
attributes within an category is fundamentally more difficult to avoid than
the former, as shown in Fig. 2, this imbalance in attributes undermines the
performance of the model in two ways [8]: I) Firstly, it weakens the accuracy
of images with tail attributes. For example, black swans are more likely to be
misclassified than white swans in the category “Swan”, even though they both
belong to the same category. II) It leads to some attributes being mistakenly
associated with certain categories, e.g. the attribute “White” may be falsely
correlated with “Swan”. Therefore, when “White” appears in images of other
birds (e.g., “Cock”), there is a high risk that the sample will be
misclassified as a “Swan”.

Figure 2: Spurious correlation of the “White” attribute with the “Swan”
category.

Current research mostly focuses on solving the inter-class long-tailed
problem, where resampling [9, 10] or loss reweighting [11, 12] methods are
often used when dealing with imbalanced data, aiming to rebalance the training
process. However, most of these methods sacrifice the performance of the head
class to improve the performance of the tail class, which is like playing a
performance seesaw, making it difficult to fundamentally improve the
performance of all classes. In addition, some methods believe that data
imbalance does not affect feature learning, so the training of the model is
divided into two stages through decoupling feature learning and classifier
learning [13, 14]. However, this adjustment is only a trade-off between
accuracy and precision [8, 15], and the confusion areas of similar attributes
in the feature space learned by the model do not change. As mentioned earlier
in this paper, it is precisely the long-tailed between attributes within a
category that leads to spurious correlations between the head attributes of
certain categories and that category. This means that these attributes
correspond to the spurious features (i.e., a confusing region in the feature
space) of the category. The method proposed in this paper aims to eliminate
the confusing region based on the invariant feature learning, which ultimately
achieves the goal of improving the model’s precision and accuracy at the same
time [16, 17, 18].

In the framework proposed in this paper, inter-class and intra-class long-
tailed problems are handled simultaneously by constructing different
environments, in which we design a new sampling method Coarse-Grained Leading
Forest (CLF), which is based on unsupervised learning that can characterize
the attribute distributions within a class and guide the data sampling in
different environments during the training process. In the experimental setup
of this paper, two environments are constructed, one of which is the original
environment without special treatment, and the other environment in which the
distributions of categories and attributes tend to be balanced. Finally, in
order to gradually eliminate confusing pseudo-features during the training
process, we design a new metric learning loss, Multi-Center Loss (MCL), which
is inspired by [8] and [19], extends the center loss to its Invariant Risk
Minimization (IRM) version, and further improves the robustness of the model
compared to the previous two, giving the model the ability to learn invariant
features. In addition, this method is not coupled with a specific backbone
model or loss function, and can be seamlessly integrated into other LT methods
for performance enhancement on top of the original.

Our contributions can be summarized as follows:

  * •

We designed a novel unsupervised learning-based sampling scheme to guide the
sampling of different environments in the IRM process, while dealing with the
inter-class long-tail problem as well as the intra-class long-tailed problem
among attributes, which is often neglected in traditional methods.

  * •

We combined the idea of invariant feature learning to design a new metric
learning loss, which can enable the model to gradually remove the influence of
pseudo-features during the training process, further improving the robustness
of the model and taking into account the precision and accuracy of the
prediction.

  * •

We conducted extensive experiments on two existing benchmarks, ImageNet-GLT
and MSCOCO-GLT, to validate the effectiveness of this framework while
improving the performance of all popular LT lineups.

##  II Related Work

###  II-A Long-Tailed Classification

The key challenge of long-tailed classification is to effectively deal with
the imbalance of data distribution to ensure that excellent classification
performance can be achieved both between the head and the tail. Current
treatments for long-tailed classification can be broadly categorized into
three groups [1]: 1) Class Re-balancing, which is the mainstream paradigm in
long-tailed learning, aims to enhance the influence of tail samples on the
model by means of re-sampling [20, 10, 21, 22], re-weighting [23, 24, 25, 26,
27] or logit adjustment [28] during the model training process, and some of
these methods [13, 14] consider that the unbalanced samples do not affect the
learning of the features, and thus divide feature learning and classifier
learning into two phases, and perform operations such as resampling only in
the classifier learning phase. 2) Information Augmentation, information
augmentation based approaches seek to introduce additional information in
model training in order to improve model performance in long-tailed learning.
There are two approaches in this method type: migration learning [29, 30, 31]
and data augmentation [32, 33, 34]. 3) Module Improvement, in addition to
class rebalancing and information augmentation, researchers have explored ways
to improve network modules in long-tailed learning such as RIDE [35] and TADE
[36], both of which deal with long-tail recognition independent of test
distribution by introducing integrated learning of multi-expert models in the
network.

In addition, a recent study proposed the concept of Generalized Long-Tailed
Classification (GLT) [8], which first pointed out the problem of long-tailed
of attributes within a class, which pointed out that the traditional long-
tailed classification methods represent the classification model as $p(Y|X)$,
i.e., predicting the label $Y$ from the input image $X$, which can be further
decomposed as $p(Y|X)\propto p(X|Y)\cdot p(Y)$, the formula identifies the
cause of class bias as $p(Y)$. However, the distribution of $p(X|Y)$ also
changes in different domains, so the classification model is extended to the
form of Equation 1 based on the Bayesian Theorem in this study.

$p(Y=k|z_{c},z_{a})=\frac{p(z_{c}|Y=k)}{p(z_{c})}\cdot\underbrace{\frac{p(z_{a}|Y=k,z_{c})}{p(z_{a}|z_{c})}}_{attribute\
bias}\cdot\underbrace{p(Y=k)}_{class\ bias},$ (1)

where $z_{c}$ is the invariant present in the category, and the attribute-
related variable $z_{a}$ is the domain-specific knowledge in different
distributions. Taking the mentioned “swan” as an example, the attribute
“color” of “Swan” belongs to $z_{a}$, while the attributes of “Swan” such as
feathers and shape belong to $z_{c}$, and it is worth noting that in practical
applications the formula does not impose the untangling assumption, i.e., it
does not assume that a perfect feature vector $z=[z_{c};z_{a}]$ can be
obtained, where $z_{a}$ and $z_{c}$ are separated.

###  II-B Invariant Risk Minimization

IRM [17] was proposed by Arjovsky et al. in 2019, and its main goal is to
build robust learning models that can have the same performance on different
data distributions. In machine learning, we usually hope that the trained
model can perform well on future data, which is called Risk Minimization.
However, in practice, there may be distributional differences between the
training data and the test data, known as Domain Shift, which causes the model
to perform poorly on new domains. The core idea of IRM is to solve the domain
adaptation problem by encouraging models to learn features that are invariant
across data domains. This means that the model should focus on those shared
features that are present in all data domains rather than overfitting a
particular data distribution.

$\begin{split}&\mathop{\min}\limits_{\begin{subarray}{c}\Phi:\mathcal{X}\rightarrow\mathcal{H}\\\
w:\mathcal{H}\rightarrow\mathcal{Y}\end{subarray}}\sum\limits_{e\in\varepsilon_{tr}}R^{e}(w\circ\Phi)\\\
subject\
to\quad&w\in\mathop{\arg\min}\limits_{\overline{w}:\mathcal{H}\rightarrow\mathcal{Y}}R^{e}(\overline{w}\circ\Phi),\
for\ all\ e\in\varepsilon_{tr}\end{split}$ (2)

As shown in 2, where $\varepsilon_{tr}$ represents all environments,
$\mathcal{X}$, $\mathcal{H}$, and $\mathcal{Y}$ represent inputs, feature
representations, and prediction results, respectively, $\Phi$ and $w$ are
feature learner and classifier, respectively, and $R^{e}$ denotes the risk
under the environment $e$, the goal of IRM is to find a common solution that
can perform stably on all environments, thus improving the model’s
generalization ability.

###  II-C Optimal Leading Forest

The CLF proposed in this paper starts with OLeaF [37], so we provide a brief
introduction to its ideas and algorithms here. The concept of optimal leading
forest originates from a clustering method based on density peak [38], and the
two most critical factors in the construction of OLeaF are the density of the
data points and the distance of the data points to their nearest neighbors
with higher densitiy. Let $I=\left\\{1,2,...,N\right\\}$ be the index set of
dataset $\mathcal{X}$, and $d_{i,j}$ represent the distance between data
points $\boldsymbol{x}_{i}$ and $\boldsymbol{x}_{j}$ (any distance metric can
be used), and let $\rho_{i}=\sum_{j\in
I\backslash\\{i\\}}exp(-(d_{i,j}/d_{c})^{2})$ be the density of data point
$\boldsymbol{x}_{i}$, where $d_{c}$ is the cut-off distance. If there exists
$\xi_{i}=argmin_{j}\left\\{d_{i,j}|\rho_{j}>\rho_{i}\right\\}$, then
$\boldsymbol{x}_{\xi_{i}}$ is said to be the leading node of
$\boldsymbol{x}_{i}$. Based on this a partial order relation can be
established, i.e., if there exists $\xi_{i}=\eta(\boldsymbol{x}_{i})$, then
$\boldsymbol{x}_{i}\prec\boldsymbol{x}_{\xi_{i}}$, and connecting every
$\boldsymbol{x}_{i}$ and $\boldsymbol{x}_{\xi_{i}}$ (for $r=argmax_{1\leq
i\leq N}\left\\{\rho_{i}\right\\}$, then $\boldsymbol{x}_{r}$ is the root
node) based on the partial order bias, a tree structure can be established and
is known as the Leading Tree. And then let $\delta_{i}=d_{\xi_{i},i}$,
$\gamma_{i}=\rho_{i}\times\delta_{i}$, then the larger $\gamma_{i}$ represents
the higher potential of the data point $\boldsymbol{x}_{i}$ to be selected as
the cluster center. Intuitively, if an object a has a large
$\boldsymbol{x}_{i}$, it means it has a lot of close neighbors, and
$\delta_{i}$ is large, it is far away from another data point with a large
$\rho_{i}$ value, so $\boldsymbol{x}_{i}$ has a good chance to become the
center of the cluster, and the root node can be selected based on the ordering
of $\gamma_{i}$. Then a Leading Tree will be partitioned in this way, and the
resulting multiple Leading Tree will be called an Optimal Leading Forest
(OLeaF).

##  III The proposed Cognisance framework

In the previous introduction of GLT [8], attribute bias and class bias were
introduced, and traditional LT methods tend to focus only on the latter, and
the resampling, reweighting, etc. designed based on this tend to focus only on
the distribution of classes. As a matter of fact, as the example given
earlier, attribute bias not only impairs the model’s performance on tail
attribute samples within a class, but also it is the spurious correlation
between some head attributes with pseudo-features and certain classes that
leads to the imbalance of the model’s performance among classes. Therefore, in
order to improve the generalization ability of the model on data with
different category distributions and different attribute distributions, this
paper proposes an framework named Cognisance based on invariant feature
learning, which firstly uses Coarse-Grained Leading Forest to construct
different environments with different data distributions, and then under
Multi-Center Loss, the model is allowed to learn invariant features in each
data domain instead of overfitting a certain distribution to solve the multi-
domain adaptation problem.

###  III-A Coarse-Grained Leading Forest

In the concept of IRM [17], the construction of different environments is a
prerequisite for training, and the challenge in this paper is to construct
controllable environments with different attribute distributions. Environments
with different category distributions are easy to construct because the labels
of the categories are explicit in the dataset, while the attributes are
implicit for most datasets, and even if the category imbalance is completely
eliminated, its attribute imbalance still exists. At the same time, because of
the nature that attributes can be continuously superimposed and combined, the
boundaries of attributes are also complex, thus we design a sampling method
based on unsupervised learning, which can portray the distribution of
attributes within the same category and can control the granularity of the
portraying of attributes according to the setting of hyperparameters.

Figure 3: Evolution of intra-class samples, the samples within the class have
delicate transitions.

Our motivation is based on a reasonable assumption that the differences
between samples within the same category are the result of the gradual
separation and evolution of attributes. This is somewhat similar to a
biological evolutionary tree, where the gap from the root node to the leaf
nodes doesn’t happen all at once, but is the result of constant evolution and
branching, while the evolution is not only reflected at a coarse-grained
level, but within the same category, the transition will be more subtle.

As shown in Fig. 3, even within the “human” category, the samples between
different ages are clustered and transition gradually. Of course, in a
manually collected dataset, not all changes within a class may be gradual, and
if the differences among more granular groups within a class are too large,
then the class cannot be portrayed by a single tree. In other words, when
there is a common ancestor node, the samples within a class can be viewed as a
whole with the characteristics of streaming transformation, but when the
ancestor node is removed, the gap between two sub-branches will be too large,
which should actually be portrayed with two or more trees instead of forcibly
connected together. Based on the above analysis, we designed a new clustering
algorithm Coarse-Grained Leading Forest in combination with the construction
of OLeaF [37], and its construction process is shown in Algorithm 1. Firstly,
the distance matrix between the sample points in the dataset X is computed
using an arbitrary distance metric, and then the density of each sample point
is computed, where the density of sample point $i$ is computed as shown in
Equation 3:

$\rho_{i}=\sum_{j\in\textbf{I}\backslash\\{i\\}\backslash\textbf{O}_{i}}exp\Big{(}-(d_{i,j}/d_{max})^{2}\Big{)},$
(3)

where $\boldsymbol{I}=\left\\{1,2,...,N\right\\}$ is the index set of dataset
$\boldsymbol{X}$, $\boldsymbol{O}_{i}$ is the set of nodes whose distance from
node $i$ exceeds $d_{max}$, $d_{i,j}$ is the distance between node $i$ and
node $j$, and $d_{max}$ is the cut-off distance. Next sort $\boldsymbol{I}$ in
descending order according to the density value, denoted as $\boldsymbol{S}$,
i.e., $\boldsymbol{S}_{i}$ is the index of the data point with the $i$-th
largest density value. The next step is to perform small-scale clustering on
the points in $\boldsymbol{S}$ sequentially, which relies on the
hyperparameter $d_{min}$ , and if the data point $\boldsymbol{S}_{i}$ has not
yet been merged into a coarse-grained node, then the data point
$\boldsymbol{S}_{i}$ and the points within $d_{min}$ distance from it are
formed into a coarse-grained node, i.e.:

$\boldsymbol{C}_{mem}=\\{\boldsymbol{S}_{i}\\}\cup\boldsymbol{K}\ \backslash\
\boldsymbol{A},\quad\boldsymbol{S}_{i}\notin\boldsymbol{A},$ (4)

where $\boldsymbol{C}_{mem}$ are members of the newly generated coarse-grained
node, $\boldsymbol{S}_{i}$ will be a agent node for that coarse-grained node,
$\boldsymbol{K}$ is the set of nodes that are within $d_{min}$ from
$\boldsymbol{S}_{i}$, i.e., $\boldsymbol{K}=\\{j\ |\ j\in\boldsymbol{I},\
d_{S_{i},j}<d_{min}\\}$, and $\boldsymbol{A}$ is the set of visited nodes,
i.e., the set of nodes that have already been merged into a particular coarse-
grained node. Note that if $\boldsymbol{S}_{i}$ itself is already in
$\boldsymbol{A}$ then it skips the creation of that new coarse-grained node
and continues to process the next node in $\boldsymbol{S}$.

Input: All training samples X of a given category.

Output: A Coarse-Grained Leading Forest clf for a given category.

Parameter : $d_{min}$ is the radius of the coarse-grained node, $d_{max}$ is
the cut-off distance.

1

2dist = calculate_distance($X$);

3 density = calculate_density(dist, $d_{\text{max}}$);

4 density_argsort = argsort(density, descend=True);

// Record the index of samples that have been visited

5 accessed = initial_vector(length(density), False);

6

7for  _$i$ in range(length(density))_ do

8 node_idx = density_argsort[i];

9 if  _accessed[node_idx]_ then

10 continue;

11

12 end if

// Combine points that are within $d_{min}$ and have not been visited into one
coarse-grained node.

13 c_members = where(accessed == False and dist[node_idx] $\leq
d_{\text{min}}$);

14 accessed[c_members] = True;

15 coarse_node = CoarseNode(id=Autoincrement, agent=node_idx,
members=c_members);

// A node becomes a root node if no denser leading node can be found within
$d_{max}$.

16 leader_node = find_leader(node_idx, $d_{\text{max}}$);

17 if  _leader_node is not null_ then

18 coarse_node.leader = leader_node;

19

20 end if

21 else

22 clf.root.append(coarse_node);

23

24 end if

25

26 end for

27

28return clf;

Algorithm 1 Construction of CLF

The next step is to find the leading node of the newly constructed coarse-
grained node. Here, the problem is transformed into finding the leading node
of the agent node $\boldsymbol{S}_{i}$ of the coarse-grained node, denoted as
$l_{i}$, and then the coarse-grained node where $l_{i}$ is located is used as
the leading node of the current coarse-grained node. The process of finding
the leading node $l_{i}$ of $\boldsymbol{S}_{i}$ can be referred to Equation
5, where $\boldsymbol{O}_{S_{i}}$ is the set of nodes whose distance from
$\boldsymbol{S}_{i}$ exceeds $d_{max}$:

$l_{S_{i}}=argmin_{j}\left\\{d_{S_{i},j}|\rho_{j}>\rho_{S_{i}}\right\\},j\in{\boldsymbol{I}\backslash\\{\boldsymbol{S}_{i}}\\}\backslash\boldsymbol{O}_{S_{i}},$
(5)

note that $l_{S_{i}}$ may not exist, and when $l_{S_{i}}$ is not found, the
coarse-grained node where $\boldsymbol{S}_{i}$ is located automatically
becomes a root in the whole coarse-grained leading forest. Also since the
density of nodes in $\boldsymbol{S}$ is decreasing, when $l_{S_{i}}$ is found,
it must have been processed and already merged into a coarse-grained node
since $l_{S_{i}}$ is denser.

###  III-B Sampling based on CLF

By constructing the CLF we can simulate the portraying of attribute
distributions within a category, and by adjusting the hyperparameters
$d_{min}$ and $d_{max}$, we can control the fineness of the portraying. Next,
we can construct the environment of different attribute distributions based on
CLF, as shown in Fig. 4, because each branch represents a new direction of
attribute evolution, and we categorize all the data points on each path from
the root node to the leaf nodes as members of a certain attribute branch.

Figure 4: The left is an example of CLF constructed for category “sand”, while
the right is an example of attribute splitting using CLF. Each path from the
root node to the leaf nodes can be considered as an attribute, and the samples
within the coarse-grained nodes are extremely similar, requiring an
appropriate reduction in sampling weights. In addition the samples within the
red and pink boxes demonstrate the potential of this method for noisy
recognition.

When we obtain the samples of each attribute, we can refer to the idea of
inter-class resampling [13] to resample on the attributes, as shown in
Equation 6, when inter-class sampling is performed, $j$ represents the class
index, $n_{j}$ represents the number of samples of class $j$, and $C$
represents the total number of classes, where $q\in[0,1]$, and inter-class
balanced sampling is performed when $q=0$.

$p_{j}=\frac{n_{j}^{q}}{\sum_{i=1}^{C}n_{i}^{q}}$ (6)

As shown in Algorithm 2, when sampling different attributes within a category,
the idea is the same as inter-class sampling, only need to regard $j$ as the
attribute index, $n_{j}$ as the number of samples of attribute $j$, and $C$ as
the total number of attributes. In addition, it should be noted that in intra-
class sampling, unlike inter-class sampling, the same sample may have more
than one attribute (e.g., the root node appears in all branches of the same
tree, so it has all the attributes that this tree can represent), so the
weight of such a sample point needs to be penalized to some extent. At the
same time, due to the concept of coarse-grained node in our algorithm, as
shown in Fig. 4, the variance of the members in the coarse-grained nodes is
extremely small, i.e., the representativeness of the sample decreases, and the
value of the information provided decreases, so the sampling weight of the
members in the coarse-grained nodes should be appropriately reduced. In this
paper, we use the method where all members within a CoarseNode share the
weight of that CoarseNode equally.

Input: A Coarse-Grained Leading Forest clf, balance factor q

Output: The sampling probability of each sample within a certain category
weights

// Initialize an array with 0 to record the sampling weights for each sample.

1 weights = initial_vector(default=0);

// Generate the path from the root node to the leaf node based on clf.

2 paths = generate_path(clf);

// Get the number of times each node appears in all paths.

3 repetitions = get_repetition(paths);

// Calculate the probability of each path being sampled based on q.

4 path_weight = get_path_weight(paths, q);

5 foreach  _path in paths_ do

// CoarsedNodes in the same path have the same sampling weight.

6 coarse_node_weight = path_weight / $\text{len}(\textit{path})$;

7 foreach  _node in path_ do

// All members within a CoarsedNode share the sampling weight of that node
equally.

8 small_node_weight = coarse_node_weight /
$\text{len}(\textit{node.members})$;

9 weights[node.members] $+=$ small_node_weight;

10

11 end foreach

12

13 end foreach

// Apply some penalty to nodes that have appeared in more than one path.

14 weights $/=\textit{repetitions}$;

15 return weights;

Algorithm 2 Sampling according to CLF

Taking the trunk tree in CLF in Fig. 4 as an example (ignoring isolated
samples for demonstration purposes), we will provide a detailed introduction
to attribute balanced sampling. Firstly, different attributes can be separated
based on CLF, as shown in Fig. 4, each path from the root node to the leaf
node can be considered as an attribute, which means that if you want to evenly
sample the samples of each attribute, you only need to assign equal sampling
weights to each attribute. Where for nodes that are repeated in multiple
paths, we simply divide their sampling weight by the number of repetitions as
a penalty. Take the example of calculating the weight of the root node,
firstly since there are three attribute groups, so $weight=\frac{1}{3}$, and
then the node’s weight in the three attribute groups are $\frac{1}{5}$,
$\frac{1}{4}$, and $\frac{1}{5}$, so the root node’s sampling weights globally
are $\frac{1}{15}$, $\frac{1}{12}$, and $\frac{1}{15}$, and summing up the
three weights yields $weight=\frac{13}{60}$, and finally an appropriate
penalty is to be applied, i.e.,
$weight=\frac{weight}{repetition}=\frac{13}{180}$. Furthermore, for a
CoarseNode containing multiple samples, the penalty is similar, setting the
sampling weight of each sample to
$weight=CoarseNode.weight/CoarseNode.length$. Taking the second coarse-grained
node in Attribute 2 as an example, the sampling weight of this node is
$weight=(\frac{1}{3}\times\frac{1}{4}+\frac{1}{3}\times\frac{1}{5})/2=\frac{3}{40}$,
and the sampling weight of each sample in this coarse-grained node is
$weight=\frac{3/40}{2}=\frac{3}{80}$.

###  III-C Multi-Center Loss

In the previous step multiple training environments for invariant feature
learning were constructed, and the next step is to train using the objective
function of IRM [17], however, since the original IRM loss has convergence
problems in realistic datasets, we designed a new goal Multi-Center Loss based
on the idea of IRM and the center loss in IFL [8], which can be formulated as
the following optimization problem:

$\begin{split}&\mathop{\min}\limits_{\theta,\
\omega}\sum\limits_{e\in\varepsilon}\sum\limits_{i\in e}L_{cls}(f(x_{i}^{e};\
\theta),\ y_{i}^{e};\ \omega)\\\
\mathrm{s.t.}&\quad\theta\in\mathop{\arg\min}\limits_{\Theta}\sum\limits_{e\in\varepsilon}\sum\limits_{i\in
e}{\|f(x_{i}^{e};\ \Theta)-C(x_{i})\|}_{2},\end{split}$ (7)

where $\Theta$ and $\omega$ are the learnable parameters of the backbone and
classifier, respectively, $x_{i}^{e}$ and $y_{i}^{e}$ are the $i$-th instance
in environment $e$ and its label, respectively, $\varepsilon$ is all the
training environments, $f(x_{i}^{e};\ \theta)$ is the feature extracted by the
backbone from $x_{i}^{e}$, $L_{cls}(f(x_{i}^{e};\ \theta),\ y_{i}^{e};\
\omega)$ is the classification loss under environment $e$ (arbitrary loss
function), and $C(x_{i})$ is the center to which $x_{i}^{e}$ belongs in all
environments $\varepsilon$. Note that the number of centers in each category
$n_{c_{y_{i}}}\geq 1$, and the center to which $x_{i}^{e}$ belongs depends on
which tree in the CLF of that category $x_{i}^{e}$ is located, i.e.,
$n_{c_{y_{i}}}=n_{t_{y_{i}}}$, where $n_{t_{y_{i}}}$ is the number of trees in
the CLF constructed from all samples of that category, and the initial value
of $C(x_{i})$ is the value of the agent of the root node of the tree where
$x_{i}^{e}$ is located. The practical version of this optimization problem is
shown in Equation 8, where $L_{IFL}={\|f(x_{i}^{e};\ \theta)-C(x_{i})\|}_{2}$
is the constraint loss for invariant feature learning and $\alpha$ is the
trade-off parameter:

$\begin{split}\mathop{\min}\limits_{\theta,\
\omega}\sum\limits_{e\in\varepsilon}\sum\limits_{i\in
e}L_{mc}&=\mathop{\min}\limits_{\theta,\
\omega}\sum\limits_{e\in\varepsilon}\sum\limits_{i\in e}L_{cls}+\alpha\cdot
L_{IFL}\\\ &=\mathop{\min}\limits_{\theta,\
\omega}\sum\limits_{e\in\varepsilon}\sum\limits_{i\in e}L_{cls}(f(x_{i}^{e};\
\theta),\ y_{i}^{e};\ \omega)\\\ &\quad\ +\alpha\cdot{\|f(x_{i}^{e};\
\theta)-C(x_{i})\|}_{2},\end{split}$ (8)

this loss is the IRM version of center loss, on the other hand, relative to
the original version of center loss increases the robustness, as in the
previous introduction of CLF, in some artificial datasets, even within the
same category there may be a situation where the gap between the samples is
extremely large, i.e., the number of trees in CLF is greater than 1, as shown
in Fig. 5, in category “garage”, it can actually be divided into three
categories: “Outside the garage”, “Inside the garage with car”, and “Inside
the garage without car”, and the features of these three subcategories vary
greatly. If only one center is used, making each category’s features gradually
approach one center during the training process will actually damage the
learning of features, and this is the starting point for using Multi-Center.

Figure 5: Multiple trees in the CLF of category “garage” (only part of it is
shown), there are multiple subclasses in the category “garage”, e.g., outside
the garage, inside the garage (parked cars), and inside the garage (no parked
cars), and there is a huge disparity among these subclasses.

###  III-D Overall Framework

The overall framework of this scheme is shown in Fig. 6. Each environment has
a pair of $(q_{cls},q_{attr})$, with the former being the balance factor for
inter-class sampling and the latter being the balance factor for intra-class
sampling. Different environments can be constructed through different
$(q_{cls},q_{attr})$ pairs, and model parameters can be shared for training
under the constraint of MCL. The portraying of attribute distribution during
sampling is achieved through CLF, while the number of centers in MCL is
determined by the number of trees in CLF of the corresponding category.

Figure 6: Overall framework diagram, where different environments have
different sampling strategies, where $q_{cls}$ and $q_{attr}$ are balancing
factors for inter-class sampling and intra-class sampling, respectively.

The algorithm process of the overall framework is shown in Algorithm 3, which
is divided into two stages: firstly, since the initial features of the samples
need to be used for clustering when constructing CLF, M-round normal sampling
training is required to obtain an initial model with imperfect predictions;
Next, the initial features are used to construct the CLF, and different
environments are constructed through the CLF and different balance factor
pairs. For example, in the experimental phase, this article sets up two
environments with balance factor pairs of
$(q_{cls}^{e_{1}}=1,q_{attr}^{e_{1}}=1)$ and
$(q_{cls}^{e_{2}}=0,q_{attr}^{e_{2}}=0)$, where the former is a normal i.i.d.
sampling environment and the latter is a balanced sampling environment for
both categories and attributes. Then, the feature learner is continuously
updated to further update the center in the CLF and MCL. Of course, the number
of epoch steps for executing updates in the second stage can be adjusted,
rather than being fixed to one update per epoch.

Input: the original training set $\\{(x,y)\\}$; balance parameter pairs
$\\{(q_{cls}^{e},q_{attr}^{e})\\}$ for different environments.

Output: backbone $f(\cdot;\theta)$, classifier $g(\cdot;w)$

1 Initialize: backbone $f(\cdot;\theta)$, classifier $g(\cdot;w)$;

2 for  _$M$ warm-up epochs_ do

3 // optimizing the model from cross-entropy classification loss;

4 $\theta,w\leftarrow\hat{\theta},\hat{w}\in\text{arg
min}_{\theta,w}L_{cls}(g(f(x;\theta);w),y)$;

5

6 end for

7$\\{F_{y}\\}=\text{ClfConstruct}(\\{(x,y)\\},\theta,w)$;

8
$\\{e_{n}\\}=\text{EnvConstruct}(\\{(q_{cls}^{e},q_{attr}^{e})\\},\\{F_{y}\\})$;

9 for  _$N$ epochs_ do

10 $\\{(x^{e},y^{e})\\}=\text{DataLoader}(\\{e_{n}\\})$;

11 $\\{C_{y}\\}=\text{ReadCenters}(\\{F_{y}\\})$;

12 $\theta,w\leftarrow\hat{\theta},\hat{w}\in\text{arg
min}_{\theta,w}L_{cls}(g(f(x;\theta);w),y)+\alpha\cdot
L_{mc}(f(x;\theta),\\{C_{y}\\})$;

13 $\\{F_{y}\\}=\text{ClfConstruct}(\\{(x,y)\\},\theta,w)$;

14
$\\{e_{n}\\}=\text{EnvConstruct}(\\{(q_{cls}^{e},q_{attr}^{e})\\},\\{F_{y}\\})$;

15

16 end for

Algorithm 3 The overall of the proposed Cognisance algorithm

##  IV Experiment

TABLE I: Evaluation of CLT and GLT Protocols on ImageNet-GLT

Protocols width 1pt |  Class-Wise Long Tail (CLT) Protocol width 1pt | Generalized Long Tail (GLT) Protocol
---|---|---
$\langle$Accuracy $|$ Precision$\rangle$ width 1pt | $Many_{C}$ | $Medium_{C}$ | $Few_{C}$ |  Overall width 1pt | $Many_{C}$ | $Medium_{C}$ | $Few_{C}$ | Overall
Re-balance | Baseline width 1pt | $58.39|38.35$ | $36|52.15$ | $13.98|55.34$ |  $41.65|47.11$ width 1pt | $50.7|32.5$ | $27.8|43.99$ | $10.18|47.7$ | $34.32|39.95$
BBN width 1pt | $62.66|43.45$ | $44.36|55.44$ | $14.66|57.57$ |  $47.22|50.96$ width 1pt | $53.37|36.45$ | $34.97|46.81$ | $10.73|46.08$ | $38.7|42.56$
BLSoftmax width 1pt | $54.03|47.07$ | $41.65|46.83$ | $28.37|37.58$ |  $44.61|45.54$ width 1pt | $46.45|40.45$ | $32.67|38.59$ | $21.38|29.57$ | $36.49|37.98$
Logit-Adj width 1pt | $53.3|49.05$ | $43.49|44.12$ | $31.86|35$ |  $45.67|44.72$ width 1pt | $45.43|41.75$ | $34.2|35.9$ | $24.58|27.78$ | $37.25|37.02$
GLTv1 width 1pt | $62.51|42.32$ | $38.69|57.23$ | $17.41|\textbf{65}$ |  $45.03|52.43$ width 1pt | $54.4|36.23$ | $30.18|49.61$ | $12.64|55.47$ | $37.24|45.14$
* Baseline + Cognisance width 1pt | $\textbf{62.84}|42.82$ | $39.62|57.25$ | $18.61|61.55$ |  $45.76|52.12$ width 1pt | $\textbf{54.53}|36.47$ | $31.3|49.99$ | $13.8|\textbf{58.27}$ | $37.97|45.82$
* BLSoftmax + Cognisance width 1pt | $58.37|53.88$ | $\textbf{44.98}|51.97$ | $33.82|39.05$ |  $\textbf{48.66}|50.8$ width 1pt | $49.92|46.89$ | $\textbf{36.11}|44.31$ | $26.14|30.03$ | $\textbf{40.14}|43.2$
* Logit-Adj + Cognisance width 1pt | $43.3|\textbf{75.13}$ | $41.19|\textbf{59.08}$ | $\textbf{45.43}|27.11$ |  $42.92|\textbf{60.7}$ width 1pt | $36.01|\textbf{70.18}$ | $32.6|\textbf{52.48}$ | $\textbf{36.26}|21.76$ | $34.51|\textbf{54.95}$
Augment | Mixup width 1pt | $60.14|38.02$ | $31.46|56.67$ | $7.59|32.82$ |  $39.35|45.63$ width 1pt | $51.68|32.21$ | $23.87|48.25$ | $5.47|28.27$ | $32.23|38.84$
RandAug width 1pt | $64.14|42.23$ | $40.1|58.27$ | $14.96|59.51$ |  $45.94|52.04$ width 1pt | $55.7|35.87$ | $31.61|50.15$ | $10.2|47.29$ | $38.03|44.01$
* Mixup + Cognisance width 1pt | $67.86|47.9$ | $45.5|62.76$ | $24.98|\textbf{67.56}$ |  $51.37|57.54$ width 1pt | $59.28|40.95$ | $36.16|54.09$ | $17.63|57.72$ | $42.63|49.38$
* RandAug + Cognisance width 1pt | $\textbf{69.12}|\textbf{49.31}$ | $\textbf{47.64}|\textbf{63.08}$ | $\textbf{26.97}|67.01$ |  $\textbf{53.13}|\textbf{58.16}$ width 1pt | $\textbf{60.85}|\textbf{42.3}$ | $\textbf{38.46}|\textbf{55.21}$ | $\textbf{19.9}|\textbf{60.11}$ | $\textbf{44.63}|\textbf{50.78}$
Ensemble | TADE width 1pt | $57.3|\textbf{55.22}$ | $46.85|50.29$ | $\textbf{34.69}|37.93$ |  $49.21|50.41$ width 1pt | $49.61|\textbf{48.19}$ | $37.55|42.59$ | $\textbf{27.52}|32.21$ | $40.87|43.28$
RIDE width 1pt | $63.18|51.44$ | $47.67|52.55$ | $29.91|47.38$ |  $51.21|51.33$ width 1pt | $54.83|44.02$ | $38.25|44.2$ | $22.77|38.12$ | $42.56|43.21$
* TADE + Cognisance width 1pt | $60.69|55.15$ | $48.15|52.22$ | $33.38|42.12$ |  $50.95|51.88$ width 1pt | $52.77|48.15$ | $39.09|44.06$ | $26.51|33.32$ | $42.68|44.08$
* RIDE + Cognisance width 1pt | $\textbf{64.83}|54.6$ | $\textbf{50.95}|\textbf{56.21}$ | $33.28|\textbf{50.15}$ |  $\textbf{53.85}|\textbf{54.66}$ width 1pt | $\textbf{57.1}|47.63$ | $\textbf{42}|\textbf{48.42}$ | $25.5|\textbf{41.23}$ | $\textbf{45.57}|{47.03}$

TABLE II: Evaluation of CLT and GLT Protocols on MSCOCO-GLT

Protocols width 1pt |  Class-Wise Long Tail (CLT) Protocol width 1pt | Generalized Long Tail (GLT) Protocol
---|---|---
$\langle$Accuracy $|$ Precision$\rangle$ width 1pt | $Many_{C}$ | $Medium_{C}$ | $Few_{C}$ |  Overall width 1pt | $Many_{C}$ | $Medium_{C}$ | $Few_{C}$ | Overall
Re-balance | Baseline width 1pt | $81.27|71.08$ | $74.13|76.06$ | $50.17|85.61$ |  $71.88|76.15$ width 1pt | $74.59|64.82$ | $66.25|69.08$ | $35.75|77.22$ | $63.1|69.14$
BBN width 1pt | $83.59|70.21$ | $76.13|76.3$ | $47.25|\textbf{90.98}$ |  $72.98|77.02$ width 1pt | $75.36|61.97$ | $68.29|68.61$ | $31.75|\textbf{81.38}$ | $63.41|68.73$
BLSoftmax width 1pt | $80.77|71.87$ | $75.67|69.94$ | $45.25|90.08$ |  $71.31|74.84$ width 1pt | $73.23|65.13$ | $68.63|62.64$ | $32.08|79.6$ | $62.81|67.09$
Logit-Adj width 1pt | $81.55|73.95$ | $76|75.55$ | $\textbf{60.83}|82.04$ |  $74.97|76.28$ width 1pt | $73.64|66.98$ | $68.71|67.61$ | $\textbf{46.58}|70.69$ | $66|68.01$
GLTv1 width 1pt | $82.45|73.09$ | $76.42|79.53$ | $55.58|86.87$ |  $74.4|78.6$ width 1pt | $76.41|67.07$ | $67.33|70.54$ | $39.75|81.02$ | $65.07|71.39$
* Baseline + Cognisance width 1pt | $\textbf{83}|73.95$ | $77.88|\textbf{80.48}$ | $55.92|90.08$ |  $75.28|\textbf{79.99}$ width 1pt | $\textbf{76.91}|68.24$ | $67.5|\textbf{71.41}$ | $39.67|80.33$ | $65.31|\textbf{72.05}$
* BLSoftmax + Cognisance width 1pt | $83|74.17$ | $78.37|76.24$ | $53.92|86.51$ |  $75.07|77.58$ width 1pt | $75.86|67.98$ | $70.46|68.14$ | $40.08|80.26$ | $66.22|70.59$
* Logit-Adj + Cognisance width 1pt | $82.68|\textbf{77.16}$ | $\textbf{80.17}|75.45$ | $59.17|83.54$ |  $\textbf{76.78}|77.77$ width 1pt | $75.14|\textbf{70.56}$ | $\textbf{71.54}|66.34$ | $45.83|73.9$ | $\textbf{67.59}|69.5$
Augment | Mixup width 1pt | $82.41|72.95$ | $73.12|79.53$ | $54.33|87.47$ |  $72.76|78.68$ width 1pt | $74.64|66.43$ | $65.33|71$ | $36|78.11$ | $62.79|70.74$
RandAug width 1pt | $84.23|74.33$ | $77.42|79.28$ | $56.33|87.07$ |  $75.64|79.01$ width 1pt | $77.91|68.56$ | $69.67|70.95$ | $39.58|78.43$ | $66.57|71.59$
* Mixup + Cognisance width 1pt | $85|76.96$ | $81.42|83.19$ | $\textbf{63.33}|88.89$ |  $79.03|82.01$ width 1pt | $79.23|\textbf{70.89}$ | $72.25|\textbf{74.84}$ | $\textbf{46.5}|80.62$ | $\textbf{69.57}|\textbf{74.54}$
* RandAug + Cognisance width 1pt | $\textbf{86.55}|\textbf{78.31}$ | $81.5|82.82$ | $62.42|\textbf{90.29}$ |  $\textbf{79.47}|\textbf{82.65}$ width 1pt | $\textbf{79.05}|69.93$ | $\textbf{72.92}|74.51$ | $44.33|\textbf{81.82}$ | $69.33|74.28$
Ensemble | TADE width 1pt | $83.55|76.47$ | $80.79|73.89$ | $50.5|89.34$ |  $75.57|78.06$ width 1pt | $77.18|68.62$ | $71.58|65.63$ | $33.5|82.32$ | $65.83|70.22$
RIDE width 1pt | $84.23|77.56$ | $82.29|77.79$ | $58.42|89.69$ |  $78.09|80.16$ width 1pt | $77.45|71.2$ | $75.17|68.67$ | $41.25|83.33$ | $69.02|72.66$
* TADE + Cognisance width 1pt | $85.59|77.48$ | $82.25|77.85$ | $55.08|90.4$ |  $77.9|80.31$ width 1pt | $79|70.54$ | $73.96|68.94$ | $40.67|\textbf{83.65}$ | $68.98|72.59$
* RIDE + Cognisance width 1pt | $\textbf{86.27}|\textbf{79.64}$ | $\textbf{83.13}|\textbf{79.89}$ | $\textbf{63.5}|\textbf{90.41}$ |  $\textbf{80.26}|\textbf{81.98}$ width 1pt | $\textbf{80.09}|\textbf{72.77}$ | $\textbf{75.17}|\textbf{71.45}$ | $\textbf{47.83}|83.03$ | $\textbf{71.38}|\textbf{74.34}$

###  IV-A Evaluation Protocols

Before conducting the experiment, it is necessary to first introduce two new
evaluation protocols: CLT Protocol and GLT Protocol, both of which were
proposed in the first baseline of GLT [8]:

  * •

Class wise Long Tail (CLT) Protocol: The samples in the training set follow a
long-tailed distribution, which means that they are normally sampled from the
LT dataset, while the samples in the test set are balanced by class. Note that
the issue of attribute distribution is not considered in CLT, as the training
and testing sets of CLT have the same attribute distribution and different
class distributions, so the effectiveness of category long tail classification
can be evaluated.

  * •

Generalized Long Tail (GLT) Protocol: Compared with the former, the difference
in attribute distribution is taken into account, that is, the attribute bias
in Equation 1 is introduced. The training set in GLT is the same as that in
CLT and conforms to the LT distribution, while the attribute distribution in
the test set tends to be balanced. Since the train set and test set in GLT
have different attribute distributions and different class distributions, it
is possible to evaluate the model’s ability to handle both inter-class long-
tailed classification and intra-class long-tailed classification.

###  IV-B Datasets and Metrics

In total, we evaluated and compared the LT methods on two benchmarks, MSCOCO-
GLT, and ImageNet-GLT, which are proposed in the first baseline of GLT [8].

ImageNet-GLT is a long-tailed subset of ImageNet [39], where the train set
contains 113k samples of 1k classes, where the number of each class ranges
from 570 to 4. The number of test set in both CLT and GLT protocols is 60k,
and the test set are divided into three subsets according to the following
class frequencies: #sample > 100 for $Many_{C}$, 100 $\geq$ #sample $\geq$ 20
for $Medium_{C}$, and #sample <20 for $Few_{C}$. Note that in constructing the
test set for attribute balancing in this dataset, the images in each category
were simply clustered into 6 groups using KMeans, and then 10 images were
sampled for each group in each category.

MSCOCO-GLT is a long-tailed subset of MSCOCO-Attribute [40], a dataset
explicitly labeled with 196 different attributes, where each object with
multiple labels is cropped as a separate image. The train set contains 144k
samples from 29 classes, where the number of each class ranges from 61k to
0.3k, and the number of test set is 5.8k in both the CLT and GLT protocols,
and the test set are divided into three subsets according to the following
category frequencies: $Index_{C}$ $\leq$ 10 for $Many_{C}$ , 22 $\geq$
$Index_{C}$ $\geq$ 10 for $Medium_{C}$ and $Index_{C}$ > 22 for $Few_{C}$ ,
where $Index_{C}$ is the index of the category in ascending order.

Evaluation Metrics. In our experiments in this paper we use two metrics to
evaluate the performance of the methods: 1) Accuracy:
$\frac{\\#CorrectPredictions}{\\#AllSamples}$, which is the Top-1 Accuracy
that has been used in traditional long-tailed methods; 2) Precision:
$\frac{1}{\\#class}\cdot\sum_{class}\frac{\\#CorrectPredictions}{\\#SamplesPredictedAsThisClass}$,
the reason why this metric is introduced is to better reveal the precision-
accuracy trade-off problem [8] that has not been paid attention to in
traditional inter-class long-tailed methods.

###  IV-C Comparisons with LT Line-up

This scheme handles both the inter-class long-tailed problem and the intra-
class long-tailed problem by eliminating the false correlation caused by
attributes, and this scheme can be seamlessly combined with other LT methods.
In the following comparison experiments we follow the classification of
current long-tailed research in [1] and [8], and classify current long-tailed
methods into three categories: 1) Class Re-balancing, 2) Information
Augmentation, and 3) Module Improvement, and take two effective methods from
each of these three categories for comparison and enhancement, in which we
chose two methods, BLSoftmax [23] and Logit-Adj [6], in the Class Re-balancing
category. For Information Augmentation we chose two methods, Mixup [34] and
RandAug [32], and for Module Improvement category we chose two methods, RIDE
[35] and TADE [36], which adopt the idea of ensemble learning and are both
current SOTA methods. In addition, we have included the first strong baseline
GLTv1 [8] in the GLT domain as a comparison, as shown in Table I and Table II,
where the methods with an asterisk are the ones that add our component, and
the bolded numbers are the optimal results in the category of the method, and
it can be seen that our proposed method gets the best results in all the
classifications, especially when combined with the method RandAug or the
method RIDE got the best results in almost all evaluation metrics in both
datasets.

In addition, this method aims to deal with both intra-class long-tailed and
inter-class long-tailed, although the starting point of this method is to
solve the problem of attribute long-tailed within the class, but it also meets
the challenge of inter-class long-tailed by eliminating the false correlation
caused by long-tailed attributes [8]. From Fig. 7, we can see more clearly the
improvements that Cognisance achieves over the existing LT approachs on the
two protocols of the two benchmarks, and we can see that the performance of
all the methods from CLT to GLT has been degraded, which also shows that the
long-tailed problem is not purely inter-class long-tailed, while intra-class
long-tailed is much more challenging, but Cognisance can still successfully
improve all existing popular LT methods.

(a) CLT protocol, ImageNet-GLT (b) GLT protocol, ImageNet-GLT (c) CLT
protocol, MSCOCO-GLT (d) GLT protocol, MSCOCO-GLT

Figure 7: Cognisance enhancements to existing LT methods

Finally, Table III records the experimental results of various methods on the
test set of long-tailed distribution, which is consistent with the
distribution of the training set, and we can see that compared with other
methods, our methods can still achieve the best results on all evaluation
metrics.

TABLE III: Evaluation of ImageNet-GLT and MSCOCO-GLT on LT test set

Benchmarks width 1pt |  ImageNet-GLT width 1pt | MSCOCO-GLT
---|---|---
Overall Evaluation width 1pt | $Acc$ | $Prec$ |  $F1$ width 1pt | $Acc$ | $Prec$ | $F1$
Re-balance | Baseline width 1pt | 53.93 | 44.46 | 48.74 width 1pt | 85.74 | 79.98 | 82.76
BBN width 1pt | 58.6 | 48.9 | 53.32 width 1pt | 84.84 | 78.04 | 81.3
BLSoftmax width 1pt | 51.73 | 41.97 | 46.34 width 1pt | 83.69 | 73.81 | 78.44
Logit-Adj width 1pt | 50.94 | 40.58 | 45.17 width 1pt | 85.28 | 73.6 | 79.01
GLTv1 width 1pt | 58.28 | 50.43 | 54.07 width 1pt | 86.67 | 81.88 | 84.21
* Baseline + Cognisance width 1pt | 58.85 | 50.98 |  54.64 width 1pt | 86.86 | 81.24 | 83.96
* BLSoftmax + Cognisance width 1pt | 56.24 | 47.58 | 51.55 width 1pt | 86.55 | 78.48 | 82.32
* Logit-Adj + Cognisance width 1pt | 43.61 | 55.15 | 48.71 width 1pt | 85.52 | 73.36 | 78.98
Augment | Mixup width 1pt | 55.25 | 46.76 | 50.66 width 1pt | 86.95 | 82 | 84.4
RandAug width 1pt | 59.88 | 51.28 | 55.25 width 1pt | 87.64 | 81.02 | 84.2
* Mixup + Cognisance width 1pt | 64.09 | 56.57 | 60.09 width 1pt | 88.81 | 81.55 | 85.03
* RandAug + Cognisance width 1pt | 65.15 | 56.85 |  60.72 width 1pt | 89.12 | 84.02 | 86.49
Ensemble | TADE width 1pt | 55.21 | 46.52 | 50.49 width 1pt | 86.57 | 79.91 | 83.1
RIDE width 1pt | 60.17 | 48.28 | 53.58 width 1pt | 88.03 | 81.79 | 84.8
* TADE + Cognisance width 1pt | 58.06 | 47.44 | 52.21 width 1pt | 88 | 81.72 | 84.74
* RIDE + Cognisance width 1pt | 62.11 | 50.8 |  55.89 width 1pt | 89.02 | 81.85 | 85.28

##  V Dicussion and Further Analyses

###  V-A Why not directly apply the OLeaF?

In this scheme, CLF is more appropriate compared to OLeaF for two very
important characteristics: 1) Automatic tree-splitting mechanism, the number
of parameters of CLF is relatively small, and the fine degree of tree-
splitting can be controlled only by adjusting the parameter $d_{max}$. The
tree-splitting scheme in OLeaF is more rigorous and detailed but requires a
certain degree of manual analysis, whereas the feature learner needs to be
iterated in this framework, and it may be possible after each iteration to re-
trigger clustering, and the dataset targeted is often large image data, so a
fully automated scheme must be needed; 2) CoarseNode, each node of CLF is a
coarse-grained node, the reason why this mechanism is designed mainly to deal
with the long-tailed problem of attributes within a category, due to the fact
that the head attribute may contain a large number of similar samples, and if
left uncontrolled, the length of the path occupied by the head attribute may
skyrocket, while each node in the same path is sampled with the same weight, a
large number of extremely similar nodes actually lose their
representativeness, and at the same time, the sampling weights of bottom nodes
are compromised, so it is necessary to carry out a small range of clustering
to be controlled, and this clustering process is mainly controlled by the
parameter $d_{min}$, and all the members of the nodes within a CoarseNode will
equally share the node’s sampling weight.

###  V-B Why not directly apply IRM loss?

In this scheme, there are two reasons why IRM loss is not used directly: 1)
the original IRM loss has convergence problems on real-world large-scale image
datasets; 2) the core in Multi-Center Loss lies in the Multi-center, which is
a mechanism that can make the model’s learning more robust because the samples
within the same category in an artificial dataset may gap tremendously, and at
this time, if we simply add the regularity of one center, it will instead
impair the learning of the features.

###  V-C About Parameter Adjustment

In this scheme, there are two types of parameters that can be tuned: 1)
overall process-related, e.g., the number of epochs for warm-up, the number of
epoch steps for updating the sampling weights of samples, and the number of
environments; and 2) clustering-related, the two parameters $d_{min}$ and
$d_{max}$ in CLF clustering, the former controlling the radius of the
CoarseNode and the latter controls the fineness of the tree splitting. The
algorithm is preset with two relatively general default values, but they can
still be adjusted according to the specific dataset. The above parameters have
little impact on the overall effect within a certain range, but there is still
room for further optimization.

###  V-D About Distance Measure

In this scheme, the distance metric is used in two places: 1) CLF
construction, the distance matrix needs to be calculated when CLF clustering
is carried out, and the distance metric here can be switched arbitrarily; 2)
Multi-Center Loss, the distance from the samples to the center of their
belonging will be calculated when optimizing the MCL, and the distance metric
here is consistent with that of CLF construction, and it can also be switched
freely. The Euclidean distance is used as the default distance metric in this
paper, switching other metrics may give better results, due to the relatively
large amount of work, only one metric is used here, but it does not affect the
elaboration of the main idea.

###  V-E About Eliminating Noise

Any real-world dataset is imperfect, and noisy samples such as sensory
glitches (e.g., low-quality or corrupted images) and human errors (e.g.,
mislabeling or ambiguous annotations) may impair model training. In this
scheme, the proposed CLF actually has the potential to perform noise
identification, as shown in Fig. 4, if there is a tree in the CLF acontains
only one node and the node contains only one sample, or the node is a leaf
node of a very deep tree, then the sample corresponding to this node have a
high probability of being noise samples, which can be further processed to
finalize whether they are noise or not, and this idea can be explored more in
future work.

##  VI Conclusion

In this study, we provide insights into the long-tailed problem at two levels
of granularity: inter-class and intra-class, and propose two important
components: the CLF (Coarse-Grained Leading Forest) and the MCL (Multi-Center
Loss). The CLF, as an unsupervised learning methodology, aims to capture the
distribution of attributes within a class in order to lead the construction of
multiple environments, thus supporting the invariant feature learning.
Meanwhile, MCL, as an evolutionary version of Center Loss, aims to replace the
traditional IRM Loss to further enhance the robustness of the model on real-
world datasets.

Through extensive experiments on existing benchmarks MSCOCO-GLT and ImageNet-
GLT, we exhaustively demonstrate the significant results of our method.
Finally, we would also like to point out the advantages of the two components,
CLF and MCL, which are designed as low-coupling plug-ins that can be
organically integrated with other long-tailed classification methods, bringing
new possibilities for model performance improvement.

## Acknowledgment

This work has been supported by the National Natural Science Foundation of
China under grants 61966005, 61936001 and 62366008, the National Key Research
and Development Program of China under grant 2020YFB1713300, the Natural
Science Foundation of Chongqing (cstc2019jcyjcxttX0002, cstc2021ycjh-
bgzxm0013), and the Key Collaboration Project of Chongqing Municipal Education
Commission(HZ2021008).

## References

  * [1] Y. Zhang, B. Kang, B. Hooi, S. Yan, and J. Feng, “Deep long-tailed learning: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
  * [2] Y. Yang, H. Wang, and D. Katabi, “On multi-domain long-tailed recognition, imbalanced domain generalization and beyond,” in European Conference on Computer Vision, pp. 57–75, Springer, 2022.
  * [3] M. Buda, A. Maki, and M. A. Mazurowski, “A systematic study of the class imbalance problem in convolutional neural networks,” Neural networks, vol. 106, pp. 249–259, 2018.
  * [4] S. Ando and C. Y. Huang, “Deep over-sampling framework for classifying imbalanced data,” in Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2017, Skopje, Macedonia, September 18–22, 2017, Proceedings, Part I 10, pp. 770–785, Springer, 2017.
  * [5] Y. Yang, G. Zhang, D. Katabi, and Z. Xu, “Me-net: Towards effective adversarial robustness with matrix estimation,” arXiv preprint arXiv:1905.11971, 2019.
  * [6] K. Cao, C. Wei, A. Gaidon, N. Arechiga, and T. Ma, “Learning imbalanced datasets with label-distribution-aware margin loss,” Advances in neural information processing systems, vol. 32, 2019.
  * [7] L. Sun, K. Wang, K. Yang, and K. Xiang, “See clearer at night: towards robust nighttime semantic segmentation through day-night image conversion,” in Artificial Intelligence and Machine Learning in Defense Applications, vol. 11169, pp. 77–89, SPIE, 2019.
  * [8] K. Tang, M. Tao, J. Qi, Z. Liu, and H. Zhang, “Invariant feature learning for generalized long-tailed classification,” in European Conference on Computer Vision, pp. 709–726, Springer, 2022.
  * [9] C. Drummond, R. C. Holte, et al., “C4. 5, class imbalance, and cost sensitivity: why under-sampling beats over-sampling,” in Workshop on learning from imbalanced datasets II, vol. 11, pp. 1–8, 2003.
  * [10] H. Han, W.-Y. Wang, and B.-H. Mao, “Borderline-smote: a new over-sampling method in imbalanced data sets learning,” in International conference on intelligent computing, pp. 878–887, Springer, 2005.
  * [11] H. He and E. A. Garcia, “Learning from imbalanced data,” IEEE Transactions on knowledge and data engineering, vol. 21, no. 9, pp. 1263–1284, 2009.
  * [12] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in Proceedings of the IEEE international conference on computer vision, pp. 2980–2988, 2017.
  * [13] B. Kang, S. Xie, M. Rohrbach, Z. Yan, A. Gordo, J. Feng, and Y. Kalantidis, “Decoupling representation and classifier for long-tailed recognition,” in International Conference on Learning Representations, 2019.
  * [14] B. Zhou, Q. Cui, X.-S. Wei, and Z.-M. Chen, “Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9719–9728, 2020.
  * [15] B. Zhu, Y. Niu, X.-S. Hua, and H. Zhang, “Cross-domain empirical risk minimization for unbiased long-tailed classification,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 3589–3597, 2022\.
  * [16] P. Cui and S. Athey, “Stable learning establishes some common ground between causal inference and machine learning,” Nature Machine Intelligence, vol. 4, no. 2, pp. 110–115, 2022.
  * [17] M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz, “Invariant risk minimization,” stat, vol. 1050, p. 27, 2020.
  * [18] E. Creager, J.-H. Jacobsen, and R. Zemel, “Environment inference for invariant learning,” in International Conference on Machine Learning, pp. 2189–2200, PMLR, 2021.
  * [19] Y. Wen, K. Zhang, Z. Li, and Y. Qiao, “A discriminative feature learning approach for deep face recognition,” in Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part VII 14, pp. 499–515, Springer, 2016.
  * [20] T. Wang, Y. Li, B. Kang, J. Li, J. Liew, S. Tang, S. Hoi, and J. Feng, “The devil is in classification: A simple framework for long-tail instance segmentation,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIV 16, pp. 728–744, Springer, 2020.
  * [21] A. Estabrooks, T. Jo, and N. Japkowicz, “A multiple resampling method for learning from imbalanced data sets,” Computational intelligence, vol. 20, no. 1, pp. 18–36, 2004.
  * [22] Z. Zhang and T. Pfister, “Learning fast sample re-weighting without reward data,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 725–734, 2021.
  * [23] J. Ren, C. Yu, X. Ma, H. Zhao, S. Yi, et al., “Balanced meta-softmax for long-tailed visual recognition,” Advances in neural information processing systems, vol. 33, pp. 4175–4186, 2020.
  * [24] C. Elkan, “The foundations of cost-sensitive learning,” in International joint conference on artificial intelligence, vol. 17, pp. 973–978, Lawrence Erlbaum Associates Ltd, 2001.
  * [25] M. A. Jamal, M. Brown, M.-H. Yang, L. Wang, and B. Gong, “Rethinking class-balanced methods for long-tailed visual recognition from a domain adaptation perspective,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7610–7619, 2020.
  * [26] J. Tan, C. Wang, B. Li, Q. Li, W. Ouyang, C. Yin, and J. Yan, “Equalization loss for long-tailed object recognition,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11662–11671, 2020.
  * [27] J. Tan, X. Lu, G. Zhang, C. Yin, and Q. Li, “Equalization loss v2: A new gradient balance approach for long-tailed object detection,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1685–1694, 2021.
  * [28] A. K. Menon, S. Jayasumana, A. S. Rawat, H. Jain, A. Veit, and S. Kumar, “Long-tail learning via logit adjustment,” arXiv preprint arXiv:2007.07314, 2020.
  * [29] P. Chu, X. Bian, S. Liu, and H. Ling, “Feature space augmentation for long-tailed data,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIX 16, pp. 694–710, Springer, 2020.
  * [30] J. Wang, T. Lukasiewicz, X. Hu, J. Cai, and Z. Xu, “Rsg: A simple but effective module for learning imbalanced datasets,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3784–3793, 2021.
  * [31] X. Hu, Y. Jiang, K. Tang, J. Chen, C. Miao, and H. Zhang, “Learning to segment the tail,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14045–14054, 2020.
  * [32] E. D. Cubuk, B. Zoph, J. Shlens, and Q. V. Le, “Randaugment: Practical automated data augmentation with a reduced search space,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 702–703, 2020.
  * [33] J. Liu, Y. Sun, C. Han, Z. Dou, and W. Li, “Deep representation learning on long-tailed data: A learnable embedding augmentation perspective,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2970–2979, 2020.
  * [34] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” arXiv preprint arXiv:1710.09412, 2017.
  * [35] X. Wang, L. Lian, Z. Miao, Z. Liu, and S. Yu, “Long-tailed recognition by routing diverse distribution-aware experts,” in International Conference on Learning Representations, 2020.
  * [36] Y. Zhang, B. Hooi, L. Hong, and J. Feng, “Self-supervised aggregation of diverse experts for test-agnostic long-tailed recognition,” Advances in Neural Information Processing Systems, vol. 35, pp. 34077–34090, 2022.
  * [37] J. Xu, G. Wang, and W. Deng, “Denpehc: Density peak based efficient hierarchical clustering,” Information Sciences, vol. 373, pp. 200–218, 2016.
  * [38] A. Rodriguez and A. Laio, “Clustering by fast search and find of density peaks,” science, vol. 344, no. 6191, pp. 1492–1496, 2014.
  * [39] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al., “Imagenet large scale visual recognition challenge,” International journal of computer vision, vol. 115, pp. 211–252, 2015.
  * [40] G. Patterson and J. Hays, “Coco attributes: Attributes for people, animals, and objects,” in Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VI 14, pp. 85–100, Springer, 2016.

|  Jinye Yang received the B.E degree from Yanshan University, Qinhuangdao,
China in 2019. He is currently a graduate student with the State Key
Laboratory of Public Big Data, Guizhou University. His current research
interests include granular computing and machine learning.
---|---
|  Ji Xu received the B.S. from Beijing Jiaotong University in 2004 and the
Ph.D. from Southwest Jiaotong University, Chengdu, China, in 2017,
respectively. Both degrees are in the major of Computer Science. Now he is an
associate professor with the State Key Laboratory of Public Big Data, Guizhou
University. His research interests include data mining, granular computing and
machine learning. He has authored and co-authored over 10 papers in refereed
international journals such as IEEE TCYB, Information Sciences, Knowledge-
Based Systems and Neurocomputing.
---|---