# Subclass-balancing Contrastive Learning
for Long-tailed Recognition

Chengkai Hou
Jilin university
<EMAIL_ADDRESS>These authors contributed equally to this work. Jieyu
Zhang11footnotemark: 1
University of Washington
<EMAIL_ADDRESS>Haonan Wang
National University of Singapore
<EMAIL_ADDRESS>Tianyi Zhou
University of Maryland, College Park
<EMAIL_ADDRESS>Corresponding author

###### Abstract

Long-tailed recognition with imbalanced class distribution naturally emerges
in practical machine learning applications. Existing methods such as data
reweighing, resampling, and supervised contrastive learning enforce the class
balance with a price of introducing imbalance between instances of head class
and tail class, which may ignore the underlying rich semantic substructures of
the former and exaggerate the biases in the latter. We overcome these
drawbacks by a novel “subclass-balancing contrastive learning (SBCL)” approach
that clusters each head class into multiple subclasses of similar sizes as the
tail classes and enforce representations to capture the two-layer class
hierarchy between the original classes and their subclasses. Since the
clustering is conducted in the representation space and updated during the
course of training, the subclass labels preserve the semantic substructures of
head classes. Meanwhile, it does not overemphasize tail class samples, so each
individual instance contribute to the representation learning equally. Hence,
our method achieves both the instance- and subclass-balance, while the
original class labels are also learned through contrastive learning among
subclasses from different classes. We evaluate SBCL over a list of long-tailed
benchmark datasets and it achieves the state-of-the-art performance. In
addition, we present extensive analyses and ablation studies of SBCL to verify
its advantages.

##  1 Introduction

In reality, the datasets often follow the Zipfian distribution over classes
with a long tail [74, 56], $i.e.$, a few classes (head classes) containing
significantly more instances than the remaining tail classes. Such tail
classes could be of great importance for high-stake applications, _e.g_.,
patient class in medical diagnosis or accident class in autonomous driving [5,
54]. However, training on such class-imbalanced datasets can result in a
severely biased model with noticeable performance drop in classification tasks
[63, 46, 70, 2, 3, 12, 67].

To overcome the challenges posed by long-tailed data, data resampling [2, 3,
8, 53] and loss reweighing [4, 5, 14, 16] have been widely applied but they
cannot fully leverage all the head-class samples. Very recent work discovered
that supervised contrastive learning (SCL) [35] can achieve state-of-the-art
(SOTA) performance on benchmark datasets of long-tailed recognition [32, 40].
Specifically, the $k$-positive contrastive learning (KCL) [32] and its
subsequent work targeted supervised contrastive learning (TSC) [40] revamp SCL
by encouraging the learned feature space to be class-balanced and uniformly
distributed. However, those methods enforcing class-balance often come with a
price of instance-imbalance, _i.e_., each individual instance of tail classes
would have much greater impact on model training than that of head classes.

Such instance-imbalance can result in significant degradation of the
performance on long-tailed recognition for several reasons. On the one hand,
the limited samples in each tail class might not be sufficiently
representative of the whole class. So even a small bias of them can be
enormously exaggerated by class-balancing methods and result in sub-optimal
learning of classifiers or representations. On the other hand, head classes
usually have more complicated semantic substructures, e.g., multiple high-
density regions of the data distribution, so simply downweighing samples of
head classes and treating them equally can easily lose critical structural
information. For example, images of a head class “cat” might be highly diverse
in breeds and colors, which need to be captured by different features but
downweighing or subsampling them may easily lose such information, while a
tail class “platypus” might only contain a few similar images that are
unlikely to cover all the representative features. Therefore, it is non-
trivial to enforce both class-balance and instance-balance simultaneously in
the same method.

Can we remove the negative impact of class-imbalance while still retain the
advantages of instance-balance? In this paper, we achieve both through
subclass-balancing contrastive learning (SBCL), a novel supervised contrastive
learning defined on subclasses, which are the clusters within each head class,
have comparable size as tail classes, and are adaptively updated during the
training. Instead of sacrificing instance-balance for class-balance, our
method achieves both instance- and subclass-balance by exploring the head-
class structure in the learned representation space of the model-in-training.
In particular, we propose a _bi-granularity contrastive loss_ that enforces a
sample (1) to be closer to samples from the same subclass than all the other
samples; and (2) to be closer to samples from a different subclass but the
same class than samples from any other subclasses. While the former learns
representations with balanced and compact subclasses, the latter preserves the
class structure on subclass level by encouraging the same class’s subslasses
to be closer to each other than to any different class’s subclasses. Hence, it
can learn an accurate classifier distinguishing original classes while enjoy
both the instance- and subclass-balance.

In this paper, we apply SBCL for several visual recognition tasks to
demonstrate SBCL superiority over other previous works (_e.g_., KCL[32], TSC
[40]). To summarize, this paper makes the following contributions:

  1. (a).

We provide a new design principal of leveraging supervised contrastive
learning for long-tailed recognition, _i.e_., aiming at achieving both
instance- and subclass-balance instead of class-balance at the expense of
instance-balance.

  2. (b).

We propose a novel instantiation of the aforementioned design principal,
subclass-balancing contrastive learning (SBCL), which consists of two major
components, namely, subclass-balancing adaptive clustering and bi-granularity
contrastive loss.

  3. (c).

Empirically, we compare the SBCL against state-of-the-art methods on three
visual tasks: image classification, object detection, and instance
segmentation to demonstrate its effectiveness on handling class imbalance. We
also conduct a series of experiments to analyze the efficacy of SBCL.

##  2 Background and notations

#### Long-tailed recognition.

Long-tailed recognition aims to learn a classifier from a training dataset
with long-tailed class distribution, _i.e_., a few classes contain many data
(head classes) while most classes contain only a few data (tail classes),
where the major challenge is to require model recognizing all classes equally
well. Let $\mathcal{D}=\left\\{x_{i},y_{i}\right\\}_{i\in[n]}$ be a long-
tailed training dataset, where $x_{i}$ denotes a sample and $y_{i}\in[C]$
denotes its label. Denote by $\mathcal{D}_{k}\subseteq\mathcal{D}$ the set of
instances belonging to class $k$ and $n_{k}=|\mathcal{D}_{k}|$ the number of
samples in class $k$ The total number of training samples over $C$ classes is
$n=\sum_{k=1}^{C}n_{k}$. Without loss of generality, we follow prior work [33,
27] to assume that the classes are sorted by cardinality in decreasing order
($i.e.$, if $i<j$, then $n_{i}\geq n_{j}$), and $n_{1}\gg n_{C}$. In addition,
we define the imbalance ratio as
$\max_{k\in[C]}(n_{k})/\min_{k\in[C]}(n_{k})=n_{1}/n_{C}$. Finally, let
$f_{\theta}(\cdot)$ be a deep feature extractor, _e.g_., a neural network,
parameterized by $\theta$ and $\mathbf{w}_{c}$is the linear classifier of
class $c$, then the classifier we aim to learn is
$h(x_{i})=\mathop{\arg\max}_{c\in[C]}\mathbf{w}_{c}^{\top}f_{\theta}(x_{i})$.

#### Supervised contrastive learning.

Recent studies have shown that supervised contrastive learning (SCL) [35]
provides a strong performance gain for long-tailed recognition and its
variants have achieved state-of-the-art (SOTA) performance [32, 40].
Specifically, SCL learns the feature extractor $f_{\theta}(\cdot)$ via
maximizing the discriminativeness of positive instances, _i.e_., instances
from the same class, and the learning objective for a single training data
$(x_{i},y_{i})$ in a batch $\mathcal{B}=\\{(x_{i},y_{i})\\}_{i\in[N]}$, is

$\displaystyle\mathcal{L}_{SCL}=\sum_{i=1}^{N}-\frac{1}{|\tilde{P_{i}}|}\sum_{z_{p}\in\tilde{P_{i}}}\log\frac{\exp(z_{i}\cdot
z_{p}^{\top}/\tau)}{\sum_{z_{a}\in\tilde{V_{i}}}\exp(z_{i}\cdot
z_{a}^{\top}/\tau)},$ (1)

where $\tau$ is the temperature hyperparameter, $z_{i}=f_{\theta}(x_{i})$ is
the feature generated from $x_{i}$,
$V_{i}=\left\\{z_{i}\right\\}_{i\in[N]}\backslash\left\\{z_{i}\right\\}$ is
the current batch of features except for $z_{i}$, $P_{i}=\left\\{z_{j}\in
V_{i}:y_{j}=y_{i}\right\\}$ is a set of instances with the same label as
$x_{i}$. Finally, let $\tilde{z_{i}}$ be the feature of $\tilde{x_{i}}$, the
augmented version of $x_{i}$, and for any set $S_{i}$ indexed by $i$, we use
$\tilde{S}_{i}=S_{i}\cup\left\\{\tilde{z_{i}}\right\\}$, _e.g_.,
$\tilde{V_{i}}=V_{i}\cup\left\\{\tilde{z_{i}}\right\\}$. However, for the
long-tailed datasets, the feature spaces is dominated by head classes and thus
have limited capability of semantic discrimination [32]. To address this, the
$k$-positive contrastive learning (KCL) [32] attempts to balance the feature
space by keeping the number of positive instances in $\tilde{P_{i}}$ equal for
each class, leading to the following loss

$\displaystyle\mathcal{L}_{KCL}=\sum_{i=1}^{N}-\frac{1}{k+1}\sum_{z_{p}\in\tilde{P_{i}^{k}}}\log\frac{\exp(z_{i}\cdot
z_{p}^{\top}/\tau)}{\sum_{z_{a}\in\tilde{V_{i}}}\exp(z_{i}\cdot
z_{a}^{\top}/\tau)}$ (2)

where $P_{i}^{k}$ is a subset of $P_{i}$ with $k$ randomly drawn instances.
Finally, the learned feature extractor $f_{\theta}(\cdot)$ is exploited in a
sequel stage of training the classifier for long-tailed recognition [32, 40].

##  3 Methodology

Figure 1: Illustration of subclass-balancing contrastive learning (SBCL). It
initially divides the head classes into multiple subclasses of comparable
size. Then, during training, SBCL builds each sample to be closer to samples
from the same subclass than samples from different subclasses but the same
class, which are also made to be closer than samples from different classes.

As mentioned above, KCL and its sequels [32, 40] balance the learning
objective of SCL by picking the same number of positive instance for each
class, _i.e_., $|P_{i}^{k}|=k$ in Eq. 2 no matter which class $x_{i}$ belongs
to, however, we argue that such a class-balancing approach would inevitably
introduce instance-imbalance: the instances of tail classes have much more
chances to be engaged in the training than that of head class. Specifically,
assume each class has no less than $k$ instances, then the probability of an
instance of class $c$ being selected as positive instance is
$p(c)=\frac{k}{n_{c}}$; if the tail class $n_{C}$ has only $k$ instances and
the imbalance ratio is $\frac{n_{1}}{n_{C}}=100$, then we have
$p(1)=\frac{k}{100k}=0.01$ while $p(C)=1$. We can see that when the instances
of head class are selected once, that of tail class may already be trained 100
times. Thus, the training is immensely biased towards the few samples in each
tail class. Besides, as tail classes only have very few instances that are not
necessarily representative, the learned feature space might be unsatisfactory
and sensitive to the training data of tail classes.

Here, we provide a new prospective of handling class-imbalance issue by
contrastive learning: instead of aiming at class-balance at the expense of
instance-imbalance, we propose to achieve both instance- and subclass-balance.
We argue that head classes typically contain more diverse instances and thus
have richer semantics in the training dataset. Therefore, it might be wise to
break down the head classes into multiple semantically coherent subclasses,
each of which consists of similar number of instance as tail classes. Built on
this spirit, we develop subclass-balancing contrastive learning (SBCL), a new
contrastive learning framework for long-tailed recognition (visualized in
Figure 1) that achieves both instance- and subclass-balance.

###  3.1 Subclass-balancing adaptive clustering

We break down the class into several "subclasses" to attack the imbalanced
phenomenon. Particularly, given a class $c$ and the associated set of data
$\mathcal{D}_{c}$, we employ a clustering algorithm of choice based on the
features extracted by current feature extractor $f_{\theta}(\cdot)$ to divide
$\mathcal{D}_{c}$ into $m_{c}$ subclasses/clusters. We use $\Gamma_{c}(x_{i})$
to denote the cluster label of an instance $x_{i}$ of class $c$. To ensure
that the number of samples for each subclass is roughly the same, we propose a
new cluster algorithm to divide the unit-length feature vectors, _i.e_., the
features output by $f_{\theta}(\cdot)$ with additional unit-length
normalization. The new proposed cluster algorithm is described in Algorithm 1.

Algorithm 1 Subclass-balancing Adaptive Clustering

Sample set $\mathcal{S}=\left\\{x_{i}\right\\}_{i=1}^{n}$; A threshold $M$;
The number of iterations $K$;

Cluster assignments for samples in $\mathcal{S}$

for $k=0$ to $K$ do

if $k=0$ then

Choose the cluster centers $y_{j}$ which are farther away from previously
selected centers.

else

Update the cluster centers $y_{j}=\frac{1}{n_{i}}\sum_{i=1}^{n_{i}}x_{i}$.
$\triangleright$ $n_{i}$ is the number of samples in a cluster

end if

Construct the cluster center set
$\mathcal{C}=\left\\{y_{j}\right\\}_{j=1}^{m}$. $\triangleright$ $m$ is the
number of cluster centers

while $\mathcal{S}\neq\phi$ do $\triangleright$ Assign samples to centers
$y_{i}$

Select the most similar pair
$(x_{i},y_{j})=\mathop{\arg\max}\limits_{x\in\mathcal{S},y\in\mathcal{C}}\text{cosine-
similarity}(x,y)$.

Assign the sample $x_{i}$ to the center $y_{j}$.

Delete the assigned sample $x_{i}$ from the sample set
$\mathcal{S}=\mathcal{S}/\left\\{x_{i}\right\\}$.

if $n_{i}\geq M$ then $\triangleright$ Sample number in a cluster exceeds the
threshold $M$

Delete the cluster center $y_{j}$ from the cluster center set
$\mathcal{C}=\mathcal{C}/\left\\{y_{j}\right\\}$.

end if

end while

end for

In Algorithm 1, during the initialization of cluster centers, we gradually
choose cluster centers, and the samples that are farther away from other
cluster centers are more likely to be selected as the next cluster center. In
the process of assigning vectors to their centers, we set a threshold $M$ as
an upper limit of sample size in a cluster, which guarantees clusters of
balanced sizes. We show the distribution of the sample size in clusters on the
benchmark dataset in Appendix A.1. Specifically, the threshold $M$ is

$\displaystyle M=\mathop{\max}(n_{C},\delta)$ (3)

where $n_{C}$ is the size of tail class and the hyperparameter $\delta$
controls the lower bound of sample size in clusters to prevent overly-small
cluster. Note that we only apply clustering algorithm to classes which contain
multiple instances while the tail classes remain unchanged. As a consequence,
the size of each resultant cluster is similar to that of tail class $n_{C}$.
In addition, instead of only clustering once at the beginning, we update the
cluster assignment adaptively based on the current feature extractor
$f_{\theta}(\cdot)$ during training process and empirically show that such
adaptive clustering outperforms only-once clustering in Section 4.5.

Then, by replacing the class labels of head classes used in SCL/KCL with the
finer-grained cluster labels, we ensure the instance-balanced, _i.e_., each
instance has similar chance of being selected regardless of its class. By
breaking down the head classes, which typically contain more diverse
instances, into multiple semantically coherent subclasses, we achieve
subclass-balanced (instead of class-balanced) while maintain the rich
semantics rendered by head classes in training dataset.

###  3.2 Bi-granularity contrastive loss

We now have two types of label for instances in head classes from different
granularities: the coarse-grained _class_ label and the fine-grained _cluster_
label. A direct consequence of replacing class label in SCL/KCL with cluster
label is that we no longer distinguish instances from different head classes,
and therefore the boundaries between classes might be blurry, leading to
optimal feature space. As a remedy, we combine the contrastive loss of both
class label and cluster label into the following one and reuse the notations
of Eq. 1:

$\displaystyle\mathcal{L}_{SBCL}=-\sum_{i=1}^{N}\Big{(}\frac{1}{|\tilde{M}_{i}|}\sum_{z_{p}\in\tilde{M_{i}}}\log\frac{\exp(z_{i}\cdot
z_{p}^{\top}/\tau_{1})}{\sum_{z_{a}\in\tilde{V_{i}}}\exp(z_{i}\cdot
z_{a}^{\top}/\tau_{1})}$ (4)
$\displaystyle+\beta\frac{1}{|\tilde{P_{i}}|-|M_{i}|}\sum_{z_{p}\in\tilde{P_{i}}/M_{i}}\log\frac{\exp(z_{i}\cdot
z_{p}^{\top}/\tau_{2})}{\sum_{z_{a}\in\tilde{V_{i}}/M_{i}}\exp(z_{i}\cdot
z_{a}^{\top}/\tau_{2})}\Big{)}$

where $M_{i}=\left\\{z_{j}\in
P_{i}:\Gamma_{y_{i}}(x_{i})=\Gamma_{y_{i}}(x_{j})\right\\}$ is a set of
instances with the same cluster label as $x_{i}$. $\beta$ is a hyperparameter
that balances these two loss terms. The first term corresponds to the SCL loss
with cluster labels, while the second term leverages the class label but does
not consider the instances of the same cluster, _i.e_., the instances in
$M_{i}$ are removed in the second term. Such a design choice reflects the two
types of positive instances for $z_{i}$: (1) the instance in the same cluster
and (2) the instance of the same class but in different clusters.

According to previous studies [60, 26, 39], the temperature $\tau$ in
contrastive loss is critical in controlling the local separation and global
uniformity of the feature distribution. Specifically, for supervised
contrastive learning, a low temperature makes relative high penalty on feature
distribution, that actually encourages the features distribute more
concentrically. As the temperature increases, the relative penalty tends to be
more uniform which uniformizes the distribution of the features. Although the
above objective explicitly considers the two types of label from different
granularities, it still treats class and cluster label similarly. Intuitively,
we expect instances of the same subclass to form a more concentrated cluster
in feature space than those of the same class, since subclass naturally
indicates finer-grained semantic coherence. To achieve this, we ensure the
temperature $\tau_{2}>\tau_{1}$ and dynamically adjust $\tau_{2}$ for each
class according to its current level of concentration of the instances’
feature. Following [39], for class $c$ we define $\phi(c)$ as

$\displaystyle\phi(c)=\frac{\sum_{i=1}^{n_{c}}\|z_{i}-t_{c}\|_{2}}{n_{c}\log(n_{c}+\alpha)}$
(5)

where $t_{c}$ is the centroid for the class $c$, $\alpha$ is a hyperparameter
to ensure that $\phi(c)$ is not overly-large, and $z_{i}$ corresponds to
instances of class $c$. From the formulation, we can see that if the current
averaged distance to the class centroid is large or the class contains fewer
data, thus the temperature will be set large to adopt the feature distribution
of class $c$ during the training process. Then we define the temperature of
class $c$ as

$\displaystyle\tau_{2}(c)=\tau_{1}\cdot\exp\left(\frac{\phi(c)}{\frac{1}{C}\sum_{i=1}^{C}\phi(i)}\right)$
(6)

such that $\tau_{2}(c)$ for class label is always larger than $\tau_{1}$ for
cluster label (since $\phi(c)>0$) and could reflect the current level of
concentration of the instances in a class. In particular, the proposed
$\tau_{2}(c)$ encourages the features of instances in class $c$ to form a less
tight cluster than that of a subclass (by $\tau_{2}(c)>\tau_{1}$) while
adaptively adjust the temperature to prevent an overly-loose/dense cluster.

###  3.3 Training algorithm

Here, we describe the overall training process of subclass-balancing
contrastive learning and the algorithm can be found in Algorithm 2. First, the
adaptive clustering (Section 3.1) could be noisy at the early stage of
training [39, 62]. Thus, we warm-up the feature extractor $f_{\theta}(\cdot)$
by a few epochs of training on ordinary SCL or KCL loss. In addition, our
algorithm involves two adaptively-adjusting parts, namely, the cluster
assignment and the temperature $\tau_{2}(c)$ for each head class $c$. Instead
of updating these every epoch, we use a hyperparameter $K$ as the update
interval, _i.e_., we update the cluster assignment and the temperature based
on the current learned $f_{\theta}(\cdot)$ every $K$ epoch.

Algorithm 2 Training Algorithm

1:Dataset $\mathcal{D}=\left\\{x_{i},y_{i}\right\\}_{i\in[n]}$; ; The update
interval of cluster assignment $K$; The number of warm-up epoch $T_{0}$; The
total number of epoch $T$; The hyperparameters $\beta$ and $\delta$.

2:A trained feature extractor $f_{\theta}(\cdot)$

3:Initialize the model parameters $\theta$

4:Train $f_{\theta}(\cdot)$ with SCL/KCL for $T_{0}$ epochs $\triangleright$
Warm-up stage

5:for $t=T_{0}$ to $T$ do

6: if $t//K==0$ then $\triangleright$ Update cluster and termperature

7: Update the cluster assignment based on the current feature extractor
$f_{\theta}(x)$

8: Update the temperture $\tau_{2}$ for each head class using Eq. 5 and Eq. 6

9: end if

10: Train $f_{\theta}(\cdot)$ using Eq. 4 $\triangleright$ Subclass-balancing
contrastive learning

11:end for

##  4 Experiment

###  4.1 Experimental setup

#### Datasets.

We consider three commonly used long-tailed recognition benchmark datasets:
CIFAR-100-LT [5], ImageNet-LT [45], and iNaturalist 2018 [59]. The
CIFAR-100-LT and ImageNet-LT datasets are artificially generated long-tailed
datasets from the class-balanced datasets [37, 51], and the iNaturalist 2018
dataset is a large-scale real-world dataset that exhibits long-tailed
imbalance.

#### Baselines.

We consider baseline methods of the following three categories: (1) class-
balancing classifiers, including $\tau$-norm, LWS and cRT [33], which fixes
the representation which trained by cross-entropy loss and trains the
classifier with class-balanced sampling; (2) one-stage balancing loss,
including CB loss [14], Focal loss [43], and LDAM loss [5]. These supervised
distribution-aware loss makes the model to pay more attention on the minority
class during training. (3) contrastive learning methods, including SCL [35],
KCL [32], SwAV [7], PCL [39] and TSC [40] which train a feature extractor with
the contrastive loss and then learn a classifier given the trained feature
extractor.

#### Evaluation protocol.

Following [32, 33, 40], we implement SBCL, as well as other contrastive
learning methods, in a two-stage framework. In the first stage, we train the
feature extractor with a contrastive learning method, while in the second
stage, we train a linear classifier on top of the learned representation.
Specifically, for CIFAR-100-LT dataset, the linear classifier is trained with
LDAM loss and class re-weighting [5]. For ImageNet-LT and iNaturalist 2018
datasets, the linear classifier is trained with CE loss and class-balanced
sampling [33]. All results are averaged over 5 trials with different random
seeds. We mainly report the overall top-1 accuracy. For the two large
datasets, ImageNet-LT and iNaturalist 2018 datasets, following the previous
work [45], we also report the accuracy of three disjoint subsets: Many-shot
classes (classes with more than 100 samples), Medium-shot classes (classes
with 20 to 100 samples), and Few-shot classes (classes under 20 samples). We
show the implementation details and important hyperparameters in the Appendix
A.2.

###  4.2 Main results

Table 1: Performance comparison on ImageNet-LT and iNaturalist 2018 datasets.
Top-1 accuracy of ResNet-50 [25] is reported. The "Many", "Medium" ,"Few" and
"All" denotes different groups. $\dagger$ denotes our reproduced results of
PCL, SwAV, and BYOL based on their official code. Other baselines’ results on
ImageNet-LT and iNaturalist 2018 are copied from Li _et al_. [40].

Backbone | ImageNet-LT | iNaturalist 2018
---|---|---
Methods | Many | Medium | Few | All | Many | Medium | Few | All
CE | 64.0 | 33.8 | 5.8 | 41.6 | 72.2 | 63.0 | 57.2 | 61.7
Focal loss [43] | 51.0 | 40.8 | 20.8 | 43.7 | - | - | - | 61.3
CB-Focal [14] | - | - | - | - | - | - | - | 61.1
LDAM-DRW [5] | 60.4 | 46.9 | 30.7 | 49.8 | - | - | - | 64.6
OLTR [45] | 35.8 | 32.3 | 21.5 | 32.2 | 59.0 | 64.1 | 64.9 | 63.9
$\tau$-norm [33] | 56.6 | 44.2 | 27.4 | 46.7 | 71.1 | 68.9 | 69.3 | 69.3
cRT [33] | 58.8 | 44.0 | 26.1 | 47.3 | 73.2 | 68.8 | 66.1 | 68.2
LWS [33] | 57.1 | 45.2 | 29.3 | 47.7 | 71.0 | 69.8 | 68.8 | 69.5
PCL${\dagger}$ [39] | 34.7 | 26.1 | 12.3 | 27.5 | 48.5 | 45.9 | 41.7 | 44.5
SwAV${\dagger}$ [7] | 37.5 | 28.3 | 15.6 | 30.1 | 51.9 | 48.4 | 43.7 | 47.0
BYOL${\dagger}$ [20] | 37.7 | 28.9 | 16.3 | 30.6 | 52.3 | 48.6 | 44.1 | 47.2
SCL [35] | 61.4 | 47.0 | 28.2 | 49.8 | - | - | - | 66.4
KCL [32] | 62.4 | 49.0 | 29.5 | 51.5 | - | - | - | 68.6
TSC [40] | 63.5 | 49.7 | 30.4 | 52.4 | 72.6 | 70.6 | 67.8 | 69.7
SBCL | 63.8 | 51.3 | 31.2 | 53.4 | 73.3 | 71.9 | 68.6 | 70.8

Table 2: Performance comparison on CIFAR-100-LT. Top-1 accuracy of the
ResNet-32 [25] under different imbalance ratios is reported. We also report
the accuracy of our re-implemented important baselines (${\dagger}$) in same
setting on CIFAR-100-LT. The columns of "Statistic (IR 100)" are the results
of different disjoint subsets on CIFAR-100-LT with imbalance ratio being 100.

Method | Imbalance Ratio | Statistic (IR 100)
---|---|---
100 | 50 | 10 | Many | Medium | Few
CE | 38.3 | 43.9 | 55.7 | 65.2 | 37.1 | 9.1
CB-CE [14] | 38.6 | 44.6 | 57.1 | - | - | -
Focal Loss [43] | 38.4 | 44.3 | 55.8 | 65.3 | 38.4 | 8.1
CB-Focal [14] | 38.7 | 45.2 | 58.0 | 65.0 | 37.6 | 10.3
CE-DRW [5] | 41.4 | 45.3 | 58.1 | - | - | -
CE-DRS [5] | 41.6 | 45.5 | 58.1 | - | - | -
LDAM [5] | 39.6 | 45.0 | 56.9 | - | - | -
LDAM-DRW [5] | 42.0 | 46.6 | 58.7 | 61.5 | 41.7 | 20.2
M2m-ERM [36] | 42.9 | - | 58.2 | - | - | -
M2m-LDAM [36] | 43.5 | - | 57.6 | - | - | -
cRT [33] | 43.3 | 46.8 | 58.1 | 64.0 | 44.8 | 18.1
LWS [33] | 43.1 | 46.4 | 58.1 | - | - | -
SCL [35] ${\dagger}$ | 42.1 | 45.2 | 54.8 | 62.8 | 42.0 | 18.4
KCL [32] | 42.8 | 46.3 | 57.6 | - | - | -
KCL${\dagger}$ | 42.8 | 46.4 | 57.5 | 63.4 | 42.5 | 19.2
TSC [40] | 43.8 | 47.4 | 59.0 | - | - | -
TSC${\dagger}$ | 43.5 | 47.6 | 58.7 | 63.7 | 43.2 | 20.4
SBCL | 44.9 | 48.7 | 57.9 | 64.4 | 45.3 | 22.2

The results on ImageNet-LT and iNaturalist 2018 are in Table 1. We can see
that SBCL outperforms the baselines with a large margin over the two datasets.
In addition, on iNaturalist 2018 dataset, SBCL outperforms the previous SOTA
method by 0.7% on Many, 1.3% on Medium, 0.8% on Few and 1.1% on All, which
shows the effectiveness of the proposed method in solving real-world long-
tailed recognition problems such as natural species classification. Besides,
SBCL is also better than existing contrastive learning method like KCL and TSC
for all class splits, which demonstrates the effectiveness of the design
principal of pursuing both instances- and subclass-balance in contrastive
learning. Table 2 summarizes the results on CIFAR-100-LT dataset. For
CIFAR-100-LT dataset, SBCL outperforms previous SOTA methods except for
imbalance ratio 10. We hypothesize that it is because the tail class of
CIFAR-100-LT with imbalance ratio 10 has multiple samples, which makes it hard
to distinguish the performance of methods on the long-tailed recognition.

###  4.3 Performance on Other Visual Tasks

Table 3: Object detection Results on PASCAL VOC.

Method | ImageNet | ImageNet-LT
---|---|---
AP50 | AP |  AP75 |  AP50 | AP |  AP75
random init. | 60.2 | 33.8 | 33.1 | 60.2 | 33.8 | 33.1
CE | 81.3 | 53.7 | 59.2 | 76.5 | 48.5 | 51.0
CL [23] | 81.3 | 56.1 | 62.7 | 78.2 | 51.5 | 56.5
KCL [32] | 82.3 | 55.5 | 62.1 | 79.7 | 52.6 | 57.9
SBCL | 81.9 | 56.2 | 62.8 | 80.6 | 53.4 | 58.8

Table 4: Ablation study on different components of SBCL.

Warm-up | Adaptive cluster | Dynamic temperature | CIFAR-100-LT
---|---|---|---
Imbalance Ratio | 100 | 50 | 10
| $\checkmark$ | $\checkmark$ | 44.0 | 47.9 | 57.2
$\checkmark$ |  | $\checkmark$ | 43.8 | 47.2 | 56.5
$\checkmark$ | $\checkmark$ |  | 43.8 | 47.8 | 57.0
$\checkmark$ | $\checkmark$ | $\checkmark$ | 44.9 | 48.7 | 57.9

There is a recent trend of using the contrastive learning to pretrain a
feature extractor for downstream visual tasks other than image classification
[23]. We are curious about two questions: (1) when the pretraining dataset is
class-imbalanced, how the downstream performance is affected? (2) In such
case, can our SBCL improve the learned feature extractor over existing
contrastive learning baselines? To answer these questions, we use the object
detection task of PASCAL VOC dataset as the evaluation suite and use
ImageNet/ImageNet-LT datasets as class-balanced/-imbalanced pretraining
datasets. Following [32, 23], we first pretrain a feature extractor on
ImageNet/ImageNet-LT then further finetune it for the downstream object
detaction tasks using Faster R-CNN [50] with R50-C4 backbone.

The experiment results are shown in Tables 3. From the results, we can see
that pretraining on class-balanced data (ImgeNet) leads to consistently better
results than that on class-imbalanced dataset (ImageNet-LT) pretraining the
model on the ImageNet and ImageNet-LT datasets by the SBCL can perform
slightly better than other baselines. In addition, the proposed SBCL
significantly outperforms baselines on class-imbalanced pretraining dataset,
while achieve comparable performance on class-balanced ones. For the
representation which trained on the full ImageNet dataset, the performance
advantage is not obvious. In Appendix A.1, we show additional experimental
results of object detection and instance segmentation on COCO [44] dataset and
SBCL also outperforms other baseline methods. Thus, we conclude that the
proposed SBCL is not only helpful for image classification, but also other
visual tasks.

###  4.4 Combining SBCL with SOTA methods

Another line of research to address the long-tailed problem is the ensemble-
based methods, such as MisLAS [71], WD [1], RIDE [61], PaCo [13], and BCL
[73]. Here we show that SBCL can also be leveraged to boost the performance of
these methods.

MisLAS use label-aware smoothing and shifted batch normalization to improve
the second stage (the classifier learning) on the long-tailed recognition,
while our method is to improve the first stage of representation learning.
Thus, MisLAS could be combined with SBCL and TSC. In Table 5, we report the
results of combing MisLAS with both SBCL and TSC. The results show that both
SBCL and TSC improve the performance of MisLAS and SBCL renders more
performance boost than TSC.

WD (weight decay) penalizes larger weights more heavily and so learns more
balanced parameters. Training a classifier on the learned features using
either/both weight decay and MaxNorm remarkably improves classification
accuracy [1]. Table 5 presents the accuracy of the classifier trained by
weight decay and MaxNorm, based on different representations learned by SBCL
and TSC. The results demonstrate that SBCL improves the performance of the WD
method [1] more effectively compared to TSC.

To implement SBCL with RIDE which incorporates multiple models in a multi-
expert framework, we follow [40] to simply replace the feature extractor on
stage-1 training in RIDE with that trained with SBCL and keep the stage-2
routing training unchange. As shown in Table 5, applying SBCL to RIDE improves
its performance with a significant gap, outperforms the combination of TSC and
RIDE on all different number of experts.

PaCo and BCL proposed new variants of supervised contrastive loss and jointly
train both the proposed loss and classification loss to improve long-tail
recognition, while we focus on the two stage pipeline, especially the first
stage of representation learning. In this experiment, we show that using
models pretrained with both TSC and SBCL as initialization could improve the
performance of both PaCo and BCL. The results can be found in Table 5, and we
can see that SBCL renders larger performance gain than TSC. The improvement
over PaCo and BCL sheds lights on potential future work to evaluate the
combination of multiple techniques of long-tail recognition to achieve new
SOTA results.

Table 5: Performance of the combination of SBCL and SOTA ensemble-based methods with ResNet-50 [25] on ImageNet-LT. ${\dagger}$ denotes the results of our re-implemented TSC.  Method | Many | Medium | Few | All
---|---|---|---|---
MiSLAS [71] | - | - | - | 52.7
WD+WD & Max [1] | 62.5 | 50.4 | 41.5 | 53.9
RIDE [61] (3 experts) | 66.2 | 51.7 | 34.9 | 54.9
PaCo [13] | 64.4 | 55.7 | 33.7 | 56.0
BCL [73] | 67.9 | 54.2 | 36.6 | 57.1
TSC+ MiSLAS | 63.7 | 50.5 | 36.0 | 53.6
TSC+WD+WD & Max | 63.8 | 51.9 | 41.6 | 55.1
TSC+ RIDE (3 experts) | 69.1 | 51.7 | 36.7 | 56.3
TSC+ RIDE (3 experts)${\dagger}$ | 68.6 | 51.4 | 36.0 | 55.9
TSC+ PaCo | 66.4 | 55.8 | 35.7 | 57.1
TSC+ BCL | 69.0 | 56.3 | 37.8 | 58.7
SBCL + MiSLAS | 64.1 | 52.0 | 36.4 | 54.5
SBCL +WD+WD & Max | 65.3 | 52.6 | 42.7 | 56.1
SBCL + RIDE (3 experts) | 69.2 | 52.4 | 36.9 | 56.8
SBCL + PaCo | 66.9 | 56.1 | 38.4 | 57.9
SBCL + BCL | 70.0 | 56.9 | 38.5 | 59.5

###  4.5 Ablation studies

#### Warm-up.

As mentioned in Section 3.3, we train the feature extractor for several epochs
using ordinary SCL or KCL as warm-up stage. As shown in Table 4, such a warm-
up stage is beneficial since the performance drops when we remove the warm-up
stage. This is likely because at the early stage of training, the extracted
feature is not well-trained and the cluster assignment could be noisy and
ineffective, hindering the efficacy of SBCL.

#### Adaptive clustering.

We are also curious about the efficacy of adaptive clustering and thus present
the performance of SBCL with clustering only once and fixing cluster
assignments during training. As shown in Table 4, without adaptive clustering,
the performance decreases in all cases. The reason could be fixed cluster
assignment is prone to noise when the model is not well-trained, while
adaptive clustering would dynamically adjust the cluster assignments based on
the current learned model, which is supposed to become better as the training
proceeds.

#### Dynamic temperature.

In Table 4, we also study the effectiveness of dynamic temperature (Section
3.2). We remove the dynamic temperature and simply set $\tau_{2}>\tau_{1}$
following [26]. With the fixed temperature $\tau_{2}$, the performance of SBCL
is significantly worse than that with dynamic temperature. We speculate that
this is because dynamic temperature could help prevent the instances of a
class to form overly large or small cluster in the feature space and therefore
lead to better learned representations. Additionally, to evaluate the impact
of dynamic temperature on other baselines, we apply the dynamic temperature on
TSC, as reported in Appendix A.1. The additional analysis of other
hyperparametets on SBCL is also shown in Appendix A.1.

###  4.6 Feature distribution of SBCL

To analyze the representation learned by SBCL, we firstly define the euclidean
distance between a given sample and other samples from the same/different
classes as intra/inter-class distance. Concretely, the euclidean distance
between a sample $z_{i}$ and a set $S$ is defined as
$\mathbf{D}(z_{i},S)=\frac{1}{|S|}\sum_{z_{j}\in S}\|z_{i}-z_{j}\|_{2}$. Then,
the intra- and inter-class distance of sample $z_{i}$ can be defined as
$\mathbf{D}(z_{i},P_{i})$ and $\mathbf{D}(z_{i},\mathcal{D}/P_{i})$
separately; and the intra- and inter-subclass distance of sample $z_{i}$ can
be defined as $\mathbf{D}(z_{i},M_{i})$ and $\mathbf{D}(z_{i},P_{i}/M_{i})$
separately.

Figure 2 visualizes the feature distribution of subclasses in a head class and
head classes in CIFAR-100-LT with imbalance ratio 100 trained by SBCL using
t-SNE [58]. The representations learned by SBCL form some separated clusters,
which be aligned with subclass labels and class labels. This suggests that
SBCL can learn an embedding space where features from same subclasses/classes
are pulled closely and features from different subclasses/class are separated
each other.

(a) Distribution of subclasses on a head class.

(b) Distribution of head classes on the dataset.

Figure 2: Feature distribution of subclasses and classes on CIFAR-100-LT with
imbalance ratio 100. Color represents suclasses/classes.

(a) Feature distance in subclasses.

(b) Feature distance in classes.

Figure 3: Feature distance of subclasses and classes on CIFAR-100-LT with
imbalance ratio 100. We randomly sample instances from many-shot and medium-
shot classes so that the size of each equals to that of few-shot classes.

Figure 3 shows the average distance between features learned by SBCL in
subclasses and classes on the CIFAR-100-LT. As shown in Figure 3(a), the
distance between samples from the same subclass is less than those from the
same class but different subclasses. Meanwhile, in Figure 3(b), the inter-
class distance is higher than the intra-class distance with stable value,
which denotes features from the different classes are obviously separated from
each other and uniformly distributed on a hypersphere. The results in all
indicate that the two-layer class hierarchy is successfully captured and
feature distribution achieves the core idea of SBCL. The specific values of
feature distance are displayed in Appendix A.1.

##  5 Related work

Traditional methods for handling long-tailed recognition problem includes re-
sampling and re-weighting. There are roughly two types of re-sampling
techniques: over-sampling the minority classes [53, 69, 3, 4] and under-
sampling the frequent classes [22, 31, 3]. The re-weighting techniques assign
adaptive weights for different classes or even different samples. The vanilla
scheme re-weights classes proportionally to the inverse of their frequency
[28, 29, 63]. For class-level re-weighting methods, many loss functions
including CB loss [14], LDAM loss [5] and Balanced softmax loss [49] were
recently proposed, while instance-level re-weighting methods include Focal
loss [43] and Influence-balanced loss [48]. Recently, two-stage algorithms
have achieved remarkable performance for long-tailed recognition, such as
classifier re-training (cRT) [33], learnable weight scaling (LWS) [33], and
Mixup Shifted Label-Aware Smoothing model (MiSLAS) [71]. Meanwhile, bilateral
branch network (BBN) [72] uses an additional network branch for re-balancing.
RIDE [61] use multiple branches named experts, each learning to specialize in
the entire classes. LADE [27] assumes the prior of test class distributions is
available and accordingly post-adjust model predictions. PaCo [13] applies
parametric class-wise learnable centers to rebalance in contrastive learning.
BCL [73] proposes a multi-branch framework to achieve class-averaging and
class-complement in the training process.

To boost the performance of the two-stage algorithms, researchers have
introduced supervised contrastive learning [35] to the first feature-learning
stage and proposed $k$-positive contrastive loss (KCL) [32] and targeted
supervised contrastive learning (TSC) [40]. While achieving the state-of-the-
art performance, these methods inject class-balance in the contrastive
learning objective, inevitably leading to instance-imbalance during training.
In this work, we instead propose to achieve both subclass- and instance-
balance in the contrastive learning object. Our method is also related to
recent studies of clustering-based deep unsupervised learning [17, 64, 42, 65,
6, 7], especially those that leverage contrastive learning [41, 39, 62, 21].
However, they target at general unsupervised representation learning scenario,
while our method is tailored for long-tailed recognition where the training
data is immensely class-imbalanced.

##  6 Conclusion

In this paper, we introduced Subclass-balancing Contrastive Learning (SBCL)
for long-tailed recognition. It breaks down the head classes into multiple
semantically-coherent subclasses via subclass-balancing adaptive clustering
and incorporates a bi-granularity contrastive loss that encourages both
subclass- and instance-balance. Extensive experiments on multiple datasets
demonstrate that SBCL achieves state-of-the-art single-model performance on
benchmark datasets for long-tailed recognition.

## References

  * [1] Shaden Alshammari, Yu-Xiong Wang, Deva Ramanan, and Shu Kong.  Long-tailed recognition via weight balancing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6897–6907, 2022.
  * [2] Shin Ando and Chun Yuan Huang.  Deep over-sampling framework for classifying imbalanced data.  In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 770–785. Springer, 2017.
  * [3] Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski.  A systematic study of the class imbalance problem in convolutional neural networks.  Neural Networks, 106:249–259, 2018.
  * [4] Jonathon Byrd and Zachary Lipton.  What is the effect of importance weighting in deep learning?  In International Conference on Machine Learning, pages 872–881. PMLR, 2019.
  * [5] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma.  Learning imbalanced datasets with label-distribution-aware margin loss.  Advances in Neural Information Processing Systems, 32, 2019.
  * [6] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze.  Deep clustering for unsupervised learning of visual features.  In European Conference on Computer Vision, 2018.
  * [7] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  Advances in Neural Information Processing Systems, 2020.
  * [8] Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer.  Smote: synthetic minority over-sampling technique.  Journal of artificial intelligence research, 16:321–357, 2002.
  * [9] Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia.  Distilling knowledge via knowledge review.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5008–5017, 2021.
  * [10] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International Conference on Machine Learning, pages 1597–1607. PMLR, 2020.
  * [11] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He.  Improved baselines with momentum contrastive learning.  arXiv preprint arXiv:2003.04297, 2020.
  * [12] Collobert, Ronan, Weston, and Jason.  A unified architecture for natural language processing: deep neural networks with multitask learning.  In International Conference on Machine Learning, 2008.
  * [13] Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia.  Parametric contrastive learning.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 715–724, 2021.
  * [14] Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie.  Class-balanced loss based on effective number of samples.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9268–9277, 2019.
  * [15] Carl Doersch, Abhinav Gupta, and Alexei A Efros.  Unsupervised visual representation learning by context prediction.  In Proceedings of the IEEE international conference on computer vision, pages 1422–1430, 2015.
  * [16] Qi Dong, Shaogang Gong, and Xiatian Zhu.  Imbalanced deep learning by minority class incremental rectification.  IEEE transactions on pattern analysis and machine intelligence, 41(6):1367–1381, 2018.
  * [17] Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox.  Discriminative unsupervised feature learning with convolutional neural networks.  Advances in Neural Information Processing Systems, 27, 2014.
  * [18] Spyros Gidaris, Praveer Singh, and Nikos Komodakis.  Unsupervised representation learning by predicting image rotations.  arXiv preprint arXiv:1803.07728, 2018.
  * [19] Priya Goyal, Dhruv Mahajan, Abhinav Gupta, and Ishan Misra.  Scaling and benchmarking self-supervised visual representation learning.  In Proceedings of the ieee/cvf International Conference on computer vision, pages 6391–6400, 2019.
  * [20] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent-a new approach to self-supervised learning.  Advances in neural information processing systems, 33:21271–21284, 2020.
  * [21] Yuanfan Guo, Minghao Xu, Jiawen Li, Bingbing Ni, Xuanyu Zhu, Zhenbang Sun, and Yi Xu.  Hcsc: Hierarchical contrastive selective coding.  Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2022.
  * [22] Haibo He and Edwardo A Garcia.  Learning from imbalanced data.  IEEE Transactions on knowledge and data engineering, 21(9):1263–1284, 2009.
  * [23] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729–9738, 2020.
  * [24] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
  * [25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016.
  * [26] David T Hoffmann, Nadine Behrmann, Juergen Gall, Thomas Brox, and Mehdi Noroozi.  Ranking info noise contrastive estimation: Boosting contrastive learning via ranked positives.  the Association for the Advance of Artificial Intelligence, 2022\.
  * [27] Youngkyu Hong, Seungju Han, Kwanghee Choi, Seokjun Seo, Beomsu Kim, and Buru Chang.  Disentangling label distribution for long-tailed visual recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6626–6636, 2021.
  * [28] Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang.  Learning deep representation for imbalanced classification.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5375–5384, 2016.
  * [29] Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang.  Deep imbalanced learning for face recognition and attribute prediction.  IEEE transactions on pattern analysis and machine intelligence, 42(11):2781–2794, 2019.
  * [30] Ahmet Iscen, André Araujo, Boqing Gong, and Cordelia Schmid.  Class-balanced distillation for long-tailed visual recognition.  Proceedings of the IEEE international conference on computer vision, 2021.
  * [31] Nathalie Japkowicz and Shaju Stephen.  The class imbalance problem: A systematic study.  Intelligent data analysis, 6(5):429–449, 2002.
  * [32] Bingyi Kang, Yu Li, Sa Xie, Zehuan Yuan, and Jiashi Feng.  Exploring balanced feature spaces for representation learning.  In International Conference on Learning Representations, 2020.
  * [33] Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis.  Decoupling representation and classifier for long-tailed recognition.  International Conference on Learning Representations, 2019.
  * [34] Alex Kendall, Yarin Gal, and Roberto Cipolla.  Multi-task learning using uncertainty to weigh losses for scene geometry and semantics.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7482–7491, 2018.
  * [35] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan.  Supervised contrastive learning.  Advances in Neural Information Processing Systems, 33:18661–18673, 2020.
  * [36] Jaehyung Kim, Jongheon Jeong, and Jinwoo Shin.  M2m: Imbalanced classification via major-to-minor translation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13896–13905, 2020.
  * [37] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [38] Harold W Kuhn.  The hungarian method for the assignment problem.  Naval research logistics quarterly, 2(1-2):83–97, 1955.
  * [39] Junnan Li, Pan Zhou, Caiming Xiong, and Steven C.H. Hoi.  Prototypical contrastive learning of unsupervised representations.  In International Conference on Learning Representations, 2021.
  * [40] Tianhong Li, Peng Cao, Yuan Yuan, Lijie Fan, Yuzhe Yang, Rogerio Feris, Piotr Indyk, and Dina Katabi.  Targeted supervised contrastive learning for long-tailed recognition.  Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
  * [41] Yunfan Li, Peng Hu, Zitao Liu, Dezhong Peng, Joey Tianyi Zhou, and Xi Peng.  Contrastive clustering.  AAAI Conference on Artificial Intelligence, 2021.
  * [42] Renjie Liao, Alex Schwing, Richard Zemel, and Raquel Urtasun.  Learning deep parsimonious representations.  Advances in Neural Information Processing Systems, 29, 2016.
  * [43] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2980–2988, 2017.
  * [44] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [45] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu.  Large-scale long-tailed recognition in an open world.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2537–2546, 2019.
  * [46] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten.  Exploring the limits of weakly supervised pretraining.  In European conference on computer vision, pages 181–196, 2018\.
  * [47] Mehdi Noroozi and Paolo Favaro.  Unsupervised learning of visual representations by solving jigsaw puzzles.  In European conference on computer vision, pages 69–84. Springer, 2016.
  * [48] Seulki Park, Jongin Lim, Younghan Jeon, and Jin Young Choi.  Influence-balanced loss for imbalanced visual classification.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 735–744, 2021.
  * [49] Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, et al.  Balanced meta-softmax for long-tailed visual recognition.  Advances in Neural Information Processing Systems, 33:4175–4186, 2020.
  * [50] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in neural information processing systems, 28, 2015.
  * [51] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  International journal of computer vision, 115(3):211–252, 2015\.
  * [52] Xie Saining, Girshick Ross, Dollar Piotr, Tu Zhuowen, and He Kaiming.  Aggregated residual transformations for deep neural networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, page 1492–1500, 2017.
  * [53] Li Shen, Zhouchen Lin, and Qingming Huang.  Relay backpropagation for effective learning of deep convolutional neural networks.  In European conference on computer vision, pages 467–482. Springer, 2016.
  * [54] Tony Shen, Ariel Lee, Carol Shen, and C Jimmy Lin.  The long tail and rare disease research: the impact of next-generation sequencing for rare mendelian disorders.  Genetics research, 97, 2015.
  * [55] Aditya Singh and Alessandro Bay.  On mixup training: Improved calibration and predictive uncertainty for deep neural networks.  Advances in Neural Information Processing Systems, 2019.
  * [56] Merrielle Spain and Pietro Perona.  Measuring and predicting importance of objects in our visual world.  2007\.
  * [57] AFM Uddin, Mst Monira, Wheemyung Shin, TaeChoong Chung, Sung-Ho Bae, et al.  Saliencymix: A saliency guided data augmentation strategy for better regularization.  International Conference on Learning Representations, 2021.
  * [58] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  Journal of machine learning research, 9(11), 2008.
  * [59] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie.  The inaturalist species classification and detection dataset.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 8769–8778, 2018.
  * [60] Feng Wang and Huaping Liu.  Understanding the behaviour of contrastive loss.  In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pages 2495–2504, 2021.
  * [61] Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella X Yu.  Long-tailed recognition by routing diverse distribution-aware experts.  International Conference on Learning Representations, 2021.
  * [62] Xudong Wang, Ziwei Liu, and Stella X Yu.  Unsupervised feature learning by cross-level instance-group discrimination.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12586–12595, 2021.
  * [63] Yu-Xiong Wang, Deva Ramanan, and Martial Hebert.  Learning to model the tail.  Advances in Neural Information Processing Systems, 30, 2017.
  * [64] Junyuan Xie, Ross Girshick, and Ali Farhadi.  Unsupervised deep embedding for clustering analysis.  In International conference on machine learning, pages 478–487. PMLR, 2016.
  * [65] Jianwei Yang, Devi Parikh, and Dhruv Batra.  Joint unsupervised learning of deep representations and image clusters.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 5147–5156, 2016.
  * [66] Yuzhe Yang and Zhi Xu.  Rethinking the value of labels for improving class-imbalanced learning.  Advances in Neural Information Processing Systems, 33:19290–19301, 2020.
  * [67] Yuzhe Yang, Guo Zhang, Dina Katabi, and Zhi Xu.  Me-net: Towards effective adversarial robustness with matrix estimation.  International Conference on Machine Learning, 2019.
  * [68] Han-Jia Ye, Hong-You Chen, De-Chuan Zhan, and Wei-Lun Chao.  Identifying and compensating for feature deviation in imbalanced deep learning.  arXiv preprint arXiv:2001.01385, 2020.
  * [69] Qiaoyong Zhong, Chao Li, Yingying Zhang, H Sun, S Yang, D Xie, and S Pu.  Towards good practices for recognition & detection.  In CVPR workshops, volume 1, page 2, 2016.
  * [70] Yaoyao Zhong, Weihong Deng, Mei Wang, Jiani Hu, Jianteng Peng, Xunqiang Tao, and Yaohai Huang.  Unequal-training for deep face recognition with long-tailed noisy data.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7812–7821, 2019.
  * [71] Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia.  Improving calibration for long-tailed recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16489–16498, 2021.
  * [72] Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen.  Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9719–9728, 2020.
  * [73] Jianggang Zhu, Zheng Wang, Jingjing Chen, Yi-Ping Phoebe Chen, and Yu-Gang Jiang.  Balanced contrastive learning for long-tailed visual recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6908–6917, 2022.
  * [74] George Kingsley Zipf.  The psycho-biology of language: An introduction to dynamic philology.  Routledge, 2013.

##  Appendix A APPENDIX

###  A.1 Additional experiment results

Figure 4: Accuracy of each classes on CIFAR100-LT. The black line is the class
distribution, and the classes in the left part are head classes while those in
the right part are tail classes.

#### Accuracy of each class on CIFAR-100-LT.

We visualize the accuracy of each class of both SCL and SBCL on CIFAR-100-LT
with imbalance ratio 100 (Figure 4). From the results, we can see that SBCL
improves performance on tail classes over SCL without the expense of the
perforamnce of the head classes.

#### The per-class weight norm of a linear classifier trained on top of
features learned by SBCL in different training stages.

Figure 5 shows the per-class weight norm of a linear classifier trained on top
of features learned by SBCL in different training stages on CIFAR-100-LT. From
the figure, we can see that as the training proceeds, the per-class weight
norm becomes model balanced even when training the linear classifier, the
original cross-entropy loss is used.

Figure 5: Change in weight norm of the linear classifier based on the
representations trained by SBCL on CIFAR-100-LT with imbalance ratio 100
during training.

#### Selection of negative instances in SBCL.

Our proposed loss in Eq. 4 consists of two supervised contrastive losses with
subclass and class labels respectively. The first term regards instances in
different subclasses as negative instead of these in different classes; In
Table 6, we show that such design choice leads to better performance than
using instances in different classes as negative, which illustrates the
effectiveness of exploiting the rich semantic in head classses.

Table 6: Different selection of negative instances in SBCL on CIFAR-100-LT with different imbalance ratio. The ‘Class label‘ row are the first term of loss function is constructed by class label and the ’Subclass label’ row are subclass label. Negative samples | Imbalance Ratio
---|---
100 | 50 | 10
Class label | 43.7 | 47.6 | 56.8
Subclass label | 44.9 | 48.7 | 57.9

#### Analysis of feature distribution.

To analyze the representation learned by SBCL, we firstly define the euclidean
distance between a given sample and other samples from the same/different
classes as intra/inter-class distance. Concretely, the euclidean distance
between a sample $z_{i}$ and a set $S$ is defined as
$\mathbf{D}(z_{i},S)=\frac{1}{|S|}\sum_{z_{j}\in S}\|z_{i}-z_{j}\|_{2}$. Then,
the intra- and inter-class distance of sample $z_{i}$ can be defined as
$\mathbf{D}(z_{i},P_{i})$ and $\mathbf{D}(z_{i},\mathcal{D}/P_{i})$
separately; and the intra- and inter-subclass distance of sample $z_{i}$ can
be defined as $\mathbf{D}(z_{i},M_{i})$ and $\mathbf{D}(z_{i},P_{i}/M_{i})$
separately.

To leverage instance semantic coherence to balance the feature space, we
expect instances of high semantic coherence to form a more concentrated
cluster than other instances in the same class. So, we embed the subclass-
balancing adaptive clustering strategy on SBCL to illustrate this on
CIFAR-100-LT with imbalance ratio 100. In Table 7, we report the intra-
subclass/inter-subclass distance on different splits. The results show that
SBCL achieves to concentrate instances from the same subclass and pulls
instances from different subclasses away on all splits.

SBCL aims at learning a compact representation space, in which representations
from different classes are far from each other and the feature space spanned
by representations of each class is invariant to the long-tailed distribution.
The average intra/inter-class distance are summarized in Table 7 and the
distances of different groups are reported separately. The results also show
SBCL clears the boundary of feature distribution on the different class
splits.

Table 7: Average intra/inter-subclass and intra/inter-class distance of features learned by SBCL. Distance | Many | Medium | Few | All
---|---|---|---|---
Intra-subclass | 0.68 | 0.76 | 0.87 | 0.70
Inter-subclass | 1.02 | 0.99 | 0.89 | 1.01
Intra-class | 1.00 | 0.94 | 0.88 | 0.99
Inter-class | 1.39 | 1.38 | 1.37 | 1.39

#### Hyperparameter analysis on CIFAR-100-LT.

(a)

(b)

(c)

(d)

Figure 6: Analysis of SBCL as a loss function of different hyperparameters on CIFAR-100-LT with imbalance ratio 100. (a): Sample number in clusters with different cluster algorithm. (b): Top-1 accuracy of SBCL/TSC as a function of different batch size. (c): Top-1 accuracy of SBCL/TSC as a function of different pretraining epochs. (d): Top-1 accuracy of SBCL/TSC as a function of different learning rates. Table 8: Distribution of sample number in subclass on CIFAR-100-LT with imbalance ratio 100. Dataset | Max | Min | Average | Std | Imbalance ratio(Max/Min)
---|---|---|---|---|---
Kmean clustering algorithm | 40 | 1 | 10.34 | 6.27 | 40
Subclass-balancing adaptive clustering algorithm | 19 | 2 | 10.34 | 1.60 | 9.5

Figure 6(a) and Table 8 show the distribute situation of sample number in
subclasses obtained by different cluster algorithms on CIFAR-100-LT with
imbalance ratio 100. For Kmean cluster algorithm, the imbalance phenomenon of
subclasses is obvious. When using our proposed cluster algorithm, the
imbalance ratio of sample number in subclasses deceases from 40 to 9.5. And
the standard deviation of sample number on CIFAR-100-LT is relatively small,
which denotes the number of samples in most subclasses keeps stable in a
certain range.

Figure 6(b) shows the impact of batch size on SBCL/TSC. We find that larger
batch sizes have a significant advantage over the smaller ones. This is
because larger batch sizes provide more negative examples to facilitate
convergence. However, the over-large batch size hurts the model performance.
And SBCL and TSC are equally sensitive to batch size on CIFAR-100-LT.

Figure 6(c) shows the curve of the accuracy of SBCL/TSC vs. the number of
training epochs. From the curve, we can see that the performance of SBCL and
TSC both converge after 800 epochs. When the model is trained with SBCL over
600 epochs, its performance already exceeds TSC.

In Figure 6(d), we display the performance of SBCL with different learning
rates on CIFAR-100-LT with imbalance ratio 100. As shown in the figure, the
learning rate has significant impact on the performance, and we set the
learning rate as 0.5 for CIFAR-100-LT.

#### Combining TSC with dynamic temperature.

According to Table 4, dynamic temperature effectively contributes to the
improvement of accuracy. We also add the dynamic temperature to the second
term of TSC [40] and the experiment results are shown in Table 9. However, the
improvement of the dynamic temperature on TSC is less significant than that on
our method, which is reasonable because we introduce dynamic temperature for
the loss to distinguish between class and subclass, while TSC does not have
subclass and therefore the dynamic temperature is less effective.

Table 9: Combination of TSC and SBCL with dynamic temperature. Dynamic temperature | CIFAR-100-LT
---|---
TSC | SBCL
Imbalance Ratio | 100 | 50 | 10 | 100 | 50 | 10
| 43.5 | 47.6 | 58.7 | 43.8 | 47.8 | 57.0
$\checkmark$ |  43.9(+0.4) |  48.0(+0.4) |  59.2(+0.5) |  44.9(+1.1) |  48.7(+0.9) |  57.9(+0.9)

#### Warm-up on ImageNet-LT.

Instead of using the SCL at the warm-up stage for CIFAR datasets, KCL is
adopted for ImageNet-LT and iNaturalist 2018 datasets to warm up the feature
extractor. As Table 10 shows, warm-up phase makes feature extractor improve
accuracy on all splits of ImageNet-LT. This is because it prevents cluster
assignment from feature random distribution at the begining and avoids using
the SCL to make the feature space dominated by the head class at the warm-up
stage.

Table 10: SBCL with and without warm-up stage on ImageNet-LT.  Method | Many | Medium | Few | All
---|---|---|---|---
SBCL without warm-up | 62.9 | 49.6 | 29.3 | 52.0
SBCL with warm-up | 63.8 | 51.3 | 31.2 | 53.4

#### Advantages of cluster validity.

Actually, previous studies [32, 40] have proven that randomly sampling
balanced instances as positive pairs (such as KCL, TSC) is better than
sampling all instances of the same class as positive pairs (such as SCL).
However, this strategy may destruct instance semantic coherence. In Table 11,
we replace the first team (regard subclasses as positive pairs) with the
balanced positive sampling strategy (KCL) to prove this on ImageNet-LT. As the
results show, subclass-balancing adaptive clustering strategy brings more
improvement to SBCL than balanced positive sampling strategy.

Table 11: Subclass-balancing adaptive clustering strategy improves more than balanced positive sampling strategy on ImageNet-LT. Method | Many | Medium | Few | All
---|---|---|---|---
FCL | 61.4 | 47.0 | 28.2 | 49.8
KCL | 62.4 | 49.0 | 29.5 | 51.5
TSC | 63.5 | 49.7 | 30.4 | 52.4
SBCL (KCL) | 63.3 | 49.5 | 30.6 | 52.2
SBCL | 63.8 | 51.3 | 31.2 | 53.4

#### COCO object detection and instance segmentation.

In this section, following the experiment setting in [23], we use Mask R-CNN
[24] to conduct the object detection and instance segmentation experiments on
COCO dataset. The schedule is the default 2× in [23]. Table 12 shows the
pretrained model trained by SBCL outperforms it learned with other contrastive
learning for the downstream tasks.

Table 12: Object detection and instance segmentation results on COCO dataset. The representation model is trained on ImageNet and ImageNet-LT. We report results in bounding-box AP (APbb) and mask AP (APmk).  Method | ImageNet | ImageNet-LT
---|---|---
AP |  AP50 |  AP75 | AP |  AP50 |  AP75
APbb | random init. | 35.6 | 54.6 | 38.2 | 35.6 | 54.6 | 38.2
CE | 40.1 | 59.8 | 43.3 | 38.1 | 57.4 | 41.2
CL [23] | 40.4 | 60.1 | 44.1 | 39.7 | 59.4 | 42.7
KCL [32] | 40.8 | 60.6 | 44.0 | 39.4 | 59.1 | 42.6
SBCL | 41.1 | 60.8 | 44.2 | 40.0 | 59.6 | 43.0
APmk | random init. | 31.4 | 51.5 | 33.5 | 31.4 | 51.5 | 33.5
CE | 34.9 | 56.6 | 37.0 | 33.3 | 54.2 | 35.4
CL [23] | 35.1 | 56.9 | 37.6 | 34.7 | 56.1 | 37.1
KCL [32] | 35.5 | 57.4 | 37.8 | 34.4 | 55.8 | 36.4
SBCL | 35.7 | 57.5 | 37.9 | 35.0 | 56.3 | 37.3

#### Hyperparameter studies.

Here, we study the effect of hyperparameters $\beta$ and $\delta$. Note that
$\beta$ controls the balance of two loss terms in Eq. 4 and $\delta$
determines the lower bound of the cluster size in Eq. 3. Specifically, on
CIFAR-100-LT with imbalance ratio 100, we vary the values of $\beta$ from
{0.1, 0.2, 0.5, 0.8, 1.0, 2.0} with $\delta=10$ and the value of $\delta$ from
{5, 10, 20, 30, 50, 100} with $\beta=0.2$. The results are summarized in Table
13. We observe that the smaller $\beta$ values (between 0.1 and 0.5) can
achieve relatively good performance, with the best being 0.2. This observation
aligns with our intuition of emphasizing the subclass-level contrastive loss,
because smaller $\beta$ is equivalent to putting more weights on the first
term of Eq. 4, which corresponds to the subclass-level contrastive. For
$\delta$, the values between 5 and 30 yield high accuracy, with the best being
10. We can see that large $\delta$ values ($\delta=50,100$) lead to
significant drop in performance. We argue that this is because large $\delta$
value would result in subclasses that contain more instance than tail classes
and therefore affect the subclass-balance, leading to suboptimal performance.
In addition, smaller $\delta$ value ($\delta=5$) also causes performance drop;
the reason could be small cluster size may let similar instance being assigned
to different clusters and therefore affect the learned representations.
Therefore, we fix $\beta=0.2$ and $\delta=10$ for all experiments.

Table 13: Hyperparameter study of $\beta$ and $\delta$ on CIFAR-100-LT with imbalance ratio 100. $\beta$ | 0.1 | 0.2 | 0.5 | 0.8 | 1.0 | 2.0
---|---|---|---|---|---|---
ACC(%) | 44.6 | 44.9 | 44.5 | 44.1 | 43.9 | 42.1
$\delta$ | 5 | 10 | 20 | 30 | 50 | 100
ACC(%) | 44.3 | 44.9 | 44.6 | 44.3 | 42.9 | 42.3

#### Visualization of generated clusters.

In Figure 7, we show the clustering results of ImageNet-LT training images
generated by subclass-balancing adaptive clustering algorithm. From the
results, we can see that the algorithm is able to find the subclasses with
similar patterns, helping the model learn semantic coherent representations.
For example, the two subclasses in the bottom-left are telephone with/without
human.

Figure 7: Visualization of subclasses generated by SBCL. Images with green and
orange boarder are randomly drawn from different subclasses within the same
classes. We can see that SBCL could produce semantically coherent subclasses.

###  A.2 Additional information

#### Benchmark datasets statistical information and Implementation Details.

Table 14: Statistics of datasets. The imbalance ratio $\rho=n_{1}/n_{C}$.

Dataset | classes | training data | test data | imbalance ratio
---|---|---|---|---
CIFAR-100-LT | 100 | 50,000 | 10,000 | $\left\\{100,50,10\right\\}$
ImageNet-LT | 1,000 | 115,846 | 50,000 | 256
iNaturalist 2018 | 8,142 | 437,513 | 24,426 | 500

Table 15: Hyperparameters used by different loss functions for benchmark datasets. The detailed hyperparameters of iNaturalist 2018 are the same as the ImageNet-LT. Hyperparameters | ImageNet-LT | CIFAR100-LT
---|---|---
TSC | SBCL | TSC | SBCL
module | MoCo | MoCo | SimCLR | SimCLR
warm-up epoch | 200 | 200 | 0 | 10
epoch | 400 | 400 | 1000 | 1000
batch size | 256 | 256 | 1024 | 1024
learning rate | 0.1 | 0.1 | 0.5 | 0.5
learning rate schedule | cosine | cosine | cosine | cosine
memory size | 65536 | 65536 | - | -
encoder momentum | 0.999 | 0.999 | - | -
feature dimension | 128 | 128 | 128 | 128
softmax temperature | 0.07 | 0.07 | 0.1 | 0.1
$k$-positive number | 6 | - | 4 | -
hyperparameter of $\beta$ | 0.2 | 0.2 | 0.2 | 0.2
hyperparameter of $\delta$ | - | 20 | - | 10
hyperparameter of $\alpha$ | - | 10 | - | 10

We summarize the statistical information of the three benchmark datasets in
Table 14. Following [33, 32, 40], we apply SBCL on the long-tailed recognition
by using a two-stage training strategy: (i) train the representation with
SBCL; (ii) learn a linear classifier on top of the fixed representation. The
training process is the same as TSC [40]. Thus, we use TSC default
hyperparameters and implementation details for the representation learning.
For CIFAR-100-LT dataset, all experiments are performed on 2 NVIDIA RTX 3090
GPUs. For ImageNet-LT and iNaturalist 2018 datasets, we perform the
experiments on 8 NVIDIA RTX 3090 GPUs. The detailed hyperparameters of TSC and
SBCL are given in Table 15.

For the classify learning, training the linear classifier strategy is the same
with TSC [40]; so, we use TSC default hyperparameters and implementation
details for the classifier learning. For the detect model learning, we follow
MoCo [23] to adopot the same setting, hyperparameters and evolution metrics
with R50-C4 backbone. For Pascal VOC dataset, we train Faster R-CNN [50] on
VOC07+12 and evaluate on the test set of VOC07. For COCO dataset, we train
Mask R-CNN [24] on train2017 set and evaluate on val2017 set.

#### Limitations.

SBCL has some limitations. First, clustering the head class in SBCL takes a
long time on the training phase, especially for ImageNet-LT and iNaturalist
2018. Second, SBCL requires knowing the number of samples in each class to
decide the cluster number; so, it is not applicable to problems where the
number of samples is unknown.

#### Social impacts.

This work aims to propose a novel representation learning to help people
resolve the bias in the real world data recognition, which might has positive
social impact. We do not foresee any form of negative social impact induced by
our work.

#### Privacy information in data.

All datesets we used in the experiment are public. The datasets only include
the pictures, which most are animals and plants. No private information is
included.

#### Baseline information.

We report the accuracy of KCL and TSC on different benchmark datasets from
[40]. For SwAV111SwAV offical implementation:
https://github.com/facebookresearch/swav. [7], PCL222PCL offical
implementation: https://github.com/salesforce/PCL. [39] and BYOL333BYOL
offical implementation: https://github.com/deepmind/deepmind-
research/tree/master/byol. [20], we use their official open-source
implementations.


* Frolov  _et al._ [2017] V. Frolov, P. Krtous, and D. Kubiznak, Black holes, hidden symmetries, and complete integrability, Living Rev. Rel. 20, 6 (2017), arXiv:1705.05482 [gr-qc] .


* [112] N. T. Varopoulos, L. Saloff-Coste, and T. Coulhon.  Analysis and geometry on groups, volume 100 of Cambridge Tracts in Mathematics.  Cambridge University Press, Cambridge, 1992.
  * [113] J. L. Vázquez.  Smoothing and decay estimates for nonlinear diffusion equations, volume 33 of Oxford Lecture Series in Mathematics and its Applications.  Oxford University Press, Oxford, 2006.
  * [114] J. L. Vázquez.  The porous medium equation. Mathematical theory.  Oxford Mathematical Monographs. The Clarendon Press, Oxford University Press, Oxford, 2007.
  * [115] J. L. Vázquez.  The mathematical theories of diffusion. Nonlinear and fractional diffusion.  In Nonlocal and Nonlinear Diffusions and Interactions: New Methods and Directions, volume 2186 of C.I.M.E. Foundation Subseries, pages 205–278. Springer International Publishing, 2017.
  * [116] J. L. Vázquez, A. de Pablo, F. Quirós, and A. Rodríguez.  Classical solutions and higher regularity for nonlinear fractional diffusion equations.  J. Eur. Math. Soc. (JEMS), 19(7):1949–1975, 2017.
  * [117] L. Véron.  Effets régularisants de semi-groupes non linéaires dans des espaces de Banach.  Ann. Fac. Sci. Toulouse Math. (5), 1(2):171–200, 1979.
  * [118] F. Xu.  A class of singular symmetric Markov processes.  Potential Anal., 38(1):207–232, 2013.