# Open-Sampling: Exploring Out-of-Distribution Data for Re-balancing Long-
tailed Datasets

Hongxin Wei  Lue Tao  Renchunzi Xie  Lei Feng  Bo An

###### Abstract

Deep neural networks usually perform poorly when the training dataset suffers
from extreme class imbalance. Recent studies found that directly training with
out-of-distribution data (i.e., open-set samples) in a semi-supervised manner
would harm the generalization performance. In this work, we theoretically show
that out-of-distribution data can still be leveraged to augment the minority
classes from a Bayesian perspective. Based on this motivation, we propose a
novel method called Open-sampling, which utilizes open-set noisy labels to re-
balance the class priors of the training dataset. For each open-set instance,
the label is sampled from our pre-defined distribution that is complementary
to the distribution of original class priors. We empirically show that Open-
sampling not only re-balances the class priors but also encourages the neural
network to learn separable representations. Extensive experiments demonstrate
that our proposed method significantly outperforms existing data re-balancing
methods and can boost the performance of existing state-of-the-art methods.

long-tailed, out-of-distribution,

##  1 Introduction

The success of deep neural networks (DNNs) heavily relies on large-scale
datasets with balanced distribution (Krizhevsky et al., 2009; Russakovsky et
al., 2015). However, in real-world applications like autonomous driving and
medical diagnosis, large-scale datasets naturally exhibit imbalanced and long-
tailed distributions, i.e., a few classes (majority classes) occupy most of
the data while most classes (minority classes) are under-represented (Zhou et
al., 2017; Van Horn et al., 2018; Lin et al., 2014). It has been shown that
training on long-tailed datasets leads to poor generalization performance,
especially on the minority classes (Zhou et al., 2020; Liu et al., 2019a; Kang
et al., 2020). Thus, designing effective algorithms to handle class imbalance
is of great practical importance.

In the literature, a popular direction in long-tailed learning is to re-
balance the data distribution by data re-sampling (Japkowicz & Stephen, 2002;
He & Garcia, 2009). For example, Over-sampling (Byrd & Lipton, 2019; Shen et
al., 2016) repeats samples from under-presented classes, but it usually causes
over-fitting to the minority classes. To alleviate the over-fitting issue,
synthesized novel samples are introduced to augment the minority classes
without repetition (Chawla et al., 2002). As a result, the model is still
error-prone due to noise in the synthesized samples (Cui et al., 2019) A
recent work (Yang & Xu, 2020) introduced unlabeled-in-distribution data to
compensate for the lack of training samples, and showed that directly adding
unlabeled data from mismatched classes (_i.e._ , out-of-distribution data) by
semi-supervised learning hurts the generalization performance. These data
augmentation methods normally require in-distribution data with precise labels
for selected classes. However, such kind of data would be extremely hard to
collect in real-world scenarios, due to the expensive labeling cost. This
fatal weakness of previous methods motivates us to explore the possibility of
using out-of-distribution (OOD) data for long-tailed imbalanced learning.

In this paper, we theoretically show that out-of-distribution data (i.e.,
open-set samples) could be leveraged to augment the minority classes from a
Bayesian perspective. Based on this motivation, we propose a simple yet
effective method called Open-sampling, which uses open-set noisy labels to re-
balance the label priors of the training dataset. For each OOD instance, the
label is sampled from our pre-defined distribution that is complementary to
the original class priors. To alleviate the over-fitting issue on the minority
classes, a class-dependent weight is used in the training objective to provide
stronger regularization on the minority classes than the majority classes. In
this way, the open-set noisy labels could be used to re-balance the class
priors while retaining their non-toxicity.

To provide a comprehensive understanding, we conduct a series of analyses to
illustrate the properties of the proposed Open-sampling method. From these
empirical analyses, we show that: 1) the Complementary Distribution is
superior to the commonly used Class Balanced distribution (CB) as the former
is closer to the uniform distribution, which reduces the harmfulness of the
open-set noisy labels; 2) real-world datasets with large sample size are the
best choices for the open-set auxiliary dataset in Open-sampling and the
diversity (i.e., number of classes) is not an important factor in the method;
3) the Open-sampling method not only re-balances the class prior, but also
promotes the neural network to learn more separable representations.

To the best of our knowledge, we are the first to explore the benefits of OOD
instances in learning from long-tailed datasets. To verify the effectiveness
of our method, we conduct experiments on four long-tailed image classification
benchmark datasets, including long-tailed CIFAR10/100 (Krizhevsky et al.,
2009), CelebA-5 (Liu et al., 2015b; Kim et al., 2020), and Places-LT (Zhou et
al., 2017). Empirical results show that our method can be easily incorporated
into existing state-of-the art methods to enhance their performance on long-
tailed imbalanced classification tasks. Furthermore, experimental results
validate that our method could also achieve impressive performance for
detecting OOD examples under class-imbalanced setting. Code and data are
publicly available at https://github.com/hongxin001/open-sampling.

##  2 Imbalanced Learning with OOD instances

###  2.1 Background

In this work, we consider a multi-class classification problem, where the
input space is denoted by $\mathcal{X}\in\mathbb{R}^{d}$ and the label space
$\mathcal{Y}$ is $\\{1,\ldots,K\\}$. We denote by
$\mathcal{D}_{\mathrm{train}}=\\{(\boldsymbol{x}_{i},y_{i})\\}^{N}_{i=1}\in\mathcal{X}\times\mathcal{Y}$
the training dataset with $N$ samples. Let $n_{j}$ be the number of samples in
class $j$, then $N=\sum^{K}_{j=1}n_{j}$. Let $P_{\mathrm{s}}(X,Y)$ define the
underlying training (source) distribution and $P_{\mathrm{t}}(X,Y)$ define the
test (target) distribution. Generally, the class imbalance problem assumes
that the test data has the same class conditional probability as the training
data, i.e., $P_{\mathrm{s}}(X|Y)=P_{\mathrm{t}}(X|Y)$, while their class
priors are different, i.e., $P_{\mathrm{s}}(Y)\neq P_{\mathrm{t}}(Y)$.

Besides, we consider an unlabelled auxiliary dataset
$\mathcal{D}_{\mathrm{out}}^{(\boldsymbol{x})}=\\{\tilde{\boldsymbol{x}}_{i}\\}^{M}_{i=1}\in\mathcal{X}$
consisting of $M$ open-set instances, and we have $M\gg N$. These open-set
instances are also known as OOD data as they are sampled from
$P_{\mathrm{out}}(X)$, which is disjoint from $P_{\mathrm{s}}(X)$. In real-
world scenarios, it is easy to obtain such auxiliary datasets, which are
commonly used in OOD detection tasks. In what follows, we may assign each
open-set instance $\tilde{\boldsymbol{x}}_{i}$ with a random label
$\tilde{y}_{i}\in\mathcal{Y}$, which is independently sampled from an
appropriate label distribution $P_{\mathrm{out}}(Y)$ over $\mathcal{Y}$. We
denote by
$\mathcal{D}_{\mathrm{out}}=\\{(\tilde{\boldsymbol{x}}_{i},y_{i})\\}^{M}_{i=1}$
the auxiliary dataset with randomly sampled noisy labels.

###  2.2 Theoretical Motivation

From a Bayesian perspective, the prediction of a Bayes classifier is generally
made as follows:

$y^{*}=\underset{y\in\mathcal{Y}}{\arg\max}~{}P(y|\boldsymbol{x})=\underset{y\in\mathcal{Y}}{\arg\max}~{}P(\boldsymbol{x}|y)P(y),$
(1)

where $P(\boldsymbol{x})$ and $P(y)$ represent $P(X=\boldsymbol{x})$ and
$P(Y=y)$, respectively. Unfortunately, the predicted posterior probability in
Eq. (1) becomes unreliable when there is a large discrepancy between the class
priors of training and test distributions. Specifically, the class prior of
the test distribution is usually balanced (i.e., a uniform distribution over
labels), while the training dataset exhibits a long-tailed class distribution.
Formally, we have $P_{\mathrm{s}}(Y=i)\neq P_{\mathrm{t}}(Y=i)$ for any class
$i\in\mathcal{Y}$, and $P_{\mathrm{s}}(Y=j)\ll P_{\mathrm{t}}(Y=j)$ for a
minority class $j\in\mathcal{Y}$.

To re-balance the class priors of the training dataset, existing re-sampling
methods mostly augment the minority classes with extra in-distribution samples
with precise labels, e.g., duplicated examples (Buda et al., 2018), synthetic
generation (He et al., 2008), and interpolation (Chawla et al., 2002).
However, due to their strict in-distribution constraints on data distribution
and label quality, generating and collecting those samples are challenging and
expensive, especially for minority classes. In this paper, we instead try to
break through the constraints by demonstrating that OOD instances with noisy
labels are actually useful for re-balancing the training dataset.

We start by presenting an intriguing fact in the following theorem (proof in
Appendix A), which demonstrates that augmenting the training data with open-
set instances and uniformly random labels can be non-toxic.

######  Theorem 2.1.

Assume that $P_{\mathrm{out}}(Y)$ is the discrete uniform distribution over
the label space $\mathcal{Y}$. Let the augmented dataset be
$\mathcal{D}_{\mathrm{mix}}=\mathcal{D}_{\mathrm{train}}\cup\mathcal{D}_{\mathrm{out}}$,
and $P_{\mathrm{mix}}(X,Y)$ be the underlying data distribution of
$\mathcal{D}_{\mathrm{mix}}$, then we have

$\underset{y\in\mathcal{Y}}{\arg\max}~{}P_{\mathrm{mix}}(\boldsymbol{x}|y)P_{\mathrm{mix}}(y)=\underset{y\in\mathcal{Y}}{\arg\max}~{}P_{\mathrm{s}}(\boldsymbol{x}|y)P_{\mathrm{s}}(y).$

Theorem 2.1 indicates that, when the labels are uniformly sampled from the in-
distribution label space, the prediction of the Bayes classifier is unchanged
after augmenting the training dataset with the OOD instances111The discovery
of the non-toxicity of OOD instances is not new and has been studied in ODNL
(Wei et al., 2021). Our unique contribution here is to theoretically prove
this property from a Bayesian perspective. We provide a detailed comparison
for these two works in Subsection 2.3. . In this way, the theoretical result
motivates us to exploit the potential value of OOD instances to repair class
imbalance.

Figure 1: The test accuracy of models under different training epochs. Blue
color denotes the model trained on long-tailed CIFAR-10 with ResNet-34, and
gray color denotes the model trained on long-tailed CIFAR-10 and OOD data. We
use instances from 300K Random Images (Hendrycks et al., 2019) as OOD data and
label as a minority class—“9”. The comparison implies that simply adding OOD
data into training may downgrade the generalization performance.

Although using the uniform distribution as $P_{\mathrm{out}}(Y)$ will not
downgrade the Bayes classifier as shown above, the resulting classifier is not
yet optimal, and its corresponding partition is still far away from the
optimal decision boundary on the test distribution, since the label
distribution $P_{\mathrm{mix}}(Y)$ still remains largely imbalanced. In the
following, we will show that a better classifier can be obtained by exploring
the trade-off between re-balancing the label distribution and keeping non-
toxicity.

###  2.3 Open-sampling

Motivated by the previous analysis, we propose to exploit the open-set
auxiliary dataset to improve the generalization under class-imbalanced
settings. With a proper label distribution, OOD instances with dynamic labels
can be used to re-balance the class priors while retaining their non-toxicity.
We start by giving the following definition.

######  Definition 2.2 (Complementary Distribution).

Complementary Distribution (CD) is a label distribution for the auxiliary
dataset to re-balance the class priors of the original dataset. In particular,
Minimum Complementary Distribution (MCD) is the complementary distribution
that requires the smallest number of auxiliary instances to re-balance the
original training set.

Designing a proper Complementary Distribution to mitigate the class imbalance
problem is a difficult problem that depends on the trade-off between re-
balancing the class priors and keeping the non-toxicity of the added noisy
labels. Intuitively, to re-balance the class priors, more OOD instances should
be allocated into the minority classes than the majority classes. Meanwhile,
the unequal number of OOD instances in different classes may shift the Bayes
classifier.

The above conflict can be understood naturally through a simple case. We
consider a binary classification task with $K=2$, and let the sample numbers
of the two classes be $n_{1}$ and $n_{2}$, respectively. Without loss of
generality, let $n_{1}>n_{2}$. It is straightforward to verify that one of the
optimal allocation to re-balance the class priors is simply appending
$n_{1}-n_{2}$ extra OOD instances into the minority class. However, in such a
way, all OOD instances are assigned the same label, thereby downgrading the
resulting classifier.

To provide a straightforward view, we show in Figure 1 the performance of
model trained with the extra OOD data. Indeed, in the extreme case where all
OOD instances are assigned the same label, their toxicity would be enlarged
during training, leading to poor generalization performance.

Complementary Sampling Rate. To find a “sweet spot”, we propose the following
sampling rate that allows us to achieve a smooth transition from the MCD to
the uniform distribution. We denote CD as $\Gamma$, MCD as $\Gamma^{m}$, and
the complementary sampling rate for class $j$ as $\Gamma_{j}$, then we have
the following proposition.

######  Proposition 2.3 (Complementary Sampling Rate).

$\Gamma_{j}=(\alpha-\beta_{j})/(K\cdot\alpha-1)$, where
$\beta_{j}=\frac{n_{j}}{\sum^{K}_{i=1}n_{i}}$. Then, (i)
$\sum^{K}_{i=1}\Gamma_{i}=1$; (ii) $\Gamma=\Gamma^{m}$ if
$\alpha=\max_{j}(\beta_{j})$; (iii) $\Gamma_{j}\to 1/K$ as $\alpha\to\infty$.

The hyperparameter $\alpha\in\mathbb{R}^{+}\geq\max_{j}(\beta_{j})$ controls
the trade-off bewteen the MCD and uniform distribution. As shown in
Proposition 2.3, when $\alpha=\max_{j}(\beta_{j})$, it recovers the MCD. With
a larger value for the $\alpha$, the label distribution of the auxiliary
dataset would be closer to a uniform distribution. The proof of Proposition
2.3 is provided in Appendix B.

(a) Original

(b) CB

(c) MCD

Figure 2: An illustration of label distributions for long-tailed CIFAR-10
dataset with imbalance ratio 100. (2(a)) Original label prior, (2(b)) Class
Balanced distribution, (2(c)) Minimum Complementary Distribution.

As shown in Figure 2, both the CB and MCD distributions exhibit an inverse
relationship with the original label prior, i.e., the minority classes possess
larger sampling rate than the majority classes. Compared with the CB
distribution, our MCD distribution is flatter so that the instances would not
be concentrated in a class. In such a manner, the harmfulness of open-set
noisy labels would be alleviated to a large extent. The advantage of the MCD
distribution is empirically supported in Section 4.

With the Complementary Sampling Rate, we can then build an auxiliary dataset
with OOD instances to re-balance the training dataset while retaining their
non-toxicity.

Training Objective. To involve the sampled open-set noisy labels into the
training, a natural idea is directly combining them with the original training
dataset, and using the standard cross entropy as the training objective
function:

$\displaystyle\mathcal{L}$
$\displaystyle=\mathbb{E}_{P_{\mathrm{mix}}(X,Y)}[\ell(f(\boldsymbol{x};\boldsymbol{\theta},y)]$
$\displaystyle=\mathbb{E}_{P_{\mathrm{mix}}(X,Y)}[-\mathbf{e}^{y}\log
f(\boldsymbol{x};\boldsymbol{\theta})],$

where $\boldsymbol{e}^{y}\in\\{0,1\\}^{K}$ denotes the one-hot vector whose
$y$-th entry of $\boldsymbol{e}^{y}$ is 1. In each epoch, the labels of
samples from the auxiliary dataset could be updated following the
Complementary Distribution $\Gamma$.

However, the naive combination would consume too much capacity of the network
on fitting the open-set noisy labels, making it hard to converge, especially
when the sample size of the auxiliary dataset is much larger than that of the
original training dataset. To handle this issue, we propose to use the loss on
the auxiliary dataset as a regularization term as shown below:

$\mathcal{L}_{\mathrm{reg}}=\mathbb{E}_{\widetilde{\boldsymbol{x}}\sim
P_{\mathrm{out}}{(X)}}\left[\ell\left(f(\widetilde{\boldsymbol{x}};\boldsymbol{\theta}),\widetilde{y}\right)\right],$

where $\widetilde{y}$ is drawn from the Complementary Distribution $\Gamma$.

To further alleviate the over-fitting issue on the minority classes without
sacrificing the performance on the majority classes, we explicitly introduce a
class-dependent weighting factor $\omega_{j}$ based on the pre-defined
Complementary Distribution. To make the total loss roughly in the same scale
after applying $\omega_{j}$, we normalize $\omega$ so that
$\sum^{K}_{j=1}\omega_{j}=K$. Then, the regularization item becomes:

$\mathcal{L}_{\mathrm{reg}}=\mathbb{E}_{\widetilde{\boldsymbol{x}}\sim
P_{\mathrm{out}}{(X)}}\left[\omega_{\widetilde{y}}\cdot\ell\left(f(\widetilde{\boldsymbol{x}};\boldsymbol{\theta}),\widetilde{y}\right)\right],$

where $\widetilde{y}\sim\Gamma$ and
$\omega_{\widetilde{y}}=\Gamma_{\widetilde{y}}\cdot K$. Now, the final
training objective function is given as follows:

$\displaystyle\begin{split}\mathcal{L}_{\mathrm{total}}&=\mathbb{E}_{((\boldsymbol{x},y)\sim
P_{\mathrm{s}}{(X,Y)})}\left[\ell\left(f(\boldsymbol{x};\boldsymbol{\theta}),y\right)\right]\\\
&+\eta\cdot\mathbb{E}_{(\widetilde{\boldsymbol{x}})\sim
P_{\mathrm{out}}{(X)}}\left[\omega_{\widetilde{y}}\cdot\ell\left(f(\widetilde{\boldsymbol{x}};\boldsymbol{\theta}),\widetilde{y}\right)\right],\end{split}$
(2)

where $\eta$ controls the strength of the regularization term. The
corresponding algorithm is provided in Appendix C.

As a data re-balancing technique, Open-sampling is orthogonal to existing
methods, including the training objective based methods (e.g., LDAM (Cao et
al., 2019), Balanced Softmax (Ren et al., 2020)), Decoupled training methods
(Kang et al., 2020), and self-supervised pre-trained methods (Yang & Xu,
2020). Our method can be easily incorporated into these algorithms to further
improve their generalization performance. Given the original learning
objective $\mathcal{L}_{\mathrm{imb}}$ of the existing methods, we can
formalize the final objective as:

$\displaystyle\mathcal{L}_{\mathrm{total}}$
$\displaystyle=\mathcal{L}_{\mathrm{imb}}+\eta\cdot\mathcal{L}_{\mathrm{reg}}.$
(3)

Relation to ODNL. Recent work (Wei et al., 2021) shows that open-set noisy
labels could be applied to enhance the robustness against inherent noisy
labels, which has some conceptual similarities to the proposed method in this
work. Here, we summarize the main differences between ODNL (Wei et al., 2021)
and our work.

  1. 1.

Problem setting: ODNL focuses on improving the robustness against noisy labels
while our work considers the problem of learning from long-tailed imbalanced
datasets.

  2. 2.

Technique: we proposed to sample the labels of OOD instances from the
Complementary Distribution and add a weight factor to their losses, while they
treats all the OOD instances equally by simply using a uniform distribution.
In particular, ODNL can be seen as a special case of our method with a large
value of $\alpha$. As analyzed in Figures 4(a), 4(b), and 4(c), our Open-
sampling consistently outperforms the variant with a uniform distribution or a
large value of $\alpha$, which demonstrates the advantage of the proposed
method.

  3. 3.

Insight: ODNL aims to consume the extra representation capacity of neural
networks to prevent over-fitting inherent noisy labels and show that their
method helps the network converge to a flat minimum as SGD noises. In our
work, OOD instances are applied to re-balance the label prior of the training
dataset and the proposed method are shown to encourage the network to learn
more separable representations.

##  3 Experiments

In this section, we evaluate our proposed method on long-tailed image
classification datasets, including long-tailed CIFAR10/100 (Krizhevsky et al.,
2009), CelebA-5 (Liu et al., 2015b; Kim et al., 2020), Places-LT (Zhou et al.,
2017). Then we analyze the impact of $\eta$ by sensitivity analysis. The
ablation study for the class-dependent weight is provided in Appendix F.
Finally, we conduct experiments to evaluate the performance of Open-sampling
on OOD detection task under class-imbalanced setting. The datasets and
implementation details are introduced in Appendix E.

Table 1: Test accuracy (%) of ResNet-32 on long-tailed CIFAR-10 and CIFAR-100
with various imbalance ratios. “$\dagger$” indicates the reported results from
(Kim et al., 2020). The bold indicates the improved results by integrating our
regularization.

Dataset | Long-tailed CIFAR-10 | Long-tailed CIFAR-100
---|---|---
Imbalance Ratio | 100 | 50 | 10 | 100 | 50 | 10
Standard | 71.61 $\pm$ 0.21 | 77.30 $\pm$ 0.13 | 86.74 $\pm$ 0.41 | 37.59 $\pm$ 0.19 | 43.20 $\pm$ 0.30 | 56.44 $\pm$ 0.12
SMOTE † | 71.50 $\pm$ 0.57 | - | 85.70 $\pm$ 0.25 | 34.00 $\pm$ 0.33 | - | 53.80 $\pm$ 0.93
CB-RW | 72.57 $\pm$ 1.30 | 78.19 $\pm$ 1.79 | 87.18 $\pm$ 0.95 | 38.11 $\pm$ 0.78 | 43.26 $\pm$ 0.87 | 56.40 $\pm$ 0.40
CB-Focal | 70.91 $\pm$ 0.39 | 77.71 $\pm$ 0.57 | 86.89 $\pm$ 0.21 | 37.84 $\pm$ 0.80 | 42.96 $\pm$ 0.77 | 56.09 $\pm$ 0.15
Ours | 77.62 $\pm$ 0.28 | 81.76 $\pm$ 0.51 | 89.38 $\pm$ 0.46 | 40.26 $\pm$ 0.65 | 44.77 $\pm$ 0.25 | 58.09 $\pm$ 0.29
LDAM-RW | 74.21 $\pm$ 0.61 | 78.86 $\pm$ 0.65 | 86.44 $\pm$ 0.78 | 29.02 $\pm$ 0.34 | 36.41 $\pm$ 0.84 | 54.23 $\pm$ 0.72
\+ Ours | 75.19 $\pm$ 0.34 | 79.76 $\pm$ 0.44 | 87.28 $\pm$ 0.61 | 35.85 $\pm$ 0.62 | 42.18 $\pm$ 0.82 | 55.48 $\pm$ 0.59
LDAM-DRW | 78.08 $\pm$ 0.38 | 81.88 $\pm$ 0.44 | 87.49 $\pm$ 0.18 | 42.84 $\pm$ 0.25 | 47.13 $\pm$ 0.28 | 57.18 $\pm$ 0.47
\+ Ours | 79.82 $\pm$ 0.31 | 82.22 $\pm$ 0.45 | 87.83 $\pm$ 0.38 | 44.07 $\pm$ 0.75 | 47.5 $\pm$ 0.24 | 57.43 $\pm$ 0.31
Balanced Softmax | 78.03 $\pm$ 0.28 | 81.63 $\pm$ 0.39 | 88.10 $\pm$ 0.32 | 42.11 $\pm$ 0.70 | 46.79 $\pm$ 0.24 | 58.06 $\pm$ 0.40
\+ Ours | 79.05 $\pm$ 0.20 | 82.76 $\pm$ 0.52 | 88.89 $\pm$ 0.21 | 42.86 $\pm$ 0.27 | 47.28 $\pm$ 0.58 | 58.80 $\pm$ 0.72
SSP | 74.58 $\pm$ 0.16 | 79.20 $\pm$ 0.43 | 88.50 $\pm$ 0.24 | 43.00 $\pm$ 0.51 | 47.04 $\pm$ 0.60 | 59.08 $\pm$ 0.46
\+ Ours | 79.38 $\pm$ 0.65 | 82.18 $\pm$ 0.33 | 88.80 $\pm$ 0.43 | 43.57 $\pm$ 0.29 | 48.66 $\pm$ 0.57 | 59.78 $\pm$ 0.91

###  3.1 Comparison methods

In this section, we verify that Open-sampling can boost the standard training
and several state-of-the-art techniques by integrating Open-sampling with the
following methods: 1) Standard: all the examples have the same weights; by
default, we use standard cross-entropy loss. 2) SMOTE (Chawla et al., 2002): a
variant of re-sampling with data augmentation. 3) CB-RW (Cui et al., 2019):
training examples are re-weighted according to the inverse of the effective
number of samples in each class, defined as $(1-\beta^{n_{i}})/(1-\beta)$. 4)
CB-Focal (Cui et al., 2019): the CB method is combined with Focal loss. 5) M2m
(Kim et al., 2020): an over-sampling method with adversarial examples. 6)
LDAM-RW (Cao et al., 2019): the method derives a generalization error bound
for the imbalanced training and uses a margin-aware multi-class weighted cross
entropy loss. 7) LDAM-DRW (Cao et al., 2019): the network is trained with LDAM
loss and deferred re-balancing training. 8) Balanced Softmax (Ren et al.,
2020): the method derives a Balanced Softmax function from the probabilistic
perspective that explicitly models the test-time label distribution shift. 9)
SSP (Yang & Xu, 2020): the method uses self-supervised learning to pre-train
the network on the auxiliary dataset before standard training. Here, we do not
expect vanilla Open-sampling to achieve state-of-the-art results compared with
many complicated methods, our method can be still a promising option in the
family of class-imbalanced learning methods, because it can outperform
existing data re-balancing methods and improve existing state-of-the-art
methods.

Table 2: Classification accuracy (%) on CIFAR-10 under the setting of Balance
Softmax (Ren et al., 2020). The bold indicates the improved results by
integrating our regularization. Baseline results are taken from Balance
Softmax(Ren et al., 2020). “ $\star$” indicates decoupled training.

Imbalance Factor | 200 | 100 | 10
---|---|---|---
Standard | 71.2 | 77.4 | 90.0
CB-RW | 72.5 | 78.6 | 90.1
LDAM-RW | 73.6 | 78.9 | 90.3
Equalization Loss (Tan et al., 2020) | 74.6 | 78.5 | 90.2
Balanced Softmax | 79.0 | 83.1 | 90.9
Ours | 80.6 | 83.6 | 90.6
cRT⋆ (Kang et al., 2020) | 76.6 | 82.0 | 91.0
LWS⋆ (Kang et al., 2020) | 78.1 | 83.7 | 91.1
Ours⋆ | 81.1 | 84.9 | 90.9

###  3.2 Main results

Results on long-tailed CIFAR. Extensive experiments are conducted on long-
tailed CIFAR datasets with three different imbalance ratios: 10, 50, and 100.
We use 300K Random Images222The dataset is published on
https://github.com/hendrycks/outlier-exposure. (Hendrycks et al., 2019) as the
open-set auxiliary dataset. The test accuracy of ResNet-32 (He et al., 2016)
on long-tailed CIFAR datasets are reported in Table 1. The results show that
our method can achieve impressive improvements on the standard training
method. Especially for long-tailed CIFAR-10 with imbalance ratio 100, an
extreme imbalance case, the vanilla Open-sampling can significantly outperform
the standard baseline by 8.39%. Besides, incorporating our method into
existing state-of-the-art methods can consistently improve their performance
under various imbalance ratios.

We also conduct experiments on long-tailed CIFAR-10 following the training
setting of Balanced Softmax (Ren et al., 2020) and present the results in
Table 2. Under this setting, our proposed method achieves the best performance
among the end-to-end methods and could also improve the performance of
decoupled training (Kang et al., 2020).

To further clarify the influence of $\eta$, we present a sensitivity analysis
on long-tailed CIFAR-10 dataset with imbalance ratio 100 in Figure 3. We
highlight the differences in the trend of test accuracy after the decay of
learning rate at the 160th epoch. From the figure, we can observe that with a
proper value of $\eta$ like 1.5, the generalization performance can be largely
improved by our proposed regularization. The result verifies that our
regularization is an effective method to improve the generalization
performance in long-tailed imbalanced learning.

Results on CelebA-5. We further verify the effectiveness of our method on
real-world class-imbalanced datasets. The CelebA dataset has a long-tailed
label distribution and the test set has a balanced label distribution. We use
300K Random Images (Hendrycks et al., 2019) as the open-set auxiliary dataset.
Table 3 summarizes test accuracy on the CelebA-5 dataset. In particular, the
proposed method outperforms existing data-rebalancing methods and is able to
consistently improve the existing state-of-the-art methods in test accuracy.

Results on Places-LT. For Places-LT, we follow the protocol of Decoupled
training (Kang et al., 2020) and start from a ResNet-152 (He et al., 2016)
backbone pre-trained on the ImageNet dataset (Russakovsky et al., 2015). We
fine-tune the backbone with Instance-balanced sampling for representation
learning and then re-train the classifier with our proposed algorithm as
decoupled training. Here, we use Places-extra69 (Zhou et al., 2017) as the
open-set auxiliary dataset. Table 4 summarizes test accuracy on the Places-LT
dataset. The proposed method achieves the best overall performance, show that
our method and decoupled training scheme are two orthogonal components and
Open-sampling is applicable for large-scale datasets.

Figure 3: Results of sensitivity analysis on long-tailed CIFAR-10 with various
values for $\eta$.

OOD detection under class imbalance. Out-of-distribution (OOD) detection is an
essential problem for the deployment of deep learning especially in safety-
critical applications (Hendrycks et al., 2019; Liu et al., 2020; Liang et al.,
2018; Yang et al., 2021) and it would be more difficult when the training
dataset exhibits long-tailed distributions. Outlier Exposure (OE) (Hendrycks
et al., 2019) is a commonly-used method to improve anomaly detection
performance with auxiliary dataset. Specifically, the training objective of OE
under class imbalance is
$\mathbb{E}_{(x,y)\sim\mathcal{D}_{\mathrm{train}}}\left[-\log
f_{y}(x)\right]+\lambda\mathbb{E}_{x\sim\mathcal{D}_{\mathrm{out}}}[H(P(Y);f(x))]$,
where $H$ is the cross entropy and $P(Y)$ is the label prior of the original
training dataset. Following OE (Hendrycks et al., 2019), we consider the
maximum softmax probability (MSP) baseline (Hendrycks & Gimpel, 2016). Here,
we conduct experiments on long-tailed CIFAR10 with imbalance rate 100 to
verify the advantage of Open-sampling on OOD detection under class imbalance.
Table 5 presents the test accuracy and the average performance over the three
types of noises and seven OOD test datasets. We can observe that our method
achieves impressive improvement on both the test accuracy and the detection
performance.

Table 3: Classification accuracy (%) on CelebA-5 with ResNet-32. “$\dagger$”
indicates the reported results are obtained from (Kim et al., 2020). The
shadow indicates the improved results.

Method | Accuracy | Method | Accuracy | Method | Accuracy
---|---|---|---|---|---
Standard | 72.7 | M2m † | 75.6 | LDAM-DRW | 74.5
SMOTE † | 72.8 | Ours | 76.8 | LDAM-DRW + Ours | 76.9
CB-RW | 73.6 | LDAM-RW | 73.1 | Balanced Softmax | 76.4
CB-Focal | 74.2 | LDAM + Ours | 75.8 | Balanced Softmax + Ours | 78.6

Table 4: Top-1 accuracy (%) on Places-LT with an ImageNet pre-trained
ResNet-152. Baseline results are taken from original papers. “DT” indicates
decoupled training.

Method | Many | Medium | Few | Overall
---|---|---|---|---
Lifted Loss (Oh Song et al., 2016) | 41.1 | 35.4 | 24.0 | 35.2
Focal Loss | 41.1 | 34.8 | 22.4 | 34.6
Range Loss (Zhang et al., 2017) | 41.1 | 35.4 | 23.2 | 35.1
OLTR (Liu et al., 2019b) | 44.7 | 37.0 | 25.3 | 35.9
cRT⋆ (Kang et al., 2020) | 42.1 | 37.6 | 24.9 | 36.7
LWS⋆ (Kang et al., 2020) | 40.6 | 39.1 | 28.6 | 37.6
Ours⋆ | 42.9 | 39.3 | 26.8 | 38.2

Table 5:  OOD detection performance comparison on long-tailed CIFAR-10. All
values are percentages and are averaged over the ten test datasets described
in Appendix E. “$\uparrow$” indicates larger values are better, and
“$\downarrow$” indicates smaller values are better. Bold numbers are superior
results. Detailed results for each OOD test dataset can be found in Appendix
F.

Method | Test Accuracy $\uparrow$ | FPR95 $\downarrow$ | AUROC $\uparrow$ | AUPR $\uparrow$
---|---|---|---|---
MSP | 71.83 | 56.1 | 75.2 | 32.71
OE | 66.74 | 32.38 | 84.15 | 36.86
Ours | 77.62 | 20.68 | 92.40 | 58.38
Ours ($\alpha=5$) | 75.16 | 22.13 | 94.26 | 75.91

(a) Label distribution.

(b) Label distribution.

(c) Alpha.

(d) Auxiliary dataset.

(e) Sample size.

(f) Number of Classes.

Figure 4: Analytical experiments of Open-sampling on long-tailed CIFAR-10:
(4(a))(4(b)) with various label distributions, (4(c)) with various values of
the $\alpha$, (4(d)) with various open-set auxiliary datasets, including
simulated noise datasets and real-world datasets. All the datasets contain
50,000 instances. (4(e)) with various sample sizes (K) of the auxiliary
dataset, (4(f)) with various number of classes in the auxiliary datasets that
are randomly sampled from CIFAR-100. Experiments in (4(b)), (4(c)), and (4(e))
are conducted under the imbalance ratio 100. The y-axis represents the test
error in all the figures. “Standard” denotes the baseline with standard cross-
entropy loss.

##  4 A Closer Look at Open-sampling under Class Imbalance

To provide a comprehensive understanding of the proposed method, we conduct a
set of analyses in this section. Firstly, we compare our defined distribution
with several alternative distributions to show the advantage of the
Complementary Distribution in our method. Secondly, the effect of $\alpha$ in
our method is thoroughly analyzed by empirical studies. Then, we present a
guideline about how to choose or collect a suitable open-set auxiliary dataset
for long-tailed imbalanced learning. Finally, we analyze the effect of the
proposed method through the lens of decision boundaries.

The advantage of Complementary Distribution. In the proposed method, the
labels of OOD instances from the auxiliary dataset are sampled from a random
label distribution. For the random label distribution, we defined a
Complementary Distribution in Proposition 2.3 and also presented several
commonly used distributions, including uniform distribution (Unif), class
balanced distribution (CB) (Cui et al., 2019), and the original class priors
of the training dataset (Original). Here, we conduct experiments to compare
the performance of the Open-sampling variants with different label
distributions. The results in Figure 4(a) show that using the Complementary
Distribution consistently achieves the best performance on the test set. In
particular, using the uniform distribution can also improve the generalization
performance while both CB and the original class priors deteriorate the
performance of the neural networks.

To further understand why CB is not a good choice in our method, we present
the per-class top-1 error on long-tailed CIFAR-10 in Figure 4(b). Although
using the CB distribution can achieve better performance on the smallest
class, it downgrades the generalization performance on the other classes. The
reason is that the CB distribution is far away from the uniform distribution,
thereby introducing too much noise to the Bayes classifier. Different from CB,
the Complementary Distribution is closer to a uniform distribution, making it
achievable to re-balance the class priors while almost keeping non-toxicity of
the open-set noisy labels.

Here, we also show the effect of $\alpha$ in Figure 4(c). As analyzed in
Section 2, the larger the value of $\alpha$ is, the Complementary Distribution
tends to be closer to a uniform distribution. Here, “M” denotes the default
value of $\alpha=(\max_{j}\beta_{j}+\min_{j}\beta_{j})$. Additionally, the MCD
variant achieves $22.51\%$ on the test error. From Figure 4(c), the test error
presented a slightly upward trend with the increasing of the value of
$\alpha$. The results verified that it is necessary to adopt a complementary
distribution, instead of simply using a uniform distribution.

(a) Standard

(b) CB-Focal

(c) CB-Resampling

(d) LDAM-RW

(e) Balanced Softmax

(f) Ours

Figure 5: t-SNE visualization of test set on long-tailed CIFAR-10 with
imbalance ratio 100. We can observe that LDAM and our method appear to learn
more separable representations than Standard training and the other
algorithms.

The choices of open-set auxiliary dataset. With proper label distribution, can
any OOD dataset be used to improve Long-tail imbalanced learning? In Figure
4(d), we show the test error on long-tailed CIFAR-10 with imbalance ratio 100
using Open-sampling with different auxiliary datasets. We can observe that
using simple noises like Gaussian noise could not achieve similar performance
than those of real-world datasets, _e.g._ , CIFAR100 and 300K Random Images
(Hendrycks et al., 2019).

The sample size is another important factor for the open-set auxiliary
dataset. In Figure 4(e), we show that the performance of Open-sampling would
be slightly better with a larger sample size of the auxiliary dataset. It is
worth noting that the phenomenon is not consistent with that of (Wei et al.,
2021), where using larger open-set auxiliary dataset could not improve the
performance on learning with noisy labels. Here, we also conduct experiments
using a variant of Open-sampling with fixed labels for OOD instances. We can
observe that the variant performs poorly with a small auxiliary dataset, but
the performance can be significantly improved by increasing the sample size of
auxiliary dataset. In particular, the variant could achieve nearly the same
improvement as the proposed Open-sampling method, with a large enough sample
size.

In addition, we also find that the diversity of the open-set auxiliary dataset
is unimportant for the effectiveness of Open-sampling. We conduct experiments
on long-tailed CIFAR-10 with imbalance ratio 100 and use the subset of
CIFAR100 with different number of classes as the open-set auxiliary dataset.
The sample size of these subsets are fixed as 5,000. The results of using
different subsets are almost the same, achieving 73.28% test accuracy. The
results are consistent with that in Figure 4(d), where using CIFAR100 as open-
set auxiliary dataset can achieve almost the same improvements as 300K Random
Images (Hendrycks et al., 2019).

The effect of Open-sampling as regularization. In addition to re-balancing the
class prior, what is the effect of Open-sampling as a regularization method?
To gain additional insight, we look at the t-SNE projection of the learnt
representations for different algorithms in Figure 5. For each method, the
projection is performed over test data. The figures show that the decision
boundaries of Open-sampling and LDAM (Cao et al., 2019) are much clearer than
those of the other methods. The phenomenon illustrates that Open-sampling can
encourage the minority classes to have a larger margin, which is similar to
the effect of the LDAM loss (Cao et al., 2019). As shown in Section 3, our
method could still boost the performance of the LDAM method, which
demonstrates the differences between our works.

##  5 Related Work

Re-sampling. Re-sampling methods aims to re-balance the class priors of the
training dataset. Under-sampling methods remove examples from the majority
classes, which is infeasible under extremely data imbalanced settings (He &
Garcia, 2009; Japkowicz & Stephen, 2002). The over-sampling method adds
repeated samples for the minority classes, usually causing over-fitting to the
minority classes (Buda et al., 2018; Byrd & Lipton, 2019; Shen et al., 2016).
Some methods utilize synthesized in-distribution samples to alleviate the
over-fitting issue but introduce extra noise (Chawla et al., 2002; He et al.,
2008; Kim et al., 2020). In contrast to in-distribution samples used in
existing methods, our approach exploits OOD instances to re-balance the class
priors of the training dataset.

Re-weighting. Re-weighting methods propose to assign adaptive weights for
different classes or samples. Generally, the vanilla scheme re-weights classes
proportionally to the inverse of their frequency (Huang et al., 2016). Focal
loss (Lin et al., 2017) assigns low weights to the well-classified examples.
Class-balanced loss (Cui et al., 2019) proposes to re-weight by the inverse
effective number of samples. However, these re-weighting methods tend to make
the optimization of DNNs difficult under extremely data imbalanced settings
(Wang et al., 2017; Mikolov et al., 2013).

Other methods for long-tailed datasets. In addition to the data re-balancing
approaches, some other solutions are also applied for class-imbalanced
learning, including transfer-learning based methods (Yin et al., 2019; Jamal
et al., 2020), two-stage training methods(Kang et al., 2020; Zhong et al.,
2021; Zhang et al., 2021; Liu et al., 2019b), training objective based methods
(Cao et al., 2019; Ren et al., 2020; Hong et al., 2021), expert methods (Wang
et al., 2021b) and self-supervised learning methods (Yang & Xu, 2020; Cui et
al., 2021). For example, transfer-learning based methods addressed the class-
imbalanced issue by transferring features learned from head classes with
abundant training instances to under-represented tail classes (Yin et al.,
2019; Jamal et al., 2020). Two-stage training methods proposed to apply
decoupled training where the classifier is re-balanced during the fine-tuning
stage (Kang et al., 2020; Zhong et al., 2021; Zhang et al., 2021; Liu et al.,
2019b). Generally, our proposed method is complementary to these existing
methods and our method can further improve their performance, which are
explicitly shown in our experiments.

Utilizing auxiliary dataset. In the deep learning community, auxiliary dataset
is utilized in various contexts, e.g., adversarial machine learning (Madry et
al., 2018; Lee et al., 2021) and weakly supervised learning (Wei et al.,
2020a, b; Zhu et al., 2021a; Cheng et al., 2021; Wang et al., 2021a; Zhu et
al., 2021b; Wei et al., 2022b; Zhu et al., 2022a, b). For example, OE
(Hendrycks et al., 2019) regularized the network to give conservative
predictions on OOD instances. OAT (Lee et al., 2021) utilized OOD instances to
improve generalization in adversarial training. ODNL (Wei et al., 2021)
proposed to use open-set auxiliary dataset to prevent the model from over-
fitting inherent noisy labels. In long-tailed imbalanced learning, recent work
(Yang & Xu, 2020) introduced unlabeled-in-distribution data to compensate for
the lack of training samples. Another work (Su et al., 2021) adopted semi-
supervised learning techniques to improve performance on long-tailed datasets
with out-of-class images from related classes, which are different from out-
of-distribution data in our method. To the best of our knowledge, we are the
first to explore the benefits of _out-of-distribution data_ in learning with
long-tailed datasets.

OOD detection. OOD detection is an essential building block for safely
deploying machine learning models in the open world (Hendrycks & Gimpel, 2016;
Hendrycks et al., 2019; Liu et al., 2020; Wei et al., 2022a). A common
baseline for OOD detection is the softmax confidence score (Hendrycks &
Gimpel, 2016). It has been theoretically shown that neural networks with ReLU
activation can produce arbitrarily high softmax confidence for OOD inputs
(Hein et al., 2019). To improve the performance, previous methods have
explored using artificially synthesized data from GANs (Goodfellow et al.,
2014) or unlabeled data (Hendrycks et al., 2019; Lee et al., 2017) as
auxiliary OOD training data. Energy scores are shown to be better for
distinguishing in- and out-of-distribution samples than softmax scores (Liu et
al., 2020). As a side effect, our method could also achieve superior
performance in OOD detection tasks even if the labels of training dataset are
noisy.

##  6 Conclusion

In this paper, we propose a simple yet effective method termed Open-sampling,
by introducing OOD instances to re-balance the class priors of the training
dataset. To the best of our knowledge, our method is the first to utilize OOD
instances in long-tailed learning. We show that our method not only re-
balances the training dataset, but also promotes the neural network to learn
more separable representations. Besides, we also present a guideline about the
open-set auxiliary datasets: the realism and sample size are more important
than the diversity. Extensive experiments demonstrate that our proposed method
can enhance the performance of existing state-of-the-art methods, and also
achieves strong OOD detection performance under class-imbalanced setting.
Overall, our method enables to exploit OOD data for long-tail learning and we
expect that our insights inspire future research to further explore the
benefits of auxiliary datasets.

## Acknowledgements

This research is supported by MOE Tier-1 project RG13/19 (S). LF is supported
by the National Natural Science Foundation of China (Grant No. 62106028) and
CAAI-Huawei MindSpore Open Fund.

## References

  * Buda et al. (2018) Buda, M., Maki, A., and Mazurowski, M. A.  A systematic study of the class imbalance problem in convolutional neural networks.  _Neural Networks_ , 106:249–259, 2018.
  * Byrd & Lipton (2019) Byrd, J. and Lipton, Z.  What is the effect of importance weighting in deep learning?  In _International Conference on Machine Learning_ , pp. 872–881. PMLR, 2019.
  * Cao et al. (2019) Cao, K., Wei, C., Gaidon, A., Arechiga, N., and Ma, T.  Learning imbalanced datasets with label-distribution-aware margin loss.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019.
  * Chawla et al. (2002) Chawla, N. V., Bowyer, K. W., Hall, L. O., and Kegelmeyer, W. P.  Smote: Synthetic minority over-sampling technique.  _Journal of Artificial Intelligence Research_ , 16:321–357, 2002.
  * Cheng et al. (2021) Cheng, H., Zhu, Z., Li, X., Gong, Y., Sun, X., and Liu, Y.  Learning with instance-dependent label noise: a sample sieve approach.  In _International Conference on Learning Representations_ , 2021.
  * Cimpoi et al. (2014) Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A.  Describing textures in the wild.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 3606–3613, 2014.
  * Cui et al. (2021) Cui, J., Zhong, Z., Liu, S., Yu, B., and Jia, J.  Parametric contrastive learning.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pp. 715–724, 2021.
  * Cui et al. (2019) Cui, Y., Jia, M., Lin, T.-Y., Song, Y., and Belongie, S.  Class-balanced loss based on effective number of samples.  In _Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition_ , pp. 9268–9277, 2019.
  * Goodfellow et al. (2014) Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y.  Generative adversarial networks.  _arXiv preprint arXiv:1406.2661_ , 2014.
  * Goyal et al. (2017) Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., and He, K.  Accurate, large minibatch sgd: Training imagenet in 1 hour.  _arXiv preprint arXiv:1706.02677_ , 2017.
  * He & Garcia (2009) He, H. and Garcia, E. A.  Learning from imbalanced data.  _IEEE Transactions on Knowledge and Data Engineering_ , 21(9):1263–1284, 2009.
  * He et al. (2008) He, H., Bai, Y., Garcia, E. A., and Li, S.  Adasyn: Adaptive synthetic sampling approach for imbalanced learning.  In _2008 IEEE International Joint Conference on Neural Networks_ , pp. 1322–1328, 2008.
  * He et al. (2016) He, K., Zhang, X., Ren, S., and Sun, J.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , pp. 770–778, 2016.
  * Hein et al. (2019) Hein, M., Andriushchenko, M., and Bitterwolf, J.  Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 41–50, 2019.
  * Hendrycks & Gimpel (2016) Hendrycks, D. and Gimpel, K.  A baseline for detecting misclassified and out-of-distribution examples in neural networks.  _arXiv preprint arXiv:1610.02136_ , 2016.
  * Hendrycks et al. (2019) Hendrycks, D., Mazeika, M., and Dietterich, T.  Deep anomaly detection with outlier exposure.  _Proceedings of the International Conference on Learning Representations_ , 2019.
  * Hong et al. (2021) Hong, Y., Han, S., Choi, K., Seo, S., Kim, B., and Chang, B.  Disentangling label distribution for long-tailed visual recognition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 6626–6636, 2021.
  * Huang et al. (2016) Huang, C., Li, Y., Loy, C. C., and Tang, X.  Learning deep representation for imbalanced classification.  In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , pp. 5375–5384, 2016.
  * Jamal et al. (2020) Jamal, M. A., Brown, M., Yang, M.-H., Wang, L., and Gong, B.  Rethinking class-balanced methods for long-tailed visual recognition from a domain adaptation perspective.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 7610–7619, 2020.
  * Japkowicz & Stephen (2002) Japkowicz, N. and Stephen, S.  The class imbalance problem: A systematic study.  _Intelligent Data Analysis_ , 6(5):429–449, 2002\.
  * Kang et al. (2020) Kang, B., Xie, S., Rohrbach, M., Yan, Z., Gordo, A., Feng, J., and Kalantidis, Y.  Decoupling representation and classifier for long-tailed recognition.  In _Eighth International Conference on Learning Representations (ICLR)_ , 2020.
  * Kim et al. (2020) Kim, J., Jeong, J., and Shin, J.  M2m: Imbalanced classification via major-to-minor translation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 13896–13905, 2020.
  * Komodakis & Gidaris (2018) Komodakis, N. and Gidaris, S.  Unsupervised representation learning by predicting image rotations.  In _International Conference on Learning Representations_ , 2018.
  * Krizhevsky et al. (2009) Krizhevsky, A., Hinton, G., et al.  Learning multiple layers of features from tiny images.  2009\.
  * Lee et al. (2017) Lee, K., Lee, H., Lee, K., and Shin, J.  Training confidence-calibrated classifiers for detecting out-of-distribution samples.  _arXiv preprint arXiv:1711.09325_ , 2017.
  * Lee et al. (2021) Lee, S., Park, C., Lee, H., Yi, J., Lee, J., and Yoon, S.  Removing undesirable feature contributions using out-of-distribution data.  _arXiv preprint arXiv:2101.06639_ , 2021.
  * Liang et al. (2018) Liang, S., Li, Y., and Srikant, R.  Enhancing the reliability of out-of-distribution image detection in neural networks.  _International Conference on Learning Representations (ICLR)_ , 2018\.
  * Lin et al. (2014) Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L.  Microsoft coco: Common objects in context.  In _European Conference on Computer Vision_ , pp. 740–755. Springer, 2014.
  * Lin et al. (2017) Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Dollár, P.  Focal loss for dense object detection.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pp. 2980–2988, 2017.
  * Liu et al. (2020) Liu, W., Wang, X., Owens, J., and Li, Y.  Energy-based out-of-distribution detection.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020\.
  * Liu et al. (2015a) Liu, Z., Luo, P., Wang, X., and Tang, X.  Deep learning face attributes in the wild.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pp. 3730–3738, 2015a.
  * Liu et al. (2015b) Liu, Z., Luo, P., Wang, X., and Tang, X.  Deep learning face attributes in the wild.  In _Proceedings of International Conference on Computer Vision_ , December 2015b.
  * Liu et al. (2019a) Liu, Z., Miao, Z., Zhan, X., Wang, J., Gong, B., and Yu, S. X.  Large-scale long-tailed recognition in an open world.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 2537–2546, 2019a.
  * Liu et al. (2019b) Liu, Z., Miao, Z., Zhan, X., Wang, J., Gong, B., and Yu, S. X.  Large-scale long-tailed recognition in an open world.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2019b.
  * Loshchilov & Hutter (2016) Loshchilov, I. and Hutter, F.  Sgdr: Stochastic gradient descent with warm restarts.  _arXiv preprint arXiv:1608.03983_ , 2016.
  * Madry et al. (2018) Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A.  Towards deep learning models resistant to adversarial attacks.  In _International Conference on Learning Representations_ , 2018.
  * Mikolov et al. (2013) Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J.  Distributed representations of words and phrases and their compositionality.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , pp. 3111–3119, 2013.
  * Netzer et al. (2011) Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y.  Reading digits in natural images with unsupervised feature learning.  2011\.
  * Oh Song et al. (2016) Oh Song, H., Xiang, Y., Jegelka, S., and Savarese, S.  Deep metric learning via lifted structured feature embedding.  In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , pp. 4004–4012, 2016.
  * Paszke et al. (2019) Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 32:8026–8037, 2019.
  * Ren et al. (2020) Ren, J., Yu, C., Sheng, S., Ma, X., Zhao, H., Yi, S., and Li, H.  Balanced meta-softmax for long-tailed visual recognition.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , Dec 2020.
  * Russakovsky et al. (2015) Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L.  ImageNet large scale visual recognition challenge.  _International Journal of Computer Vision_ , 115(3):211–252, 2015.
  * Shen et al. (2016) Shen, L., Lin, Z., and Huang, Q.  Relay backpropagation for effective learning of deep convolutional neural networks.  In _European Conference on Computer Vision_ , pp. 467–482. Springer, 2016.
  * Su et al. (2021) Su, J.-C., Cheng, Z., and Maji, S.  A realistic evaluation of semi-supervised learning for fine-grained classification.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 12966–12975, 2021.
  * Tan et al. (2020) Tan, J., Wang, C., Li, B., Li, Q., Ouyang, W., Yin, C., and Yan, J.  Equalization loss for long-tailed object recognition.  In _Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition_ , pp. 11662–11671, 2020.
  * Van Horn et al. (2018) Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., and Belongie, S.  The inaturalist species classification and detection dataset.  In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , pp. 8769–8778, 2018.
  * Wang et al. (2021a) Wang, J., Guo, H., Zhu, Z., and Liu, Y.  Policy learning using weak supervision.  _Advances in Neural Information Processing Systems_ , 34, 2021a.
  * Wang et al. (2021b) Wang, X., Lian, L., Miao, Z., Liu, Z., and Yu, S.  Long-tailed recognition by routing diverse distribution-aware experts.  In _International Conference on Learning Representations_ , 2021b.
  * Wang et al. (2017) Wang, Y.-X., Ramanan, D., and Hebert, M.  Learning to model the tail.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , pp. 7032–7042, 2017.
  * Wei et al. (2020a) Wei, H., Feng, L., Chen, X., and An, B.  Combating noisy labels by agreement: A joint training method with co-regularization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 13726–13735, 2020a.
  * Wei et al. (2020b) Wei, H., Feng, L., Wang, R., and An, B.  Metainfonet: Learning task-guided information for sample reweighting.  _arXiv preprint arXiv:2012.05273_ , 2020b.
  * Wei et al. (2021) Wei, H., Tao, L., Xie, R., and An, B.  Open-set label noise can improve robustness against inherent label noise.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2021.
  * Wei et al. (2022a) Wei, H., Renchunzi, X., Hao, C., Feng, L., An, B., and Yixuan, L.  Mitigating neural network overconfidence with logit normalization.  In _International Conference on Machine Learning (ICML)_. PMLR, 2022a.
  * Wei et al. (2022b) Wei, J., Zhu, Z., Cheng, H., Liu, T., Niu, G., and Liu, Y.  Learning with noisy labels revisited: A study using real-world human annotations.  In _International Conference on Learning Representations_ , 2022b.
  * Xiao et al. (2010) Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A.  Sun database: Large-scale scene recognition from abbey to zoo.  In _proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 3485–3492. IEEE, 2010.
  * Xu et al. (2015) Xu, P., Ehinger, K. A., Zhang, Y., Finkelstein, A., Kulkarni, S. R., and Xiao, J.  Turkergaze: Crowdsourcing saliency with webcam based eye tracking.  _arXiv preprint arXiv:1504.06755_ , 2015.
  * Yang et al. (2021) Yang, J., Zhou, K., Li, Y., and Liu, Z.  Generalized out-of-distribution detection: A survey.  _arXiv preprint arXiv:2110.11334_ , 2021.
  * Yang & Xu (2020) Yang, Y. and Xu, Z.  Rethinking the value of labels for improving class-imbalanced learning.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020.
  * Yin et al. (2019) Yin, X., Yu, X., Sohn, K., Liu, X., and Chandraker, M.  Feature transfer learning for face recognition with under-represented data.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 5704–5713, 2019.
  * Yu et al. (2015) Yu, F., Seff, A., Zhang, Y., Song, S., Funkhouser, T., and Xiao, J.  Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop.  _arXiv preprint arXiv:1506.03365_ , 2015.
  * Zhang et al. (2021) Zhang, S., Li, Z., Yan, S., He, X., and Sun, J.  Distribution alignment: A unified framework for long-tail visual recognition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 2361–2370, 2021.
  * Zhang et al. (2017) Zhang, X., Fang, Z., Wen, Y., Li, Z., and Qiao, Y.  Range loss for deep face recognition with long-tailed training data.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pp. 5409–5418, 2017.
  * Zhong et al. (2021) Zhong, Z., Cui, J., Liu, S., and Jia, J.  Improving calibration for long-tailed recognition.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2021\.
  * Zhou et al. (2017) Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., and Torralba, A.  Places: A 10 million image database for scene recognition.  _IEEE transactions on Pattern Analysis and Machine Intelligence_ , 40(6):1452–1464, 2017.
  * Zhou et al. (2020) Zhou, B., Cui, Q., Wei, X.-S., and Chen, Z.-M.  Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 9719–9728, 2020.
  * Zhu et al. (2021a) Zhu, Z., Liu, T., and Liu, Y.  A second-order approach to learning with instance-dependent label noise.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2021a.
  * Zhu et al. (2021b) Zhu, Z., Song, Y., and Liu, Y.  Clusterability as an alternative to anchor points when learning with noisy labels.  In _Proceedings of the 38th International Conference on Machine Learning_ , 2021b.
  * Zhu et al. (2022a) Zhu, Z., Dong, Z., and Liu, Y.  Detecting corrupted labels without training a model to predict.  In _International Conference on Machine Learning (ICML)_ , 2022a.
  * Zhu et al. (2022b) Zhu, Z., Wang, J., and Liu, Y.  Beyond images: Label noise transition matrix estimation for tasks with lower-quality features.  In _International Conference on Machine Learning (ICML)_ , 2022b.

##  Appendix A Proof of Theorem 2.1

Since
$\mathcal{D}_{\mathrm{mix}}=\mathcal{D}_{\mathrm{train}}\cup\mathcal{D}_{\mathrm{out}}$,
the underlying data distribution of $\mathcal{D}_{\mathrm{mix}}$ will be a
linear combination of the training distribution $P_{\mathrm{s}}(X,Y)$ and the
OOD distribution $P_{\mathrm{out}}(X,Y)$:

$P_{\mathrm{mix}}(\boldsymbol{x},y)=\frac{N}{M+N}P_{\mathrm{s}}(\boldsymbol{x},y)+\frac{M}{M+N}P_{\mathrm{out}}(\boldsymbol{x},y),$
(4)

where $N$ is the size of $\mathcal{D}_{\mathrm{train}}$ and $M$ is the size of
$\mathcal{D}_{\mathrm{out}}$.

By the virtue of Bayes’ theorem, we have

$\displaystyle P_{\mathrm{mix}}(\boldsymbol{x}|y)P_{\mathrm{mix}}(y)$ (5)
$\displaystyle=$
$\displaystyle\frac{N}{M+N}P_{\mathrm{s}}(\boldsymbol{x}|y)P_{\mathrm{s}}(y)+\frac{M}{M+N}P_{\mathrm{out}}(\boldsymbol{x}|y)P_{\mathrm{out}}(y)$
$\displaystyle=$
$\displaystyle\frac{N}{M+N}P_{\mathrm{s}}(\boldsymbol{x}|y)P_{\mathrm{s}}(y)+\frac{M}{M+N}P_{\mathrm{out}}(\boldsymbol{x})P_{\mathrm{out}}(y)$
$\displaystyle=$
$\displaystyle\frac{N}{M+N}P_{\mathrm{s}}(\boldsymbol{x}|y)P_{\mathrm{s}}(\boldsymbol{x},y)+\frac{1}{K}\cdot\frac{M}{M+N}P_{\mathrm{out}}(\boldsymbol{x}),$

where the second equality follows the fact that
$P_{\mathrm{mix}}(\boldsymbol{x}|y)=P_{\mathrm{mix}}(\boldsymbol{x})$ since
the label $y$ is independent of the instance $x$ for the OOD data, and the
third equality is simply the fact that $P_{\mathrm{out}}(y)=1/K$.

Then, by taking the maximum of both sides, we have

$\displaystyle\underset{y\in\mathcal{Y}}{\arg\max}~{}P_{\mathrm{mix}}(\boldsymbol{x}|y)P_{\mathrm{mix}}(y)$
(6) $\displaystyle=$
$\displaystyle\underset{y\in\mathcal{Y}}{\arg\max}~{}\left\\{\frac{N}{M+N}P_{\mathrm{s}}(\boldsymbol{x}|y)P_{\mathrm{s}}(\boldsymbol{x},y)+\frac{M/K}{M+N}P_{\mathrm{out}}(\boldsymbol{x})\right\\}$
$\displaystyle=$
$\displaystyle\underset{y\in\mathcal{Y}}{\arg\max}~{}\frac{N}{M+N}P_{\mathrm{s}}(\boldsymbol{x}|y)P_{\mathrm{s}}(\boldsymbol{x},y).$
$\displaystyle=$
$\displaystyle\underset{y\in\mathcal{Y}}{\arg\max}~{}P_{\mathrm{s}}(\boldsymbol{x}|y)P_{\mathrm{s}}(\boldsymbol{x},y).$

Thus Theorem 2.1 is proved. ∎

##  Appendix B Proof of Proposition 2.3

By definition, $\sum^{K}_{i=1}\Gamma_{i}=1$ naturally holds for any $\alpha$.

If $\alpha=\max_{j}(\beta_{j})$, then
$\Gamma_{j}=(\max_{j}(\beta_{j})-\beta_{j})/(K\cdot\max_{j}(\beta_{j})-1)$. In
particular, for $k=\arg\max_{i}(\beta_{i})$, we have $\Gamma_{k}=0$.

In the case of $\alpha\to\infty$, let us denote $f(\alpha)=\alpha-\beta_{j}$
and $g(\alpha)=K\cdot\alpha-1$. Since
$\lim_{\alpha\to\infty}f(\alpha)=\lim_{\alpha\to\infty}g(\alpha)=\infty$,
$g^{\prime}(\alpha)=K\neq 0$, and
$\lim_{\alpha\to\infty}f^{\prime}(\alpha)/g^{\prime}(\alpha)=1/K$ exists,
using L’Hôpital’s rule, we have:

$\lim_{\alpha\to\infty}\Gamma_{j}=\lim_{\alpha\to\infty}\frac{f(\alpha)}{g(\alpha)}=\lim_{\alpha\to\infty}\frac{f^{\prime}(\alpha)}{g^{\prime}(\alpha)}=1/K$

Thus Proposition 2.3 is proved. ∎

##  Appendix C Algorithm Details

The details of Open-sampling are provided below:

Algorithm 1 Open-sampling

0: Training dataset $\mathcal{D}_{\mathrm{train}}$. Open-set auxiliary dataset
$\mathcal{D}^{(x)}_{\mathrm{out}}$;

1: for each iteration do

2: Sample a mini-batch of original training samples
$\\{(\boldsymbol{x}_{i},y_{i})\\}^{n}_{i=0}$ from
$\mathcal{D}_{\mathrm{train}}$;

3: Sample a mini-batch of open-set instances
$\\{\widetilde{\boldsymbol{x}}_{i}\\}^{m}_{i=0}$ from
$\mathcal{D}^{(x)}_{\mathrm{out}}$;

4: Generate random noisy label $\widetilde{y}_{i}\sim\Gamma$ for each open-set
instance $\widetilde{\boldsymbol{x}}_{i}$;

5: Perform gradient descent on $f$ with $\mathcal{L}_{\mathrm{total}}$ from
Equation (2);

6: end for

##  Appendix D Datasets and empirical settings

Long-Tailed CIFAR. The original version of CIFAR-10 and CIFAR-100 contains
50,000 training images and 10,000 validation images of size $32\times 32$ with
10 and 100 classes, respectively. To create their long-tailed version, we
reduce the number of training examples per class according to an exponential
function $n=n_{j}\mu^{j}$, where $j$ is the class index, $n_{j}$ is the
original number of training images, and $\mu\in(0,1)$. Besides, the validation
set and the test set are kept unchanged. The imbalance ratio of a dataset is
defined as the number of training samples in the largest class divided by that
of the smallest.

CelebA-5. CelebFaces Attributes (CelebA) dataset is a real-world long-tailed
dataset. It is originally composed of 202,599 number of RGB face images with
40 binary attributes annotations per image. Note that CelebA is originally a
multi-labeled dataset, we port it to a 5-way classification task by filtering
only the samples with five non-overlapping labels about hair colors. We also
subsampled the full dataset by 1/20 while maintaining the imbalance ratio as
10.7, following (Kim et al., 2020). In particular, We pick out 50 and 100
samples in each class for validation and testing. We denote the resulting
dataset by CelebA-5.

Places-LT. Places-LT is a long-tailed version of the large-scale scene
classification dataset Places (Zhou et al., 2017). It consists of 184.5K
images from 365 categories with class cardinality ranging from 5 to 4,980.

Auxiliary datasets. 300K Random Images (Hendrycks et al., 2019) is a cleaned
and debiased dataset with 300K natural images. In this dataset, Images that
belong to CIFAR classes from it, images that belong to Places or LSUN classes,
and images with divisive metadata are removed so that
$\mathcal{D}_{\mathrm{train}}$ and $\mathcal{D}_{\mathrm{out}}$ are disjoint.
We use the dataset as the open-set auxiliary dataset for experiments with
CIFAR-10/100 and CelebA-5. For Experiments on Places-LT, we use the training
set of Places-Extra69 (Zhou et al., 2017) as the open-set auxiliary dataset.

OOD test datasets. Following OE (Hendrycks et al., 2019), we comprehensively
evaluate OOD detectors on artificial and real anomalous distributions,
including: Gaussian, Rademacher, Blobs, Textures (Cimpoi et al., 2014), SVHN
(Netzer et al., 2011), Places365 (Zhou et al., 2017), LSUN-Crop (Yu et al.,
2015), LSUN-Resize (Yu et al., 2015), iSUN (Xu et al., 2015). For experiments
on CIFAR-10, we also use CIFAR-100 as OOD test dataset. Gaussian noises have
each dimension i.i.d. sampled from an isotropic Gaussian distribution.
Rademacher noises are images where each dimension is $-1$ or $1$ with equal
probability, so each dimension is sampled from a symmetric Rademacher
distribution. Blobs noises consist in algorithmically generated amorphous
shapes with definite edges. Textures (Cimpoi et al., 2014) is a dataset of
describable textural images. SVHN dataset (Netzer et al., 2011) contains 32 ×
32 color images of house numbers. There are ten classes comprised of the
digits 0-9. Places365 (Zhou et al., 2017) consists in images for scene
recognition rather than object recognition. LSUN (Yu et al., 2015) is another
scene understanding dataset with fewer classes than Places365. Here we use
LSUN-Crop and LSUN-Resize to denote the cropped and resized version of the
LSUN dataset respectively. iSUN (Xu et al., 2015) is a large-scale eye
tracking dataset, selected from natural scene images of the SUN database (Xiao
et al., 2010).

##  Appendix E Implementation details

For experiments on Long-Tailed CIFAR-10/100 (Krizhevsky et al., 2009) and
CelebA-5 (Liu et al., 2015a), we perform training with ResNet-32 (He et al.,
2016) for 200 epochs, using SGD with a momentum of 0.9, and a weight decay of
0.0002. We set the initial learning rate as 0.1, then decay by 0.01 at the
160th epoch and again at the 180th epoch. For fair comparison, We also use
linear warm-up learning rate schedule (Goyal et al., 2017) for the first 5
epochs. For data augmentation in training, we use the commonly used version: 4
pixels are padded on each side, and a $32\times 32$ crop is randomly sampled
from the padded image or its horizontal flip. For experiments under the
setting of Balanced Softmax (Ren et al., 2020), we use Nesterov SGD with
momentum 0.9 and weight-decay 0.0005 for training. We use a total mini-batch
size of 512 images on a single GPU. The learning rate increased from 0.05 to
0.1 in the first 800 iterations. Cosine scheduler (Loshchilov & Hutter, 2016)
is applied afterward, with a minimum learning rate of 0. Our augmentation
follows Balanced Softmax and Equalization Loss (Tan et al., 2020). In testing,
the image size is $32\times 32$. In end-to-end training, the model is trained
for 13K iterations. In decoupled training experiments, we fix the Softmax
model, i.e., the instance-balanced baseline model obtained from the standard
end-to-end training, as the feature extractor. And the classifier is trained
for 2K iterations.

For Places-LT, we choose ResNet-152 as the backbone network and pretrain it on
the full ImageNet-2012 dataset, following Decoupled training (Kang et al.,
2020). We use SGD optimizer with momentum 0.9, batch size 512, cosine learning
rate schedule (Loshchilov & Hutter, 2016). Similar to Decoupled training, we
fine-tune the backbone with Instance-balanced sampling for representation
learning and then re-train the classifier with our proposed algorithm with
class-balanced sampling. Through the paper, we refer to decoupled training as
training the last linear classifier on a fixed feature extractor obtained from
instance-balanced training. For experiments with the SSP method, we use
rotation prediction (Komodakis & Gidaris, 2018) as self-supervised learning
method, where an image is rotated by a random multiple of 90 degrees and a
model is trained to predict the rotation degree as a 4-way classification
problem.

We conduct all the experiments on NVIDIA GeForce RTX 3090, and implement all
methods with default parameters by PyTorch (Paszke et al., 2019). All
experiments are repeated five times with different seeds and we report the
average test accuracy. We tune the hyperparameter $\eta$ on the validation
set, then train the model on the full training set. The $\alpha$ is fixed as
$(\max_{j}\beta_{j}+\min_{j}\beta_{j})$ by default and we find this value
performs well overall. For the $\eta$ in the training objective, the best
value depends on the dataset, imbalance ratio, network architecture, and the
integrated method.

For OOD detection task, we measure the following metrics: (1) the false
positive rate (FPR95) of OOD examples when the true positive rate of in-
distribution examples is at 95%; (2) the area under the receiver operating
characteristic curve (AUROC); and (3) the area under the precision-recall
curve (AUPR).

##  Appendix F More empirical results

###  F.1 Ablation study

To show the effect of the class-dependent weighting factor $w_{j}$ in Open-
sampling, we conduct ablation study on long-tailed CIFAR-10 with imbalanced
rate 100. As shown in Table 6, the class-dependent weighting factor generally
improves the performance, when compared with the variant without the weight.

Table 6: Results of ablation study on long-tailed CIFAR-10 for the class-
dependent weighting factor $w_{j}$.

Imbalance Factor |  100 |  50 |  10
---|---|---|---
Standard |  71.61 |  77.30 |  86.74
Ours w/o $w_{j}$ |  76.57 |  81.18 |  88.57
Ours |  77.62 |  81.76 |  89.38

###  F.2 Detailed results for OOD detection.

Table 7 presents the detailed results of OOD detection performance on long-
tailed CIFAR-10 with imbalanced rate 100 with various OOD test datasets. From
the results, we show that our method can outperform OE (Hendrycks et al.,
2019) in OOD detection tasks when training dataset is class-imbalanced.

Table 7:  OOD detection performance comparison on long-tailed CIFAR-10 with imbalanced rate 100. $\uparrow$ indicates larger values are better and $\downarrow$ indicates smaller values are better. OOD test dataset | Method | FPR95 $\downarrow$ | AUROC $\uparrow$ | AUPR $\uparrow$
---|---|---|---|---
Gaussian | MSP | 43.02 | 68.62 | 21.54
OE | 13.71 | 87.94 | 40.61
Ours | 7.52 | 93.37 | 53.82
Ours $(\alpha=5)$ | 1.20 | 99.55 | 95.72
Rademacher | MSP | 38.33 | 81.44 | 32.97
OE | 13.88 | 87.17 | 39.50
Ours | 7.93 | 92.91 | 52.39
Ours $(\alpha=5)$ | 3.25 | 97.90 | 78.00
Blob | MSP | 49.03 | 80.88 | 39.51
OE | 44.29 | 76.55 | 27.28
Ours | 9.42 | 93.33 | 55.45
Ours $(\alpha=5)$ | 33.65 | 91.50 | 66.51
Textures | MSP | 66.53 | 72.19 | 29.27
OE | 38.00 | 84.92 | 38.14
Ours | 23.29 | 92.53 | 57.99
Ours $(\alpha=5)$ | 24.64 | 94.91 | 78.43
SVHN | MSP | 63.74 | 69.11 | 24.06
OE | 43.46 | 84.17 | 37.06
Ours | 22.09 | 93.53 | 63.92
Ours $(\alpha=5)$ | 28.89 | 92.12 | 67.60
CIFAR-100 | MSP | 70.50 | 70.24 | 28.27
OE | 61.15 | 77.88 | 32.77
Ours | 55.32 | 84.09 | 45.81
Ours $(\alpha=5)$ | 54.73 | 84.75 | 54.57
LSUN-C | MSP | 40.68 | 85.52 | 50.07
OE | 22.60 | 86.80 | 39.06
Ours | 14.73 | 96.70 | 80.81
Ours $(\alpha=5)$ | 13.66 | 96.25 | 82.77
LSUN-R | MSP | 59.27 | 76.86 | 36.88
OE | 17.91 | 87.36 | 39.86
Ours | 15.29 | 93.87 | 60.49
Ours $(\alpha=5)$ | 10.68 | 97.10 | 82.44
iSUN | MSP | 61.86 | 74.66 | 32.68
OE | 20.02 | 87.44 | 40.37
Ours | 15.33 | 93.66 | 58.85
Ours $(\alpha=5)$ | 11.47 | 96.92 | 82.40
Places365 | MSP | 67.98 | 72.47 | 31.82
OE | 48.77 | 81.24 | 33.90
Ours | 35.91 | 89.98 | 54.23
Ours $(\alpha=5)$ | 39.16 | 91.60 | 70.69


* [26] N. Paskal, M. Cameron, An efficient jet marcher for computing the quasipotential for 2d sdes, Journal of Scientific Computing 91 (30) (2022).
  * [27] L. Kikuchi, R. Singh, M. E. Cates, R. Adhikari, Ritz method for transition paths and quasipotentials of rare diffusive events, arXiv: Statistical Mechanics (2020).
  * [28] W. E, E. Vanden-Eijnden, Towards a theory of transition paths, Journal of Statistical Physics (2006).
URL https://link.springer.com/article/10.1007/s10955-005-9003-9

  * [29] W. E, E. Vanden-Eijnden, Transition-path theory and path-finding algorithms for the study of rare events, Annual Review of Physical Chemistry 61 (1) (2010) 391–420, pMID: 18999998.  doi:10.1146/annurev.physchem.040808.090412.
  * [30] Y. Khoo, J. Lu, L. Ying, Solving for high dimensional committor functions using artificial neural networks, CoRR abs/1802.10275 (2018).  arXiv:1802.10275.
URL http://arxiv.org/abs/1802.10275

  * [31] Q. Li, B. Lin, W. Ren, Computing committor functions for the study of rare events using deep learning, The Journal of Chemical Physics 151 (2019) 054112.
URL https://aip.scitation.org/doi/10.1063/1.5110439

  * [32] G. M. Rotskoff, E. Vanden-Eijnden, Learning with rare data: Using active importance sampling to optimize objectives dominated by rare events, arXiv: Data Analysis, Statistics and Probability (2020).
  * [33] R. Banisch, Z. Trstanova, A. Bittracher, S. Klus, P. Koltai, Diffusion maps tailored to arbitrary non-degenerate itô processes, Applied and Computational Harmonic Analysis 48 (1) (2020) 242–265.
  * [34] L. Evans, M. K. Cameron, P. Tiwary, Computing committors in collective variables via mahalanobis diffusion maps, Applied and Computational Harmonic Analysis 64 (2023) 62–101.  doi:https://doi.org/10.1016/j.acha.2023.01.001.
  * [35] L. Evans, M. Cameron, P. Tiwary, Computing committors via mahalanobis diffusion maps with enhanced sampling data, The Journal of Chemical Physics 157 (2022) 214107\.
  * [36] Y. Chen, J. Hoskins, Y. Khoo, M. Lindsey, Committor functions via tensor networks, Journal of Computational Physics 472 (2023) 111646\.  doi:https://doi.org/10.1016/j.jcp.2022.111646.
URL https://www.sciencedirect.com/science/article/pii/S0021999122007094

  * [37] M. Freidlin, Some remarks on the smoluchowski-kramers approximation, Journal of Statistical Physics 117 (11) (2004) 617–634.
  * [38] M. I. Freidlin, A. D. Wentzell, Random Perturbations of Dynamical Systems, Springer, Heidelberg New York Dordrecht London, 2012.
  * [39] M. Cameron, Finding the quasipotential for nongradient sdes, Physica D: Nonlinear Phenomena 241 (18) (2012) 1532–1550.
  * [40] D. Dahiya, M. Cameron, An ordered line integral method for computing the quasi-potential in the case of variable anisotropic diffusion, Physica D: Nonlinear Phenomena 382 (2018) 33–45.
  * [41] C. Schuette, S. Winkelmann, C. Hartmann, Optimal control of molecular dynamics using markov state models, Mathematical Programming 134 (2012) 259–282.
  * [42] C. Hartmann, R. Banisch, M. Sarich, T. Badowski, C. Schütte, Characterization of rare events in molecular dynamics, Entropy 16 (1) (2014) 350–376.
URL https://www.mdpi.com/1099-4300/16/1/350

  * [43] F. Legoll, T. Lelievre, Effective dynamics using conditional expectations, Nonlinearity 23 (9) (2010) 2131.  doi:10.1088/0951-7715/23/9/006.
URL https://dx.doi.org/10.1088/0951-7715/23/9/006

  * [44] J. Yuan, Github: Optimal stochastic control and neural network-based solvers for the committor problem (2023).
URL https://github.com/margotyjx/Optimal_control.git

  * [45] M. Cameron, Github: Transition path theory, fem, distmesh (2023).
URL
https://github.com/mar1akc/transition_path_theory_FEM_distmesh/blob/main/README.md

  * [46] M. Raissi, P. Perdikaris, G. Karniadakis, Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, Journal of Computational Physics 378 (2019) 686–707.  doi:https://doi.org/10.1016/j.jcp.2018.10.045.
  * [47] B. Dickson, D. Makarov, Pitfalls of choosing an order parameter for rare event calculations, The Journal of chemical physics 131 (2009) 074108.  doi:10.1063/1.3204008.
  * [48] M. K. Cameron, E. Vanden-Eijnden, Flows in complex networks: Theory, algorithms, and application to lennard–jones cluster rearrangement, Journal of Statistical Physics 156 (2014) 427–454.
  * [49] W. M. Fleming, H. Soner, Controlled Markov Processes and Viscosity Solutions, Springer Science, 2006.
  * [50] L. C. Evans, An introduction to mathematical optimal control theory, https://math.berkeley.edu/~evans/control.course.pdf, lecture notes. Version 0.2.
  * [51] M. Baudel, A. Guyader, T. Lelievre, On the hill relation and the mean reaction time for metastable processes, Stochastic Processes and their Applications 155 (2023) 393–436.  doi:https://doi.org/10.1016/j.spa.2022.10.014.
URL https://www.sciencedirect.com/science/article/pii/S0304414922002319

  * [52] T. Lelievre, M. Ramil, J. Reygner, Estimation of statistics of transitions and hill relation for langevin dynamics, Annales de l’Institut Henri Poincare (2023).
URL https://arxiv.org/abs/2206.13264

  * [53] T. S. van Erp, D. Moroni, P. G. Bolhuis, A novel path sampling method for the calculation of rate constants, The Journal of Chemical Physics 118 (17) (2003) 7762–7774.  doi:10.1063/1.1562614.
  * [54] R. J. Allen, P. B. Warren, P. R. ten Wolde, Sampling rare switching events in biochemical networks, Phys. Rev. Lett. 94 (2005) 018104.  doi:10.1103/PhysRevLett.94.018104.
  * [55] L. Donati, B. Keller, Girsanov reweighting for metadynamics simulations, The Journal of Chemical Physics 149 (2018) 072335.  doi:10.1063/1.5027728.
  * [56] W. Zhang, C. Hartmann, C. Schütte, Effective dynamics along given reaction coordinates and reaction rate theory, Faraday Discuss. 195 (10 2016).  doi:10.1039/C6FD00147E.
  * [57] I. Lagaris, A. Likas, D. Fotiadis, Artificial neural networks for solving ordinary and partial differential equations, IEEE Transactions on Neural Networks 9 (5) (1998) 987–1000.  doi:10.1109/72.712178.
  * [58] M. Crosskey, M. Maggioni, Atlas: A geometric approach to learning high-dimensional stochastic systems near manifolds, Multiscale Modeling & Simulation 15 (1) (2017) 110–156.  doi:10.1137/140970951.
URL https://doi.org/10.1137/140970951

  * [59] L. Evans, M. K. Cameron, P. Tiwary, Computing committors via mahalanobis diffusion maps with enhanced sampling data, The Journal of Chemical Physics 157 (21) (2022) 214107.  arXiv:https://doi.org/10.1063/5.0122990, doi:10.1063/5.0122990.
URL https://doi.org/10.1063/5.0122990

  * [60] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, CoRR abs/1412.6980 (2014).
  * [61] G. E. Karniadakis, I. G. Kevrekidis, L. L. 5, P. Perdikaris, S. Wang, L. Yang, Physics-informed machine learning, Nature (2021).
URL https://www.nature.com/articles/s42254-021-00314-5

  * [62] P.-O. Persson, G. Strang, A simple mesh generator in matlab, SIAM Review 46 (2) (2004) 329–345.  doi:10.1137/S0036144503429121.
  * [63] D. Engwirda, Mesh2d-automatic mesh generation, available online on MatlabCentral: http://www. mathworks. com/matlabcentral/fileexchange/25555-mesh2d-automatic-mesh-generation (2009).
  * [64] C. Dellago, P. G. Bolhuis, D. W. Chandler, Efficient transition path sampling: Application to lennard-jones cluster rearrangements, Journal of Chemical Physics 108 (1998) 9236–9245.
  * [65] D. Passerone, M. Parrinello, Action-derived molecular dynamics in the study of rare events, Physical Review Letters 87 (10) (2001) 108302.
  * [66] D. J. Wales, Discrete path sampling, Molecular Physics 100 (20) (2002) 3285–3305.  arXiv:https://doi.org/10.1080/00268970210162691, doi:10.1080/00268970210162691.
URL https://doi.org/10.1080/00268970210162691

  * [67] R. R. Coifman, I. G. Kevrekidis, S. Lafon, M. Maggioni, B. Nadler, Diffusion maps, reduction coordinates, and low dimensional representation of stochastic systems, Multiscale Modeling & Simulation 7 (2) (2008) 842–864.  arXiv:https://doi.org/10.1137/070696325, doi:10.1137/070696325.
URL https://doi.org/10.1137/070696325

  * [68] G. A. Tribello, M. Ceriotti, M. Parrinello, A self-learning algorithm for biased molecular dynamics, Proceedings of the National Academy of Sciences of the United States of America 107 (41) (2010) 17509–17514.
URL http://www.jstor.org/stable/20780489

  * [69] S.-T. Tsai, Z. Smith, P. Tiwary, Reaction coordinates and rate constants for liquid droplet nucleation: Quantifying the interplay between driving force and memory., The Journal of chemical physics 151 15 (2019) 154106.
  * [70] G. O. Roberts, R. L. Tweedie, Exponential convergence of langevin distributions and their discrete approximations, Bernoulli 2 (4) (1996) 341 – 363.  doi:bj/1178291835.
  * [71] B. Øksendal, Stochastic Differential Equations: An Introduction with Applications, Springer, 2003.
  * [72] J. Alberty, C. Carstensen, S. A. Funken, Remarks around 50 lines of matlab: short finite element implementation, Numerical Algorithms 20 (1999) 117–137.
URL https://doi.org/10.1023/A:1019155918070

  * [73] K. W. Morton, Finite element methods for non-self-adjoint problems, in: P. R. Turner (Ed.), Topics in Numerical Analysis, Springer, Lancaster, England, 1981, pp. 113–148.