11institutetext: State Key Laboratory for Novel Software Technology, Nanjing
University, China
11email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
# Logit Normalization for Long-tail Object Detection

Liang Zhao* Yao Teng* Limin Wang🖂

###### Abstract

Real-world data exhibiting skewed distributions pose a serious challenge to
existing object detectors. Moreover, the samplers in detectors lead to shifted
training label distributions, while the tremendous proportion of background to
foreground samples severely harms foreground classification. To mitigate these
issues, in this paper, we propose Logit Normalization (LogN), a simple
technique to self-calibrate the classified logits of detectors in a similar
way to batch normalization. In general, our LogN is training- and tuning-free
(i.e. require no extra training and tuning process), model- and label
distribution-agnostic (i.e. generalization to different kinds of detectors and
datasets), and also plug-and-play (i.e. direct application without any bells
and whistles). Extensive experiments on the LVIS dataset demonstrate superior
performance of LogN to state-of-the-art methods with various detectors and
backbones. We also provide in-depth studies on different aspects of our LogN.
Further experiments on ImageNet-LT reveal its competitiveness and
generalizability. Our LogN can serve as a strong baseline for long-tail object
detection and is expected to inspire future research in this field. Code and
trained models will be publicly available at https://github.com/MCG-NJU/LogN.

###### Keywords:

Long-tail object detection, normalization

††*: Equal contribution. 🖂: Corresponding author.

##  1 Introduction

Large-scale datasets collected manually such as MS COCO [22] and Open Images
[18] have delivered tremendous success on object detection. However, in
comparison with data from realistic scenarios, they tend to be relatively
balanced, since realistic data often exhibit skewed distributions with a long
tail [29, 24]. In this sense, a handful of head (frequent) classes containing
massive samples dominate, yet a lot of tail (rare) classes containing a few
samples are thus concealed. Such a class imbalance poses considerable
challenges for object detectors.

Apart from the skewed class distribution, we also summarize the other two
challenges from the perspective of detection: (1) The samplers in object
detectors lead to a shift between label frequencies [50, 27] (i.e. the
frequency of samples in each category) calculated in advance and those
calculated during training, as shown in Fig. 1a. In this paper, we term the
frequency calculated in advance as prior label distribution and the frequency
calculated during training as training label distribution. Obviously, for one
detector, its intrinsic label assignment [51, 11] which is based on an
internal sampler, fluctuates the original prior label distribution.
Furthermore, with the commonly attached external sampler such as RFS [12], the
final training label distribution is gradually shifted from head to tail
categories. (2) The tremendous proportion of background proposals depicted in
Fig. 1b severely harms the capability of object detectors in classifying
foreground proposals. Unlike regular image classifiers, the classifier in
object detector is required to precisely discriminate the foreground proposals
from the background. However, for each foreground category, the extreme
foreground-background class imbalance [21] severely affects its
classification.

(a) Differences between logarithms of training label distributions (with and
without external sampler, RFS [12]) and logarithm of prior label distribution.

(b) Per and cumulative ratios of the number of samples in foreground class
($N_{FG}$) relative to the number of samples in the background class
($N_{BG}$).

Figure 1:  The x-axis indicates the category index sorted according to the
number of samples included. (a) Relative to prior label distribution, internal
sampler causes fluctuations and external sampler causes shifts, co-leading to
the alternations of training label distributions. (b) Tremendous ratios of
foreground to background samples from head to tail classes could influence the
foreground classification.

Based on these challenges, there are diverse techniques for tackling long-tail
object detection [38, 37, 43, 9, 30]. Re-sampling [12], an intuitive method
that re-balances the number of training samples from head to tail classes, has
not gained enough favor in community. This technique not only leads to higher
training complexity, but also easily causes the issues of under-fitting to
head classes and over-fitting to tail classes. Far more methods [38, 33, 30]
aim at re-weighting training loss by modifying the logits, i.e. the classified
outputs, with class priors, e.g. the label frequency of training set. Although
these methods are theoretically proven to be well suited for classification
tasks [33], unfortunately, they are often not well adapted to object detectors
due to the shift caused by samplers [19]. There are also several approaches
[37, 43, 9] that seek to dynamically accommodate long-tail distribution with
statistics, but they contain lots of parameters. Moreover, in long-tail object
detection, most methods lack an adaptive and unified design for the background
class like the foreground classes.

(a) Logit Mean.

(b) Logit Variance.

Figure 2: Logit statistics and logarithms of training label distributions,
with and without external sampler. (a) Logit mean consistently reflects the
inter-class long-tail tendency of the dynamic training label distributions.
(b) Logit variance is also correlated with the long-tail distribution.

In this paper, we propose Logit Normalization (LogN), a conceptually simple
yet quite effective post-hoc calibration method for long-tail object
detection. By shifting the operation of batch normalization (BN) to the
logits, we are able to rectify the predictions of detectors biased by long-
tail distribution, in a highly simple and versatile style. Our LogN is based
on two observations: (1) The statistics (i.e. mean and variance) at logits are
capable of characterizing the inter-class long-tail tendency with label
distribution, as illustrated in Fig. 2. This phenomenon, in conjunction with
the prior normalization-based work [41, 8, 15] for style translation,
motivates us to leverage the predictions themselves for self-calibration, so
as to realize a sort of Long-tailedness Normalization. As a BN-like
calibration method, it can be flexibly fitted to arbitrary detectors or
distributions, without injecting any label-related priors. (2) Improving the
discriminability of classifier on background instead contributes to its
foreground classification, as analyzed in Sec. 3.3. On this basis, we increase
the margin between foreground and background logits via adding a negative
scalar to the logit of each foreground class, so as to accomplish an
equivalent background calibration. Practically, we provide an adaptive
solution for this margin, which allows our LogN to be parameter-free.

Overall, LogN comprises the following three compelling characteristics: (1)
Training-free and tuning-free: as a post-hoc normalization technique, LogN
requires no independent training as in [38, 37, 9], and also requires no
extensive parametric adjustments as in [37, 48], being highly efficient and
resource-saving. (2) Model-agnostic and label distribution-agnostic: when both
the network structure and label distribution are invisible, LogN is still
capable of delivering promising results, guaranteeing maximum security and
privacy [36] of the model and data compared to previous methods. (3) Plug-and-
play: it can be easily implemented with a few codes appended to the predicted
logits, and simply deployed to various platforms, without any bells and
whistles.

Extensive experiments have been conducted on the challenging long-tail object
detection dataset, LVIS [12], and the results demonstrate superior performance
to the state-of-the-art models with various detectors including Mask R-CNN
[13] and Cascade Mask R-CNN [4], using diverse backbones including
ResNet-50-FPN [14, 20] and ResNet-101-FPN. We further experiment on the long-
tail image classification dataset ImageNet-LT [24], and the competitive
performance illustrates the favorable generality of LogN.

##  2 Related Work

Normalization. Generally, normalization techniques can be divided into two
groups, targeting weights [28, 32, 34, 44] and features [16, 7, 47, 1, 45, 41,
8]. This work is a generalization of feature normalization, thus we focus on
the latter.

Ioffe and Szegedy [16] pioneered a feature-level batch normalization (BN). By
normalizing activations using feature statistics within a mini-batch, BN
conditions the training with high learning rates and hence improves
generalization accuracy effectively. Afterwards, various subsequent methods
shift their attention from the batch dimension to (1) fine-grained batch
dimension: Cross-GPU BN [31] on larger batch size across multiple GPUs, Ghost
Normalization [7] on small virtual batch and Cross-Iteration BN [47] on recent
iterations of batch; (2) channel dimension: Layer Normalization [1] on all
channels, Group Normalization [45] on groups of channels, and Instance
Normalization (IN) [41] on each channel; (3) combined dimension: Switchable
Normalization series [25, 26, 35] on mixtures of multiple dimensions including
batch, channel, etc. In contrast to these methods that overly focus on
intermediate features, our method operates normalization during inference on
the logits for the first time, and is devoted to standardizing the biased
predictions of detectors suffered from long-tail learning.

It also worth noting that IN and its subsequent methods (e.g. Conditional IN
[8] and Adaptive IN [15]) not only achieve impressive results for practical
cross-domain learning, but also prompt a rethinking of normalization in terms
of transfer learning. Huang et al. [15] found that instance-wise normalization
is essentially a kind of style purification, while the subsequent operation of
scale and shift is the adaptation. In this paper, we continue this thought,
but focus on label distribution rather than on feature distribution that
characterize styles. We purify biased predictions by simply normalizing them
with logit statistics, enabling the detectors to better cope with relatively
balanced test distribution.

Long-Tail Learning. A great number of methods have been proposed for long-tail
learning. Lin et al. [21] proposed focal loss to mine the hardness of
classification correlated with the label imbalance, and then online modified
the penalty on each prediction. Kang et al. [17] utilized decoupled training
strategy to disentangle the representation learning with the classifier
training, and separately trained a balanced classifier to rectify the decision
boundaries. Logit Adjustment (LogA) [33, 27] leveraged the label frequency to
adjust logits and theoretically guaranteed its optimality w.r.t. the ideal
image classification tasks.

Object detection suffers more severely from the long-tail category
distribution [12]. Up to now, considerable methods have been proposed to
address this challenge. Repeat factor sampling (RFS) [12] is a re-sampling
strategy applied to this task in early research and its core is oversampling
the rare categories. Balanced Group Softmax (BAGS) [19] split the object
categories into multiple groups and modified the application of softmax. Tan
et al. [38, 37] proposed a series of equalization loss for balancing the
weights of frequent and rare classes from a gradient competition view. Pan et
al. [30] introduced directly LogA into long-tail object detection and kept the
background score intact. Recently, several efforts [37, 9, 43] start to inject
statistics into classification loss to dynamically accommodate the training
distribution, for harmonizing the training of each class more accurately.
Among them, Seesaw loss [43] balanced the training on different categories via
online accumulated instance numbers and predicted probability. EQL v2 [37]
dynamically equalized gradients between positives and negatives with the
gradient statistics. Feng et al. [9] brought statistics of classification
scores into long-tail foreground object classification and set a hyper-
parameter to adjust the background score. Problematically however, these
methods typically exhibit complicated forms with lots of parameters. Moreover,
they overly focus on the imbalance within foreground object categories yet
overlooked the appropriate treatment of background classification. This paper
utilizes a format of normalization to calibrate the logits in the phase of
testing, ensuring both the efficiency and simplicity. We further analyze in
detail the critical role of background on long-tail detectors and provide a
concise and adaptive solution.

##  3 Logit Normalization

Most real-world data is long-tail in nature, and generally the number of
samples per class diminishes exponentially from head to tail classes. In this
context, a general classifier based on random sampling suffers from a training
label distribution that is highly consistent with the long-tail prior label
distribution, resulting in samples from head classes predominant and those
from tail classes under-learned. Thus a model, especially a detector trained
on a long-tail training set, often underperforms on a class-balanced test set.
Various methods [33, 27, 30] calibrate such a biased distribution by injecting
explicitly the prior label distribution, or namely label frequencies into
logits. Despite great success in image classification, their performance in
object detection is poorly guaranteed [19].

Different from image classification, the training label distribution in object
detection is normally inconsistent with prior distribution, as illustrated in
Fig. 1a. It can be observed that there is always a difference between the
prior and training label distribution. We ascribe this phenomenon to the
samplers within object detectors. Regular detectors normally utilize the label
assignment mechanism [51, 11], an implicit internal sampler, for handling the
label co-occurrence issue [49]. Yet this sampler implicitly oscillates label
frequencies and hence renders a fluctuating training label distribution.
Additionally, some detectors further introduce external samplers [12, 9] that
re-sample images or instances to alleviate severe long-tail bias, explicitly
shifting the label distribution. These samplers co-lead to an altered training
label distribution. Currently, in response to the shift of distribution,
several recent methods [37, 43, 9] propose to dynamically adapt, in terms of
the statistics on logit. They mostly specialize in the relevance between the
label distribution and the mean of classification scores or logits, which
consistently reflects the inter-class long-tail tendency, as shown in Fig. 2a,
but ignore the more in-depth 2-order statistic, variance, which is also
correlated with the training label distribution, as shown in Fig. 2b.
Furthermore, logit variance is adequate to represent the intra-class magnitude
[40, 2], or namely the hardness [42] of categories, and is able to balance the
decision boundaries of classifiers like $\tau$-norm [17].

These statistics (i.e. the mean and variance) on features are often
investigated in [16, 1, 45] for normalizing activations. They are also
commonly considered as a surrogate to image style [41, 8, 15] and facilitate
the practice of style normalization. However, the association between them and
the categories, or the label distribution is still poorly explored. Recently,
Frosst et al. [10] found that networks typically persist in developing class-
independent representations until the final layer, where class-dependent
logits are exploited for discriminating, indicating a disagreement between the
statistics on features and logits w.r.t. the categories. These works motivate
us to explicitly develop the correlation between logit statistics and the
label distribution.

Another critical issue in detection is the background class. Unlike image
classifiers, a well-trained object detector is capable of distinguishing the
foreground objects from the background. Generally, detectors treat the
background as a specific background category, and the built-in classifiers are
usually designed with $C+1$ categories, where $C$ denotes the number of
foreground object categories. In long-tail object detection, the ability of
object detectors in classifying foreground objects is always constrained by
the proportion of background proposals. As depicted in Fig. 1b, the number of
training instances of each foreground category is greatly less than those of
the background. For this problem, past practices either directly overlook the
background class [38, 37, 30] or manually calibrate it with extra parameters
[9]. A proper analytical and adaptive treatment towards the background class
is urgently needed.

Inspired by the above, we propose a novel post-hoc logit normalization for
long-tail object detection.

###  3.1 Formulation

We first describe a general formulation of our logit normalization on the
classification task, and then present its formulation when applied to object
detection. For the $C$-d logits output by a classifier, the normalization
performs on the whole vector in the testing phase. Its general formulation is
as follows:

$\hat{x}=\frac{x-\mu}{\sqrt{\sigma^{2}+\varepsilon}},$ (1)

where $x\in\mathbb{R}^{C}$ indicates the logits and
$\mu,\sigma\in\mathbb{R}^{C}$ denote their mean and standard deviation post-
hoc calculated from the training samples. $\varepsilon$ is equipped for
numerical stability. To efficiently estimate these statistics, we employ
exponential moving average in practice to accumulate global statistics from
batch-wise statistics as in BN [16]. Notably, each statistic is calculated
over all samples.

When it comes to the object detection task, networks first extract features
for each ROI and then categorize them into batch of logits,
$X\in\mathbb{R}^{N\times(C+1)}$, each of which comprises logits on $C$
foreground classes as well as one background class. Contrary to the
calibration in image classification, it is detrimental to object detection
performance if we directly mimic this form of calibration on all classes. In
experiments, we find the reason behind this phenomenon is the statistics of
the background category. Thus, we additionally consider a background
calibration to repair the calibration on background. Concretely, we use a
specific scalar to consistently modify the logits of all foreground classes,
according to the equivalence of the operation on background logit and all
foreground logits. This operation can be viewed as imposing a certain margin
between each foreground logit and the background one. Therefore, the
formulation of the normalization on long-tail object detection becomes as
follows:

$\hat{x}=\frac{x-\mu+\beta}{\sqrt{\sigma^{2}+\varepsilon}},$ (2)

where $\beta\in\mathbb{R}$ denotes the background calibration scalar, and it
can be assigned with the minimum value of the mean of foreground logits.
$x\in\mathbb{R}^{C}$ indicates foreground logits and
$\mu,\sigma\in\mathbb{R}^{C}$ denote their mean and standard deviation.

In Algorithm 1, we illustrate the PyTorch-like implementation on object
detection of our LogN. First, we offline traverse the training set and
aggregate the global statistics. Second, we equivalently perform the
background calibration on all foreground logits. Third, we apply our LogN on
classified logits ${x}$, and these transformed logits are for scores. Since
the overall algorithm is exclusively executed after training, we also refer to
it as the post-hoc Logit Normalization.

Algorithm 1 LogN Pseudocode, PyTorch-like

⬇

# detector: a well-trained object detector

# running_mean: global mean, running_var: global variance

# beta: background calibration scalar

# eps: 1e-5 for numerical stability, EMA: Exponential Moving Average

detector.eval() # the following steps are during **testing**

# 1. traverse training samples for calculating statistics

for x in train_loader: # load a training batch x

feat = detector.extractor(x) # extract (ROI) features

logit = detector.cls_head(feat) # output classifications

mean = logit.mean(dim=0), var = logit.var(dim=0) # compute statistics in a
mini-batch

# update global mean and variance with EMA

running_mean = EMA(running_mean, mean), running_var = EMA(running_var, var)

# 2. background calibration

beta = running_mean.min() # an adaptive beta value

running_mean = running_mean - beta

running_mean[0] = 0, running_var[0] = 1 # index 0 as the background

# 3. normalize logit for testing samples

for x in test_loader: # load a testing batch x

feat = detector.extractor(x) # extract (ROI) features

logit = detector.cls_head(feat) # output classifications

normed_logit = (logit - running_mean) / sqrt(running_var + eps) # Logit
Normalization

###  3.2 Discussion

Relation to BN. The proposed LogN is formally close to BN [16], i.e. they both
use statistics for normalization. This ensures the versatility of LogN, since
by post-hoc enforcing BN on logit, it can be conveniently adapted to any
detectors and distributions in a plug-and-play manner, without training or
tuning. However, differently, BN normalizes intermediate activations with
feature statistics to accelerate network training; while LogN normalizes final
activations with logit statistics to cope with long-tail learning. Meanwhile,
BN functions both in training and testing phases, to ensure the homogeneity in
the input distribution of the data in both phases; yet LogN functions solely
in testing phase, to handle their heterogeneity in the label distribution.
Additionally, LogN does not include scale and shift, as the normalized data
exhibit a basically uniform label distribution, which is also guaranteed in
test set by a regular long-tail setting.

Relation to Prior-based Methods. Existing methods [38, 33, 27, 30] focus on
the application of dataset priors. They typically inject label frequencies
into the classified logits, hence fail in accommodating the re-sampled
training label distribution within detectors. In contrast, our LogN that
normalizes logits with post-hoc calculated statistics is capable of
dynamically adapting this altered distribution. Moreover, the BN-like
formulation renders LogN as a non-parametric self-calibration approach, and
thus is more flexible than the parametric prior-based methods such as LogA
[27]. Additionally, LogN also imposes calibration of background class, which
is overlooked by previous prior-based methods.

Relation to Statistic-based Methods. Several recent methods [37, 43, 9]
propose to accommodate the dynamic distribution within detectors in terms of
statistics. Despite the favorable performance achieved, most of them are
formally complicated and overly parametric. By contrast, our LogN enforces
post-hoc BN on classified logits, calibrating the biased detections without
any training or tuning procedure. Moreover, existing practices [9] rely
heavily on a manual tuning hyper-parameter to adjust the weight of background
logit. In comparison, we analyze the processing of background logit and
provide an adaptive choice for $\beta$ selection, sidestepping the tuning on
the background logit calibration. The experiments demonstrate the
effectiveness of resilient background calibration.

###  3.3 Analysis

Why Post-hoc? Compared with the primitive BN, we remove the batch-wise
normalization during training. We conjecture that this is because the
normalization of the training phase increases the shift between the learned
posterior distribution and the training label distribution. Specifically, the
minimization of the objective function describing the consistency between
these two distributions drives the former to adapt to the latter. Therefore,
forcing the normalized posterior distribution to fit the long-tail bias will
eventually lead to more biased predictions. Two natural choices exist at this
point, we can either delay the normalization until the testing phase (i.e. our
post-hoc LogN), or inverse the normalization process in the training phase as
follows:

$\hat{x}=x\cdot\sqrt{\sigma^{2}+\varepsilon}+\mu-\beta.$ (3)

Here, the $\mu,\sigma$ are the persistent statistics accumulated over the
training set, rather than the temporary batch-wise statistics used in BN. This
online alternative renders the modified posterior distribution more skewed and
thereby easily adapted to the long-tail target distribution, hence keeping the
original posterior distribution more uniform. As indicated in Sec. 4.4, this
online inverse LogN also provides relatively balanced predictions.

Why with $\beta$? Considering the case of no foreground calibration, in a
$(C+1)$-d score vector, the score on the $i$-th category with background
calibration is:

$\begin{gathered}\hat{s}_{i}=\frac{e^{x_{i}}}{e^{x_{0}-\beta}+\sum_{j=1}^{C}{e^{x_{j}}}}=\frac{e^{x_{i}+\beta}}{e^{x_{0}}+\sum_{j=1}^{C}{e^{x_{j}+\beta}}},\end{gathered}$
(4)

where the 0th class is set as the background. Eq. 4 reveals that both the
foreground and background logits are involved in the calculation of scores and
they jointly affect the confidence ranking for the final evaluation [39, 23].
Meanwhile, a single scalar $\beta$ is sufficient to calibrate the impact of
the background, and this background calibration is equivalent to the
consistent calibration on all foreground categories. We further reformulate
the foreground score with logarithmic operation since it does not change the
monotonicity:

$\small\log\hat{s}_{i}=\log(\frac{1}{1+e^{-\beta}\cdot
B})+\log(\frac{e^{x_{i}}}{\sum_{j=1}^{C}{e^{x_{j}}}}),\\\ $ (5) $\small
B=\frac{e^{x_{0}}}{\sum_{j=1}^{C}{e^{x_{j}}}}=\frac{e^{x_{0}}}{\sum_{j=0}^{C}{e^{x_{j}}}}\cdot\frac{1}{1-\frac{e^{x_{0}}}{\sum_{j=0}^{C}{e^{x_{j}}}}},$
(6)

where $B$ is a monotonic function of the initial background score,
$\frac{e^{x_{0}}}{\sum_{j=0}^{C}{e^{x_{j}}}}$. In Eq. 5, when $\beta<0$, its
first item becomes more sensitive to the change of $B$ than the case of
$\beta\geq 0$, leading to a stronger attention on background.

$\beta$ determines the contribution of background to detection results, and is
dependent on the characteristics of datasets. Generally, a dataset with more
classes such as LVIS [12] contains fewer samples per class compared to a
dataset with fewer classes such as MS COCO [22]. Detectors trained on datasets
like LVIS [12] tend to have a more imprecise discrimination of foreground
class relative to the background. Therefore, we are inclined to allocate more
attention to background scores, making full use of the superior
discriminability of classifier on the background, thereby eventually rectify
the ranking of foreground proposals, and delivering better detection results.
In this case, a negative $\beta$ is appropriate. However, in practice, the
statistical mean of the background is often too large to reflect the function
of $\beta$ and hampers the calibration. Consequently, we set $\beta$ to the
minimum value of the average logits that is always negative to provide an
adaptive solution, and the ablation study in Sec. 4.3 validates its
effectiveness.

##  4 Experiments

###  4.1 Experimental Setup

Datasets and evaluation metric. We conduct experiments on the long-tail Large
Vocabulary Instance Segmentation (LVIS) [12] dataset to validate the
effectiveness of the proposed LogN. We use the latest version v1.0, which
provides precise bounding box and mask annotations for 1,203 categories. Its
training set with around 100k images and 1.3M instances is employed for
training and the val set with around 19.8k images and 244k instances is then
taken for evaluation. Particularly, all categories are divided into three
groups according to how many images they appear in training set: rare (1-10
images), common (11-100 images), and frequent ($>$100 images). Hence for
evaluation, apart from the commonly-used metric $\textbf{AP}^{b}$ (for object
detection) and AP (for instance segmentation), the fine-grained metrics
including $\textbf{AP}_{r}$ (for rare classes), $\textbf{AP}_{c}$ (for common
classes), and $\textbf{AP}_{f}$ (for frequent classes) are also reported.

Implementation details. Our LogN calibrates predictions during evaluation for
pretrained baselines, which are implemented with MMDetection [5] and identical
to [43] in training details for a fair comparison. Concretely, two-stage
detectors Mask R-CNN [13] and Cascade Mask R-CNN [4] with backbone ResNet-FPN
[14, 20] pretrained on ImageNet [6] are employed and we train them using the
2x training schedule [5], i.e. the initial learning rate is 0.02 and decreases
by 90% after the 16th and 22nd epochs, respectively. All models are optimized
using SGD with a momentum of 0.9 and a weight decay of 0.0001 and a batch size
of 16 on 8 GPUs for 24 epochs. Following the convention, scale jitter and
horizontal flipping are used in training and no test time augmentation is
used. As in [43, 9], we adopt normalized linear activation for both category
and mask prediction. In addition to the internal sampler that randomly samples
training images, the repetition factor sampler (RFS) [12] is also
experimentally evaluated as the external sampler. RFS oversamples categories
appearing in less than 0.1% of the total images and effectively enhances the
overall AP. Unless specific, all ablation studies and further analysis are
conducted on Mask R-CNN with ResNet-101-FPN with RFS.

Table 1: Comparison with state-of-the-art methods on LVIS v1 dataset [12]. All
methods are grouped according to their framework and training sampler, and we
additionally report whether models perform training and the number of training
epochs. Mask: Mask R-CNN [13]; Cascade: Cascade Mask R-CNN [4]; R50:
ResNet-50-FPN [14, 20]; R101: ResNet-101-FPN [14, 20]; Random: no external
sampler employed; RFS: Repeat Factor Sampler [12]; MFS: Memory-augmented
Feature Sampling [9]; +: using decouple training pipeline [17]; -: merely
post-hoc calibration.

Framework | Method | Sampler | Train | Epochs | $\textbf{AP}^{b}$ | AP | $\textbf{AP}_{r}$ | $\textbf{AP}_{c}$ | $\textbf{AP}_{f}$
---|---|---|---|---|---|---|---|---|---
Mask-R50 | CE | Random | ✓ | 24 | 19.9 | 19.0 | 1.3 | 16.7 | 29.3
EQL [38] | Random | ✓ | 24 | 22.5 | 21.6 | 3.8 | 21.7 | 29.2
EQL v2 [37] | Random | ✓ | 24 | 26.1 | 25.5 | 17.7 | 24.3 | 30.2
Seesaw Loss [43] | Random | ✓ | 24 | 25.6 | 25.4 | 15.9 | 24.7 | 30.4
LOCE [9] | Random | ✓ | 24+6 | 24.0 | 23.8 | 8.3 | 23.7 | 30.7
LogA [27, 30] | Random | ✗ | - | 26.5 | 26.2 | 20.6 | 25.9 | 29.1
LogN | Random | ✗ | - | 26.8 | 26.5 | 19.2 | 25.7 | 30.4
CE |  RFS [12] | ✓ | 24 | 24.7 | 23.7 | 13.5 | 22.8 | 29.3
Seesaw Loss [43] |  RFS [12] | ✓ | 24 | 27.6 | 26.8 | 19.8 | 26.3 | 30.5
LOCE [9] |  MFS [9] | ✓ | 24+6 | 27.4 | 26.6 | 18.5 | 26.2 | 30.7
LogA [27, 30] |  RFS [12] | ✗ | - | 27.4 | 26.8 | 22.0 | 27.0 | 28.7
LogN |  RFS [12] | ✗ | - | 28.1 | 27.5 | 21.8 | 27.1 | 30.4
Mask-R101 | CE | Random | ✓ | 24 | 21.7 | 20.6 | 0.8 | 19.3 | 30.7
EQL [38] | Random | ✓ | 24 | 24.2 | 22.9 | 3.7 | 23.6 | 30.7
BAGS [19] | Random | ✓ | 24 | 26.4 | 25.6 | 17.3 | 25.0 | 30.1
EQL v2 [38] | Random | ✓ | 24 | 27.9 | 27.2 | 20.6 | 25.9 | 31.4
Seesaw Loss [43] | Random | ✓ | 24 | 27.4 | 27.1 | 18.7 | 26.3 | 31.7
LogA [27, 30] | Random | ✗ | - | 27.9 | 27.5 | 21.3 | 27.3 | 30.4
LogN | Random | ✗ | - | 28.4 | 28.0 | 21.0 | 27.2 | 31.9
CE |  RFS [12] | ✓ | 24 | 26.6 | 25.5 | 16.6 | 24.5 | 30.6
EQL [38] |  RFS [12] | ✓ | 24 | 27.6 | 26.2 | 17.0 | 26.2 | 30.2
BAGS [19] |  RFS [12] | ✓ | 24 | 26.5 | 25.8 | 16.5 | 25.7 | 30.1
Seesaw Loss [43] |  RFS [12] | ✓ | 24 | 28.9 | 28.1 | 20.0 | 28.0 | 31.8
LOCE [9] |  MFS [9] | ✓ | 24+6 | 29.0 | 28.0 | 19.5 | 27.8 | 32.0
LogA [27, 30] |  RFS [12] | ✗ | - | 29.0 | 28.2 | 23.2 | 28.6 | 30.0
LogN |  RFS [12] | ✗ | - | 29.8 | 29.0 | 22.9 | 28.8 | 31.8
Cascade-R101 | CE | Random | ✓ | 24 | 25.5 | 22.6 | 2.4 | 22.0 | 32.2
EQL [38] | Random | ✓ | 24 | 27.2 | 24.5 | 4.1 | 25.8 | 32.0
BAGS [19] | Random | ✓ | 24 | 31.5 | 27.9 | 19.6 | 27.7 | 31.6
EQL v2 [38] | Random | ✓ | 24 | 32.3 | 28.8 | 22.3 | 27.8 | 32.8
Seesaw Loss [9] | Random | ✓ | 24 | 32.7 | 29.6 | 20.3 | 29.3 | 34.0
LogA [27, 30] | Random | ✗ | - | 32.7 | 29.9 | 22.8 | 30.5 | 32.3
LogN | Random | ✗ | - | 33.5 | 30.6 | 22.7 | 30.7 | 34.0
CE |  RFS [12] | ✓ | 24 | 30.3 | 27.0 | 16.6 | 26.7 | 32.0
EQL [38] |  RFS [12] | ✓ | 24 | 30.4 | 27.1 | 17.0 | 27.2 | 31.4
BAGS [19] |  RFS [12] | ✓ | 24 | 30.2 | 27.0 | 16.9 | 26.9 | 31.7
Seesaw Loss [9] |  RFS [12] | ✓ | 24 | 32.8 | 30.1 | 21.4 | 30.0 | 33.9
LogA [27, 30] |  RFS [12] | ✗ | - | 32.7 | 30.0 | 23.9 | 30.8 | 31.7
LogN |  RFS [12] | ✗ | - | 33.8 | 30.9 | 23.9 | 31.1 | 33.8

###  4.2 Comparison with State of the Arts

To display the effectiveness of LogN on long-tail object detection, we compare
LogN with the state-of-the-art methods on LVIS v1.0 dataset. We present the
results with and without external sampler to show the effectiveness of LogN at
multiple circumstances. As shown in Tab. 1, our LogN outperforms all state-of-
the-art methods with and without external sampler on various detection
frameworks. Specifically, with both Cascade Mask R-CNN and RFS, LogN
outperforms Seesaw Loss [43] by 1.0% and 0.8% on $\textbf{AP}^{b}$ and AP,
respectively. LogN also achieves higher $\textbf{AP}_{r}$ and
$\textbf{AP}_{c}$ than Seesaw Loss by 2.5% and 1.1% and is only 0.1% poorer
than it on $\textbf{AP}_{f}$, which demonstrates our method greatly reduces
the bias in long-tail learning. Notably, similar to LogA [27, 30], LogN is
executed solely in the testing phase, eliminating the requirement for a time-
consuming and laborious training process as in [38, 37, 9, 43]. More
importantly, our method further consistently outperforms LogA, especially with
the external sampler. In particular, with RFS, LogN outperforms LogA by 1.1%
$\textbf{AP}^{b}$ on Cascade-R101, 0.8% $\textbf{AP}^{b}$ on Mask-R101 and
0.7% $\textbf{AP}^{b}$ on Mask-R50, respectively. This indicates that our LogN
is more compatible with external samplers and is indeed more adaptive to the
alternation of label distribution.

###  4.3 Ablation Study

Table 2: Study on components of LogN. $\mu$: mean. $\sigma$: standard deviation. $\beta$: background calibration scalar. $\mu$ | $\sigma$ | $\beta$ | $\textbf{AP}^{b}$ | AP | $\textbf{AP}_{r}$ | $\textbf{AP}_{c}$ | $\textbf{AP}_{f}$
---|---|---|---|---|---|---|---
|  |  | 27.5 | 26.8 | 17.4 | 26.2 | 31.7
✓ |  |  | 28.7 | 28.2 | 21.5 | 27.9 | 31.5
| ✓ |  | 29.1 | 28.5 | 21.4 | 28.3 | 31.9
|  | ✓ | 29.0 | 28.0 | 19.3 | 27.4 | 32.4
✓ | ✓ | ✓ | 29.8 | 29.0 | 22.9 | 28.8 | 31.8

Study on components. We conduct ablation studies on the components in our
LogN, e.g. , mean $\mu$, standard deviation $\sigma$ and background
calibration scalar $\beta$. As shown in Tab. 2, $\mu$ pushes the performance
on $\textbf{AP}_{r}$ from 17.4% to 21.5%. This performance gain also
contributes to significant increases on both $\textbf{AP}^{b}$ and AP. In
comparison, $\sigma$ delivers greater improvements on both common and frequent
classes, and also leads to better final results. This indicates that both
statistics bring promising effects, yet surprisingly, compared to logit mean,
which better aligns with the training label distribution, dividing variance
corrects the long-tail bias more effectively and thoroughly. We further
validate the effectiveness of $\beta$, i.e. the fourth line. Its impressive
results, especially on $\textbf{AP}^{b}$ and $\textbf{AP}_{r}$, suggesting the
background calibration greatly alleviates the impact caused by foreground-
background class imbalance. Finally, LogN achieves the best results with all
three components included.

Study on selection of $\beta$. We carry out ablation studies on the selection
of $\beta$ in LogN. Based on our analysis, we find it is suitable to assign it
with a negative value. In Tab. 4, we investigate how its value influence the
performance quantitatively. We report the results with the original
statistical mean value of the background category, $\mu_{0}$ ($\approx
13.44$), and unsurprisingly, we find its performance is the worst. When using
$\mu^{+}_{\max}$ ($\approx 3.11$), it still performs bad under each metric.
However, when employing $\mu^{+}_{\mathrm{avg}}$ ($\approx-1.09$) and
$\mu^{+}_{\min}$ ($\approx-5.35$), the performance comes to a higher level.
Additionally, we try to take $\beta$ as a constant value, and manually select
two values (e.g. -4 and -12). Generally, the comparison in Tab. 4 suggests
that $\beta$ has a certain value range and, $\mu^{+}_{\min}$ is suitable for
it as an adaptive solution.

Study on object detection frameworks. We conduct experiments on various
detectors with diverse backbones. As shown in Tab. 1, LogN achieves the state-
of-the-art performance under multiple metrics. Furthermore, we observe
incremental gains by scaling up models.

Table 3: Study on the selection of $\beta$. $\mu_{0}$: mean of the background logit. $\mu^{+}_{\max}$: maximum of the foreground logits. $\mu^{+}_{\mathrm{avg}}$: average of the foreground logits. $\mu^{+}_{\min}$: minimum of the foreground logits. $\beta$ | $\textbf{AP}^{b}$ | AP | $\textbf{AP}_{r}$ | $\textbf{AP}_{c}$ | $\textbf{AP}_{f}$
---|---|---|---|---|---
$-4$ | 29.6 | 28.8 | 22.9 | 28.8 | 31.3
$-12$ | 29.3 | 28.3 | 20.8 | 28.0 | 31.9
$\mu_{0}$ | 12.1 | 12.5 | 14.1 | 14.5 | 9.6
$\mu^{+}_{\max}$ | 25.5 | 25.4 | 20.9 | 25.7 | 27.0
$\mu^{+}_{\mathrm{avg}}$ | 29.4 | 28.7 | 23.5 | 29.0 | 30.7
$\mu^{+}_{\min}$ | 29.8 | 29.0 | 22.9 | 28.8 | 31.8

Table 4: Comparison with different methods on ImageNet-LT dataset [24]. Method | Many | Medium | Few | Overall
---|---|---|---|---
Softmax | 65.9 | 37.5 | 7.7 | 44.4
Focal Loss [21] | 64.5 | 36.3 | 7.8 | 43.3
cRT [17] | 61.8 | 46.2 | 27.4 | 49.6
$\tau$-norm [17] | 59.1 | 46.9 | 30.7 | 49.4
LWS [17] | 60.2 | 47.2 | 30.3 | 49.9
EQL [38] | 61.7 | 42.5 | 13.8 | 46.0
B-Softmax [33] | 62.2 | 48.8 | 29.8 | 51.4
Seesaw Loss [43] | 67.1 | 45.2 | 21.4 | 50.4
LogA [27] | 63.4 | 47.9 | 31.5 | 51.6
LogN | 59.1 | 50.3 | 35.0 | 51.6

###  4.4 Further Analysis

Why not calibrate online? As analyzed in Sec. 3.3, there is also an online
equivalent form of our LogN. In particular, for each batch of training logits,
we first accumulate the global statistics as in BN [16], and then apply them
to scale and shift the logits as in Eq. 3, before calculating the objective,
e.g. cross-entropy loss. Without any processing of the logits during
inference, the per-group detection results of two instantiations
($\beta=\mu^{+}_{\min}$ and $\beta=0$) of the online LogN are shown in Fig. 3.
Interestingly, the former degrades the performance of baseline, suggesting a
significant difference of calibration on background between training and
testing phases; whereas the latter enhances baseline’s performance, validating
our analysis. However, there is still a small performance gap between it and
our post-hoc LogN, because of the imprecision of statistics in the early
training period, which is why we employ the post-hoc LogN as the final
solution.

Figure 3: Per-group results of online and post-hoc LogN and their baseline.

How LogN calibrate detections? To investigate the influence of LogN on the
detection results at a finer scale, we conduct an error analysis of the
detections before and after the execution of LogN with the recently proposed
TIDE [3]. As depicted in Fig. 4, LogN greatly reduces the classification error
in long-tail object detection, with a large decrease on false negative rate.
However, among these improvements, LogN leads to slightly higher false
positive rate. We speculate it is because the negative samples also have low
confidence as well as the candidates of rare classes, however, the operation
of subtracting statistical mean in LogN decreases the confidence in each
candidate, mixing up the candidates originally from various score bands, and
causing an increase on false positive rate.

(a) Baseline

(b) Our LogN

Figure 4:  Error analysis for LogN and its baseline. According to TIDE [3],
the figure shows the six types of errors, the pie chart shows the relative
contribution of each error, while the bar plots show their absolute
contribution. Cls: classification error; Loc: localization error; Both: both
cls and loc error; Dupe: duplicate detection error; Bkg: background error;
Miss: missed ground truth error. FP: false positive; FN: false negative.

###  4.5 Long-tail Image Classification

We further conduct experiments on ImageNet-LT [24] to demonstrate the
competitiveness and versatility of LogN. ImageNet-LT is a long-tail version of
the large-scale object classification dataset ImageNet [6], and contains
115.8k images of 1,000 categories, with class cardinality ranging from 5 to
1,280. Following [24], we also divide the categories of ImageNet-LT into three
groups according to the number of samples in each class: Few (1-19 images),
Medium (20-100 images), and Many ($>$100 images). Thus for evaluation, we
report top-1 accuracy of the Overall evaluation results, as well as results on
each group. The experimental implementation and mainly reported results follow
[43].

It can be observed in Tab. 4 that LogN yields 51.6% top-1 overall accuracy,
which is competitive among diverse previous methods. Notably, the performance
of LogN is roughly on par with some recent prior-based methods [33, 27] that
have been proven to be theoretically optimal. This fully demonstrates that in
a simple image classification task with only random sampling, LogN is still
able to precisely capture and effectively de-bias the long-tail training label
distribution that is consistent with the prior label distribution. Moreover,
our method also outperforms EQL [38] and Seesaw Loss [43] devised for object
detection and instance segmentation. This is in line with Sec. 4.2 and well
showcases the superior generalization of LogN. It is noteworthy that, contrary
to the results in Sec. 4.2, LogN offers better performance on tail classes and
worse performance on head classes compared to LogA [27]. We speculate that
this stems from the behavioral difference of variance, which is indeed
introduced by the differences in datasets and tasks. For classification in
ImageNet [6], variance actually balances the magnitudes of logits to some
degree, which equals to give classifiers more balanced decision boundaries
just like $\tau$-norm [17], as verified by their similar many-shot accuracy.
For more details, please refer to supplementary material.

##  5 Conclusion

In this paper, we propose Logit Normalization (LogN), a simple technique to
self-calibrate the classified logits of object detectors like batch
normalization. Extensive experiments on long-tail object detection and
classification demonstrate its impressive effectiveness and versatility.
Overall, LogN is based on statistics and thus sufficient to dynamically adapt
to various distributions and networks. It is also independent of training and
tuning and can be directly applied without any bells and whistles.

## Appendix

##  Appendix 0.A More Implementation Details

On Fig. 1 and Fig. 2, all data are captured from the training set of LVIS v1
[12] via Mask R-CNN [13] with ResNet-101-FPN [14, 20]. And all results on Fig.
3 and Fig. 4 are obtained from the same detector evaluated on the val set.

For experiments on long-tail object detection, we adopt multi-scale with
horizontally flip augmentation during training, following the practices in
mmdetection [5] and prior efforts [43, 38, 37, 43]. In specific, during
training, we randomly resize the shorter edge of the image within 640, 672,
704, 736, 768, 800 pixels and maintain the longer edge smaller than 1333
pixels without changing the aspect ratio. During testing, we then employ
single-scale testing with image size of 1333 × 800 pixels.

For experiments on long-tail classification, all implementation details follow
[43]. Specifically, we employ ResNeXt-50 [46] as the backbone network and SGD
with momentum of 0.9 and weight decay of $5\times 10^{-4}$ as the optimizer to
end-to-end train the model for 90 epochs. In line with Seesaw Loss [43], we
also adopt the normalized linear activation for Logit Adjustment (LogA) [27]
and our LogN.

##  Appendix 0.B More Analysis

###  0.B.1 How $\mu,\sigma$ and $\beta$ function?

To further clarify the functioning mechanism of each component within LogN, we
also conduct error analysis of detections with and without each component,
just like our ablation study. The results are shown in Fig. 0.B.1. Comparing
the results from Fig. 0.B.1a and Fig. 0.B.1b, the logit mean leads to a great
decrease on classification error with a slight increase on the false positive,
illustrating it mostly improves the accuracy of model prediction. Comparing
the results from Fig. 0.B.1a and Fig. 0.B.1c, the statistical variance also
reduces the classification error. In quantity, it reduces it from 24.17 dAP to
17.10 dAP. As for the last comparison, the presence of $\beta$ consistently
reduces all kinds of errors, revealing the significance of balancing the
foreground-background imbalance. Quantitatively, it reduces the false positive
rate from 10.52 dAP to 10.30 dAP, the false negative rate from 28.10 dAP to
26.21 dAP, and the classification error from 24.17 dAP to 22.05 dAP.

(a) Baseline

(b) Baseline+$\mu$

(c) Baseline+$\sigma$

(d) Baseline+$\beta$

(e) Baseline+$\mu,\sigma,\beta$

Figure 0.B.1:  Error analysis for components of LogN. According to TIDE [3],
the figure shows the six types of errors, the pie chart shows the relative
contribution of each error, while the bar plots show their absolute
contribution. Cls: classification error; Loc: localization error; Both: both
cls and loc error; Dupe: duplicate detection error; Bkg: background error;
Miss: missed ground truth error. FP: false positive; FN: false negative.

###  0.B.2 How $\beta$ influences background?

In this section, we illustrate how $\beta$ amplify the effect of the initial
background score.

Assuming $y(s_{b},s_{f},\beta)=\log(\frac{1}{1+e^{-\beta}\cdot
B(s_{b})})+\log(s_{f})$, where $B(s_{b})=\frac{s_{b}}{1-s_{b}}$ and
$s_{b},s_{f}\in(0,1)$. Then, we calculate the changing rate of $y$, that is,
$k(\Delta s_{b},s_{b},s_{f},\beta)$, on the component $s_{b}$:

$\displaystyle k(\Delta s_{b},s_{b},s_{f},\beta)=\frac{y(s_{b}+\Delta
s_{b},s_{f},\beta)-y(s_{b},s_{f},\beta)}{\Delta s_{b}}$ (7)
$\displaystyle=\frac{-1}{\Delta s_{b}}\log(1+e^{-\beta}\frac{B(s_{b}+\Delta
s_{b})-B(s_{b})}{1+e^{-\beta}B(s_{b})})$ $\displaystyle=\frac{-1}{\Delta
s_{b}}\log(1+e^{-\beta}\frac{\frac{\Delta s_{b}}{(1-s_{b})(1-s_{b}-\Delta
s_{b})}}{1+e^{-\beta}B(s_{b})}).$

Therefore, according to Eq. 7, when $\beta<0$, the inequality $|k(\Delta
s_{b},s_{b},s_{f},\beta)|>|k(\Delta s_{b},s_{b},s_{f},0)|$ holds. Meanwhile,
$\beta$ does not influence the change on $s_{f}$, and $y(s_{b},s_{f},\beta)$
is continuous and differentiable.

As for the task of classification in object detection, $s_{b}$ from Eq. 7 is
the probability of background prediction, that is, $\Pr(c=0|x;\theta)$.
$s_{f}$ indicates the conditional probability of one class over foreground
categories, that is, $\Pr(c=i|c>0,x;\theta)$. Thus, in the confidence ranking,
comparing two candidates predicting the same category, the background
confidence are crucial when $\beta<0$. In a nutshell, $\beta<0$ amplifies the
effect of the initial background score on the classification of object
detection.

###  0.B.3 Why calculate statistics on all samples like BN?

As emphasized in main body, the statistics for each class are calculated over
all samples, which is consistent with that in BN [16]. However, another
intuitive way to accumulate statistics for each category is to use only
positive samples. Specifically, for one proposal, only its logit on the
correctly predicted class is accumulated while the logits on the other classes
are masked. Problematically, this operation neglects the margins between
classes and leads to a more unstable distribution, shown in Fig. 0.B.2. These
statistics cannot represent the original long-tail distribution, and their
trends are more uniform. Additionally, they have much more fluctuations on the
tail classes, causing unstable estimations of statistics on these categories.
In general, with such statistics for normalization, the logits on head classes
cannot be correctly inhibited, leading to a worse performance for calibration.

In contrast, our strategy, i.e. using all logits of every proposal for
calculating the statistics, properly preserves the margins between classes.
Therefore, statistics on tail classes benefited from the abundant logits of
proposals become stable and robust.

Figure 0.B.2: Statistics accumulated from only positive samples.

###  0.B.4 Why results in detection and classification are inconsistently
improved?

Throughout our experimental results, we find that compared to LogA [27], LogN
is worse on tail classes and better on head classes in object detection, yet
is better on tail classes and worse on head classes in classification.

We attribute this to the behavioral differences in variance across tasks and
datasets. As shown in Tab. 0.B.1, for image classification on ImageNet-LT
[24], it is evident that, both statistics deliver significant performance
gains, i.e. +5.6% and +6.5% relatively to baseline (training with Softmax).
But among the gains of performance, $\mu$ enhances the performance of all
classes simultaneously, while $\sigma$ is more of a trade-off, reducing the
accuracy on many-shot classes yet improving the accuracy on both medium- and
few-shot classes. Thus, we speculate that the variance acts like re-balancing
the performance on each group of classes, just like $\tau$-norm [17], which
specializes in providing more balanced classifier weight magnitudes.

Object detection on LVIS [12] is a much more challenging task compared to
classification. The complex coupling of multiple modules and samplers as well
as the task of distinguishing the background proposal make it difficult to use
statistics to dynamically estimate the training label distribution. The
problem is that, although the two statistics provide efficient and comparable
boosts, as shown in Tab. 2, the variance fails in effectively equalizing the
performance on the different groups of classes. Specifically, the magnitude of
variance is too small to inhibit the confidence on head classes and increase
that on tail classes. Therefore, we try to square the standard deviation of
LogN. On Mask R-CNN 101 with RFS [12], we find that after this operation, the
performance of LogN on rare classes can achieve 23.1% which is very close to
that of LogA, i.e. 23.2%. However, it also leads to a performance drop on the
total mAP, i.e. -0.5%. Thus, we do not include such additional operations in
our method.

Table 0.B.1: Study on components of LogN on ImageNet-LT [24]. $\mu$ | $\sigma$ | Many | Medium | Few | Overall
---|---|---|---|---|---
✓ |  | 65.8 | 45.0 | 22.8 | 50.0
| ✓ | 63.0 | 48.4 | 25.9 | 50.9
✓ | ✓ | 59.1 | 50.3 | 35.0 | 51.6

## References

  * [1] Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization. arXiv preprint arXiv:1607.06450 (2016)
  * [2] Balles, L., Hennig, P.: Dissecting adam: The sign, magnitude and variance of stochastic gradients. In: ICML. Proceedings of Machine Learning Research, vol. 80, pp. 413–422. PMLR (2018)
  * [3] Bolya, D., Foley, S., Hays, J., Hoffman, J.: TIDE: A general toolbox for identifying object detection errors. In: ECCV (3). Lecture Notes in Computer Science, vol. 12348, pp. 558–573. Springer (2020)
  * [4] Cai, Z., Vasconcelos, N.: Cascade R-CNN: high quality object detection and instance segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 43(5), 1483–1498 (2021)
  * [5] Chen, K., Wang, J., Pang, J., Cao, Y., Xiong, Y., Li, X., Sun, S., Feng, W., Liu, Z., Xu, J., Zhang, Z., Cheng, D., Zhu, C., Cheng, T., Zhao, Q., Li, B., Lu, X., Zhu, R., Wu, Y., Dai, J., Wang, J., Shi, J., Ouyang, W., Loy, C.C., Lin, D.: MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155 (2019)
  * [6] Deng, J., Dong, W., Socher, R., Li, L., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: CVPR. pp. 248–255. IEEE Computer Society (2009)
  * [7] Dimitriou, N., Arandjelovic, O.: A new look at ghost normalization. arXiv preprint arXiv:2007.08554 (2020)
  * [8] Dumoulin, V., Shlens, J., Kudlur, M.: A learned representation for artistic style. arXiv preprint arXiv:1610.07629 (2016)
  * [9] Feng, C., Zhong, Y., Huang, W.: Exploring classification equilibrium in long-tailed object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3417–3426 (2021)
  * [10] Frosst, N., Papernot, N., Hinton, G.E.: Analyzing and improving representations with the soft nearest neighbor loss. In: ICML. Proceedings of Machine Learning Research, vol. 97, pp. 2012–2020. PMLR (2019)
  * [11] Gao, Z., Wang, L., Wu, G.: Mutual supervision for dense object detection. In: ICCV. pp. 3641–3650. Computer Vision Foundation / IEEE (2021)
  * [12] Gupta, A., Dollár, P., Girshick, R.B.: LVIS: A dataset for large vocabulary instance segmentation. In: CVPR. pp. 5356–5364. Computer Vision Foundation / IEEE (2019)
  * [13] He, K., Gkioxari, G., Dollár, P., Girshick, R.B.: Mask R-CNN. In: ICCV. pp. 2980–2988. IEEE Computer Society (2017)
  * [14] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR. pp. 770–778. IEEE Computer Society (2016)
  * [15] Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive instance normalization. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 1501–1510 (2017)
  * [16] Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: ICML. pp. 448–456. PMLR (2015)
  * [17] Kang, B., Xie, S., Rohrbach, M., Yan, Z., Gordo, A., Feng, J., Kalantidis, Y.: Decoupling representation and classifier for long-tailed recognition. In: ICLR. OpenReview.net (2020)
  * [18] Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J.R.R., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Duerig, T., Ferrari, V.: The open images dataset V4: unified image classification, object detection, and visual relationship detection at scale. CoRR abs/1811.00982 (2018)
  * [19] Li, Y., Wang, T., Kang, B., Tang, S., Wang, C., Li, J., Feng, J.: Overcoming classifier imbalance for long-tail object detection with balanced group softmax. In: CVPR. pp. 10988–10997. Computer Vision Foundation / IEEE (2020)
  * [20] Lin, T., Dollár, P., Girshick, R.B., He, K., Hariharan, B., Belongie, S.J.: Feature pyramid networks for object detection. In: CVPR. pp. 936–944. IEEE Computer Society (2017)
  * [21] Lin, T., Goyal, P., Girshick, R.B., He, K., Dollár, P.: Focal loss for dense object detection. In: ICCV. pp. 2999–3007. IEEE Computer Society (2017)
  * [22] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: European conference on computer vision. pp. 740–755. Springer (2014)
  * [23] Liu, J., Li, D., Zheng, R., Tian, L., Shan, Y.: Rankdetnet: Delving into ranking constraints for object detection. In: CVPR. pp. 264–273. Computer Vision Foundation / IEEE (2021)
  * [24] Liu, Z., Miao, Z., Zhan, X., Wang, J., Gong, B., Yu, S.X.: Large-scale long-tailed recognition in an open world. In: CVPR. pp. 2537–2546. Computer Vision Foundation / IEEE (2019)
  * [25] Luo, P., Zhang, R., Ren, J., Peng, Z., Li, J.: Switchable normalization for learning-to-normalize deep representation. IEEE transactions on pattern analysis and machine intelligence 43(2), 712–728 (2019)
  * [26] Luo, P., Zhanglin, P., Wenqi, S., Ruimao, Z., Jiamin, R., Lingyun, W.: Differentiable dynamic normalization for learning deep representation. In: International Conference on Machine Learning. pp. 4203–4211. PMLR (2019)
  * [27] Menon, A.K., Jayasumana, S., Rawat, A.S., Jain, H., Veit, A., Kumar, S.: Long-tail learning via logit adjustment. In: ICLR. OpenReview.net (2021)
  * [28] Miyato, T., Kataoka, T., Koyama, M., Yoshida, Y.: Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957 (2018)
  * [29] Newman, M.E.: Power laws, pareto distributions and zipf’s law. Contemporary physics 46(5), 323–351 (2005)
  * [30] Pan, T., Zhang, C., Li, Y., Hu, H., Xuan, D., Changpinyo, S., Gong, B., Chao, W.: On model calibration for long-tailed object detection and instance segmentation. CoRR abs/2107.02170 (2021)
  * [31] Peng, C., Xiao, T., Li, Z., Jiang, Y., Zhang, X., Jia, K., Yu, G., Sun, J.: Megdet: A large mini-batch object detector. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 6181–6189 (2018)
  * [32] Qiao, S., Wang, H., Liu, C., Shen, W., Yuille, A.: Weight standardization. CoRR (2019)
  * [33] Ren, J., Yu, C., Sheng, S., Ma, X., Zhao, H., Yi, S., Li, H.: Balanced meta-softmax for long-tailed visual recognition. In: NeurIPS (2020)
  * [34] Salimans, T., Kingma, D.P.: Weight normalization: A simple reparameterization to accelerate training of deep neural networks. Advances in neural information processing systems 29, 901–909 (2016)
  * [35] Shao, W., Meng, T., Li, J., Zhang, R., Li, Y., Wang, X., Luo, P.: Ssn: Learning sparse switchable normalization via sparsestmax. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 443–451 (2019)
  * [36] Shokri, R., Shmatikov, V.: Privacy-preserving deep learning. In: Allerton. pp. 909–910. IEEE (2015)
  * [37] Tan, J., Lu, X., Zhang, G., Yin, C., Li, Q.: Equalization loss v2: A new gradient balance approach for long-tailed object detection. In: CVPR. pp. 1685–1694. Computer Vision Foundation / IEEE (2021)
  * [38] Tan, J., Wang, C., Li, B., Li, Q., Ouyang, W., Yin, C., Yan, J.: Equalization loss for long-tailed object recognition. In: CVPR. pp. 11659–11668. Computer Vision Foundation / IEEE (2020)
  * [39] Tan, Z., Nie, X., Qian, Q., Li, N., Li, H.: Learning to rank proposals for object detection. In: ICCV. pp. 8272–8280. IEEE (2019)
  * [40] Tieleman, T., Hinton, G.: Rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning, Lecture 6.5 (2012)
  * [41] Ulyanov, D., Vedaldi, A., Lempitsky, V.: Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022 (2016)
  * [42] Wang, H., Wang, Y., Zhou, Z., Ji, X., Gong, D., Zhou, J., Li, Z., Liu, W.: Cosface: Large margin cosine loss for deep face recognition. In: CVPR. pp. 5265–5274. Computer Vision Foundation / IEEE Computer Society (2018)
  * [43] Wang, J., Zhang, W., Zang, Y., Cao, Y., Pang, J., Gong, T., Chen, K., Liu, Z., Loy, C.C., Lin, D.: Seesaw loss for long-tailed instance segmentation. In: CVPR. pp. 9695–9704. Computer Vision Foundation / IEEE (2021)
  * [44] Wu, F., Fan, A., Baevski, A., Dauphin, Y.N., Auli, M.: Pay less attention with lightweight and dynamic convolutions. arXiv preprint arXiv:1901.10430 (2019)
  * [45] Wu, Y., He, K.: Group normalization. In: Proceedings of the European conference on computer vision (ECCV). pp. 3–19 (2018)
  * [46] Xie, S., Girshick, R.B., Dollár, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. In: CVPR. pp. 5987–5995. IEEE Computer Society (2017)
  * [47] Yao, Z., Cao, Y., Zheng, S., Huang, G., Lin, S.: Cross-iteration batch normalization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12331–12340 (2021)
  * [48] Zhang, S., Li, Z., Yan, S., He, X., Sun, J.: Distribution alignment: A unified framework for long-tail visual recognition. In: CVPR. pp. 2361–2370. Computer Vision Foundation / IEEE (2021)
  * [49] Zhang, Y., Kang, B., Hooi, B., Yan, S., Feng, J.: Deep long-tailed learning: A survey. CoRR abs/2110.04596 (2021)
  * [50] Zhou, Z., Liu, X.: Training cost-sensitive neural networks with methods addressing the class imbalance problem. IEEE Trans. Knowl. Data Eng. 18(1), 63–77 (2006)
  * [51] Zhu, B., Wang, J., Jiang, Z., Zong, F., Liu, S., Li, Z., Sun, J.: Autoassign: Differentiable label assignment for dense object detection. CoRR abs/2007.03496 (2020)