capbtabboxtable[][] [table]capposition=top

# A Ranking-based, Balanced Loss Function Unifying Classification and
Localisation in Object Detection

Kemal Oksuz, Baris Can Cam, Emre Akbas∗, Sinan Kalkan
Department of Computer Engineering
Middle East Technical University
Ankara, Turkey
{kemal.oksuz, can.cam, eakbas<EMAIL_ADDRESS>Equal contribution for
senior authorship.

###### Abstract

We propose average Localization-Recall-Precision (aLRP), a unified, bounded,
balanced and ranking-based loss function for both classification and
localisation tasks in object detection. aLRP extends the Localization-Recall-
Precision (LRP) performance metric (Oksuz et al., 2018) inspired from how
Average Precision (AP) Loss extends precision to a ranking-based loss function
for classification (Chen et al., 2020). aLRP has the following distinct
advantages: (i) aLRP is the first ranking-based loss function for both
classification and localisation tasks. (ii) Thanks to using ranking for both
tasks, aLRP naturally enforces high-quality localisation for high-precision
classification. (iii) aLRP provides provable balance between positives and
negatives. (iv) Compared to on average $\sim$6 hyperparameters in the loss
functions of state-of-the-art detectors, aLRP has only one hyperparameter,
which we did not tune in practice. On the COCO dataset, aLRP improves its
ranking-based predecessor, AP Loss, more than $4$ AP points and outperforms
all one-stage detectors. The code is available at:
https://github.com/kemaloksuz/aLRPLoss.

##  1 Introduction

Object detection requires jointly optimizing a classification objective
($\mathcal{L}_{c}$) and a localisation objective ($\mathcal{L}_{r}$) combined
conventionally with a balancing hyperparameter ($w_{r}$) as follows:

$\displaystyle\mathcal{L}=\mathcal{L}_{c}+w_{r}\mathcal{L}_{r}.$ (1)

Optimizing $\mathcal{L}$ in this manner has three critical drawbacks: (D1) It
does not correlate the two tasks, and hence, does not guarantee high-quality
localization for high-precision examples (see Fig. 1). (D2) It requires a
careful tuning of $w_{r}$ [8, 25, 31], which is prohibitive since a single
training may last on the order of days and ends up with a sub-optimal constant
$w_{r}$ [4, 12]. (D3) It is adversely impeded by the positive-negative
imbalance in $\mathcal{L}_{c}$ and inlier-outlier imbalance in
$\mathcal{L}_{r}$, thus it requires sampling strategies [13, 14] or
specialized loss functions [10, 21], introducing more hyperparameters (Table
1).

A recent solution for D3 is to directly maximize Average Precision (AP) with a
loss function called AP Loss [7]. AP Loss is a ranking-based loss function to
optimize the ranking of the classification outputs and provides balanced
training between positives and negatives.

In this paper, we extend AP Loss to address the three drawbacks (D1-D3) in
one, unified loss function called average Localisation Recall Precision (aLRP)
Loss. In analogy with the link between precision and AP Loss, we formulate
aLRP Loss as the average of LRP values [19] of the positive examples on the
Recall-Precision (RP) curve. aLRP has the following benefits: (i) It exploits
ranking for both classification and localisation, enforcing high-precision
detections to have high-quality localisation (Fig. 1). (ii) aLRP has a single
hyperparameter (which we did not need to tune) as opposed to $\sim$6 in state-
of-the-art loss functions (Table 1). (iii) The network is trained by a single
loss function that provides provable balance between positives and negatives.

Our contributions are:

  1. 1.

We develop a generalized framework to optimize non-differentiable ranking-
based functions by extending the error-driven optimization of AP Loss.

  2. 2.

We prove that ranking-based loss functions conforming to this generalized form
provide a natural balance between positive and negative samples.

  3. 3.

We introduce aLRP Loss (and its gradients) as a special case of this
generalized formulation. Replacing AP and SmoothL1 losses by aLRP Loss
improves the performance by up to $4.3$AP, and outperforms all existing one-
stage detectors with significant margin.

Figure 1: aLRP Loss enforces high-precision detections to have high-IoUs, while others do not. (a) Classification and three possible localisation outputs for $10$ anchors and the rankings of the positive anchors with respect to (wrt) the scores (for $C$) and IoUs (for $R_{1}$, $R_{2}$ and $R_{3}$). Since the regressor is only trained by positive anchors, “–” is assigned for negative anchors. (b,c) Performance and loss assignment comparison of $R_{1}$, $R_{2}$ and $R_{3}$ when combined with $C$. When correlation between the rankings of classifier and regressor outputs decreases, performance degrades up to $17$ AP (b). While any combination of $\mathcal{L}_{c}$ and $\mathcal{L}_{r}$ cannot distinguish them, aLRP Loss penalizes the outputs accordingly (c). The details of the calculations are presented in Appendix A.  Table 1: State-of-the-art loss functions have several hyperparameters ($6.4$ on avg.). aLRP Loss has only one for step-function approximation (Sec. 2.1). See Appendix B for descriptions of the required hyperparameters. FL: Focal Loss, CE: Cross Entropy, SL1: Smooth L1, H: Hinge Loss.  Method | $\mathcal{L}$ | Number of hyperparameters
---|---|---
AP Loss [7] | AP Loss+$\alpha$ SL1 | 3
Focal Loss [14] | FL+ $\alpha$ SL1 | 4
FCOS [27] | FL+$\alpha$ IoU+$\beta$ CE | 4
DR Loss [23] | DR Loss+$\alpha$ SL1 | 5
FreeAnchor [31] |  $\alpha\log(\max(e^{\text{CE}}\times e^{\beta\text{SL1}}))$+$\gamma$ FL | 8
Faster R-CNN [24] | CE+$\alpha$ SL1+$\beta$CE+$\gamma$ SL1 | 9
Center Net [8] | FL+FL+$\alpha$ L2+$\beta$ H+$\gamma$ (SL1+SL1) | 10
Ours | aLRP Loss | 1

###  1.1 Related Work

Balancing $\mathcal{L}_{c}$ and $\mathcal{L}_{r}$ in Eq. (1), an open problem
in object detection (OD) [20], bears important challenges: Disposing $w_{r}$,
and correlating $\mathcal{L}_{c}$ and $\mathcal{L}_{r}$. Classification-aware
regression loss [3] links the branches by weighing $\mathcal{L}_{r}$ of an
anchor using its classification score. Following Kendall et al. [12], LapNet
[4] tackled the challenge by making $w_{r}$ a learnable parameter based on
homoscedastic uncertainty of the tasks. Other approaches [11, 28] combine the
outputs of two branches during non-maximum suppression (NMS) at inference.
Unlike these methods, aLRP Loss considers the ranking wrt scores for both
branches and addresses the imbalance problem naturally.

Ranking-based objectives in OD: An inspiring solution for balancing classes is
to optimize a ranking-based objective. However, such objectives are discrete
wrt the scores, rendering their direct incorporation challenging. A solution
is to use black-box solvers for an interpolated AP loss surface [22], which,
however, provided only little gain in performance. AP Loss [7] takes a
different approach by using an error-driven update mechanism to calculate
gradients (see Sec. 2). An alternative, DR Loss [23], employs Hinge Loss to
enforce a margin between the scores of the positives and negatives. Despite
promising results, these methods are limited to classification and leave
localisation as it is. In contrast, we propose a single, balanced, ranking-
based loss to train both branches.

##  2 Background

###  2.1 AP Loss and Error-Driven Optimization

AP Loss [7] directly optimizes the following loss for AP with intersection-
over-union (IoU) thresholded at 0.50:

$\displaystyle\mathcal{L}^{\mathrm{AP}}=1-\mathrm{\mathrm{AP}}_{50}=1-\frac{1}{|\mathcal{P}|}\sum\limits_{i\in\mathcal{P}}\mathrm{precision}(i)=1-\frac{1}{|\mathcal{P}|}\sum\limits_{i\in\mathcal{P}}\frac{\mathrm{rank}^{+}(i)}{\mathrm{rank}(i)},$
(2)

where $\mathcal{P}$ is the set of positives; $\mathrm{rank}^{+}(i)$ and
$\mathrm{rank}(i)$ are respectively the ranking positions of the $i$th sample
among positives and all samples. $\mathrm{rank}(i)$ can be easily defined
using a step function $H(\cdot)$ applied on the difference between the score
of $i$ ($s_{i}$) and the score of each other sample:

$\displaystyle\mathrm{rank}(i)=1+\sum\limits_{j\in\mathcal{P},j\neq
i}H(x_{ij})+\sum\limits_{j\in\mathcal{N}}H(x_{ij}),$ (3)

where $x_{ij}=-(s_{i}-s_{j})$ is positive if $s_{i}<s_{j}$; $\mathcal{N}$ is
the set of negatives; and $H(x)=1$ if $x\geq 0$ and $H(x)=0$ otherwise. In
practice, $H(\cdot)$ is replaced by $x/2\delta+0.5$ in the interval
$[-\delta,\delta]$ (in aLRP, we use $\delta=1$ as set by AP Loss [7]
empirically; this is the only hyperparameter of aLRP – Table 1).
$\mathrm{rank}^{+}(i)$ can be defined similarly over $j\in\mathcal{P}$. With
this notation, $\mathcal{L}^{\mathrm{AP}}$ can be rewritten as follows:

$\displaystyle\mathcal{L}^{\mathrm{AP}}=\frac{1}{|\mathcal{P}|}\sum\limits_{i\in\mathcal{P}}\sum\limits_{j\in\mathcal{N}}\frac{H(x_{ij})}{\mathrm{rank}(i)}=\frac{1}{|\mathcal{P}|}\sum\limits_{i\in\mathcal{P}}\sum\limits_{j\in\mathcal{N}}L_{ij}^{\mathrm{AP}},$
(4)

where $L_{ij}^{\mathrm{AP}}$ is called a primary term which is zero if
$i\notin\mathcal{P}$ or $j\notin\mathcal{N}$ 111By setting
$L_{ij}^{\mathrm{AP}}=0$ when $i\notin\mathcal{P}$ or $j\notin\mathcal{N}$, we
do not require the $y_{ij}$ term used by Chen et al. [7]..

Note that this system is composed of two parts: (i) The differentiable part up
to $x_{ij}$, and (ii) the non-differentiable part that follows $x_{ij}$. Chen
et al. proposed that an error-driven update of $x_{ij}$ (inspired from
perceptron learning [26]) can be combined with derivatives of the
differentiable part. Consider the update in $x_{ij}$ that minimizes
$L^{\mathrm{AP}}_{ij}$ (and hence $\mathcal{L}^{\mathrm{AP}}$): $\Delta
x_{ij}={L^{{\mathrm{AP}}*}_{ij}}-L^{\mathrm{AP}}_{ij}=0-L^{\mathrm{AP}}_{ij}=-L^{\mathrm{AP}}_{ij}$,
with the target, $L^{{\mathrm{AP}}*}_{ij}$, being zero for perfect ranking.
Chen et al. showed that the gradient of $\mathcal{L}^{\mathrm{AP}}$ wrt
$x_{ij}$ can be taken as $-\Delta x_{ij}$ and the gradient on the scores can
be calculated as follows:

$\displaystyle\frac{\partial\mathcal{L}^{\mathrm{AP}}}{\partial
s_{i}}=\sum\limits_{j,k}\frac{\partial\mathcal{L}^{\mathrm{AP}}}{\partial
x_{jk}}\frac{\partial x_{jk}}{\partial s_{i}}=-\sum\limits_{j,k}\Delta
x_{jk}\frac{\partial x_{jk}}{\partial s_{i}}=\sum\limits_{j}\Delta
x_{ij}-\sum\limits_{j}\Delta x_{ji}.$ (5)

###  2.2 Localization-Recall-Precision (LRP) Performance Metric

LRP [19] is a metric that quantifies classification and localisation
performances jointly. Given a detection set thresholded at a score ($s$) and
their matchings with the ground truths, LRP aims to assign an error value
within $[0,1]$ by considering localization, recall and precision:

$\displaystyle\mathrm{LRP}(s)=\frac{1}{N_{FP}+N_{FN}+{N_{TP}}}\left(N_{FP}+N_{FN}+\sum\limits_{k\in{TP}}\mathcal{E}_{loc}(k)\right),$
(6)

where $N_{FP},N_{FN}$ and $N_{TP}$ are the number of false positives (FP),
false negatives (FN) and true positives (TP); A detection is a TP if
$\mathrm{IoU}(k)\geq\tau$ where $\tau=0.50$ is the conventional TP labeling
threshold, and a TP has a localisation error of
$\mathcal{E}_{loc}(k)=(1-\mathrm{IoU}(k))/(1-\tau)$. The detection performance
is, then, $\min\limits_{s}(\mathrm{LRP}(s))$ on the precision-recall (PR)
curve, coined as the optimal LRP (oLRP).

##  3 A Generalisation of Error-Driven Optimization for Ranking-Based Losses

Generalizing the error-driven optimization technique of AP Loss [7] to other
ranking-based loss functions is not trivial. In particular, identifying the
primary terms is a challenge especially when the loss has components that
involve only positive examples, such as the localization error in LRP loss.

Given a ranking-based loss function,
$\mathcal{L}=\frac{1}{Z}\sum_{i\in\mathcal{P}}\ell(i)$, defined as a sum over
individual losses, $\ell(i)$, at positive examples (e.g., Eq. (2)), with $Z$
as a problem specific normalization constant, our goal is to express
$\mathcal{L}$ as a sum of primary terms in a more general form than Eq. (4):

######  Definition 1.

The primary term $L_{ij}$ concerning examples $i\in\mathcal{P}$ and
$j\in\mathcal{N}$ is the loss originating from $i$ and distributed over $j$
via a probability mass function $p(j|i)$. Formally,

$\displaystyle
L_{ij}=\begin{cases}\ell(i)p(j|i),&\mathrm{for}\;i\in\mathcal{P},j\in\mathcal{N}\\\
0,&\mathrm{otherwise}.\end{cases}$ (7)

Then, as desired, we can express
$\mathcal{L}=\frac{1}{Z}\sum_{i\in\mathcal{P}}\ell(i)$ in terms of $L_{ij}$:

######  Theorem 1.

$\mathcal{L}=\frac{1}{Z}\sum\limits_{i\in\mathcal{P}}\ell(i)=\frac{1}{Z}\sum\limits_{i\in\mathcal{P}}\sum\limits_{j\in\mathcal{N}}L_{ij}$.
See Appendix C for the proof.

Eq. (7) makes it easier to define primary terms and adds more flexibility on
the error distribution: e.g., AP Loss takes $p(j|i)=H(x_{ij})/N_{FP}(i)$,
which distributes error uniformly (since it is reduced to $1/N_{FP}(i)$) over
$j\in\mathcal{N}$ with $s_{j}\geq s_{i}$; though, a skewed $p(j|i)$ can be
used to promote harder examples (i.e. larger $x_{ij}$). Here,
$N_{FP}(i)=\sum_{j\in\mathcal{N}}H(x_{ij})$ is the number of false positives
for $i\in\mathcal{P}$.

Now we can identify the gradients of this generalized definition following
Chen et al. (Sec. 2.1): The error-driven update in $x_{ij}$ that would
minimize $\mathcal{L}$ is $\Delta x_{ij}={L_{ij}}^{*}-L_{ij}$, where
${L_{ij}}^{*}$ denotes “the primary term when $i$ is ranked properly”. Note
that ${L_{ij}}^{*}$, which is set to zero in AP Loss, needs to be carefully
defined (see Appendix G for a bad example). With $\Delta x_{ij}$ defined, the
gradients can be derived following Eq. (5).

This optimization provides balanced training for ranking-based losses
conforming to Theorem 1:

######  Theorem 2.

Training is balanced between positive and negative examples at each iteration;
i.e. the summed gradient magnitudes of positives and negatives are equal (see
Appendix C for the proof):

$\displaystyle\sum\limits_{i\in\mathcal{P}}\mathinner{\\!\left\lvert\frac{\partial\mathcal{L}}{\partial
s_{i}}\right\rvert}=\sum\limits_{i\in\mathcal{N}}\mathinner{\\!\left\lvert\frac{\partial\mathcal{L}}{\partial
s_{i}}\right\rvert}.$ (8)

Deriving AP Loss. Let us derive AP Loss as a case example for this generalized
framework: $\ell^{\mathrm{AP}}(i)$ is simply
$1-\mathrm{precision(i)}=N_{FP}(i)/\mathrm{rank}(i)$, and $Z=|\mathcal{P}|$.
$p(j|i)$ is assumed to be uniform, i.e. $p(j|i)=H(x_{ij})/N_{FP}(i)$. These
give us
$L_{ij}^{\mathrm{AP}}=\frac{N_{FP}(i)}{\mathrm{rank}(i)}\frac{H(x_{ij})}{N_{FP}(i)}=\frac{H(x_{ij})}{\mathrm{rank}(i)}$
(c.f. $L_{ij}^{\mathrm{AP}}$ in Eq. (4)). Then, since
${L_{ij}^{\mathrm{AP}}}^{*}=0$, $\Delta
x_{ij}=0-L^{\mathrm{AP}}_{ij}=-L^{\mathrm{AP}}_{ij}$ in Eq. (5).

Deriving Normalized Discounted Cumulative Gain Loss [17]: See Appendix D

##  4 Average Localization-Recall-Precision (aLRP) Loss

Figure 2: aLRP Loss assigns gradients to each branch based on the outputs of
both branches. Examples on the PR curve are in sorted order wrt scores ($s$).
$\mathcal{L}$ refers to $\mathcal{L}^{\mathrm{aLRP}}$. (a) A
$\mathrm{p_{i}}$’s gradient wrt its score considers (i) localization errors of
examples with larger $s$ (e.g. high $\mathcal{E}_{loc}(\mathrm{p_{1}})$
increases the gradient of $s_{\mathrm{p_{2}}}$ to suppress $\mathrm{p_{1}}$),
(ii) number of negatives with larger $s$. (b) Gradients wrt $s$ of the
negatives: The gradient of a $\mathrm{p}_{i}$ is uniformly distributed over
the negatives with larger $s$. Summed contributions from all positives
determines the gradient of a negative. (c) Gradients of the box parameters:
While $\mathrm{p_{1}}$ (with highest $s$) is included in total localisation
error on each positive, i.e.
$\mathcal{L}^{\mathrm{aLRP}}_{loc}(i)=\frac{1}{\mathrm{rank}(i)}(\mathcal{E}_{loc}(i)+\sum\limits_{k\in\mathcal{P},k\neq
i}\mathcal{E}_{loc}(k)H(x_{ik}))$, $\mathrm{p_{3}}$ is included once with the
largest $\mathrm{rank}(\mathrm{p_{i}})$.

Similar to the relation between precision and AP Loss, aLRP Loss is defined as
the average of LRP values ($\ell^{\mathrm{LRP}}(i)$) of positive examples:

$\displaystyle\mathcal{L}^{\mathrm{aLRP}}\mathrel{\mathop{\mathchar
58\relax}}=\frac{1}{|\mathcal{P}|}\sum\limits_{i\in\mathcal{P}}\ell^{\mathrm{LRP}}(i).$
(9)

For LRP, we assume that anchors are dense enough to cover all ground-truths,
i.e. $N_{FN}=0$. Also, since a detection is enforced to follow the label of
its anchor during training, TP and FP sets are replaced by the thresholded
subsets of $\mathcal{P}$ and $\mathcal{N}$ for $i\in\mathcal{P}$,
respectively. This is applied by $H(\cdot)$, and
$\mathrm{rank}(i)=N_{TP}+N_{FP}$ from Eq. (6). Then, following the definitions
in Sec. 2.1, $\ell^{\mathrm{LRP}}(i)$ is:

$\displaystyle\ell^{\mathrm{LRP}}(i)=\frac{1}{\mathrm{rank(i)}}\left(N_{FP}(i)+\mathcal{E}_{loc}(i)+\sum\limits_{k\in\mathcal{P},k\neq
i}\mathcal{E}_{loc}(k)H(x_{ik})\right).$ (10)

Note that Eq. (10) allows using robust forms of IoU-based losses (e.g.
generalized IoU (GIoU) [25]) only by replacing IoU Loss (i.e.
$1-\mathrm{IoU}(i)$) in $\mathcal{E}_{loc}(i)$ and normalizing the range to
$[0,1]$.

In order to provide more insight and facilitate gradient derivation, we split
Eq. (9) into two as localisation and classification components such that
$\mathcal{L}^{\mathrm{aLRP}}=\mathcal{L}^{\mathrm{aLRP}}_{cls}+\mathcal{L}^{\mathrm{aLRP}}_{loc}$,
where

$\displaystyle\mathcal{L}^{\mathrm{aLRP}}_{cls}$
$\displaystyle=\frac{1}{|\mathcal{P}|}\sum\limits_{i\in\mathcal{P}}\frac{N_{FP}(i)}{\mathrm{rank}(i)}\text{,
and
}\mathcal{L}^{\mathrm{aLRP}}_{loc}=\frac{1}{|\mathcal{P}|}\sum\limits_{i\in\mathcal{P}}\frac{1}{\mathrm{rank}(i)}\left(\mathcal{E}_{loc}(i)+\sum\limits_{k\in\mathcal{P},k\neq
i}\mathcal{E}_{loc}(k)H(x_{ik})\right).$ (11)

###  4.1 Optimization of the aLRP Loss

$\mathcal{L}^{\mathrm{aLRP}}$ is differentiable wrt the estimated box
parameters, $B$, since $\mathcal{E}_{loc}$ is differentiable [25, 29] (i.e.
the derivatives of $\mathcal{L}^{\mathrm{aLRP}}_{cls}$ and
$\mathrm{rank}(\cdot)$ wrt $B$ are $0$). However,
$\mathcal{L}^{\mathrm{aLRP}}_{cls}$ and $\mathcal{L}^{\mathrm{aLRP}}_{loc}$
are not differentiable wrt the classification scores, and therefore, we need
the generalized framework from Sec. 3.

Using a uniform distribution for $p(\cdot|\cdot)$ in Eq. (7), the primary term
of aLRP for $i\in\mathcal{P}$ and $j\in\mathcal{N}$ is:

$\displaystyle
L^{\mathrm{aLRP}}_{ij}=\ell^{\mathrm{LRP}}(i)\frac{H(x_{ij})}{N_{FP}(i)}=\frac{1}{\mathrm{rank(i)}}\left(N_{FP}(i)+\mathcal{E}_{loc}(i)+\sum\limits_{k\in\mathcal{P},k\neq
i}\mathcal{E}_{loc}(k)H(x_{ik})\right)\frac{H(x_{ij})}{N_{FP}(i)}.$ (12)

which reduces to a target primary term describing perfect ranking for
$i\in\mathcal{P}$:

$\displaystyle{L^{\mathrm{aLRP}}_{ij}}^{*}=\frac{1}{\mathrm{rank(i)}}\left({\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\cancelto{0}{N_{FP}(i)}}+\mathcal{E}_{loc}(i)+{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\cancelto{0}{\sum\limits_{k\in\mathcal{P},k\neq
i}\mathcal{E}_{loc}(k)H(x_{ik})}}\right)\frac{H(x_{ij})}{N_{FP}(i)}=\frac{\mathcal{E}_{loc}(i)}{\mathrm{rank}(i)}\frac{H(x_{ij})}{N_{FP}(i)},$
(13)

which includes localisation error and can be non-zero when $s_{i}<s_{j}$,
unlike AP Loss. Then, the error-driven update for $x_{ij}$ is

$\displaystyle\Delta
x_{ij}={L^{\mathrm{aLRP}}_{ij}}^{*}-{L^{\mathrm{aLRP}}_{ij}}=-\frac{1}{\mathrm{rank}(i)}\left(N_{FP}(i)+\sum\limits_{k\in\mathcal{P},k\neq
i}\mathcal{E}_{loc}(k)H(x_{ik})\right)\frac{H(x_{ij})}{N_{FP}(i)}.$ (14)

Finally, ${\partial\mathcal{L}^{\mathrm{aLRP}}}/{\partial s_{i}}$ can be
obtained with Eq. (5). Our algorithm to compute the loss and gradients is
presented in Appendix E in detail and has the same time&space complexity with
the AP Loss.

Figure 3: aLRP Loss and its components. The localisation component is self-
balanced.

Interpretation of the Components: A distinctive property of aLRP loss is that
classification and localisation errors are handled in a unified manner: i.e.
with aLRP, both classification and localisation branches use the entire output
of the detector, instead of working in their separate domains as
conventionally done. As shown in Fig. 2(a,b),
$\mathcal{L}^{\mathrm{aLRP}}_{cls}$ takes into account localization errors of
detections with larger scores ($s$) and promotes the detections with larger
IoUs to have higher $s$, or suppresses the detections with high-$s$&low-IoU.
Similarly, $\mathcal{L}^{\mathrm{aLRP}}_{loc}$ inherently weighs each positive
based on its classification rank (see Appendix F for the weights): the
contribution of a positive increases if it has a larger $s$. To illustrate, in
Fig. 2(c), while $\mathcal{E}_{loc}(p_{1})$ (i.e. with largest $s$)
contributes to each $\mathcal{L}^{\mathrm{aLRP}}_{loc}(i)$;
$\mathcal{E}_{loc}(p_{3})$ (i.e. with the smallest $s$) only contributes once
with a very low weight due to its rank normalizing
$\mathcal{L}^{\mathrm{aLRP}}_{loc}(\mathrm{p}_{3})$. Hence, the localisation
branch effectively focuses on detections ranked higher wrt $s$.

###  4.2 A Self-Balancing Extension for the Localisation Task

LRP metric yields localisation error only if a detection is classified
correctly (Sec. 2.2). Hence, when the classification performance is poor (e.g.
especially at the beginning of training), the aLRP loss is dominated by the
classification error ($N_{FP}(i)/\mathrm{rank}(i)\approx 1$ and
$\ell^{\mathrm{LRP}}(i)\in[0,1]$ in Eq. (10)). As a result, the localisation
head is hardly trained at the beginning (see Fig. 3). To alleviate this, we
propose a simple self-balancing (SB) strategy using the gradient magnitudes:
note that
$\frac{1}{|\mathcal{P}|}\sum_{i\in\mathcal{P}}\mathinner{\\!\left\lvert{\partial\mathcal{L}}/{\partial
s_{i}}\right\rvert}=\frac{1}{|\mathcal{P}|}\sum_{i\in\mathcal{N}}\mathinner{\\!\left\lvert{\partial\mathcal{L}}/{\partial
s_{i}}\right\rvert}\approx\mathcal{L}^{\mathrm{aLRP}}$ (see Appendix F). Then,
assuming that the gradients wrt scores and boxes are proportional to their
contributions to the $\mathrm{aLRP}$ loss, we multiply
${\partial\mathcal{L}}/{\partial B}$ by the average
$\mathcal{L}^{\mathrm{aLRP}}/\mathcal{L}^{\mathrm{aLRP}}_{loc}$ of the
previous epoch.

##  5 Experiments

Table 2: Ablation analysis on COCO minival. For optimal LRP (oLRP), lower is better. Method |  Rank-Based $\mathcal{L}_{c}$ |  Rank-Based $\mathcal{L}_{r}$ | SB | $\mathrm{AP}$ | $\mathrm{AP_{50}}$ | $\mathrm{AP_{75}}$ | $\mathrm{AP_{90}}$ | $\mathrm{oLRP}$ | $\rho$
---|---|---|---|---|---|---|---|---|---
AP Loss [7] | ✓ |  |  | $35.5$ | $58.0$ | $37.0$ | $9.0$ | $71.0$ | $0.45$
aLRP Loss | ✓ | ✓(w IoU) |  | $36.9$ | $57.7$ | $38.4$ | $13.9$ | $69.9$ | $0.49$
✓ | ✓(w IoU) | ✓ | $38.7$ | $58.1$ | $40.6$ | $17.4$ | $68.5$ | $0.48$
✓ | ✓(w GIoU) | ✓ | $38.9$ | $58.5$ | $40.4$ | $17.4$ | $68.4$ | $0.48$

Dataset: We train all our models on the COCO trainval35K set [15] (115K
images). The analysis uses the minival set (5k images) for testing whereas the
comparisons with the state-of-the-art are performed on the test-dev set (20K
images).

Performance Measures: COCO-style AP [15] and when possible optimal LRP [19]
(Sec. 2.2) are used for comparison. For more insight into aLRP Loss, we use
Pearson correlation coefficient ($\rho$) to measure correlation between the
rankings of classification and localisation, averaged over classes.

Implementation Details: As basis, we take the official implementation of AP
Loss [5], which uses RetinaNet [14]. Keeping all design choices fixed, unless
stated, we just replace AP & smooth L1 losses by aLRP Loss. For training, we
use $4$ GPUs (v100 or RTX2080Ti). The batch size is 32 for training with
$512\times 512$ images (aLRPLoss500), whereas it is 16 for $800\times 800$
images (aLRPLoss800). Following AP Loss, our models are trained for 100 epochs
using stochastic gradient descent with a momentum factor of $0.9$. We use a
learning rate of $0.008$ for aLRPLoss500 and $0.004$ for aLRPLoss800, each
decreased by factor $0.1$ at epochs 60 and 80. Similar to previous work [7,
8], standard data augmentation methods from SSD [16] are used. Ablation
analysis is performed using aLRPLoss500 on ResNet-50 backbone. At test time,
we rescale shorter sides of images to $500$ (aLRPLoss500) or $800$
(aLRPLoss800) pixels by ensuring that the longer side does not exceed
$1.66\times$ of the shorter side. NMS is applied to $1000$ top-scoring
detections using $0.50$ as the IoU threshold.

###  5.1 Ablation Study

Effect of using ranking for localisation: Table 2 shows that using a ranking
loss for localisation improves AP (from $35.5$ to $36.9$). For better insight,
$\mathrm{AP_{90}}$ is also included in Table 2, which shows $\sim$5 points
increase despite similar $\mathrm{AP_{50}}$ values. This confirms that aLRP
Loss does produce high-quality outputs for both branches, and boosts the
performance for larger IoUs.

Effect of Self-Balancing (SB): Section 4.2 and Fig. 3 discussed how
$\mathcal{L}^{\mathrm{aLRP}}_{cls}$ and $\mathcal{L}^{\mathrm{aLRP}}_{loc}$
behave during training and introduced self-balancing to improve training of
the localisation branch. Table 2 shows that SB provides +1.8AP gain, similar
$\mathrm{AP}_{50}$ and +8.4 points in $\mathrm{AP}_{90}$ against AP Loss.

Using GIoU: Table 2 suggests robust IoU-based regression (GIoU) improves
performance slightly.

Table 3: Effect of correlating rankings. $\mathcal{L}$ | $\rho$ | $\mathrm{AP}$ | $\mathrm{AP_{50}}$ | $\mathrm{AP_{75}}$ | $\mathrm{AP_{90}}$
---|---|---|---|---|---
aLRP Loss | $0.48$ | $38.7$ | $58.1$ | $40.6$ | $17.4$
Lower Bound | $-1.00$ | $28.6$ | $58.1$ | $23.6$ | $5.6$
Upper Bound | $1.00$ | $48.1$ | $58.1$ | $51.9$ | $33.9$

###  5.2 More insight on aLRP Loss

Potential of Correlating Classification and Localisation. We analyze two
bounds: (i) A Lower Bound where localisation provides an inverse ranking
compared to classification. (ii) An Upper Bound where localisation provides
exactly the same ranking as classification. Table 3 shows that correlating
ranking can have a significant effect (up to $20$ AP) on the performance
especially for larger IoUs. Therefore, correlating rankings promises
significant improvement (up to $\sim 10$AP). Moreover, while $\rho$ is $0.44$
and $0.45$ for Focal Loss (results not provided in the table) and AP Loss
(Table 2), respectively, aLRP Loss yields higher correlation ($0.48$).

Figure 4: (left) The rate of the total gradient magnitudes of negatives to
positives. (right) Loss values.

Analysing Balance Between Positives and Negatives. For this analysis, we
compare Cross Entropy Loss (CE), Focal Loss (FL) and aLRP Loss on RetinaNet
trained for 12 epochs. The average results over 10 runs are presented. Fig. 4
experimentally confirms Theorem 2 for aLRP loss
($\mathcal{L}^{\mathrm{aLRP}}_{cls}$), as it exhibits perfect balance between
the gradients throughout training. However, we see large fluctuations in
derivatives of CE and FL (left), which biases training towards positives or
negatives alternately across iterations. As expected, imbalance impacts CE
more as it quickly drops (right), overfitting in favor of negatives since it
is dominated by the error and gradients of these large amount of negatives.

###  5.3 Comparison with State of the Art (SOTA)

Different from ablation analysis, here aLRPLoss500 is trained for 160 epochs
with learning rate decay at epochs 90, 110 & 140 like previous work [2, 8,
18]. Table 4 presents the results:

Ranking-based Losses. aLRP Loss yields significant gains over other ranking-
based solutions: e.g., compared with AP Loss, aLRP Loss provides +4.3AP for
scale 500 and +3.3AP for scale 800. Similarly, for scale 800, aLRP Loss
performs up to 2.8AP better than DR Loss.

Methods combining branches. Although a direct comparison is not fair since
different conditions are used, we observe a significant margin (more than 5AP
in scale 800) compared to other approaches that combine localisation and
classification.

Comparison on scale 500. We see that, even with ResNet-101, aLRPLoss500
outperforms all other methods with 500 test scale except HSD with ResNext-101.
With ResNext-101, aLRP Loss outperforms its closest counterpart (HSD) by 1.5AP
and also in all sizes ($\mathrm{AP_{S}}$-$\mathrm{AP_{L}}$).

Comparison on scale 800. For 800 scale, aLRPLoss achieves $44.1$ and $45.9$ on
ResNet-101 and ResNeXt-101 backbones respectively. Also in this scale, aLRP
Loss consistently outperforms its closest counterparts (i.e. FreeAnchor and
CenterNet) by 1.0AP and reaches the highest results wrt all performance
measures.

Running time. aLRP Loss does not affect inference time and provides $\sim$15
frames/sec for 800 scale on ResNet-101 with a v100 GPU [6].

Table 4: Comparison with the SOTA detectors on COCO test-dev. $S,\times 1.66$
implies that the image is rescaled such that its longer side cannot exceed
$1.66\times S$ where $S$ is the size of the shorter side. R:ResNet, X:ResNeXt,
H:HourglassNet, D:DarkNet, De:DeNet.

Method | Backbone | Training Size | Test Size | $\mathrm{AP}$ | $\mathrm{AP_{50}}$ | $\mathrm{AP_{75}}$ | $\mathrm{AP_{S}}$ | $\mathrm{AP_{M}}$ | $\mathrm{AP_{L}}$
---|---|---|---|---|---|---|---|---|---
One-Stage Methods |  |  |  |  |  |  |  |  |
RefineDet [30]‡ | R-101 | $512\times 512$ | $512\times 512$ | $36.4$ | $57.5$ | $39.5$ | $16.6$ | $39.9$ | $51.4$
EFGRNet [18]‡ | R-101 | $512\times 512$ | $512\times 512$ | $39.0$ | $58.8$ | $42.3$ | $17.8$ | $43.6$ | $54.5$
HSD [2]‡ | R-101 | $512\times 512$ | $512\times 512$ | $40.2$ | $59.4$ | $44.0$ | $20.0$ | $44.4$ | $54.9$
ExtremeNet [32]∗‡ | H-104 | $511\times 511$ | original | $40.2$ | $55.5$ | $43.2$ | $20.4$ | $43.2$ | $53.1$
RetinaNet800 [14] | X-101 | $800,\times 1.66$ | $800,\times 1.66$ | $40.8$ | $61.1$ | $44.1$ | $24.1$ | $44.2$ | $51.2$
FCOS [27]† | R-101 | $(640,800),\times 1.66$ | $800,\times 1.66$ | $41.5$ | $60.7$ | $45.0$ | $24.4$ | $44.8$ | $51.6$
HSD [2] ‡ | X-101 | $512\times 512$ | $512\times 512$ | $41.9$ | $61.1$ | $46.2$ | $21.8$ | $46.6$ | $57.0$
FreeAnchor [31]† | R-101 | $(640,800),\times 1.66$ | $800,\times 1.66$ | $43.1$ | $62.2$ | $46.4$ | $24.5$ | $46.1$ | $54.8$
FCOS [27]† | X-101 | $(640,800),\times 1.66$ | $800,\times 1.66$ | $44.7$ | $64.1$ | $48.4$ | $27.6$ | $47.5$ | $55.6$
CenterNet [8]∗‡ | H-104 | $511\times 511$ | original | $44.9$ | $62.4$ | $48.1$ | $25.6$ | $47.4$ | $57.4$
FreeAnchor [31]† | X-101 | $(640,800),\times 1.66$ | $800,\times 1.66$ | $44.9$ | $64.3$ | $48.5$ | $26.8$ | $48.3$ | $55.9$
Ranking Losses |  |  |  |  |  |  |  |  |
AP Loss500 [7]‡ | R-101 | $512\times 512$ | $500,\times 1.66$ | $37.4$ | $58.6$ | $40.5$ | $17.3$ | $40.8$ | $51.9$
AP Loss800 [7]‡ | R-101 | $800\times 800$ | $800,\times 1.66$ | $40.8$ | $63.7$ | $43.7$ | $25.4$ | $43.9$ | $50.6$
DR Loss [23]† | R-101 | $(640,800),\times 1.66$ | $800,\times 1.66$ | $41.7$ | $60.9$ | $44.8$ | $23.5$ | $44.9$ | $53.1$
DR Loss [23]† | X-101 | $(640,800),\times 1.66$ | $800,\times 1.66$ | $43.1$ | $62.8$ | $46.4$ | $25.6$ | $46.2$ | $54.0$
Combining Branches |  |  |  |  |  |  |  |  |
LapNet [4] | D-53 | $512\times 512$ | $512\times 512$ | $37.6$ | $55.5$ | $40.4$ | $17.6$ | $40.5$ | $49.9$
Fitness NMS [28] | De-101 | $512,\times 1.66$ | $768,\times 1.66$ | $39.5$ | $58.0$ | $42.6$ | $18.9$ | $43.5$ | $54.1$
Retina+PISA [3] | R-101 | $800,\times 1.66$ | $800,\times 1.66$ | $40.8$ | $60.5$ | $44.2$ | $23.0$ | $44.2$ | $51.4$
Ours |  |  |  |  |  |  |  |  |
aLRP Loss500‡ | R-101 | $512\times 512$ | $500,\times 1.66$ | $41.7$ | $61.5$ | $44.1$ | $21.2$ | $45.2$ | $56.8$
aLRP Loss500‡ | X-101 | $512\times 512$ | $500,\times 1.66$ | $43.4$ | $63.4$ | $46.0$ | $23.0$ | $47.3$ | $\mathbf{58.2}$
aLRP Loss800‡ | R-101 | $800\times 800$ | $800,\times 1.66$ | $44.1$ | $64.8$ | $47.0$ | $26.4$ | $47.3$ | $56.2$
aLRP Loss800‡ | X-101 | $800\times 800$ | $800,\times 1.66$ | $\mathbf{45.9}$ | $\mathbf{66.8}$ | $\mathbf{48.8}$ | $\mathbf{28.8}$ | $\mathbf{49.3}$ | $57.4$

†: multiscale training, ‡: SSD-like augmentation, ∗: Soft NMS [1] and flip
augmentation at test time

##  6 Conclusion

In this paper, we provided a general framework for the error-driven
optimization of ranking-based functions. As a special case of this
generalization, we introduced aLRP Loss, a ranking-based, balanced loss
function which handles the classification and localisation errors in a unified
manner. aLRP Loss has only one hyperparameter which we did not need to tune,
as opposed to around 6 in SOTA loss functions. We showed that using aLRP
improves its baseline up to $4.3$ points in AP, and outperforms all one-stage
detectors.

## Broader Impact

We anticipate our work to significantly impact the following domains:

  1. 1.

Object detection: Our loss function is unique in many important aspects: It
unifies localisation and classification in a single loss function. It uses
ranking for both classification and localisation. It provides provable balance
between negatives and positives, similar to AP Loss.

These unique merits will contribute to a paradigm shift in the object
detection community towards more capable and sophisticated loss functions such
as ours.

  2. 2.

Other computer vision problems with multiple objectives: Problems including
multiple objectives (such as instance segmentation, panoptic segmentation –
which actually has classification and regression objectives) will benefit
significantly from our proposal of using ranking for both classification and
localisation.

  3. 3.

Problems that can benefit from ranking: Many vision problems can be easily
converted into a ranking problem. They can then exploit our generalized
framework to easily define a loss function and to determine the derivatives.

Our paper does not have direct social implications. However, it inherits the
following implications of object detectors: Object detectors can be used for
surveillance purposes for the betterness of society albeit privacy concerns.
When used for detecting targets, an object detector’s failure may have severe
consequences depending on the application (e.g. self-driving cars). Moreover,
such detectors are affected by the bias in data, although they will not try to
exploit them for any purposes.

## Acknowledgments and Disclosure of Funding

This work was partially supported by the Scientific and Technological Research
Council of Turkey (TÜBİTAK) through a project titled “Object Detection in
Videos with Deep Neural Networks” (grant number 117E054). Kemal Öksüz is
supported by the TÜBİTAK 2211-A National Scholarship Programme for Ph.D.
students. The numerical calculations reported in this paper were performed at
TUBITAK ULAKBIM High Performance and Grid Computing Center (TRUBA), and
Roketsan Missiles Inc. sources.

## References

  * Bodla et al. [2017] Bodla N, Singh B, Chellappa R, Davis LS (2017) Soft-nms – improving object detection with one line of code. In: The IEEE International Conference on Computer Vision (ICCV)
  * Cao et al. [2019a] Cao J, Pang Y, Han J, Li X (2019a) Hierarchical shot detector. In: The IEEE International Conference on Computer Vision (ICCV)
  * Cao et al. [2019b] Cao Y, Chen K, Loy CC, Lin D (2019b) Prime Sample Attention in Object Detection. arXiv 1904.04821
  * Chabot et al. [2019] Chabot F, Pham QC, Chaouch M (2019) Lapnet : Automatic balanced loss and optimal assignment for real-time dense object detection. arXiv 1911.01149
  * Chen [Last Accessed: 14 May 2020] Chen K (Last Accessed: 14 May 2020) Ap-loss. https://githubcom/cccorn/AP-loss
  * Chen et al. [2019] Chen K, Wang J, Pang J, Cao Y, Xiong Y, Li X, Sun S, Feng W, Liu Z, Xu J, Zhang Z, Cheng D, Zhu C, Cheng T, Zhao Q, Li B, Lu X, Zhu R, Wu Y, Dai J, Wang J, Shi J, Ouyang W, Loy CC, Lin D (2019) MMDetection: Open mmlab detection toolbox and benchmark. arXiv 1906.07155
  * Chen et al. [2020] Chen K, Lin W, j li, See J, Wang J, Zou J (2020) Ap-loss for accurate one-stage object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence pp 1–1
  * Duan et al. [2019] Duan K, Bai S, Xie L, Qi H, Huang Q, Tian Q (2019) Centernet: Keypoint triplets for object detection. In: The IEEE International Conference on Computer Vision (ICCV)
  * Everingham et al. [2010] Everingham M, Van Gool L, Williams CKI, Winn J, Zisserman A (2010) The pascal visual object classes (voc) challenge. International Journal of Computer Vision (IJCV) 88(2):303–338
  * Girshick [2015] Girshick R (2015) Fast R-CNN. In: The IEEE International Conference on Computer Vision (ICCV)
  * Jiang et al. [2018] Jiang B, Luo R, Mao J, Xiao T, Jiang Y (2018) Acquisition of localization confidence for accurate object detection. In: The European Conference on Computer Vision (ECCV)
  * Kendall et al. [2018] Kendall A, Gal Y, Cipolla R (2018) Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
  * Li et al. [2019] Li B, Liu Y, Wang X (2019) Gradient harmonized single-stage detector. In: AAAI Conference on Artificial Intelligence
  * Lin et al. [2020] Lin T, Goyal P, Girshick R, He K, Dollár P (2020) Focal loss for dense object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 42(2):318–327
  * Lin et al. [2014] Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick CL (2014) Microsoft COCO: Common Objects in Context. In: The European Conference on Computer Vision (ECCV)
  * Liu et al. [2016] Liu W, Anguelov D, Erhan D, Szegedy C, Reed SE, Fu C, Berg AC (2016) SSD: single shot multibox detector. In: The European Conference on Computer Vision (ECCV)
  * Mohapatra et al. [2018] Mohapatra P, Rolínek M, Jawahar C, Kolmogorov V, Pawan Kumar M (2018) Efficient optimization for rank-based loss functions. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
  * Nie et al. [2019] Nie J, Anwer RM, Cholakkal H, Khan FS, Pang Y, Shao L (2019) Enriched feature guided refinement network for object detection. In: The IEEE International Conference on Computer Vision (ICCV)
  * Oksuz et al. [2018] Oksuz K, Cam BC, Akbas E, Kalkan S (2018) Localization recall precision (LRP): A new performance metric for object detection. In: The European Conference on Computer Vision (ECCV)
  * Oksuz et al. [2020] Oksuz K, Cam BC, Kalkan S, Akbas E (2020) Imbalance problems in object detection: A review. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) pp 1–1
  * Pang et al. [2019] Pang J, Chen K, Shi J, Feng H, Ouyang W, Lin D (2019) Libra R-CNN: Towards balanced learning for object detection. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
  * Pogančić et al. [2020] Pogančić MV, Paulus A, Musil V, Martius G, Rolinek M (2020) Differentiation of blackbox combinatorial solvers. In: International Conference on Learning Representations (ICLR)
  * Qian et al. [2020] Qian Q, Chen L, Li H, Jin R (2020) Dr loss: Improving object detection by distributional ranking. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
  * Ren et al. [2017] Ren S, He K, Girshick R, Sun J (2017) Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 39(6):1137–1149
  * Rezatofighi et al. [2019] Rezatofighi H, Tsoi N, Gwak J, Sadeghian A, Reid I, Savarese S (2019) Generalized intersection over union: A metric and a loss for bounding box regression. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
  * Rosenblatt [1958] Rosenblatt F (1958) The perceptron: A probabilistic model for information storage and organization in the brain. Psychological Review pp 65–386
  * Tian et al. [2019] Tian Z, Shen C, Chen H, He T (2019) Fcos: Fully convolutional one-stage object detection. In: The IEEE International Conference on Computer Vision (ICCV)
  * Tychsen-Smith and Petersson [2018] Tychsen-Smith L, Petersson L (2018) Improving object localization with fitness nms and bounded iou loss. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
  * Yu et al. [2016] Yu J, Jiang Y, Wang Z, Cao Z, Huang T (2016) Unitbox: An advanced object detection network. In: The ACM International Conference on Multimedia
  * Zhang et al. [2018] Zhang S, Wen L, Bian X, Lei Z, Li SZ (2018) Single-shot refinement neural network for object detection. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
  * Zhang et al. [2019] Zhang X, Wan F, Liu C, Ji R, Ye Q (2019) Freeanchor: Learning to match anchors for visual object detection. In: Advances in Neural Information Processing Systems (NeurIPS)
  * Zhou et al. [2019] Zhou X, Zhuo J, Krahenbuhl P (2019) Bottom-up object detection by grouping extreme and center points. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

## APPENDIX

##  Appendix A Details of Figure 1: Comparison of Loss Functions on a Toy
Example

This section aims to present the scenario considered in Figure 1 of the main
paper. Section A.1 explains the scenario, Section A.2 and Section A.3 clarify
how the performance measures (AP, AP50, etc.) and loss values (cross-entropy,
AP loss, aLRP loss, etc.) are calculated.

###  A.1 The Scenario

We assume that the scenario in Figure 1(a) of the paper includes five ground
truths in the mini-batch of which four of them are detected as true positives
with different Intersection-over-Union (IoU) overlaps by three different
detectors (i.e. $\mathrm{C\&R_{1}}$, $\mathrm{C\&R_{2}}$,
$\mathrm{C\&R_{3}}$). Each detector has different ranking of these true
positives with respect to their IoUs. In addition, the output of each detector
contains the same six detections with different scores as false positives.
Note that the IoUs of these false positives are marked with "–" in Figure 1(a)
since they do not match with any ground truth and therefore their IoUs are not
being considered neither by performance (i.e. Average Precision) nor by loss
computation.

Figure A.5: Visualization of anchor boxes given in Figure 1. Green boxes are
foreground anchors, red boxes are ground truth boxes.

###  A.2 Performance Evaluation

There are different ways to calculate Average Precision (AP) and loss values.
For example, in PASCAL [9] and COCO [15] datasets, the recall domain is
divided into 11 and 101 evenly spaced points, respectively, and the precision
values at these points are averaged to compute AP for a single IoU threshold.

Here, we present how Average Precision ($\mathrm{AP}$) is calculated in Figure
1(b). Similar to the widely adopted performance metric, COCO-style AP, we use
$\mathrm{AP}_{\mathrm{IoU}}$ with different $\mathrm{IoU}$ thresholds. In
order to keep things simple but provide the essence of the performance metric,
we use four samples with $0.15$ increments (i.e. $\\{0.50,0.65,0.80,0.95\\}$)
instead of ten samples with $0.05$ increments as done by original COCO-style
AP.

In order to compute a single average precision with an IoU as the conventional
true positive labelling threshold, denoted by $\mathrm{AP}_{\mathrm{IoU}}$,
the approaches vary according to how they sample individual precision values
on a PR curve. The PR curves corresponding for each
detector-$\mathrm{AP}_{\mathrm{IoU}}$ pair are presented in Fig. A.6. While
drawing these curves, similar to Pascal-VOC and COCO, we also adopt
interpolation on the PR curve, which requires keeping the larger precision
value in the case that the larger one resides in a lower recall. Then, again
similar to what these common methods do for a single AP threshold, we check
the precision values on different recall values after splitting the recall
axis equally. Here, we use 10 recall points between 0.1 and 1.0 in 0.1
increments. Then, based on the PR curves in Fig. A.6, we check the precision
under these different recall values and present them in Table A.5. Having
generated these values in Table A.5 for each $\mathrm{AP}_{\mathrm{IoU}}$s,
the computation is trivial: Just averaging over these precisions (i.e. row-
wise average) yields $\mathrm{AP}_{\mathrm{IoU}}$s. Finally, averaging over
these four $\mathrm{AP}_{\mathrm{IoU}}$s produces the final detection
performance as $0.37$, $0.29$ and $0.20$ for $\mathrm{C\&R_{1}}$,
$\mathrm{C\&R_{2}}$, $\mathrm{C\&R_{3}}$ respectively (see Table A.5 – below).

|  |  |
---|---|---|---
|  |  |
|  |  |

(a) C&R1 $AP_{50}$

(b) C&R1 $AP_{65}$

(c) C&R1 $AP_{80}$

(d) C&R1 $AP_{95}$

(e) C&R2 $AP_{50}$

(f) C&R2 $AP_{65}$

(g) C&R2 $AP_{80}$

(h) C&R2 $AP_{95}$

(i) C&R3 $AP_{50}$

(j) C&R3 $AP_{65}$

(k) C&R3 $AP_{80}$

(l) C&R3 $AP_{95}$

Figure A.6: PR curve of each detector output-$\mathrm{AP}_{IoU}$ pair. Rows and columns correspond to different $\mathrm{AP}_{IoU}$ and detector outputs respectively. PR curves are interpolated (see the text for more detail). Table A.5: Precision of each detector output-$\mathrm{AP}_{IoU}$ pair for evenly spaced recall values. This table is based on the PR curves presented in Fig. A.6. IoU | Output | Precisions for Different Recalls (R) | $\mathrm{AP_{\mathrm{IoU}}}$
---|---|---|---
R=$0.1$ | R=$0.2$ | R=$0.3$ | R=$0.4$ | R=$0.5$ | R=$0.6$ | R=$0.7$ | R=$0.8$ | R=$0.9$ | R=$1.0$
0.50 | $C\&R_{1}$ | 1.00 | 1.00 | 0.67 | 0.67 | 0.50 | 0.50 | 0.40 | 0.40 | 0.00 | 0.00 | 0.51
$C\&R_{2}$ | 1.00 | 1.00 | 0.67 | 0.67 | 0.50 | 0.50 | 0.40 | 0.40 | 0.00 | 0.00 | 0.51
$C\&R_{3}$ | 1.00 | 1.00 | 0.67 | 0.67 | 0.50 | 0.50 | 0.40 | 0.40 | 0.00 | 0.00 | 0.51
0.65 | $C\&R_{1}$ | 1.00 | 1.00 | 0.67 | 0.67 | 0.50 | 0.50 | 0.00 | 0.00 | 0.00 | 0.00 | 0.43
$C\&R_{2}$ | 1.00 | 1.00 | 0.67 | 0.67 | 0.30 | 0.30 | 0.00 | 0.00 | 0.00 | 0.00 | 0.39
$C\&R_{3}$ | 0.33 | 0.33 | 0.33 | 0.33 | 0.30 | 0.30 | 0.00 | 0.00 | 0.00 | 0.00 | 0.19
0.80 | $C\&R_{1}$ | 1.00 | 1.00 | 0.67 | 0.67 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.33
$C\&R_{2}$ | 1.00 | 1.00 | 0.20 | 0.20 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.24
$C\&R_{3}$ | 0.20 | 0.20 | 0.20 | 0.20 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.08
0.95 | $C\&R_{1}$ | 1.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.20
$C\&R_{2}$ | 0.10 | 0.10 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.02
$C\&R_{3}$ | 0.10 | 0.10 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.02

###  A.3 Computation of Loss Values

In this section, calculation of the loss values in Figure 1(c) of the paper is
presented in detail. Each section is devoted to a loss function presented in
the Figure 1(c). To keep things simple, without loss of generality, we make
the following assumptions in this section during the calculation of the
classification and localisation losses:

  1. 1.

The classifier has sigmoid non-linearity at the top.

  2. 2.

There is only one foreground class.

  3. 3.

Similar to how localisation losses deal with scale- and translation-variance
within an image, we assume that each ground truth box and its assigned
prediction is normalized to $\hat{B}=[0,0,1,1]$.

  4. 4.

For each loss, the average of its contributors are reported.

####  A.3.1 Cross-entropy loss

Cross entropy loss of the $i$th example is defined as:

$\displaystyle\mathcal{L}^{\mathrm{CE}}(s_{i})=-\mathcal{I}[i\in\mathcal{P}]\log(s_{i})-\mathcal{I}[i\in\mathcal{N}]\log(1-s_{i}),$
(A.15)

such that $s_{i}$ is the score of the $i$th example obtained by applying the
sigmoid activation to the classification logit, and $\mathcal{I}[\mathrm{Q}]$
is the Iverson bracket which is $1$ if the predicate $\mathrm{Q}$ is true; or
else it is $0$.

Seeing that all the detector outputs, $C\&R1$, $C\&R2$ and $C\&R3$, involves
the same classification output, we apply Eq. (A.15) for each anchor on the
$C$, and then find their average as follows:

$\displaystyle\mathcal{L}^{\mathrm{CE}}$
$\displaystyle=\frac{1}{|\mathcal{P}|+|\mathcal{N}|}\sum_{s_{i}}\mathcal{L}^{\mathrm{CE}}(s_{i}),$
(A.16)
$\displaystyle=-\frac{1}{10}\left(\log(1.00)+\log(1-0.90)+\log(0.80)+\log(1-0.70)+\log(1-0.60)+\log(0.50)\right.$
(A.17)
$\displaystyle\left.+\log(1-0.40)+\log(1-0.30)+\log(1-0.20)+\log(0.10)\right),$
(A.18) $\displaystyle=0.87.$ (A.19)

####  A.3.2 Average precision (AP) loss

The computation of AP loss is very similar to the $\mathrm{AP}_{\mathrm{IoU}}$
for $\mathrm{IoU}=50$ computation described in Section A.2 except that
precision is calculated on (and also averaged over) the positive examples
instead of the recall values. With this intuition the precision values on the
four positives are $1.00,0.67,0.50,0.40$ respectively. Then, the AP Loss for
the output $C$ in Figure 1 regardless of the localisation output that it is
combined with is:

$\displaystyle\mathcal{L}^{\mathrm{AP}}=1-\mathrm{\mathrm{AP}}_{50}$
$\displaystyle=1-\frac{1}{|\mathcal{P}|}\sum\limits_{i\in\mathcal{P}}\mathrm{precision}(i),$
$\displaystyle=1-\frac{1}{4}\times\left(1.00+0.67+0.50+0.40\right)=0.36.$

####  A.3.3 L1 loss

For a single ground truth,
$\hat{B}_{i}=[\hat{x}_{1},\hat{y}_{1},\hat{x}_{2},\hat{y}_{2}]$, and its
corresponding detection, $B_{i}=[x_{1},y_{1},x_{2},y_{2}]$, L1 loss is defined
simply by averaging over the L1 norm between the difference of the parameters
of the detection boxes from its corresponding ground:

$\displaystyle\mathcal{L}^{\mathrm{L1}}(\hat{B}_{i},B_{i})=\mathinner{\\!\left\lvert\hat{x}_{1}-x_{1}\right\rvert}+\mathinner{\\!\left\lvert\hat{y}_{1}-y_{1}\right\rvert}+\mathinner{\\!\left\lvert\hat{x}_{2}-x_{2}\right\rvert}+\mathinner{\\!\left\lvert\hat{y}_{2}-y_{2}\right\rvert},$
(A.20)

Then, the average L1 loss is:

$\displaystyle\mathcal{L}^{\mathrm{L1}}$
$\displaystyle=\frac{1}{|\mathcal{P}|}\sum_{i\in\mathcal{P}}\mathcal{L}^{\mathrm{L1}}(\hat{B}_{i},B_{i}),$
(A.21)
$\displaystyle=\frac{1}{4}\left((0.00+0.00+0.00+0.05)+(0.00+0.00+0.00+0.25)\right.$
(A.22)
$\displaystyle\left.+(0.00+0.00+0.00+0.35)+(0.00+0.00+0.00+0.50)\right),$
(A.23) $\displaystyle=0.29.$ (A.24)

####  A.3.4 IoU loss

For a single example, IoU loss is simply $1-\mathrm{IoU}(\hat{B}_{i},B_{i})$.
Then, for all the three outputs in the scenario (also an instance is
illustrated in Fig. A.5), seeing that the IoU distributions are all equal, the
average IoU loss of this detection set is:

$\displaystyle\mathcal{L}^{\mathrm{IoU}}$
$\displaystyle=\frac{1}{|\mathcal{P}|}\sum\limits_{i\in\mathcal{P}}1-\mathrm{IoU}(\hat{B}_{i},B_{i}),$
$\displaystyle=\frac{1}{4}\left((1-0.95)+(1-0.80)+(1-0.65)+(1-0.50)\right)=0.28.$

####  A.3.5 aLRP loss

This section calculates aLRP loss value in the scenario, and therefore we
believe that at the same time it is also a toy example to present insight on
the aLRP Loss.

First, let us recall the definition of the aLRP loss from the paper to
simplify tracking this section. aLRP Loss is defined as:

$\displaystyle\mathcal{L}^{\mathrm{aLRP}}\mathrel{\mathop{\mathchar
58\relax}}=\frac{1}{|\mathcal{P}|}\sum\limits_{i\in\mathcal{P}}\ell^{\mathrm{LRP}}(i),$
(A.25)

such that

$\displaystyle\ell^{\mathrm{LRP}}(i)=\frac{1}{\mathrm{rank(i)}}\left(N_{FP}(i)+\mathcal{E}_{loc}(i)+\sum\limits_{k\in\mathcal{P},k\neq
i}\mathcal{E}_{loc}(k)H(x_{ik})\right),$ (A.26)

where $\mathcal{E}_{loc}(k)=(1-\mathrm{IoU}(k))/(1-\tau)$. Here, we take
$H(x)$ as a step function instead of its approximation for simplicity.

Table A.6 presents the computation of aLRP values including all by-products
for each of the four positive anchors in $C\&R_{1}$, $C\&R_{2}$ and
$C\&R_{3}$. Given the table presented in Figure 1(a) in the paper, we present
how each column is derived in the following steps:

  1. 1.

$1-\mathrm{IoU(i)}$ is simply the IoU Loss of the positive anchors after
prediction.

  2. 2.

$\mathcal{E}_{loc}(i)=(1-\mathrm{IoU}(i))/(1-\tau)$ such that $\tau=0.5$.

  3. 3.

Define a cumulative sum:
$\mathrm{cumsum(\mathcal{E}_{loc})}(i)=\mathcal{E}_{loc}(i)+\sum\limits_{k\in\mathcal{P},k\neq
i}\mathcal{E}_{loc}(k)H(x_{ik})$ (see Eq. 10). Note that this simply
corresponds to a cumulative sum on a positive example using the examples with
larger scores and itself. Accordingly, in Table A.6,
$\mathrm{cumsum(\mathcal{E}_{loc})}(i)$ is calculated by summing
$\mathcal{E}_{loc})(i)$ column over anchors until (and including) $i$th
example.

  4. 4.

$N_{FP}(i)$ is the number of negative examples with larger scores than the
$i$th positive anchor. (See Section 3 for the formal definition.)

  5. 5.

$\mathrm{rank}(i)$ is the rank of an example within positives and negatives.
(See Section 2 for the formal definition.)

  6. 6.

Then using $\mathrm{cumsum(\mathcal{E}_{loc})}(i)$, $N_{FP}(i)$ and
$\mathrm{rank}(i)$, LRP error on a positive example can be computed as:

$\displaystyle\ell^{\mathrm{LRP}}(i)=\frac{N_{FP}(i)+\mathrm{cumsum(\mathcal{E}_{loc})}(i)}{\mathrm{rank}(i)}.$
(A.27)

  7. 7.

In the rightmost column, aLRP Loss of a detector, $\mathcal{L}^{aLRP}$, is
determined simply averaging over these single LRP values (i.e.
$\ell^{\mathrm{LRP}}(i)$ ) on positives.

Table A.6: Per-box calculation of $\mathcal{L}_{aLRP}$ Output | Anchor | $\mathrm{1-IoU(i)}$ | $\mathcal{E}_{loc}(i)$ | $\mathrm{cumsum(\mathcal{E}_{loc})}(i)$ | $N_{FP}(i)$ | $\mathrm{rank(i)}$ | $\ell^{\mathrm{LRP}}(i)$ | $\mathcal{L}^{\mathrm{aLRP}}$
---|---|---|---|---|---|---|---|---
C&R1 | $a_{1}$ | $0.05$ | $0.10$ | $0.10$ | $0.00$ | $1.00$ | $0.10$ | $0.53$
$a_{3}$ | $0.20$ | $0.40$ | $0.50$ | $1.00$ | $3.00$ | $0.50$
$a_{6}$ | $0.35$ | $0.70$ | $1.20$ | $3.00$ | $6.00$ | $0.70$
$a_{10}$ | $0.50$ | $1.00$ | $2.20$ | $6.00$ | $10.00$ | $0.82$
C&R2 | $a_{1}$ | $0.20$ | $0.40$ | $0.40$ | $0.00$ | $1.00$ | $0.40$ | $0.69$
$a_{3}$ | $0.35$ | $0.70$ | $1.10$ | $1.00$ | $3.00$ | $0.70$
$a_{6}$ | $0.50$ | $1.00$ | $2.10$ | $3.00$ | $6.00$ | $0.85$
$a_{10}$ | $0.20$ | $0.40$ | $2.50$ | $6.00$ | $10.00$ | $0.82$
C&R3 | $a_{1}$ | $0.50$ | $1.00$ | $1.00$ | $0.00$ | $1.00$ | $1.00$ | $0.89$
$a_{3}$ | $0.35$ | $0.70$ | $1.70$ | $1.00$ | $3.00$ | $0.90$
$a_{6}$ | $0.20$ | $0.40$ | $2.10$ | $3.00$ | $6.00$ | $0.85$
$a_{10}$ | $0.05$ | $0.10$ | $2.20$ | $6.00$ | $10.00$ | $0.82$

##  Appendix B Details of Table 1: Hyperparameters of the Loss Functions and
Models

This section presents the hyperparameters of the common loss functions in
object detection and how they are combined by different models in Table 1.

###  B.1 Hyperparameters of the Individual Loss Functions

Table A.7 presents common loss functions and their hyperparameters. Note that
since any change in these hyperparameter change the value of the loss function
and affects its contribution to the multi-task learning nature of object
detection, and, therefore $w_{r}$ also needs to be retuned.

Table A.7: Common loss functions and the hyperparameters in their definitions. | Loss Function | Type | Number & Usage of the Hyper-parameters
---|---|---|---
$\mathcal{L}_{c}$ | Cross entropy [16, 24] | Score-based | 0 | Sampling methods are required
$\alpha$-bal. cross entropy[14] | Score-based | 1 | The weight of the foreground anchors
Focal loss [14] | Score-based | 2 | The weight of the foreground anchors
Modulating factor for hard examples
AP Loss [7] | Ranking-based | 1 | Smoothness of the step function
DR loss [23] | Ranking-based | 3 | Regularizer for foreground distribution
Regularizer for background distribution
Smoothness of the loss
$\mathcal{L}_{r}$ | Smooth $L_{1}$ [10] |  $l_{p}$-based | 1 | Cut-off from $L_{1}$ loss to $L_{2}$ loss
Balanced $L_{1}$ [21] | $l_{p}$-based | 2 | The weight of the inlier anchors
Upper bound of the loss value
IoU loss [25] | IoU-based | 0 | -

###  B.2 Hyperparameters of the Loss Functions of the Models

This section discusses the loss functions of the methods discussed in Table 1
in the paper. Obviously, AP Loss [7], Focal Loss [14] and DR Loss [23] follow
the formulation in Equation 1. Hence using Table 1, their total number of
hyperparameters is easy to see. For example, DR Loss with three hyper-
parameters is combined with Smooth L1, which has one hyperparameter. Including
the weight of the localisation component, five hyper-parameters are required
to be tuned.

Other architectures in Table 1 use more than two loss functions in order to
learn different aspects to improve the performance:

  * •

FCOS [27] includes an additional centerness branch to predict the centerness
of the pixels, which is trained by an additional cross entropy loss.

  * •

Free Anchor [31] aims simultaneously to learn the assignment of the anchors to
the ground truths by modeling the loss function based on maximum likelihood
estimation. In Table 1, one can easily identify six hyper-parameters from the
loss formulation of the Free Anchor and exploiting Table 1. Moreover, the
inputs of the focal loss are subject to a saturated linear function with two
hyperparameters, which makes eight in total.

  * •

A different set of approaches, an example of which is Faster R-CNN [24], uses
directly cross entropy loss. However, cross entropy loss requires to be
accompanied by a sampling method by which a set of positive and negative
examples are sampled from the set of labelled anchors to alleviate the
significant class imbalance. Even for random sampler, two of the following
needs to be tuned in order to ensure stable training: (i) Number of positive
examples (ii) Number of negative examples (iii) The rate between positives and
negatives. Moreover, for a two-stage detector, these should be tuned for both
stages, which brings about additional four hyper-parameters. That‘s why Faster
R-CNN [24] in Table 1 requires nine hyperparameters.

  * •

Finally, CenterNet [8] , as a state-of-the-art bottom-up method, employs has a
loss function with several components while learning to predict the centers
and the corners. It combines six individual losses, one of which is Hinge Loss
with one hyperparameter. Considering the type of each, the loss function of
CenterNet [8] has 10 hyper-parameters in total.

##  Appendix C Proofs of Theorem 1 and Theorem 2

This section presents formal proofs for the theorems presented in our paper.

######  Theorem 1.

$\mathcal{L}=\frac{1}{Z}\sum\limits_{i\in\mathcal{P}}\ell(i)=\frac{1}{Z}\sum\limits_{i\in\mathcal{P}}\sum\limits_{j\in\mathcal{N}}L_{ij}$.

###### Proof.

The ranking function is defined as:

$\displaystyle\mathcal{L}=\frac{1}{Z}\sum_{i\in\mathcal{P}}\ell(i).$ (A.28)

Since $\forall i\sum\limits_{j\in\mathcal{N}}p(j|i)=1$, we can rewrite the
definition as follows:

$\displaystyle\frac{1}{Z}\sum\limits_{i\in\mathcal{P}}\ell(i)\left(\sum\limits_{j\in\mathcal{N}}p(j|i)\right).$
(A.29)

Reorganizing the terms concludes the proof as follows:

$\displaystyle\frac{1}{Z}\sum\limits_{i\in\mathcal{P}}\sum\limits_{j\in\mathcal{N}}\ell(i)p(j|i)=\frac{1}{Z}\sum\limits_{i\in\mathcal{P}}\sum\limits_{j\in\mathcal{N}}L_{ij}$
(A.30)

∎

######  Theorem 2.

Training is balanced between positive and negative examples at each iteration;
i.e. the summed gradient magnitudes of positives and negatives are equal:

$\displaystyle\sum\limits_{i\in\mathcal{P}}\mathinner{\\!\left\lvert\frac{\partial\mathcal{L}}{\partial
s_{i}}\right\rvert}=\sum\limits_{i\in\mathcal{N}}\mathinner{\\!\left\lvert\frac{\partial\mathcal{L}}{\partial
s_{i}}\right\rvert}.$ (A.31)

###### Proof.

In Eq. 5 in the paper, the gradients of a ranking based loss function are
derived as:

$\displaystyle\frac{\partial\mathcal{L}^{\mathrm{AP}}}{\partial
s_{i}}=\sum\limits_{j}\Delta x_{ij}-\sum\limits_{j}\Delta x_{ji}.$ (A.32)

such that $\Delta x_{ij}$ is the update for $x_{ij}$s and defined as $\Delta
x_{ij}=L^{*}_{ij}-L_{ij}$. Note that both $L_{ij}$ and $L^{*}_{ij}$ can be
non-zero only if $i\in\mathcal{P}$ and $j\in\mathcal{N}$ following the
definition of the primary term. Hence, the same applies to $\Delta x_{ij}$: if
$i\notin\mathcal{P}$ or $j\notin\mathcal{N}$, then $\Delta x_{ij}=0$. Then
using these facts, we can state in Eq. A.32 that if $i\in\mathcal{P}$, then
$\sum\limits_{j}\Delta x_{ji}=0$; and if $i\in\mathcal{N}$, then
$\sum\limits_{j}\Delta x_{ij}=0$. Then, we can say that, only one of the terms
is active in Eq. A.32 for positives and negatives:

$\displaystyle\frac{\partial\mathcal{L}^{\mathrm{AP}}}{\partial
s_{i}}=\underbrace{\sum\limits_{j}\Delta x_{ij}}_{\text{Active if
$i\in\mathcal{P}$}}-\underbrace{\sum\limits_{j}\Delta x_{ji}}_{\text{Active if
$i\in\mathcal{N}$}}.$ (A.33)

Considering that the value of a primary term can not be less than its target,
we have $\Delta x_{ij}\leq 0$, which implies
$\frac{\partial\mathcal{L}}{\partial s_{i}}\leq 0$. So, we can take the
absolute value outside of summation:

$\displaystyle\sum\limits_{i\in\mathcal{P}}\mathinner{\\!\left\lvert\frac{\partial\mathcal{L}}{\partial
s_{i}}\right\rvert}=\mathinner{\\!\left\lvert\sum\limits_{i\in\mathcal{P}}\frac{\partial\mathcal{L}}{\partial
s_{i}}\right\rvert},$ (A.34)

and using the fact identified in Eq. A.33 (i.e. for $i\in\mathcal{P}$,
$\frac{\partial\mathcal{L}^{\mathrm{AP}}}{\partial
s_{i}}=\sum\limits_{j}\Delta x_{ij}$):

$\displaystyle\mathinner{\\!\left\lvert\sum\limits_{i\in\mathcal{P}}\sum\limits_{j\in\mathcal{N}}\Delta
x_{ij}\right\rvert}.$ (A.35)

Simply interchanging the indices and order of summations respectively yields:

$\displaystyle\mathinner{\\!\left\lvert\sum\limits_{j\in\mathcal{P}}\sum\limits_{i\in\mathcal{N}}\Delta
x_{ji}\right\rvert}=\mathinner{\\!\left\lvert\sum\limits_{i\in\mathcal{N}}\sum\limits_{j\in\mathcal{P}}\Delta
x_{ji}\right\rvert}$ (A.36)

Note that for $i\in\mathcal{N}$,
$\frac{\partial\mathcal{L}^{\mathrm{AP}}}{\partial
s_{i}}=-\sum\limits_{j\in\mathcal{P}}\Delta x_{ji}$, and hence
$\sum\limits_{j\in\mathcal{P}}\Delta
x_{ji}=-\frac{\partial\mathcal{L}^{\mathrm{AP}}}{\partial s_{i}}$. Replacing
$\sum\limits_{j\in\mathcal{P}}\Delta x_{ji}$;

$\displaystyle\mathinner{\\!\left\lvert\sum\limits_{i\in\mathcal{N}}-\frac{\partial\mathcal{L}^{\mathrm{AP}}}{\partial
s_{i}}\right\rvert}.$ (A.37)

Since, for $i\in\mathcal{N}$,
$\frac{\partial\mathcal{L}^{\mathrm{AP}}}{\partial
s_{i}}=-\sum\limits_{j\in\mathcal{P}}\Delta x_{ji}$ is greater or equal to
zero, the proof follows:

$\displaystyle\mathinner{\\!\left\lvert\sum\limits_{i\in\mathcal{N}}-\frac{\partial\mathcal{L}^{\mathrm{AP}}}{\partial
s_{i}}\right\rvert}=\mathinner{\\!\left\lvert\sum\limits_{i\in\mathcal{N}}\frac{\partial\mathcal{L}^{\mathrm{AP}}}{\partial
s_{i}}\right\rvert}=\sum\limits_{i\in\mathcal{N}}\mathinner{\\!\left\lvert\frac{\partial\mathcal{L}^{\mathrm{AP}}}{\partial
s_{i}}\right\rvert}.$ (A.38)

∎

##  Appendix D Normalized Discounted Cumulative Gain (NDCG) Loss and Its
Gradients: Another Case Example for our Generalized Framework:

In the following we define and derive the gradients of the NDCG Loss [17]
following our generalized framework presented in Section 3 of our main paper.

The NDCG loss is defined as:

$\displaystyle\mathcal{L}^{NDCG}=1-\frac{\sum\limits_{i\in\mathcal{P}}G(i)}{G_{max}}=\frac{G_{max}-\sum\limits_{i\in\mathcal{P}}G(i)}{G_{max}}=\sum\limits_{i\in\mathcal{P}}\frac{G_{max}/|\mathcal{P}|-G(i)}{G_{max}}.$
(A.39)

Note that different from AP Loss and aLRP Loss, here $Z$ turns out to be $1$,
which makes sense since NDCG is normalized by definition.
$\ell(i)=\frac{G_{max}/|\mathcal{P}|-G(i)}{G_{max}}$ such that
$G(i)=\frac{1}{\log_{2}(1+\mathrm{rank}(i))}$ and
$G_{max}=\sum\limits_{i=1}^{|\mathcal{P}|}\log_{2}(1+i)$.

Setting $p(j|i)=\frac{H(x_{ij})}{N_{FP}(i)}$, the primary term of the NDCG
Loss is:

$\displaystyle
L_{ij}^{\mathrm{NDCG}}=\frac{G_{max}/|\mathcal{P}|-G(i)}{G_{max}}\times\frac{H(x_{ij})}{N_{FP}(i)}.$
(A.40)

When the positive example $i$ is ranked accordingly (i.e.
$\textrm{rank}(i)=1$), $G(i)=\frac{1}{\log_{2}(1+1)}=1$ and the local error in
this case is $\ell(i)=\frac{G_{max}/|\mathcal{P}|-1}{G_{max}}$. Then, the
target is

$\displaystyle{L_{ij}^{\mathrm{NDCG}}}^{*}=\frac{G_{max}/|\mathcal{P}|-1}{G_{max}}\times\frac{H(x_{ij})}{N_{FP}(i)}.$
(A.41)

Hence, the update is:

$\displaystyle\Delta x_{ij}$
$\displaystyle={L_{ij}^{\mathrm{NDCG}}}^{*}-L^{\mathrm{NDCG}}_{ij}$ (A.42)
$\displaystyle=\frac{G_{max}/|\mathcal{P}|-G(i)}{G_{max}}\times\frac{H(x_{ij})}{N_{FP}(i)}-\frac{G_{max}/|\mathcal{P}|-1}{G_{max}}\times\frac{H(x_{ij})}{N_{FP}(i)}$
(A.43)
$\displaystyle=\frac{1-G(i)}{G_{max}}\times\frac{H(x_{ij})}{N_{FP}(i)}.$
(A.44)

##  Appendix E Computing the aLRP Loss and its Gradients

This section presents the algorithm to compute aLRP Loss in detail along with
an analysis of space and time complexity. For better understanding, bold font
denotes multi-dimensional data structures (which can be implemented by
vectors, matrices or tensors). Algorithm A.1 describes the steps to compute
the aLRP Loss along with the gradients for a given mini-batch.

Description of the inputs: $\mathbf{S}$ is the raw output of the
classification branch, namely logits. For localisation, as done by IoU-based
localisation losses [29, 25], the raw localisation outputs need to be
converted to the boxes, which are denoted by $\mathbf{B}$. We assume that
$\mathbf{M}$ stores $-1$ for ignored anchors and $0$ for negative anchors. For
positive anchors, the index of the ground truth (i.e.
$\\{1,...,|\mathbf{\hat{B}}|\\}$, where $\mathbf{\hat{B}}$ is a list of ground
boxes for the mini-batch) is stored. Hence, we can find the corresponding
ground truth for a positive anchor only by using $\mathbf{M}$. $\delta$ is the
smoothness of the piecewise linear function defined in Eq. A.45 and set to $1$
following AP Loss. We use the self-balance ratio,
$\frac{\mathcal{L}^{\mathrm{aLRP}}}{\mathcal{L}^{\mathrm{aLRP}}_{cls}}$, by
averaging over its values from the previous epoch.

Part 1: Initializing Variables: Lines 2-10 aim to initialize the necessary
data from the inputs. While this part is obvious, please note that line 8
determines a threshold to select the relevant negative outputs. This is simply
due to Eq. A.45 and the gradients of these negative examples with scores under
this threshold are zero. Therefore, for the sake of time and space efficiency,
they are ignored.

Part 2: Computing Unnormalized Localisation Errors: Lines 12-14 compute
unnormalized localisation error on each positive example. Line 12 simply finds
the localisation error of each positive example and line 13 sorts these errors
with respect to their scores in descending order, and Line 14 computes the
cumulative sum of the sorted errors with $\mathrm{cumsum}$ function. In such a
way, the example with the larger scores contributes to the error computed for
each positive anchor with smaller scores. Note that while computing the
nominator of the $\mathcal{L}^{\mathrm{aLRP}}_{loc}$, we employ the step
function (not the piecewise linear function), since we can safely use
backpropagation.

Part 3: Computing Gradient and Error Contribution from Each Positive: Lines
16-32 compute the gradient and error contribution from each positive example.
To do so, Line 16 initializes necessary data structures. Among these data
structures, while $\mathbf{\mathcal{L}^{\mathrm{LRP}}_{loc}}$,
$\mathbf{\mathcal{L}^{\mathrm{LRP}}_{cls}}$ and
$\frac{\partial\mathcal{L}^{\mathrm{aLRP}}}{\partial\mathbf{S_{+}}}$ are all
with size $|\mathcal{P}|$,
$\frac{\partial\mathcal{L}^{\mathrm{aLRP}}}{\partial\mathbf{S_{-}}}$ has size
$|\hat{\mathcal{N}}|$, where $\hat{\mathcal{N}}$ is the number of negative
examples after ignoring the ones with scores less than $\tau$ in Line 8, and
obviously $|\hat{\mathcal{N}}|\leq|\mathcal{N}|$. The loop iterates over each
positive example by computing LRP values and gradients since aLRP is defined
as the average LRP values over positives (see Eq. 9 in the paper). Lines 18-22
computes the relation between the corresponding positive with positives and
relevant negatives, each of which requires the difference transformation
followed by piecewise linear function:

$\displaystyle H(x)=\begin{cases}0,&x<-\delta\\\
\frac{x}{2\delta}+0.5,&-\delta\leq x\leq\delta\\\ 1,&\delta<x.\end{cases}$
(A.45)

Then, using these relations, lines 23-25 compute the rank of the $i$th
examples within positive examples, number of negative examples with larger
scores (i.e. false positives) and rank of the example. Lines 26 and 27 compute
the aLRP classification and localisation errors on the corresponding positive
example. Note that to have a consistent denominator for total aLRP, we use
$\mathrm{rank}$ to normalize both of the components. Lines 28-30 compute the
gradients. While the local error is enough to determine the unnormalized
gradient of a positive example, the gradient of a negative example is
accumulated through the loop.

Part 4: Computing the aLRP Loss and Gradients: Lines 34-40 simply derive the
final aLRP value by averaging over LRP values (lines 34-36), normalize the
gradients (lines 37-38) and compute gradients wrt the boxes (line 39) and
applies self balancing (line 40).

Time Complexity:

  * •

First 16 lines of Algorithm A.1 require time between
$\mathcal{O}(|\mathcal{P}|)$ and $\mathcal{O}(|\mathcal{N}|)$. Since for the
object detection problem, the number of negative examples is quite larger than
number of positive anchors (i.e. $|\mathcal{P}|<<|\mathcal{N}|$), we can
conclude that the time complexity of first 13 lines is
$\mathcal{O}(|\mathcal{N}|)$.

  * •

The bottleneck of the algorithm is the loop on lines 17-32. The loop iterates
over each positive example, and in each iteration while lines 21, 24 and 30
are executed for relevant negative examples, the rest of the lines is executed
for positive examples. Hence the number of operations for each iteration is
$\max(|\mathcal{P}|,|\hat{\mathcal{N}}|)$ (i.e. number of relevant negatives,
see lines 8-9), and overall these lines require
$\mathcal{O}(|\mathcal{P}|\times\max(|\mathcal{P}|,|\hat{\mathcal{N}}|))$.
Note that, while in the early training epochs,
$|\hat{\mathcal{N}}|\approx|\mathcal{N}|$, as the training proceeds, the
classifier tends to distinguish positive examples from negative examples very
well, and $|\hat{\mathcal{N}}|$ significantly decreases implying faster mini-
batch iterations.

  * •

The remaining lines between 26-33 again require time between
$\mathcal{O}(|\mathcal{P}|)$ and $\mathcal{O}(|\mathcal{N}|)$.

Hence, we conclude that the time complexity of Algorithm A.1 is
$\mathcal{O}(|\mathcal{N}|+|\mathcal{P}|\times\max(|\mathcal{P}|,|\hat{\mathcal{N}}|))$.

Compared to AP Loss;

  * •

aLRP Loss includes an extra computation of aLRP localisation component (i.e.
lines 12-14, 27. Each of these lines requires $\mathcal{O}(|\mathcal{P}|)$)

  * •

aLRP Loss includes an additional summation while computing the gradients with
respect to the scores of the positive examples in line 29 requiring
$\mathcal{O}(|\mathcal{P}|^{2})$

  * •

aLRP Loss discards interpolation (i.e. using interpolated AP curve), which can
take up to $\mathcal{O}(|\mathcal{P}|\times|\hat{\mathcal{N}}|)$.

Algorithm A.1 The algorithm to compute aLRP Loss for a mini-batch.

Input: $\mathbf{S}$: Logit predictions of the classifier for each anchor,

$\mathbf{B}$: Box predictions of the localization branch from each anchor,

$\mathbf{\hat{B}}$: Ground truth (GT) boxes,

$\mathbf{M}$: Matching of the anchors with the GT boxes.

$\delta$: Smoothness of the piece-wise linear function ($\delta=1$ by
default).

$w_{ASB}$: ASB weight, computed using
$\frac{\mathcal{L}^{\mathrm{aLRP}}}{\mathcal{L}^{\mathrm{aLRP}}_{cls}}$ values
from previous epoch.

Output: $\mathcal{L}^{\mathrm{aLRP}}$: aLRP loss,
$\frac{\partial\mathcal{L}^{\mathrm{aLRP}}}{\partial\mathbf{S}}$: Gradients
wrt logits, $\frac{\partial\mathcal{L}^{\mathrm{aLRP}}}{\partial\mathbf{B}}$:
Gradients wrt boxes.

1:// Part 1: Initializing Variables

2:$\mathbf{idx}_{+}\mathrel{\mathop{\mathchar 58\relax}}=$ The indices of
$\mathbf{M}$ where $\mathbf{M}>0$.

3:$\mathbf{M}_{+}\mathrel{\mathop{\mathchar 58\relax}}=$ The values of
$\mathbf{M}$ where $\mathbf{M}>0$.

4:$\mathbf{B_{+}}\mathrel{\mathop{\mathchar 58\relax}}=$ The values of
$\mathbf{B}$ at indices $\mathbf{idx}_{+}$.

5:$\mathbf{S_{+}}\mathrel{\mathop{\mathchar 58\relax}}=$ The values of
$\mathbf{S}$ at indices $\mathbf{idx}_{+}$.

6:$\mathbf{idx^{sorted}_{+}}\mathrel{\mathop{\mathchar 58\relax}}=$ The
indices of $\mathbf{S_{+}}$ once it is sorted in descending order.

7:$\mathbf{S^{sorted}_{+}}\mathrel{\mathop{\mathchar 58\relax}}=$ The values
of $\mathbf{S_{+}}$ when ordered according to $\mathbf{idx^{sorted}_{+}}$.

8:$\tau=\min(\mathbf{S_{+}})-\delta$.

9:$\mathbf{idx}_{-}\mathrel{\mathop{\mathchar 58\relax}}=$ The indices of
$\mathbf{M}$ where $\mathbf{M}=0$ and $s_{j}\geq\tau$ (i.e. relevant negatives
only).

10:$\mathbf{S_{-}}\mathrel{\mathop{\mathchar 58\relax}}=$ The values of
$\mathbf{S}$ at indices $\mathbf{idx}_{-}$.

11:// Part 2: Computing Unnormalized Localisation Errors

12:$\mathbf{\mathcal{E}_{Loc}}=\frac{1-\mathrm{IoU}(\mathbf{B_{+}},\mathbf{\hat{B}_{+}})}{1-\tau}$.
(or
$\mathbf{\mathcal{E}_{Loc}}=\frac{(1-\mathrm{GIoU}(\mathbf{B_{+}},\mathbf{\hat{B}_{+}}))/2}{1-\tau}$
for $\mathrm{GIoU}$ Loss [25].)

13:$\mathbf{\mathcal{E}_{Loc}^{sorted}}\mathrel{\mathop{\mathchar 58\relax}}=$
The values of $\mathbf{\mathcal{E}_{Loc}}$ when ordered according to
$\mathbf{idx^{sorted}_{+}}$.

14:$\mathbf{\mathcal{E}_{Loc}^{cumsum}}=\mathrm{cumsum}(\mathbf{\mathcal{E}_{Loc}^{sorted}})$

15:// Part 3: Computing Gradient and Error Contribution from Each Positive

16:Initialize , $\mathbf{\mathcal{L}^{\mathrm{LRP}}_{loc}}$,
$\mathbf{\mathcal{L}^{\mathrm{LRP}}_{cls}}$,
$\frac{\partial\mathcal{L}^{\mathrm{aLRP}}}{\partial\mathbf{S_{+}}}$ and
$\frac{\partial\mathcal{L}^{\mathrm{aLRP}}}{\partial\mathbf{S_{-}}}$.

17:for each $s_{i}\in\mathbf{S^{sorted}_{+}}$ do

18: $\mathbf{X_{+}}\mathrel{\mathop{\mathchar 58\relax}}=$ Difference
transform of $s_{i}$ with the logit of each positive example.

19: $\mathbf{R_{+}}\mathrel{\mathop{\mathchar 58\relax}}=$ The relation of
$i\in\mathcal{P}$ with each $j\in\mathcal{P}$ using Eq. A.45 with input
$\mathbf{X_{+}}$.

20: $\mathbf{R_{+}}[i]=0$

21: $\mathbf{X_{-}}\mathrel{\mathop{\mathchar 58\relax}}=$ Difference
transform of $s_{i}$ with the logit of each negative example.

22: $\mathbf{R_{-}}\mathrel{\mathop{\mathchar 58\relax}}=$ The relation of
$i\in\mathcal{P}$ with each $j\in\mathcal{N}$ using Eq. A.45 with input
$\mathbf{X_{+}}$.

23: $\mathrm{rank}_{+}=1+\mathrm{sum}(\mathbf{R_{+}})$

24: $\mathrm{FP}=\mathrm{sum}(\mathbf{R_{-}})$

25: $\mathrm{rank}=\mathrm{rank}_{+}+\mathrm{FP}$

26: $\mathbf{\mathcal{L}^{\mathrm{LRP}}_{cls}}[i]=\mathrm{FP}/\mathrm{rank}$

27:
$\mathbf{\mathcal{L}^{\mathrm{LRP}}_{loc}}[i]=\mathbf{\mathcal{E}_{Loc}^{cumsum}}[i]/\mathrm{rank}$

28: if $\mathrm{FP}\geq\epsilon$ then //For stability set $\epsilon$ to a
small value (e.g. $1e-5$)

29:
$\frac{\partial\mathcal{L}^{\mathrm{aLRP}}}{\partial\mathbf{S_{+}}}[i]=-\left(\mathrm{FP}+\sum\limits_{i\in
P}\mathbf{R_{+}}[i]\times\mathcal{E}_{Loc}^{cumsum}[i]\right)/\mathrm{rank}$

30:
$\frac{\partial\mathcal{L}^{\mathrm{aLRP}}}{\partial\mathbf{S_{-}}}+=\left(-\mathbf{\frac{\partial\mathcal{L}^{\mathrm{aLRP}}}{\partial
S_{+}}}[i]\times\frac{\mathbf{R_{-}}}{\mathrm{FP}}\right)$

31: end if

32:end for

33:// Part 4: Computing the aLRP Loss and Gradients

34:$\mathcal{L}^{\mathrm{aLRP}}_{cls}=\mathrm{mean}(\mathbf{\mathcal{L}^{\mathrm{LRP}}_{cls}})$

35:$\mathcal{L}^{\mathrm{aLRP}}_{loc}=\mathrm{mean}(\mathbf{\mathcal{L}^{\mathrm{LRP}}_{loc}})$

36:$\mathcal{L}^{\mathrm{aLRP}}=\mathcal{L}^{\mathrm{aLRP}}_{cls}+\mathcal{L}^{\mathrm{aLRP}}_{loc}$

37:Place $\frac{\partial\mathcal{L}^{\mathrm{aLRP}}}{\partial\mathbf{S_{+}}}$
and $\frac{\partial\mathcal{L}^{\mathrm{aLRP}}}{\partial\mathbf{S_{-}}}$ into
$\mathbf{\frac{\partial\mathcal{L}^{\mathrm{aLRP}}}{\partial S}}$ also by
setting the gradients of remaining examples to $0$.

38:$\mathbf{\frac{\partial\mathcal{L}^{\mathrm{aLRP}}}{\partial
S}}/=|\mathcal{P}|$

39:Compute $\frac{\partial\mathcal{L}^{\mathrm{aLRP}}}{\partial\mathbf{B}}$
(possibly using autograd property of a deep learning library or refer to the
supp. mat. of [25] for the gradients of GIoU and IoU Losses.

40:$\frac{\partial\mathcal{L}^{\mathrm{aLRP}}_{loc}}{\partial\mathbf{B}}\times=w_{ASB}$

41:return $\frac{\partial\mathcal{L}^{\mathrm{aLRP}}}{\partial\mathbf{S}}$,
$\frac{\partial\mathcal{L}^{\mathrm{aLRP}}}{\mathbf{\partial\mathbf{B}}}$ and
$\mathcal{L}^{\mathrm{aLRP}}$.

Space Complexity: Algorithm A.1 does not require any data structure larger
than network outputs (i.e. $\mathbf{B}$, $\mathbf{S}$). Then, we can safely
conclude that the space complexity is similar to all of the common loss
functions that is $\mathcal{O}(|\mathbf{S}|)$.

##  Appendix F Details of the aLRP Loss

This section provides details for the aLRP Loss.

###  F.1 A Soft Sampling Perspective for aLRP Localisation Component

In sampling methods, the contribution ($w_{i}$) of the $i$th bounding box to
the loss function is adjusted as follows:

$\mathcal{L}=\sum\limits_{i\in\mathcal{P}\cup\mathcal{N}}w_{i}\mathcal{L}(i),$
(A.46)

where $\mathcal{L}(i)$ is the loss of the $i$th example. Hard and soft
sampling approaches differ on the possible values of $w_{i}$. For the hard
sampling approaches, $w_{i}\in\\{0,1\\}$, thus a BB is either selected or
discarded. For soft sampling approaches, $w_{i}\in[0,1]$, i.e. the
contribution of a sample is adjusted with a weight and each BB is somehow
included in training. While this perspective is quite common to train the
classification branch [3, 14]; the localisation branch is conventionally
trained by hard sampling with some exceptions (e.g. CARL [3] sets
$w_{i}=s_{i}$ where $s_{i}$ is the classification score).

Here, we show that, in fact, what aLRP localisation component does is soft
sampling. To see this, first let us recall the definition of the localisation
component:

$\displaystyle\mathcal{L}^{aLRP}_{loc}$
$\displaystyle=\frac{1}{|\mathcal{P}|}\sum\limits_{i\in\mathcal{P}}\frac{1}{\mathrm{rank}(i)}\left(\mathcal{E}_{loc}(i)+\sum\limits_{k\in\mathcal{P},k\neq
i}\mathcal{E}_{loc}(k)H(x_{ik})\right),$ (A.47)

which is differentiable with respect to the box parameters as discussed in the
paper. Then, setting $\mathcal{L}(i)=\mathcal{E}_{loc}(i)$ in Eq. A.46 and
accordingly taking Eq. A.47 in $\mathcal{E}_{loc}(i)$ paranthesis, the weights
of the positive examples (i.e. $w_{i}=0$ for negatives for the localisation
component) are:

$\displaystyle
w_{i}=\frac{1}{|\mathcal{P}|}\left(\left({\sum\limits_{k\in\mathcal{P},k\neq
i}}\frac{H(x_{ik})}{\mathrm{rank}(k)}\right)+\frac{1}{\mathrm{rank}(i)}\right).$
(A.48)

###  F.2 The Relation between aLRP Loss Value and Total Gradient Magnitudes

Here, we identify the relation between the loss value and the total magnitudes
of the gradients following the generalized framework due to the fact that it
is a basis for our self-balancing strategy introduced in Section 4.2 as
follows:

$\displaystyle\frac{1}{|\mathcal{P}|}\sum_{i\in\mathcal{P}}\mathinner{\\!\left\lvert\frac{\partial\mathcal{L}}{\partial
s_{i}}\right\rvert}=\frac{1}{|\mathcal{P}|}\sum_{i\in\mathcal{N}}\mathinner{\\!\left\lvert\frac{\partial\mathcal{L}}{\partial
s_{i}}\right\rvert}\approx\mathcal{L}^{\mathrm{aLRP}}.$ (A.49)

Since we showed in Section C that
$\frac{1}{|\mathcal{P}|}\sum_{i\in\mathcal{P}}\mathinner{\\!\left\lvert\frac{\partial\mathcal{L}}{\partial
s_{i}}\right\rvert}=\frac{1}{|\mathcal{P}|}\sum_{i\in\mathcal{N}}\mathinner{\\!\left\lvert\frac{\partial\mathcal{L}}{\partial
s_{i}}\right\rvert}$, here we show that the loss value is approximated by the
total magnitude of gradients. Recall from Eq. A.35 that total gradients of the
positives can be expressed as:

$\displaystyle\frac{1}{|\mathcal{P}|}\sum_{i\in\mathcal{P}}\mathinner{\\!\left\lvert\frac{\partial\mathcal{L}}{\partial
s_{i}}\right\rvert}=\mathinner{\\!\left\lvert\sum\limits_{i\in\mathcal{P}}\sum\limits_{j\in\mathcal{N}}\Delta
x_{ij}\right\rvert}.$ (A.50)

Since $\Delta x_{ij}\leq 0$, we can discard the absolute value by multiplying
it by $-1$:

$\displaystyle-\sum\limits_{i\in\mathcal{P}}\sum\limits_{j\in\mathcal{N}}\Delta
x_{ij}.$ (A.51)

Replacing the definition of the $\Delta x_{ij}$ with $L_{ij}^{*}-L_{ij}$
yields:

$\displaystyle-\sum\limits_{i\in\mathcal{P}}\sum\limits_{j\in\mathcal{N}}(L_{ij}^{*}-L_{ij})=-\left(\sum\limits_{i\in\mathcal{P}}\sum\limits_{j\in\mathcal{N}}L_{ij}^{*}-\sum\limits_{i\in\mathcal{P}}\sum\limits_{j\in\mathcal{N}}L_{ij}\right)=\sum\limits_{i\in\mathcal{P}}\sum\limits_{j\in\mathcal{N}}L_{ij}-\sum\limits_{i\in\mathcal{P}}\sum\limits_{j\in\mathcal{N}}L_{ij}^{*}$
(A.52)

Using Theorem 1, the first part (i.e
$\sum\limits_{i\in\mathcal{P}}\sum\limits_{j\in\mathcal{N}}L_{ij}$) yields the
loss value, $\mathcal{L}$. Hence:

$\displaystyle\mathcal{L}-\sum\limits_{i\in\mathcal{P}}\sum\limits_{j\in\mathcal{N}}L_{ij}^{*}$
(A.53)

Then, the difference between the total gradients of positives (or negatives,
since they are equal – see Theorem 2) and the loss values itself is the sum of
the targets:

$\displaystyle\mathcal{L}-\frac{1}{|\mathcal{P}|}\sum_{i\in\mathcal{P}}\mathinner{\\!\left\lvert\frac{\partial\mathcal{L}}{\partial
s_{i}}\right\rvert}=\sum\limits_{i\in\mathcal{P}}\sum\limits_{j\in\mathcal{N}}L_{ij}^{*}.$
(A.54)

Compared to the primary terms, the targets are very small values (if not $0$).
For example, for AP Loss ${L_{ij}^{\mathrm{AP}}}^{*}=0$, and losses and the
gradient sums are equal:
$\mathcal{L}=\frac{1}{|\mathcal{P}|}\sum_{i\in\mathcal{P}}\mathinner{\\!\left\lvert\frac{\partial\mathcal{L}}{\partial
s_{i}}\right\rvert}$.

As for the aLRP Loss, the target of a primary term is
$\frac{\mathcal{E}_{loc}(i)}{\mathrm{rank}(i)}\frac{H(x_{ij})}{N_{FP}(i)}$,
hence if $H(x_{ij})=0$, then the target is also $0$. Else if $H(x_{ij})=1$,
then it implies that there are some negative examples with larger scores, and
$\mathrm{rank}(i)$ and $N_{FP}(i)$ are getting larger depending on these
number of negative examples, which causes the denominator to grow, and hence
yielding a small target as well. Then ignoring this term, we conclude that:

$\displaystyle\frac{1}{|\mathcal{P}|}\sum_{i\in\mathcal{P}}\mathinner{\\!\left\lvert\frac{\partial\mathcal{L}}{\partial
s_{i}}\right\rvert}=\frac{1}{|\mathcal{N}|}\sum_{i\in\mathcal{N}}\mathinner{\\!\left\lvert\frac{\partial\mathcal{L}}{\partial
s_{i}}\right\rvert}\approx\mathcal{L}^{\mathrm{aLRP}}$ (A.55)

##  Appendix G Using a Wrong Target for the Primary Term in the Error-driven
Update Rule

Figure A.7: (left) The rate of the total gradient magnitudes of negatives to
positives. (right) Loss values.

As discussed in our paper (Section 4.1, Equation 13), ${L_{ij}}^{*}$, the
target value of the primary term ${L_{ij}}$ is non-zero due to the
localization error. It is easy to overlook this fact and assume that the
target is zero. Fig. A.7 presents this case where ${L_{ij}}^{*}$ is set to 0
(i.e. minimum value of aLRP). In such a case, the training continues properly,
similar to that of the correct case, up to a point and then diverges. Note
that this occurs when the positives start to be ranked properly but are still
assigned gradients since ${L_{ij}}^{*}-{L_{ij}}\neq 0$ due to the nonzero
localization error. This causes
$\sum\limits_{i\in\mathcal{P}}\mathinner{\\!\left\lvert\frac{\partial\mathcal{L}}{\partial
s_{i}}\right\rvert}>\sum\limits_{i\in\mathcal{N}}\mathinner{\\!\left\lvert\frac{\partial\mathcal{L}}{\partial
s_{i}}\right\rvert}$, violating Theorem 2 (compare min-rate and max-rate in
Fig. A.7). Therefore, assigning proper targets as indicated in Section 3 in
the paper is crucial for balanced training.