# Rethinking the Localization in Weakly Supervised Object Localization

Rui Xu  School of Computer Science,
Wuhan University,WuhanChina<EMAIL_ADDRESS>,  Yong Luo  Wuhan University &
Hubei Luojia Laboratory,WuhanChina<EMAIL_ADDRESS>,  Han Hu  Beijing
Institute of Technology,BeijingChina<EMAIL_ADDRESS>,  Bo Du  Wuhan University
&
Hubei Luojia Laboratory,WuhanChina<EMAIL_ADDRESS>,  Jialie Shen  City,
University of London,LondonUnited Kingdom<EMAIL_ADDRESS>and  Yonggang Wen
Nanyang Technological University,SingaporeSingapore<EMAIL_ADDRESS>
(2023)

###### Abstract.

Weakly supervised object localization (WSOL) is one of the most popular and
challenging tasks in computer vision. This task is to localize the objects in
the images given only the image-level supervision. Recently, dividing WSOL
into two parts (class-agnostic object localization and object classification)
has become the state-of-the-art pipeline for this task. However, existing
solutions under this pipeline usually suffer from the following drawbacks: 1)
they are not flexible since they can only localize one object for each image
due to the adopted single-class regression (SCR) for localization; 2) the
generated pseudo bounding boxes may be noisy, but the negative impact of such
noise is not well addressed. To remedy these drawbacks, we first propose to
replace SCR with a binary-class detector (BCD) for localizing multiple
objects, where the detector is trained by discriminating the foreground and
background. Then we design a weighted entropy (WE) loss using the unlabeled
data to reduce the negative impact of noisy bounding boxes. Extensive
experiments on the popular CUB-200-2011 and ImageNet-1K datasets demonstrate
the effectiveness of our method.

Weakly supervised, object localization, binary-class detector, weighted
entropy, noisy label

††copyright: acmcopyright††journalyear: 2023††copyright:
acmlicensed††conference: Proceedings of the 31st ACM International Conference
on Multimedia; October 29-November 3, 2023; Ottawa, ON, Canada††booktitle:
Proceedings of the 31st ACM International Conference on Multimedia (MM ’23),
October 29-November 3, 2023, Ottawa, ON, Canada††price: 15.00††doi:
10.1145/3581783.3611959††isbn: 979-8-4007-0108-5/23/10††ccs: Computing
methodologies Object detection

##  1\. Introduction

The development of deep neural networks has greatly advanced a wide range of
multimedia applications. Yet the training of deep learning models usually
requires a large amount of accurately annotated data. The collection of
labeled data could be extremely labor intensive and time consuming, especially
when fine-grained annotation is required, e.g., the object-level labels for
localization and pixel-level labels for segmentation. In order to mitigate
this issue, there is an increasing interest on designing weakly supervised
approaches. In this paper, we mainly focus on the task of weakly supervised
object localization (WSOL), which aims to localize the objects given the
image-level labels. It is one of the most fundamental and challenging tasks in
computer vision, and can facilitate many multimedia applications.

Figure 1. (a) The localization method based on SCR in current separated
localization-classification WSOL solution, which provides only one output
bounding box; (b) Our BCD predicts multiple bounding boxes together with
highly confident foreground probabilities; t denotes the predicted locations
of the objects and p denotes the predicted foreground probabilities.

There have been dozens of WSOL approaches during the past decades. Recently,
the separated localization-classification pipeline (Lu et al., 2020; Zhang et
al., 2020a; Guo et al., 2021; Xie et al., 2022), which regards WSOL as two
separate sub-tasks: class-agnostic object localization and object
classification, achieves the state-of-the-art (SOTA) performance, and results
in paradigm shift from the previous unified localization-classification
methods (Zhou et al., 2016; Choe and Shim, 2019; Meng et al., 2021; Zhang et
al., 2020c; Pan et al., 2021; Xie et al., 2021) to the separated ones for
WSOL. The object classification is performed using the off-the-shelf
classification models. As for the class-agnostic object localization, pseudo
ground-truth bounding boxes are usually first generated (Zhang et al., 2020a;
Guo et al., 2021; Xie et al., 2022). For example, in PSOL (Zhang et al.,
2020a), the deep descriptor transformation (DDT) (Wei et al., 2019) is
directly adopted to generate the bounding boxes. The SLT-Net (Guo et al.,
2021) dynamically improves the accuracy of the pseudo bounding boxes by
strengthening the learning tolerance to the semantic errors and image
transformations. The C2AM (Xie et al., 2022) proposes cross-image foreground-
background contrastive learning to generate accurate pseudo bounding boxes
without any supervision. Then a single-class regression (SCR) (Sermanet et
al., 2014) is trained using these generated bounding boxes.

Despite the SOTA performance of the separated localization-classification
pipeline, a major drawback is the adopted simple SCR (Sermanet et al., 2014)
for localization. SCR (Sermanet et al., 2014) is only able to provide one
output bounding box for an image, and thus inadequate and inflexible when
coping with images containing multiple objects, which are common in the real-
world applications. In addition, the generated pseudo bounding boxes may be
inaccurate or even completely wrong, but such noisy labels are directly
utilized for training without any careful consideration.

To overcome these drawbacks, we propose a novel WSOL method termed Weighted
ENtropy guided binary-class Detector (WEND). In particular, we first propose
to replace SCR (Sermanet et al., 2014) with a binary-class detector (BCD),
which can naturally output multiple bounding boxes, and be trained in a binary
classification manner by discriminating the foreground and background. This
can improve both the flexibility and accuracy since any competitive detectors
could be incorporated. Then to alleviate the negative impacts of the noisy
bounding boxes on the training of detector, we further propose a weighted
entropy (WE) loss by making use of the large amounts of unlabeled data. The
entropy minimization is able to reduce the uncertainty when discriminating the
foreground and background. Considering that the background parts are usually
much more than the foreground objects, we re-weight the entropy loss to down-
weight the overly confident background and make the detector pay more
attention on the less confident foreground. As shown in Figure 1, compared
with the localization approach adopted in the current separated localization-
classification pipeline, our WEND is able to predict multiple bounding boxes,
and the predictions are of low entropy, i.e., high confidence.

Our contributions can be summarized as follows:

  * •

We rethink and analyze the localization in the SOTA separated localization-
classification pipeline for WSOL, and propose a novel method to enhance the
localization performance.

  * •

We propose to utilize a detector trained in a binary-classification manner to
localize multiple objects. The detector is more flexible and reliable than the
single-class regression, which can only localize one object.

  * •

We design a weighted entropy loss as an additional unsupervised constraint to
reduce the negative effect of noisy pseudo bounding boxes and deal with the
foreground-background imbalance in the training of the binary-class detector.

Extensive experiments are conducted on two widely used WSOL datasets:
CUB-200-2011 (Wah et al., 2011) and ImageNet-1K (Russakovsky et al., 2015).
The results demonstrate that our method significantly outperforms the SOTA
WSOL approaches.

##  2\. Related Works

In weakly supervised object localization (WSOL), the localization of objects
is usually conducted under the supervision of image-level labels. A popular
solution for WSOL is to utilize CAM (Zhou et al., 2016), which generates class
activation maps by aggregating deep feature maps using the final fully
connected layer of the classifier. However, the classifier mainly focuses on
the discriminative parts of the objects, and this may lead to inaccurate
localization. To tackle this issue, many approaches (Choe and Shim, 2019; Meng
et al., 2021; Zhang et al., 2020c; Pan et al., 2021) are proposed by, such as,
hiding the most discriminative part and highlighting the informative region of
deep feature maps (Choe and Shim, 2019). While these approaches mostly utilize
the deep feature maps, the low level features are leveraged in (Xie et al.,
2021) for class activation map generation. Some other works conducted WSOL
under weaker supervision. For example, SCDA (Wei et al., 2017), MO (Zhang et
al., 2019), DDT (Wei et al., 2019), and PSY (Baek et al., 2020) just utilize
the models pre-trained on ImageNet (Russakovsky et al., 2015). In (Su et al.,
2022), the authors design a siamese network to encode the transformed image
pairs, and propose a joint graph partition mechanism to enhance the object co-
occurrent regions. There also exist some completely unsupervised approaches
(Cho et al., 2015; Zhang et al., 2018; Chen et al., 2019; Zhao et al., 2021)
or some weakly-supervised approaches without utilizing pre-trained weights,
but the performance are usually unsatisfactory.

Different from the approaches that unifies the classification and localization
in a single framework, several recent works divide WSOL into two sub-parts:
object classification and class-agnostic object localization (Lu et al., 2020;
Zhang et al., 2020a; Guo et al., 2021; Xie et al., 2022), where some ready-
made classifiers are directly employed for classification. In regard to the
localization, GC-Net (Lu et al., 2020) trains a regression using the geometry
constraint and a pre-trained classifier to distinguish the obtained foreground
and background. PSOL (Zhang et al., 2020a) directly generates pseudo ground-
truth bounding boxes using DDT (Wei et al., 2019) for training. In (Guo et
al., 2021), the authors propose to strengthen the network’s tolerance to the
classification errors and image transformations for consistently optimizing
the generated pseudo bounding boxes. While the image-level labels are involved
in these approaches, C2AM (Xie et al., 2022) uses unsupervised contrastive
learning to discriminate the cross-image foreground objects and backgrounds
for pseudo bounding box generation. These approaches have been demonstrated to
usually achieve very competitive performance, and the separated localization-
classification pipeline has become a new paradigm.

Unlike the regressor commonly used in WSOL tasks which merely and directly
regresses the four coordinates of the object location, the object detector
regresses the location of the object from multiple references that are close
to the object. In this work, we regard modern detectors as two types, NMS-
based detectors and NMS-free detectors. Their main difference exists in that
during training how they match the predictions with the ground truths. NMS-
based detectors (Kong et al., 2020; Zhang et al., 2020b; Tian et al., 2019;
Law and Deng, 2018; Lin et al., 2017; Ren et al., 2015; Xu et al., 2022a)
first generate a large number of predetermined references (anchors or points)
and label each of them as positive or negative based on its overlap with the
ground truths. This kind of mechanism cannot prevent an object from being
predicted by multiple references, leading to duplicates and requiring complex
post-processing. NMS-free detectors (Roh et al., 2022; Zhu et al., 2021; Dai
et al., 2021; Sun et al., 2021; Carion et al., 2020) utilize the attention
mechanism to capture the pairwise relations between the references, and
perform one-to-one matching between them and the ground truths, thus avoiding
duplicates.

Figure 2. Overview of our Weighted ENtropy guided binary-class Detector (WEND)
for localization. In the training phase, a multi-output detector (such as
RPN+R-CNN) is trained in a binary-classification manner by discriminating the
foreground and the background, where the class-agnostic pseudo bounding boxes
generated by existing approaches (such as C2AM) are utilized as the ground-
truth labels. The output positive (blue) and negative (red) probabilities
$\\{p_{i}\\}$ and locations $\\{t_{i}\\}$, (as well as the localization
quality $\\{c_{i}\\}$ if possible) are compared with the pseudo bounding boxes
for training. In addition, an unsupervised weighted entropy (WE) constraint is
applied to the classifier of the binary-class detector (BCD) for dealing with
the negative impact of the noisy pseudo labels. By enforcing the weighted
entropy minimization on the predicted probabilities, the foregrounds can be
identified with higher confidence, and the bounding boxes can be further
refined.

##  3\. Methodology

In this section, we analyze the localization in the separated localization-
classification framework (Lu et al., 2020; Zhang et al., 2020a; Guo et al.,
2021; Xie et al., 2022) for weakly supervised object localization (WSOL). Then
we introduce our proposed Weighted ENtropy guided binary-class Detector (WEND)
to improve the localization performance.

###  3.1. Analysis

The highly effective separated localization-classification pipeline treats
WSOL as two sub-tasks: object classification and class-agnostic object
localization. For the former sub-task, existing classification model is
directly adopted. In regard to the latter localization sub-task, pseudo
ground-truth bounding boxes are first generated. Afterwards, the single-class
regression (SCR) (Sermanet et al., 2014) is performed using the training
images and the corresponding generated pseudo bounding boxes.

Although this separated localization-classification pipeline achieves the SOTA
performance in WSOL, it is limited in that:

  * •

The simple localization model SCR (Sermanet et al., 2014) can only provide one
bounding box output per image. For datasets such as the ImageNet-1K
(Russakovsky et al., 2015), some images may have more than one ground-truth
bounding box. Therefore, the single output may be inappropriate.

  * •

It is inevitable that some generated pseudo ground-truth bounding boxes, e.g.,
by utilizing either DDT (Wei et al., 2019) or C2AM (Xie et al., 2022), are not
optimal or even completely wrong, while there is no design in the simple SCR
(Sermanet et al., 2014) to deal with such noisy labels.

Therefore, there is still much room for improvement in the separated
localization-classification pipeline.

###  3.2. The Proposed Method

To deal with the aforementioned two drawbacks, we propose the Weighted ENtropy
guided binary-class Detector (WEND). Following the separated localization-
classification pipeline, our WEND consists of two separate branches, where the
object classification branch directly adopts an existing model. Our class-
agnostic object localization branch is illustrated in Figure 2, where we first
adopt the high-performing C2AM (Xie et al., 2022) to generate pseudo bounding
boxes. Then for the localization, we propose to employ a binary-class detector
(BCD) and design an auxiliary unsupervised weighted entropy (WE) loss. The
BCD, as a replacement of the SCR, is able to output multiple localization
boxes, and trained by discriminating the foreground objects from the
backgrounds. Any existing detector can be incorporated in BCD. In order to
handle the noisy labels, we propose a WE loss to reduce the uncertainty in the
prediction, and the imbalance between the foregrounds and backgrounds are
considered by assigning different weights. More details are depicted as
follows.

####  3.2.1. Binary-Class Detector

After generating pseudo bounding boxes, the object localization sub-task
becomes a pseudo supervised problem, where the separated localization-
classification pipeline performs SCR (Sermanet et al., 2014) on these pseudo
bounding boxes and suffers the above-mentioned weaknesses. Here we address the
problem of inflexible outputs by simply introducing a detector, which can be
any modern object detectors. The key is how to train such detector given the
pseudo bounding boxes.

In order to realize the class-agnostic localization, we regard the generated
bounding boxes as pseudo ground-truth boxes and assign the same category of
the foregrounds to them. In this way, the localization can be realized through
the standard detection steps. The feature map of the input image is first
extracted using a conventional CNN backbone, which can be the VGG (Simonyan
and Zisserman, 2015), ResNet (He et al., 2016), Inception (Szegedy et al.,
2016), or GoogLeNet (Szegedy et al., 2015). Then in our work, the detection
head is directly applied to the feature map for generating the predictions,
which are comprised of the predicted classification probabilities (as well as
the localization qualities if possible) and the predicted bounding boxes. It
should be noted that here we treat the encoder-decoder transformer (Vaswani et
al., 2017) in the transformer-based detectors as a part of the detection head.
Some configurations of the detection head are adjusted in terms of the number
of classes, the design of anchors or the number of queries according to our
kind of ground-truth assignments and the characteristics of the WSOL task,
where most of the images are object-centric. Then during the training process,
the pseudo ground-truth bounding boxes are compared with the predictions of
the detector through Intersection-over-Union (IoU) overlap or one-to-one
matching for assigning the labels. For instance, for the anchor-based
detectors, an anchor is assigned a positive label if it has an IoU overlap
equal to or higher than a predetermined foreground IoU threshold with any
pseudo ground-truth boxes. If there are no positive anchors that meet this
criterion, the anchor with the highest IoU overlap with a pseudo ground-truth
box is considered as positive. On the flipside, an anchor is assigned a
negative label if is has an IoU less than a predetermined background IoU
threshold with all the pseudo ground-truth boxes. All the other anchors are
ignored and do not contribute to the training process. The matching results
are utilized to compute the pseudo supervised loss, which consists of the
classification loss and the regression loss:

$\displaystyle
L_{sup}=L(\\{p_{i}\\},\\{t_{i}\\})=\lambda_{1}\frac{1}{N_{cls}}\sum_{i}L_{cls}(p_{i},p_{i}^{*})$
(1)
$\displaystyle+\lambda_{2}\frac{1}{N_{reg}}\sum_{i}p_{i}^{*}L_{reg}(t_{i},t_{i}^{*}),$

where $i$ is the index of a sample in a mini-batch. This mini-batch is a
random selection of the training samples, and the ratio of positive to
negative samples are also specified manually. $p_{i}$ is the predicted class
probability of the $i$-th sample being a foreground object. $p_{i}^{*}$ is the
ground-truth label of the $i$-th sample; it is 1 if the sample is positive,
and is 0 if the sample is negative. $t_{i}$ and $t_{i}^{*}$ are vectors
denoting the 4 parameterized coordinates of the predicted bounding box and of
the corresponding pseudo ground-truth box, respectively. The $L_{cls}$, which
can be the binary cross-entropy loss, measures the classification error
between the assigned foreground/background labels and the predicted class
probabilities, and the $L_{reg}$, which can be the smooth $L_{1}$ loss or GIoU
loss, measures the localization error between the pseudo ground-truth boxes
and the predicted boxes. These two terms are then normalized by the numbers of
samples $N_{cls}$ and $N_{reg}$ separately, and weighted by the balancing
parameter $\lambda_{1}$ and $\lambda_{2}$. Some detectors also utilize an
auxiliary branch to predict the localization quality, e.g. the overlap of the
predicted bounding box and the ground-truth box, which is usually denoted by
IoU, or the deviation between the center points of the predicted bounding box
and the ground-truth box, which is often denoted by centerness. This branch is
trained using the binary cross-entropy loss, which measures the difference
between the predicted localization quality and the ground truth.

Table 1. Comparison of the parameters and running speeds. Method |  #Params |  runtime (s/batch)
---|---|---
SCR |  24.55M |  0.331
200-class FCOS |  102.79M |  0.382
1000-class FCOS |  117.54M |  0.408
BCD (Ours) |  99.12M |  0.245

Since the detector naturally gives multiple bounding box outputs, the drawback
of inflexible outputs is easily removed. It is noteworthy that the previous
works suggest that detection models are too heavy for the object localization
task. On the one hand, the calculation burden of the current detectors
themselves is heavy. On the other hand, datasets like ImageNet (Russakovsky et
al., 2015) in the object localization task have much more classes than the
ones in the object detection task, e.g., 1000 vs. 200. But our work subtly
adjusts the number of samples for training in the detector, and proposes to
only classify the foreground objects and the backgrounds, thus reducing the
computational burden. In this way, the detection models can be applied in the
large-scale localization field. To illustrate our computation reduction
quantitatively, we conduct the model’s parameter and speed analysis of the
SCR, the detectors that could be used for the WSOL tasks, and our BCD. The
results are listed in Table 1. The results of the 200-class FCOS and
1000-class FCOS show that introducing an additional detector increases the
runtime. However, by using our operations, our BCD even runs faster than the
SCR.

####  3.2.2. Weighted Entropy Loss

In order to reduce the negative impact of the noisy pseudo bounding boxes, we
propose a WE loss as an additional unsupervised loss. Entropy minimization is
a popular strategy for training the unlabeled data in the semi-supervised and
unsupervised image classification tasks (Grandvalet and Bengio, 2004;
Berthelot et al., 2019; Dhillon et al., 2020; Wang et al., 2021a; Liang et
al., 2022). It encourages the classifier to divide the unlabeled data into
well separated categories. In this work, we utilize the entropy loss (Shannon,
1948) as an additional constraint for the unlabeled data’s foreground-
background classification. This constraint is independent of the fixed
generated pseudo bounding boxes, and is only related to the predictions.
Besides, we assume that due to the weakly localization ability of the
classifier, the entropy loss (Shannon, 1948) is also beneficial for dealing
with the offsets of the boxes. However, the original entropy minimization
often assumes a uniform distribution of the categories, while there exists a
foreground-background imbalance problem in the object localization using a
detector (Oksuz et al., 2021). The predictions of the background are dominant
and overly confident, whereas the predictions of the foreground are the
opposite. Simply adopting the entropy loss (Shannon, 1948) can bias the
detector’s predictions even further. Inspired by the success of Focal loss
(Lin et al., 2017), we come up with an elegant solution to deal with this
foreground-background imbalance issue in localization. Namely, we design an
extra WE loss aiming at down-weighting the easy samples and focusing on the
hard ones for the classifier of the detector:

(2) $\displaystyle
L_{unsup}=L(\\{p_{i}\\})=\frac{1}{N_{cls}}\sum_{i}H_{cls}(p_{i},\tau_{1},\tau_{2}),$
$\displaystyle
H_{cls}(p_{i},\tau_{1},\tau_{2})=-(\mathbbm{1}(p_{i}<\tau_{1})(1-\alpha)p_{i}^{\gamma}$
(3)
$\displaystyle+\mathbbm{1}(p_{i}>\tau_{2})\alpha(1-p_{i})^{\gamma})p_{i}\log(p_{i}),$

where $\tau_{1}$ and $\tau_{2}$ are the confidence thresholds for the choice
of the modulating factor $p_{i}^{\gamma}$ or $(1-p_{i})^{\gamma}$, which is
also solely related to the predicted probability rather than the generated
pseudo bounding box labels. The $\tau_{1}$ and $\tau_{2}$ ensure the
modulating factor being proper from two aspects. First, they help to determine
the well-classified samples. Second, at the beginning of the detector
training, many samples which have predicted class probabilities between
$\tau_{1}$ and $\tau_{2}$ are neglected, and not involved in calculating the
WE loss. This alleviates the perturbation of the unsupervised WE loss on the
initial training and avoids the collapse of training. As the training
progresses, more and more samples are used with appropriate modulating factor.
It is noteworthy that our proposed modulating factor choice mechanism here
actually follow the strategy of self-paced learning (Xu et al., 2015; Wang et
al., 2021b). Then the selected modulating factor is able to down-weight the
well-classified samples’ entropy and focus on the hard ones. As in (Lin et
al., 2017), $\gamma$ is a tunable focusing parameter utilized to control the
degree of down-weighting, and $\alpha$ is another hyperparameter for dealing
with the class imbalance.

To summarize, given the pseudo ground-truth bounding boxes generated by the
effective C2AM (Xie et al., 2022) and the training images, we train a BCD for
the binary class-agnostic object localization sub-task by minimizing the
following loss:

(4) $L_{loc}=\eta L_{sup}+L_{unsup},$

where $\eta\geq 0$ is a balancing hyper-parameter.

##  4\. Experiments

###  4.1. Experimental Setups

Datasets. We evaluate our method on two widely used WSOL datasets: the
CUB-200-2011 (Wah et al., 2011) and ImageNet-1K (Russakovsky et al., 2015).
The CUB-200-2011 (Wah et al., 2011) is a challenging dataset of 200 bird
species, which is comprised of 5,994 images for training and 5,794 images for
test. Each image in the CUB-200-2011 (Wah et al., 2011) is annotated with a
bounding box and a class. The ImageNet-1K (Russakovsky et al., 2015) is a
large-scale visual recognition dataset of 1000 categories, which consists of
1,281,167 images for training and 50,000 images for validation. Each image in
the ImageNet-1K (Russakovsky et al., 2015) is assigned one class, and each
validation image is labeled one or more bounding boxes. In the setting of
WSOL, only the image-level supervision, namely the class label, is allowed to
be used during training. In our work, it is utilized for the object
classification sub-task, and we actually do not need any supervision for the
object localization sub-task.

Evaluation Metrics. Since our localization method provides multiple bounding
box outputs, and for fair comparison, we first only use our highest confident
bounding box and follow (Zhou et al., 2016) to adopt Top-1 localization (Top-1
Loc), Top-5 localization (Top-5 Loc), and GT-known localization accuracy (GT-
known Loc) for evaluation. Then we revise these criteria to take the multiple
output bounding boxes into consideration. Top-1 Loc is unchanged, and it is
correct when both the Top-1 classification and Top-1 localization results are
correct. Top-5 Loc is correct when one of the Top-5 predicted classes and one
of the Top-5 predicted bounding boxes are both correct. GT-known Loc is
correct when there exists one predicted bounding box that has an IoU overlap
over $0.5$ with any ground-truth bounding boxes, supposing that the ground-
truth class is given. These results are reported separately and differentiated
by underlining in the tables.

Table 2. Ablation study for the proposed binary-class detector (BCD) head and weighted entropy (WE) loss (%). The results listed come from the Top-1 output bounding box, whereas the underlined results are from multiple output bounding boxes. ⋆ denotes the re-implementation results of C2AM (Xie et al., 2022) by us. Method | Loc Bac. | CUB-200-2011 | ImageNet-1K
---|---|---|---
Top-1 Loc | Top-5 Loc | GT-known Loc | Top-1 Loc | Top-5 Loc | GT-known Loc
C2AM (Xie et al., 2022) | DenseNet161 | 81.76 | 91.11 | 92.88 | 59.56 | 67.05 | 68.53
w/ Faster R-CNN | DenseNet161 | 81.95 |  91.49 / 91.74 |  93.36 / 93.61 | 60.39 |  68.00 / 69.92 |  69.51 / 71.51
w/ Faster R-CNN+WE | DenseNet161 | 83.24 |  92.95 / 93.07 |  94.87 / 94.99 | 60.69 |  68.36 / 70.80 |  69.89 / 72.39
C2AM (Xie et al., 2022) | ResNet50 |  81.28⋆ |  90.69⋆ |  92.48⋆ |  59.24⋆ |  66.65⋆ |  68.10⋆
w/ FCOS | ResNet50 | 82.70 |  92.33 / 92.52 |  94.24 / 94.42 | 60.11 |  67.69 / 68.66 |  69.19 / 70.18
w/ FCOS+WE | ResNet50 | 83.05 |  92.71 / 92.88 |  94.60 / 94.77 | 60.18 |  67.76 / 68.69 |  69.27 / 70.22
w/ Faster R-CNN | ResNet50 | 82.23 |  91.79 / 91.92 |  93.63 / 93.77 | 60.17 |  67.75 / 70.08 |  69.24 / 71.65
w/ Faster R-CNN+WE | ResNet50 | 83.77 |  93.53 / 93.84 |  95.47 / 95.78 | 60.56 |  68.18 / 70.66 |  69.67 / 72.25
w/ CLN | ResNet50 | 82.87 |  92.43 / 92.58 |  94.23 / 94.41 | 60.68 |  68.30 / 69.58 |  69.80 / 71.14
w/ CLN+WE | ResNet50 | 84.15 |  93.98 / 94.25 |  95.91 / 96.19 | 60.70 |  68.36 / 69.65 |  69.86 / 71.19
w/ Deformable DETR | ResNet50 | 81.48 |  90.94 / 95.54 |  92.81 / 98.39 | 60.50 |  68.07 / 78.64 |  69.58 / 83.85
w/ Deformable DETR+WE | ResNet50 | 82.32 |  91.83 / 95.86 |  93.69 / 98.65 | 60.57 |  68.16 / 79.04 |  69.65 / 84.15

The implementation details are deferred to the appendix.

###  4.2. Results and Analyses

In this section, we present comprehensive experimental results and perform
analyses on these results to demonstrate the superiority of our method.

Figure 3. Top-1 localization result’s GT-known Loc accuracies on the CUB-200-2011 after each training epoch for the SOTA baseline and our methods (DenseNet161-Faster R-CNN). ⋆ denotes the accuracy curve from our re-implementation. Table 3. Ablation study for our WE loss using different $\gamma$ and $\alpha$ (%) (DenseNet161-Faster R-CNN). $\eta$ is set to be $1$. $\gamma$ | $\alpha$ | CUB-200-2011
---|---|---
Top-1 Loc | Top-5 Loc | GT-known Loc
2 | 0.5 | 81.73 |  91.12 / 91.24 |  92.94 / 93.06
2 | 0.25 | 82.13 |  91.63 / 91.74 |  93.49 / 93.61
4 | 0.25 | 82.23 |  91.74 / 91.89 |  93.56 / 93.71
4 | 0.1 | 82.25 |  91.77 / 91.92 |  93.58 / 93.77
6 | 0.1 | 82.31 |  91.93 / 92.03 |  93.79 / 93.90
6 | 0.05 | 82.00 |  91.54 / 91.66 |  93.41 / 93.53
Table 4. Ablation study for the proposed WEND using different $\eta$ (%) (DenseNet161-Faster R-CNN). $\gamma$ and $\alpha$ are set to be 6 and 0.1 respectively. $\eta$ | CUB-200-2011
---|---
Top-1 Loc | Top-5 Loc | GT-known Loc
4 | 82.20 |  91.81 / 91.92 |  93.64 / 93.76
2 | 82.08 |  91.61 / 91.85 |  93.43 / 93.67
1 | 82.31 |  91.93 / 92.03 |  93.79 / 93.90
0.5 | 82.26 |  91.83 / 91.93 |  93.76 / 93.85
0.25 | 82.90 |  92.54 / 92.72 |  94.43 / 94.60
0.125 | 83.24 |  92.95 / 93.07 |  94.87 / 94.99
0.0625 | 82.61 |  92.22 / 92.67 |  94.02 / 94.49

####  4.2.1. Ablation Study

We first analyze the effectiveness of our proposed binary-class detector (BCD)
and weighted entropy (WE) loss. C2AM (Xie et al., 2022) is utilized as our
baseline since we use the pseudo bounding boxes provided by it and the same
backbones for classification and localization. The results are reported in
Table 2. From the results, we observe that: 1) When using the DenseNet161
(Huang et al., 2017) as the localization backbone and the Faster R-CNN (Ren et
al., 2015) based detector, the results on the CUB-200-2011 (Wah et al., 2011)
are improved by over $1.3\%$ after applying our proposed WE loss. Meanwhile,
the results on the ImageNet-1K (Russakovsky et al., 2015) are improved by
around $1\%$ when evaluated using the Top-1 localization results, and up to
about $3\%$ when evaluated using multiple localization results after applying
the proposed detection head. The WE loss also improves the performance up to
around $1\%$ when evaluated using multiple outputs. These results demonstrate
that our proposed BCD and WE loss are beneficial to the WSOL task. We also
evaluate the Top-1 output bounding box in terms of the GT-known Loc accuracy
after each training epoch for dataset CUB-200-2011 (Wah et al., 2011). From
the results shown in Figure 3, we can see that BCD brings consistent
performance improvements, while the WE loss leads to small perturbations at
first and high performance in the end; 2) It is noted that the WE loss
contributes more on the dataset CUB-200-2011 (Wah et al., 2011), while the BCD
is more critical on ImageNet-1K (Russakovsky et al., 2015). This is because
that CUB-200-2011 (Wah et al., 2011) is a fine-grained classification dataset
and its low inter-class variance makes the discrimination between the
foreground and the background more feasible and valuable. On the other hand,
the object dataset ImageNet-1K (Russakovsky et al., 2015) vary in a wider
range of sizes and numbers, making the flexible localization more significant;
3) We observe that although the Deformable DETR (Zhu et al., 2021) usually
yields more competitive performance than the Faster R-CNN (Ren et al., 2015),
it is less effective in our setting in terms of the Top-1 outputs due to the
negative effect of the noisy pseudo bounding boxes. Yet its performance of the
multiple outputs is quite excellent. This reveals that despite the inaccurate
coordinates of the pseudo bounding boxes, the Deformable DETR (Zhu et al.,
2021) can still learn the object patterns. The label noise mainly confuses the
classifier because the pseudo bounding boxes contain much background. Our
proposed WE loss is able to deal with this problem to some extent.

Figure 4. Top-1 localization result’s GT-known Loc accuracies on dataset CUB-200-2011 after each training epoch for the proposed WEND using different $\eta$ (DenseNet161-Faster R-CNN). $\gamma$ and $\alpha$ are set to be 6 and 0.1 respectively. Table 5. Comparison of our WEND and the SOTA approaches on the CUB-200-2011 test set and ImageNet-1K validation set (%). Loc Bac. indicates the localization backbone, and Cls Bac. signifies the classification backbone. The results presented are from the Top-1 output bounding box, whereas the underlined results come from multiple output bounding boxes. ∗ denotes the results provided in C2AM (Xie et al., 2022). ⋆ denotes the re-implementation results of C2AM (Xie et al., 2022) by us. Method | Loc Bac. | Cls Bac. | CUB-200-2011 | ImageNet-1K
---|---|---|---|---
Top-1 Loc | Top-5 Loc | GT-known Loc | Top-1 Loc | Top-5 Loc | GT-known Loc
CAM (Zhou et al., 2016) | VGG16 | 44.15 | 52.16 | 56.0 | 42.80 | 54.86 | 59.00
CAM (Zhang et al., 2020a) | DenseNet161 | 29.81 | 39.85 | - | 39.61 | 50.40 | 52.54
ADL (Choe and Shim, 2019) | VGG16 | 52.36 | - | - | 44.92 | - | -
I2C (Zhang et al., 2020c) | InceptionV3 | 55.99 | 68.34 | 72.60 | 53.11 | 64.13 | 68.50
SPA (Pan et al., 2021) | VGG16 | 60.27 | 72.5 | 77.29 | 49.56 | 61.32 | 65.05
FAM (Meng et al., 2021) | VGG16 | 69.26 | - | 89.26 | 51.96 | - | 71.73
ORNet (Xie et al., 2021) | VGG16 | 67.74 | 80.77 | 86.2 | 52.05 | 63.94 | 68.27
BAS (Wu et al., 2022) | ResNet50 | 77.25 | 90.08 | 95.13 | 57.18 | 68.44 | 71.77
PPD (Xu et al., 2022b) | ViT-S | 78.8 | - | 97.0 | 55.0 | - | 67.5
GC-Net (Lu et al., 2020) | VGG16 | VGG16 | 63.24 | 75.54 | 81.10 | - | - | -
GC-Net (Lu et al., 2020) | InceptionV3 | InceptionV3 | - | - | - | 49.06 | 58.09 | -
PSOL (Zhang et al., 2020a) | ResNet50 | ResNet50 | 70.68 | 86.64 |  90.00∗ | 53.98 | 63.08 | 65.44
PSOL (Zhang et al., 2020a) | DenseNet161 | EfficientNet-B7 |  80.89∗ |  89.97∗ |  91.78∗ | 58.00 | 65.02 | 66.28
SLT-Net (Guo et al., 2021) | ResNet50 | ResNet50 | 72.3 | - | 90.7 | 56.2 | - | 68.5
SLT-Net (Guo et al., 2021) | DenseNet161 | DenseNet161 | 75.8 | - | 93.4 | 57.1 | - | 69.0
C2AM (Xie et al., 2022) | ResNet50 | EfficientNet-B7 |  81.28⋆ |  90.69⋆ |  92.48⋆ |  59.24⋆ |  66.65⋆ |  68.10⋆
C2AM (Xie et al., 2022) | DenseNet161 | EfficientNet-B7 | 81.76 | 91.11 | 92.88 | 59.56 | 67.05 | 68.53
WEND (Ours) | ResNet50 | EfficientNet-B7 | 83.77 |  93.53 / 93.84 |  95.47 / 95.78 | 60.56 |  68.18 / 70.66 |  69.67 / 72.25
WEND (Ours) | DenseNet161 | EfficientNet-B7 | 83.24 |  92.95 / 93.07 |  94.87 / 94.99 | 60.69 |  68.36 / 70.80 |  69.89 / 72.39
Table 6. Results of WEND’s localization sub-part whose pre-trained backbone is freezed on the CUB-200-2011 test set and ImageNet-1K validation set (%) (ResNet50). † denotes the localization sub-part initialized by the unsupervised pre-training (moco). Method | CUB-200-2011 | ImageNet-1K
---|---|---
Top-1 Loc | Top-5 Loc | GT-known Loc | Top-1 Loc | Top-5 Loc | GT-known Loc
SCR | 70.37 | 77.80 | 79.17 | 56.14 | 63.34 | 64.77
SCR |  64.69† |  71.89† |  73.22† |  55.68† |  62.80† |  64.20†
FCOS | 79.01 |  88.04 / 88.25 |  89.76 / 89.97 | 57.78 |  65.15 / 67.15 |  66.63 / 68.72
FCOS |  58.75† |  66.16 / 69.78† |  67.65 / 71.38† |  58.75† |  66.16 / 69.78† |  67.65 / 71.38†
Faster R-CNN | 79.69 |  88.76 / 89.72 |  90.39 / 91.39 | 57.81 |  65.14 / 68.29 |  66.61 / 69.84
Faster R-CNN |  68.30† |  75.87 / 77.66† |  77.42 / 79.23† |  58.33† |  65.64 / 65.92† |  67.12 / 67.40†
CLN | 73.91 |  82.24 / 82.22 |  83.79 / 83.77 | 60.85 |  68.44 / 69.16 |  69.94 / 70.68
CLN |  83.26† |  92.85 / 93.61† |  94.72 / 95.47† |  61.09† |  68.72 / 69.55† |  70.26 / 71.11†
Deformable DETR | 81.25 |  90.71 / 96.45 |  92.56 / 99.34 | 60.56 |  68.20 / 79.27 |  69.71 / 84.61
Deformable DETR |  81.27† |  90.82 / 96.48† |  92.59 / 99.19† |  60.59† |  68.21 / 79.29† |  69.72 / 84.51†
Figure 5. Visual comparison of our WEND and the C2AM (Xie et al., 2022)
(DenseNet161). The red boxes, blue boxes, and green boxes denote the ground-
truth bounding boxes, boxes predicted by the C2AM (Xie et al., 2022), and
boxes predicted by our method, respectively.

####  4.2.2. Hyper-parameter Analysis

We first analyze the choice of different hyper-parameters $\gamma$ and
$\alpha$ in the WE loss. Results on the CUB-200-2011 (Wah et al., 2011) for
various combination of $\gamma$ and $\alpha$ are reported in Table 3. It can
be seen from the results that the localization performance of our BCD is
largely improved after applying the WE loss using proper hyper-parameters. Our
method is insensitive to the change of $\gamma$ and $\alpha$.

Then we analyze the hyper-parameter $\eta$ for balancing of the pseudo
supervised objective and the unsupervised objective. Results for various
$\eta$ are reported in Table 4. We can see that the performance first improve
and then drop with an increasing $\eta$, and the best performance is achieved
at $\eta=0.125$.

In addition, we check the Top-1 output bounding boxes’ GT-known Loc accuracies
after each epoch of training for different $\eta$, and the results are plotted
in Figure 4. It is observed that the GT-known Loc curve of $\eta=0.125$ is
smooth and works the best. The curves of $\eta$ larger than or equal to one
are similar. Presumably, the big $\eta$ diminishes the role of the WE loss,
and the results mostly come from the pseudo supervision. In contrast, smaller
$\eta$ puts more weight on the unsupervised loss; the performance at the
beginning is perturbed but it finally reaches a high level owing to the
unsupervised loss’ ability of handling with the noisy pseudo labels. This
observation inspires us that we may combine the two losses in a better way to
reduce the perturbation at the beginning of the training and further enhance
the localization accuracy. We leave this as our future work.

####  4.2.3. Comparison with Existing Approaches

Table 5 reports the comparison results of our proposed method and other
approaches in WSOL on the two datasets. The results are reported in Table 5,
and it can be seen that our method consistently outperforms the existing works
in terms of all criteria.

On the CUB-200-2011 (Wah et al., 2011), our method outperforms the second best
approach C2AM (Xie et al., 2022) by $2.01\%$, $2.42\%$, and $2.59\%$,
respectively. When using all the output bounding boxes for evaluation, the
performance of our method further improves, which outperforms the SLT-Net (Guo
et al., 2021) by 2.38% in terms of the GT-known Loc. It is noteworthy that our
method performs much better when using the ResNet50 (He et al., 2016) as the
localization backbone than using DenseNet161 (Huang et al., 2017) on the
CUB-200-2011 dataset. This is different from other separated localization-
classification approaches. Since the last feature of the ResNet50 (He et al.,
2016) has more clear semantic meaning than that of DenseNet161 (Huang et al.,
2017), this further indicates that our method can well reduce the negative
effect of the noisy labels.

On the ImageNet-1K dataset, our method obtains $60.69\%$ Top-1 Loc, $68.36\%$
Top-5 Loc, and $69.89\%$ GT-known Loc accuracies for the Top-1 class-agnostic
bounding box output, which improves the C2AM (Xie et al., 2022) by $1.13\%$,
$1.31\%$, and $1.36\%$, respectively. Since the images of ImageNet-1K
(Russakovsky et al., 2015) may contain multiple ground-truth bounding boxes,
our method performs much better when multiple output bounding boxes are
considered for evaluation. In particular, our method achieves $70.80\%$ Top-5
Loc and $72.39\%$ GT-known Loc, which outperforms C2AM (Xie et al., 2022) by
$3.75\%$ and $3.86\%$ respectively. These results demonstrate that our method
significantly outperforms other approaches on different datasets.

####  4.2.4. Feature Preservation.

We freeze the ImageNet pre-trained backbone weights to explore the impact of
feature preservation for training using the noisy labels. Intuitively, the BCD
of our proposed have higher model capacity than the simple SCR; and thus
better performance should be achieved than SCR when the backbone is freezed.
Results reported in Table 6 verify this intuition. Furthermore, comparing with
the results of fine-tuning the backbone reported in Table 2, our WEND
(utilizing CLN and Deformable DETR) performs slightly better when the backbone
is freezed on the ImageNet-1K dataset. This reveals the potential of our
method in enhancing the localization performance and reducing the computation
cost to even less than that of the simple SCR (WEND-Deformable DETR + freeze
backbone: 16.45M $\textless$ SCR: 24.55M).

####  4.2.5. Visualization Comparison

In Figure 5, we visualize of the ground-truth bounding boxes, and the boxes
predicted by the C2AM (Xie et al., 2022) counterpart and our WEND. For the
CUB-200-2011 dataset, it can be seen in Figure 5 (a) that the predicted boxes
of our method are consistent with the ground truth. Yet the predicted boxes of
the C2AM (Xie et al., 2022) either focuses on the discriminative parts or are
affected by the confusing backgrounds. This indicates that our method has a
better ability to distinguish the foreground objects and the backgrounds, thus
predicting more complete and precise bounding box outputs. For the dataset
ImageNet-1K (Russakovsky et al., 2015), when it comes to some images with more
than one ground truth boxes, C2AM (Xie et al., 2022) fails to predict the
objects, while our method is able to correctly locates all the objects, as
shown in the first row of Figure 5 (b). For some other images with multiple
ground truths, the predicted boxes obtained by C2AM have large offsets, while
our results are more accurate. These results indicate that the performance of
C2AM (Xie et al., 2022) is largely degraded when there are more than one
object in the images. It may offset or completely mislocate the objects, since
multiple objects confuse the SCR (Sermanet et al., 2014). On the contrary, our
method can provide multiple outputs to mitigate this confusion. Besides, our
method also shows a better discrimination ability of the foreground and
background on the ImageNet-1K dataset.

##  5\. Conclusions and Discussions

In this paper, we rethink the localization of the state-of-the-art separated
localization-classification pipeline in WSOL, and identify two major
drawbacks: infeasibility of locating multiple objects and dealing with noisy
pseudo labels. To remedy these drawbacks, we propose a novel localization
method termed WEND, which consists of a binary-class detector for predicting
multiple outputs, and an unsupervised weighted entropy (WE) loss to reduce the
negative impact of the noisy labels. Comprehensive experiments show that our
WEND significantly outperforms the existing approaches, and both the detection
head and WE loss are beneficial and critical.

In the future, we intend to look deeper into the localization network design,
and apply our method to more visual applications, e.g., video understanding,
medical image analysis, and remote sensing imagery analysis.

###### Acknowledgements.

This research was supported in part by the National Key Research and
Development Program of China under No. 2021YFC3300200, the Special Fund of
Hubei Luojia Laboratory under Grant 220100014, the National Natural Science
Foundation of China (Grant No. 62276195, 62141112, and 62225113), the National
Research Foundation Singapore and DSO National Laboratories under the AI
Singapore Programme (AISG Award No: AISG2-GC-2023-006), and the Fundamental
Research Funds for the Central Universities (No. 2042023kf1033).

## References

  * (1)
  * Baek et al. (2020) Kyungjune Baek, Minhyun Lee, and Hyunjung Shim. 2020\.  PsyNet: Self-Supervised Approach to Object Localization Using Point Symmetric Transformation. In _AAAI_. 10451–10459.
  * Berthelot et al. (2019) David Berthelot, Nicholas Carlini, Ian J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. 2019.  MixMatch: A Holistic Approach to Semi-Supervised Learning. In _NIPS_. 5050–5060.
  * Carion et al. (2020) Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020.  End-to-End Object Detection with Transformers. In _ECCV_ , Vol. 12346. 213–229.
  * Chen et al. (2019) Mickaël Chen, Thierry Artières, and Ludovic Denoyer. 2019.  Unsupervised Object Segmentation by Redrawing. In _NIPS_. 12705–12716.
  * Cho et al. (2015) Minsu Cho, Suha Kwak, Cordelia Schmid, and Jean Ponce. 2015\.  Unsupervised object discovery and localization in the wild: Part-based matching with bottom-up region proposals. In _CVPR_. 1201–1210.
  * Choe and Shim (2019) Junsuk Choe and Hyunjung Shim. 2019.  Attention-Based Dropout Layer for Weakly Supervised Object Localization. In _CVPR_. 2219–2228.
  * Dai et al. (2021) Xiyang Dai, Yinpeng Chen, Jianwei Yang, Pengchuan Zhang, Lu Yuan, and Lei Zhang. 2021\.  Dynamic DETR: End-to-End Object Detection with Dynamic Attention. In _ICCV_. IEEE, 2968–2977.
  * Dhillon et al. (2020) Guneet Singh Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. 2020.  A Baseline for Few-Shot Image Classification. In _ICLR_.
  * Grandvalet and Bengio (2004) Yves Grandvalet and Yoshua Bengio. 2004.  Semi-supervised Learning by Entropy Minimization. In _NIPS_. 529–536.
  * Guo et al. (2021) Guangyu Guo, Junwei Han, Fang Wan, and Dingwen Zhang. 2021\.  Strengthen Learning Tolerance for Weakly Supervised Object Localization. In _CVPR_. 7403–7412.
  * He et al. (2020) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. 2020.  Momentum Contrast for Unsupervised Visual Representation Learning. In _CVPR_. 9726–9735.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep Residual Learning for Image Recognition. In _CVPR_. 770–778.
  * Huang et al. (2017) Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. 2017.  Densely Connected Convolutional Networks. In _CVPR_. 2261–2269.
  * Kong et al. (2020) Tao Kong, Fuchun Sun, Huaping Liu, Yuning Jiang, Lei Li, and Jianbo Shi. 2020\.  FoveaBox: Beyound Anchor-Based Object Detection.  _IEEE Trans. Image Process._ 29 (2020), 7389–7398.
  * Law and Deng (2018) Hei Law and Jia Deng. 2018\.  CornerNet: Detecting Objects as Paired Keypoints. In _ECCV_ , Vol. 11218. 765–781.
  * Liang et al. (2022) Jian Liang, Dapeng Hu, Yunbo Wang, Ran He, and Jiashi Feng. 2022.  Source Data-Absent Unsupervised Domain Adaptation Through Hypothesis Transfer and Labeling Transfer.  _IEEE Trans. Pattern Anal. Mach. Intell._ 44, 11 (2022), 8602–8617.
  * Lin et al. (2017) Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. 2017.  Focal Loss for Dense Object Detection. In _ICCV_. 2999–3007.
  * Liu et al. (2021) Yen-Cheng Liu, Chih-Yao Ma, Zijian He, Chia-Wen Kuo, Kan Chen, Peizhao Zhang, Bichen Wu, Zsolt Kira, and Peter Vajda. 2021\.  Unbiased Teacher for Semi-Supervised Object Detection. In _ICLR_.
  * Lu et al. (2020) Weizeng Lu, Xi Jia, Weicheng Xie, Linlin Shen, Yicong Zhou, and Jinming Duan. 2020\.  Geometry Constrained Weakly Supervised Object Localization. In _ECCV_ , Vol. 12371. 481–496.
  * Meng et al. (2021) Meng Meng, Tianzhu Zhang, Qi Tian, Yongdong Zhang, and Feng Wu. 2021.  Foreground Activation Maps for Weakly Supervised Object Localization. In _ICCV_. 3365–3375.
  * Oksuz et al. (2021) Kemal Oksuz, Baris Can Cam, Sinan Kalkan, and Emre Akbas. 2021\.  Imbalance Problems in Object Detection: A Review.  _IEEE Trans. Pattern Anal. Mach. Intell._ 43, 10 (2021), 3388–3415.
  * Pan et al. (2021) Xingjia Pan, Yingguo Gao, Zhiwen Lin, Fan Tang, Weiming Dong, Haolei Yuan, Feiyue Huang, and Changsheng Xu. 2021\.  Unveiling the Potential of Structure Preserving for Weakly Supervised Object Localization. In _CVPR_. 11642–11651.
  * Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. 2015\.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. In _NIPS_. 91–99.
  * Roh et al. (2022) Byungseok Roh, Jaewoong Shin, Wuhyun Shin, and Saehoon Kim. 2022\.  Sparse DETR: Efficient End-to-End Object Detection with Learnable Sparsity. In _ICLR_.
  * Russakovsky et al. (2015) Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015\.  ImageNet Large Scale Visual Recognition Challenge.  _IJCV_ 115, 3 (2015), 211–252.
  * Sermanet et al. (2014) Pierre Sermanet, David Eigen, Xiang Zhang, Michaël Mathieu, Rob Fergus, and Yann LeCun. 2014\.  OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks. In _ICLR_.
  * Shannon (1948) Claude E. Shannon. 1948\.  A mathematical theory of communication.  _Bell Syst. Tech. J._ 27, 3 (1948), 379–423.
  * Simonyan and Zisserman (2015) Karen Simonyan and Andrew Zisserman. 2015.  Very Deep Convolutional Networks for Large-Scale Image Recognition. In _ICLR_.
  * Su et al. (2022) Yukun Su, Guosheng Lin, Yun Hao, Yiwen Cao, Wenjun Wang, and Qingyao Wu. 2022\.  Self-Supervised Object Localization with Joint Graph Partition. In _AAAI_. 2289–2297.
  * Sun et al. (2021) Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang, and Ping Luo. 2021.  Sparse R-CNN: End-to-End Object Detection With Learnable Proposals. In _CVPR_. 14454–14463.
  * Szegedy et al. (2015) Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015.  Going deeper with convolutions. In _CVPR_. 1–9.
  * Szegedy et al. (2016) Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. 2016\.  Rethinking the Inception Architecture for Computer Vision. In _CVPR_. 2818–2826.
  * Tan and Le (2019) Mingxing Tan and Quoc V. Le. 2019.  EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. In _ICML_ , Vol. 97. 6105–6114.
  * Tian et al. (2019) Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. 2019\.  FCOS: Fully Convolutional One-Stage Object Detection. In _ICCV_. 9626–9635.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017\.  Attention is All you Need. In _NIPS_. 5998–6008.
  * Wah et al. (2011) C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. 2011.  _The Caltech-UCSD Birds-200-2011 Dataset_.  Technical Report CNS-TR-2011-001. California Institute of Technology.
  * Wang et al. (2021a) Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno A. Olshausen, and Trevor Darrell. 2021a.  Tent: Fully Test-Time Adaptation by Entropy Minimization. In _ICLR_.
  * Wang et al. (2021b) Yunke Wang, Chang Xu, and Bo Du. 2021b.  Robust Adversarial Imitation Learning via Adaptively-Selected Demonstrations. In _IJCAI_. 3155–3161.
  * Wang et al. (2023) Zhenyu Wang, Yali Li, Xi Chen, Ser-Nam Lim, Antonio Torralba, Hengshuang Zhao, and Shengjin Wang. 2023.  Detecting Everything in the Open World: Towards Universal Object Detection.  _CoRR_ abs/2303.11749 (2023).
  * Wei et al. (2017) Xiu-Shen Wei, Jian-Hao Luo, Jianxin Wu, and Zhi-Hua Zhou. 2017\.  Selective Convolutional Descriptor Aggregation for Fine-Grained Image Retrieval.  _IEEE Trans. Image Process._ 26, 6 (2017), 2868–2881.
  * Wei et al. (2019) Xiu-Shen Wei, Chen-Lin Zhang, Jianxin Wu, Chunhua Shen, and Zhi-Hua Zhou. 2019.  Unsupervised object discovery and co-localization by deep descriptor transformation.  _Pattern Recognit._ 88 (2019), 113–126.
  * Wu et al. (2022) Pingyu Wu, Wei Zhai, and Yang Cao. 2022.  Background Activation Suppression for Weakly Supervised Object Localization. In _CVPR_. 14228–14237.
  * Xie et al. (2021) Jinheng Xie, Cheng Luo, Xiangping Zhu, Ziqi Jin, Weizeng Lu, and Linlin Shen. 2021\.  Online Refinement of Low-level Feature Based Activation Map for Weakly Supervised Object Localization. In _ICCV_. 132–141.
  * Xie et al. (2022) Jinheng Xie, Jianfeng Xiang, Junliang Chen, Xianxu Hou, Xiaodong Zhao, and Linlin Shen. 2022\.  C${}^{\mbox{2}}$ AM: Contrastive learning of Class-agnostic Activation Map for Weakly Supervised Object Localization and Semantic Segmentation. In _CVPR_. 979–988.
  * Xu et al. (2015) Chang Xu, Dacheng Tao, and Chao Xu. 2015.  Multi-view Self-Paced Learning for Clustering. In _IJCAI_. 3974–3980.
  * Xu et al. (2022b) Jingyuan Xu, Hongtao Xie, Chuanbin Liu, and Yongdong Zhang. 2022b.  Proxy Probing Decoder for Weakly Supervised Object Localization: A Baseline Investigation. In _ACM MM_. 4185–4193.
  * Xu et al. (2022a) Rui Xu, Yong Luo, Bo Du, Kaiming Kuang, and Jiancheng Yang. 2022a.  LSSANet: A Long Short Slice-Aware Network for Pulmonary Nodule Detection. In _MICCAI_ , Vol. 13431. 664–674.
  * Zhang et al. (2020a) Chen-Lin Zhang, Yun-Hao Cao, and Jianxin Wu. 2020a.  Rethinking the Route Towards Weakly Supervised Object Localization. In _CVPR_. 13457–13466.
  * Zhang et al. (2018) Jing Zhang, Tong Zhang, Yuchao Dai, Mehrtash Harandi, and Richard I. Hartley. 2018.  Deep Unsupervised Saliency Detection: A Multiple Noisy Labeling Perspective. In _CVPR_. 9029–9038.
  * Zhang et al. (2019) Runsheng Zhang, Yaping Huang, Mengyang Pu, Qingji Guan, Jian Zhang, and Qi Zou. 2019\.  Mining Objects: Fully Unsupervised Object Discovery and Localization From a Single Image.  _CoRR_ abs/1902.09968 (2019).
  * Zhang et al. (2020b) Shifeng Zhang, Cheng Chi, Yongqiang Yao, Zhen Lei, and Stan Z. Li. 2020b.  Bridging the Gap Between Anchor-Based and Anchor-Free Detection via Adaptive Training Sample Selection. In _CVPR_. 9756–9765.
  * Zhang et al. (2020c) Xiaolin Zhang, Yunchao Wei, and Yi Yang. 2020c.  Inter-Image Communication for Weakly Supervised Localization. In _ECCV_ , Vol. 12364. 271–287.
  * Zhao et al. (2021) Nanxuan Zhao, Zhirong Wu, Rynson W. H. Lau, and Stephen Lin. 2021\.  Distilling Localization for Self-Supervised Representation Learning. In _AAAI_. 10990–10998.
  * Zhou et al. (2016) Bolei Zhou, Aditya Khosla, Àgata Lapedriza, Aude Oliva, and Antonio Torralba. 2016\.  Learning Deep Features for Discriminative Localization. In _CVPR_. 2921–2929.
  * Zhu et al. (2021) Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. 2021\.  Deformable DETR: Deformable Transformers for End-to-End Object Detection. In _ICLR_.

##  Appendix A Implementation Details

For our method, we directly use the pre-trained EfficientNet (Tan and Le,
2019) for the object classification sub-task111The pre-trained weights for
dataset CUB-200-2011 are downloaded from https://github.com/CVI-SZU/CCAM. In
regard to the object localization sub-task, we use the generated pseudo
bounding boxes provided in C2AM (Xie et al., 2022) for training the detector.
We choose the ResNet50 (He et al., 2016) and DenseNet161 (Huang et al., 2017)
pre-trained on the ImageNet as our backbone, and implement the object
detectors of various types, the popular single-stage FCOS (Tian et al., 2019),
two-stage Faster R-CNN (Ren et al., 2015) and CLN (Wang et al., 2023), and
recent transformer-based Deformable DETR (Zhu et al., 2021). Our proposed
weighted entropy (WE) loss is implemented in the classifier of the FCOS (Tian
et al., 2019) and CLN (Wang et al., 2023), the classifier of the R-CNN (Liu et
al., 2021) for the Faster R-CNN (Ren et al., 2015), and incorporated in every
stage of the iterative loss function for the Deformable DETR (Zhu et al.,
2021). The input images are first resized to 256$\times$256\. Then during
training, we randomly crop the images to 224$\times$224 and apply horizontal
flipping for data augmentation. For testing, the images are center cropped to
224$\times$224\. All images are normalized.

For all the experiments, we train our localization model 50 epochs on the
CUB-200-2011 (Wah et al., 2011) and 10 epochs on the ImageNet-1K (Russakovsky
et al., 2015) with a training batch size of 256. For the FCOS (Tian et al.,
2019), Faster R-CNN (Ren et al., 2015) and CLN (Wang et al., 2023), the
Stochastic Gradient Descent (SGD) optimizer is used. The momentum and the
weight decay are set to 0.9 and 0.0001 respectively. The initialization
learning rate is 0.004 for the backbone and doubled for the detection head;
the cosine annealing policy is applied to schedule the learning rate. For the
Deformable DETR (Zhu et al., 2021), the Adam optimizer is used with
$\beta_{1}=0.9$, $\beta_{2}=0.999$, and weight decay of 0.0001, respectively.
The initialization learning rate is $2\times 10^{-4}$, and the learning rates
of the backbone and the linear projections, which are used for predicting
object query reference and sampling offsets, are reduced to 0.1; all learning
rates are scheduled using the cosine annealing policy. Our implementation is
in PyTorch on 4 NVIDIA GeForce RTX 3090 GPUs with 24GB memory.

##  Appendix B Additional Experiments

###  B.1. Strict Unsupervised Localization

Since we adopt the pseudo bounding boxes generated via unsupervised learning
(Xie et al., 2022), our localization sub-part can be converted to a strict
unsupervised setting by initializing the localization backbone using the
unsupervised pre-training, moco (He et al., 2020). We compare our strict
unsupervised localization sub-part with C2AM (Xie et al., 2022), and the
results are reported in Table 7. It can be seen that our method also
outperforms C2AM (Xie et al., 2022) in this strict unsupervised setting, and
especially achieves a significant $3\%$ improvement on the CUB-200-2011 (Wah
et al., 2011) dataset.

Table 7. The GT-known Loc of our WEND’s localization sub-part initialized by the unsupervised pre-training (moco) and the unsupervised C2AM (Xie et al., 2022) on the CUB-200-2011 test set and ImageNet-1K validation set (%). The results in the first line are performance of C2AM (Xie et al., 2022) utilizing moco to generate the pseudo bounding boxes; ⋆ denotes the re-implementation results of the C2AM (Xie et al., 2022) by us, which uses moco to initialize the localization network. Method | Loc Bac. | CUB-200-2011 | ImageNet-1K
---|---|---|---
C2AM (Xie et al., 2022) | ResNet50 | 89.90 | 66.51
C2AM (Xie et al., 2022) | ResNet50 |  91.97⋆ |  68.44⋆
Faster R-CNN | ResNet50 |  94.59 / 94.97 |  69.65 / 71.36
CLN | ResNet50 |  94.48 / 94.95 |  69.92 / 71.42
Deformable DETR | ResNet50 |  93.00 / 97.84 |  69.52 / 84.11
Table 8. Ablation study for the proposed WEND using different $\eta$ (%) (DenseNet161-Faster R-CNN); $\gamma$, $\alpha$, and $\tau$ are set to be $4$, $0.25$, and $0.1$ respectively. $\eta$ | ImageNet-1K
---|---
Top-1 Loc | Top-5 Loc | GT-known Loc
4 | 60.49 |  68.11 / 70.07 |  69.63 / 71.66
1 | 60.51 |  68.12 / 70.04 |  69.64 / 71.63
0.5 | 60.43 |  68.02 / 69.99 |  69.53 / 71.57
0.125 | 60.39 |  68.01 / 70.02 |  69.53 / 71.60
0.0625 | 60.55 |  68.18 / 70.82 |  69.70 / 72.42
0.03125 | 60.69 |  68.36 / 70.80 |  69.89 / 72.39
0.015625 | 60.46 |  68.10 / 70.16 |  69.63 / 71.76
Table 9. Ablation study for the proposed WEND using different $\tau$ (%) (DenseNet161-Faster R-CNN), where $\gamma$, $\alpha$, and $\eta$ are set to be $6$, $0.1$, and $0.125$ respectively. $\tau$ | CUB-200-2011
---|---
Top-1 Loc | Top-5 Loc | GT-known Loc
0.1 | 83.17 |  92.82 / 93.05 |  94.72 / 94.98
0.2 | 83.13 |  92.78 / 92.96 |  94.66 / 94.85
0.3 | 83.24 |  92.95 / 93.07 |  94.87 / 94.99
0.4 | 83.04 |  92.70 / 92.87 |  94.59 / 94.78
0.5 | 83.29 |  92.95 / 93.03 |  94.85 / 94.94
Table 10. Ablation study for the standard entropy loss and the proposed weighted entropy (WE) loss (%) (DenseNet161-Faster R-CNN). The baseline is the binary-class detector (BCD), $\eta$ is set to be $0.125$ for CUB-200-2011 and $0.03125$ for ImageNet-1K. Method |  CUB-200-2011 |  ImageNet-1K
---|---|---
Top-1 Loc |  Top-5 Loc |  GT-known Loc |  Top-1 Loc |  Top-5 Loc |  GT-known Loc
BCD |  81.95 |  91.49 / 91.74 |  93.36 / 93.61 |  60.39 |  68.00 / 69.92 |  69.51 / 71.51
w/ Entropy |  82.83 |  92.42 / 92.85 |  94.28 / 94.72 |  60.65 |  68.30 / 70.62 |  69.84 / 72.23
w/ WE |  83.24 |  92.95 / 93.07 |  94.87 / 94.99 |  60.69 |  68.36 / 70.80 |  69.89 / 72.39
Table 11. Ablation study for the proposed WEND using different $\tau$ (%) (ResNet50-Deformable DETR), where $\gamma$, $\alpha$, and $\eta$ are set to be $6$, $0.1$, and $1$ respectively. $\tau$ | CUB-200-2011
---|---
Top-1 Loc | Top-5 Loc | GT-known Loc
0.5 | 81.44 |  90.99 / 95.70 |  92.81 / 98.78
0.7 | 81.52 |  90.99 / 95.52 |  92.86 / 98.45
0.9 | 82.13 |  91.64 / 95.67 |  93.43 / 98.47
Table 12. Ablation study for the proposed WEND using different positive-to-negative ratios (%) (DenseNet161-Faster R-CNN), where $\gamma$, $\alpha$, $\eta$, and $\tau$ are set to be $6$, $0.1$, $0.125$, and $0.3$ respectively. $r$ | CUB-200-2011
---|---
Top-1 Loc | Top-5 Loc | GT-known Loc
1:1 | 83.24 |  92.95 / 93.07 |  94.87 / 94.99
2:3 | 83.38 |  93.06 / 93.22 |  94.92 / 95.09
1:3 | 83.39 |  93.12 / 93.31 |  94.99 / 95.18
1:4 | 83.44 |  93.14 / 93.41 |  95.05 / 95.33
1:5 | 82.72 |  92.19 / 92.49 |  94.06 / 94.36

###  B.2. More Ablation Studies

The performance of our method w.r.t. different choices of the balancing hyper-
parameter $\eta$ on the ImageNet-1K (Russakovsky et al., 2015) dataset are
reported in Table 8. The trend is the same as that on the CUB-200-2011 (Wah et
al., 2011) presented in the main body of our paper. The best overall
performance is achieved at the value of $2^{-5}=0.03125$ for $\eta$.

We discuss the effect of choosing different confidence thresholds $\tau_{1}$
and $\tau_{2}$, which are utilized for inducing the modulating factor. To
include more samples for training, we enforce $\tau_{1}=\tau_{2}$, and denote
them using the same $\tau$. The results are reported in Table 9 and Table 11.
From the results, we can see that the binary-class detector (BCD) can be
improved in a wide range of $\tau$, and our method is insensitive to this
hyper-parameter. This indicates that as long as the easy samples are down-
weighted, our weighted entropy (WE) loss is able to largely elevate the
performance, and there is no need for carefully chosen hyper-parameters.

To further verify the effectiveness of our proposed WE loss, we compare the
performance of adopting the standard entropy loss and our WE loss in the BCD.
As reported in Table 10, adopting the standard entropy loss can also improve
the performance, and adopting our WE loss can always achieve better
performance due to the ability to deal with the foreground-background
imbalance in BCD. On the CUB-200-2011 (Wah et al., 2011), our WE loss
surpasses the standard entropy loss by $0.41\%$, $0.53\%$, and $0.59\%$ in
terms of Top-1 Loc, Top-5 Loc, and GT-known Loc accuracies, respectively, when
using the Top-1 localization results for evaluation.

Additionally, we tune the implicit foreground-background balancing factor in
some detectors, e.g., the sampling ratio of positive to negative samples of
the Faster R-CNN (Ren et al., 2015). Results reported in Table 12 shows that
our proposed WE loss can enable this kind of detectors to use more negative
samples for training and learn more discriminative features.

###  B.3. Visualization Comparison

We provide more visual results on the ImageNet-1K (Russakovsky et al., 2015)
in Figure 6. These results demonstrate that our WEND can achieve more accurate
localization than C2AM (Xie et al., 2022) in a wide variety of object
categories.

Figure 6. Visual comparison of our WEND and C2AM (Xie et al., 2022) on the
ImageNet-1K (DenseNet161-Faster R-CNN). The red, blue, and green boxes denote
the ground-truth bounding boxes, boxes predicted by C2AM (Xie et al., 2022),
and boxes predicted by our method, respectively. Figure 7. Failure cases of
our WEND (DenseNet161-Faster R-CNN).

##  Appendix C More Discussions

We also visualize some failure cases of our WEND to explore the focus in
future work. As shown in Figure 7, on the CUB-200-2011 (Wah et al., 2011), our
WEND fails when the backgrounds are confusing. Thus a future work may be to
discover the theme of the images to reduce distractions. It is the same for
the more complicated dataset ImageNet-1K (Russakovsky et al., 2015): when
there exist more than one category of objects or confusing backgrounds, the
algorithm has to be capable of finding the main target. In addition, the
occlusion problem needs to be addressed.