# Discriminative Sampling of Proposals in Self-Supervised Transformers
for Weakly Supervised Object Localization

Shakeeb Murtaza1, Soufiane Belharbi1, Marco Pedersoli1, Aydin Sarraf2, and
Eric Granger1
1 LIVIA, École de Technologie Supérieure, Montreal, Canada
2 Ericsson, Global AI Accelerator, Montreal, Canada
<EMAIL_ADDRESS>
###### Abstract

Self-supervised vision transformers can generate accurate localization maps of
the objects in an image. However, since they decompose the scene into multiple
maps containing various objects, and they do not rely on any explicit
supervisory signal, they cannot distinguish between the object of interest
from other objects, as required in weakly-supervised object localization
(WSOL). To address this issue, we propose leveraging the multiple maps
generated by the different transformer heads to acquire pseudo-labels for
training a WSOL model. In particular, a new Discriminative Proposals Sampling
(DiPS) method is introduced that relies on a pretrained CNN classifier to
identify discriminative regions. Then, foreground and background pixels are
sampled from these regions in order to train a WSOL model for generating
activation maps that can accurately localize objects belonging to a specific
class. Empirical results111Our code is available:
https://github.com/shakeebmurtaza/dips on the challenging CUB, OpenImages, and
ILSVRC benchmark datasets indicate that our proposed approach can outperform
state-of-art methods over a wide range of threshold values. DiPS provides
class activation maps with a better coverage of foreground object regions
w.r.t. the background.

##  1 Introduction

Deep learning (DL) models have achieved state-of-the-art performance in many
visual recognition applications such as image classification [26], object
detection [36], and segmentation [15, 37]. However, large annotated datasets
are required for supervised training of these models [65]. In computer vision,
the weakly supervised learning (WSL) paradigm has received considerable
attention in object localization, allowing to avoid a costly bounding-box or
pixel-label annotation process to train a DL model [64]. In particular, weakly
supervised object localization (WSOL) involves locating an object of interest
in an image based on training data with only images-class labels.

Figure 1: Top: Mechanism for pseudo-label extraction. First, localization maps
are converted into binary maps to extract region of interest. From each map,
all connected areas are then extracted with a minimum size $S_{m}$. Based on
extracted areas, the background regions of input image blurred, and classified
to select top $P$ proposals, along with their underlying map. Finally, one map
is randomly selected from top $P$ maps for sampling of pseudo pixels. (A
detailed flow diagram is displayed in Fig.3.) Bottom: Examples of maps
generated using our method compared against SST ([10]) class tokens that are
used for pseudo-label sampling.

Class activation mapping (CAMs) methods are commonly employed to perform WSOL
with convolutional neural networks (CNNs) [6, 14, 27, 34, 39, 48, 49, 56, 57,
58, 60, 62]. These methods allow producing a spatial activation map highlight
features belonging to a particular class using features from the penultimate
layer [64] of a CNN. Strong activation of a map indicates potential presence
of the corresponding class, allowing for object localization. To improve the
robustness of these maps, several techniques have been proposed to obtain maps
from different layers by utilizing gradient information [17, 38, 11]. Despite
their popularity, CAM methods have limitations leading to inaccurate
localizations. Activation maps tend to focus on small discriminant area of
objects – common to different instance of an object belonging to a specific
class. Current efforts in WSOL task are focusing on improving the CAMs by
covering the full object.

Recently, self-supervised transformers (SSTs) [10] has attracted much
attention for WSOL tasks. Using only self-supervision, these models can yield
impressive saliency maps. Given their long-range receptive fields, SSTs
analyze an entire input image, allowing to build saliency maps that cover full
objects. Transformers are able to identify multiple objects by decomposing the
scene into different spatial saliency maps. Such localization information is
accumulated in maps, _i.e._ class tokens, at the top layer (see Fig. 2(b)).
However, without any class supervision, these tokens are arbitrary and class-
agnostic, and objects of interest cannot be dissociated from others. Each
token focuses on a random object with semantics that differ from one image to
another, making them less reliable for localization. Unless ground truth
localizations are provided to select the best token [10], their application in
WSOL remains limited.

Some recent work have been proposed to extract discriminative localization
from class tokens. In particular, TS-CAM [18] has been proposed to leverage
localization in transformers, where the average of all class tokens is
multiplied with a semantic-aware map to produce the final activation map.
Since TS-CAM averages an abundance of class-agnostic attention maps, it is
prone to localization error by including non-discriminative regions.
Additionally, maps from earlier layers attend to background regions, as shown
in Fig.2(a). Therefore, accumulating all maps introduces noise in the final
activation map that can reduce the model’s localization accuracy. In addition,
different transformer-based methods have been proposed[3, 12, 20, 41]. [3]
propose a spatial calibration module to capture semantic relationships and
spatial similarities. Similarly, ViTOL [20] proposes to incorporate a patch-
based attention dropout layer into the transformer attention blocks to improve
localization maps. [12] proposes a relational patch-attention module to
enhance local perception quality, and to retain global information. [41]
introduces a mechanism to suppress background regions and focus on the target
object. All these methods are unable to minimize loss over a generated
activation map, limiting their ability to fully cover an object, and force us
to search for an optimal threshold.

In this paper, a new Discriminative Proposals Sampling (DiPS) method is
proposed to leverage localization information from SSTs for accurate WSOL. In
particular, class token maps are extracted from the top block of pretrained a
SST. Given a limited number of high-quality attention maps (class tokens)
covering different objects, we employ a CNN classifier to localize potential
discriminative regions (see Fig. 2(a)), and sample pixel-wise pseudo-labels
from these regions to train a U-Net style localization network. To benefit
from these tokens, we propose a method for collecting appropriate pseudo-
labels, and sampling foreground/background regions from them to train our
localization network such that a particular object can be localized with high
confidence (Fig.3). More specifically, with the help of a CNN classifier, we
identify sampling areas for building effective pseudo-labels by determining
foreground and background regions in class tokens obtained from SST. From
these areas, foreground pixels are sampled, while the entire image was used to
sample pixels from background regions according to the criteria defined in
Section 3. Using these pseudo-pixels, the localization network is trained by
optimizing log-barrier [8] method on the selected pixels. Additionally, the
localization network parameters are regularized by class and CRF loss [42] to
generating threshold insensitive. In the end, only localization network is
retained for inference of localization maps.

Figure 2: (a): Attention of each class token learned by TS-CAM [18] map from
the first two and last three blocks. The first half of the blocks attend
different features, including background regions, but TS-CAM is able to
accumulate all of them. Attention maps of each class token is shown in
Supplementary Material. (b): Attention map of class token corresponding to
each token head from the last layer of transformer learned in a self-
supervised fashion. These maps show that the last layer is capable of
accumulating fine-grained localization of the concerned object.

Our main contributions are summarized as follows.

(1) A new DiPS method is proposed to leverage the emerging saliency maps in
SST. Since this localization lacks discriminative information, a pretrained
classifier is employed to produce discriminant proposals based on saliency
maps from top layer of pretrained SST. To avoid the risk of fitting incorrect
proposals, the top-n proposals are considered, from which a single proposal is
randomly selected for sampling pseudo-labels.

(2) Using a single proposal, pixel-wise pseudo-labels are sampled. Foreground
pixels are sampled within the bounding box of the proposal, while background
pixels are considered to be outside the box. We rely on activation magnitude
for sampling, where high activations are more likely to represent foreground
regions, while low activations are considered as background regions.

(3) Our DiPS training architecture is comprised of pretrained SST and
classifier models that remain frozen. During training, the weights a U-Net
style localization network are updated, and learn to produce full resolution
foreground and background CAMs through parametric upscaling [6]. The loss is
composed of local and global constraints. The local constraint uses sampled
pixel-wise pseudo-labels to guide the network via partial cross-entropy loss.
In addition, a CRF loss [42] is employed to yield consistent CAMs that respect
object boundaries. To ensure that the correct object is localized, we maximize
the classifier response w.r.t. the image-class label, over the image obtained
by masking the input image with the output foreground CAM.

(4) An extensive set of experiments was conducted on three challenging WSOL
benchmark datasets – CUB-200-2011, OpenImages, and ILSVRC. Visual results show
that our method produce CAMS with a better coverage of the entire foreground
object regions, and a clearer distinction between foreground and background
regions. DiPS can outperform state-of-art methods in terms of localization
accuracy, with less sensitivity to threshold values.

##  2 Related Work

Class Activation Mapping (CAM) Methods: A common way to harvest localization
maps from network activations is by aggregating the information presented in a
specific CNN layer. The representative method for aggregating activations is
using CAM methods [64], which weighs each pixel according to its influence on
the class prediction. Several methods have been proposed to improve the
mechanism for harvesting the activation maps [11, 17, 35, 38]. Networks are
used for generating CAMs are solely trained for image classification, and thus
focus on discriminative image regions. They typically underestimate the object
size. To mitigate this issue, [39, 58] employ adversarial perturbation to
erase discriminative parts, and for the network to look beyond the
discriminative areas. Similarly, in [14, 61] discriminative features are
erased, and adversarial learning is adopted to encroach upon non-
discriminative regions of the concerned object. In addition, fusion-based
methods improve on CAM methods by combining different activation maps based on
the classifier’s response [31, 44, 45].

Different model-dependent techniques have been proposed to alleviate the poor
coverage of objects [27, 34, 56, 57, 60]. In [50], authors utilized CAMs from
multiple convolution layers with different receptive fields to enlarge the
discriminative regions. [57] combines CAMs of different classes to identify
foreground and background regions in an image. [52, 67] propose to suppress
background regions to help to generate a foreground map with high confidence,
guided by area constraints. For a robust training, [29] incorporates an
encoder-decoder layer between the shallow layers, and a generator to mask a
part of the image. Low-level feature based activation map (FAM) [53] utilizes
multiple classifiers to generate a foreground map that is decomposed into
multiple part regions. This is used to train a network to produce a final
semantic-agnostic map. Self-produced guidance (SPG) [62] separates foreground
and background regions for guiding shallow learning while expanding to less
discriminative areas. Shallow feature-aware pseudo supervised object
localization (SPOL) [48] employs a multiplicative fusion strategy for
harvesting highly confident regions, and then using those masks as pseudo-
labels for training a segmentation network. Inter-image communication (I2C)
[63] increases the robustness of localization maps by considering the
correlation of similar images within a class. Structure-preserving activation
(SPA) [33] seeks to preserve the object’s structure within a CAM, although
this method still produces blobby boundaries. Similarly, Strengthen learning
tolerance (SLT) [19] groups images of similar classes to increase the
network’s tolerance, and force the localization map to expand towards the
object’s boundaries. Also, the full resolution CAM (F-CAM) [6] has been
proposed to expand activation maps to cover full object beyond discriminative
regions. Compared to F-CAM, our proposed method is able to define sampling
regions for building efficient pseudo-label with the help of classifier. This
helps our model to focus only on target object instead of expanding the
activation to other objects in the concerned image.

Most of the above methods rely on internal activations acquired by utilizing
activation maps at different layers of the networks. Given their intrinsic
properties (e.g., local receptive field), CNNs decompose object into local
semantic elements [4, 59]. These intrinsic properties prevent the CNN from
forming global relationships between different receptive fields, limiting
their ability to localize whole object, and producing coarse blobby maps
focused on discriminant regions. One solution to form long-range dependencies
is to find global cues by calculating pixel-similarities [46, 47, 60, 63].
More recently, [22] propose a method based on non-local attention blocks to
build long-range relationships and efficiently localize objects. It enhances
attention maps by incorporating spatial similarity.

Vision Transformer (ViT) Methods: Numerous transformer based methods have
recently been proposed for WSOL [3, 12, 20, 28, 41]. A pioneering work for
object localization is the token semantic coupled attention map (TS-CAM) [18],
which is capable of localizing objects at a fine-grained level by fusing
attention maps with semantic-aware maps. (The detailed working TS-CAM model is
presented in supplementary material.) Recently, [41] proposed a re-attention
mechanism for suppressing background regions. Layer-wise relevance propagation
[20] and clustering-based [12] methods have also been proposed to accumulate
attention maps produced by ViTs. For instance, [20] introduces a patch-based
attention dropout layer in the attention block, and incorporate an attention
roll-out method to improve localization performance.

Transformer-based based methods have had a significant impact on WSOL
literature. However, they accumulate all attention maps from each layer,
leading to a noisy CAM, or a suppression of different regions within the
object [18] (see Fig.2). Also, most techniques provide a heat map without
sharp boundaries. In contrast, our proposed approach can produce high-
resolution maps with sharper boundaries, only using attention from the last
layer of the transformer block. Methods based on pseudo-labels fix their map
for all epochs [48], generating a map very close to the pseudo-labels. Our
method samples only a few pixels per epoch to learn foreground and background
maps that continue to evolve based on a bi-nominal distribution. Sampling only
a few pixels as pseudo-labels helps the network to explore different parts of
the object, which leads to coverage of the whole object with sharper
boundaries. Since these methods are typically computationally intensive during
training, we harvest maps in a self-supervised way using a pretrained
transformer, allowing to obtain clues about potential objects and train our
localization network. Our method leverages long-range dependencies as pseudo-
labels are harvested from a transformer, and convolution inductive bias to
suppress background noise.

##  3 Proposed Method

###  3.1 Background

Vision transformers (ViTs) are recognized for their success in image
classification [16, 43] and WSOL [18]. They consist of $T$ cascaded encoder
blocks, each containing multi-headed attention, followed by a multi-layer
perceptron (MLP). First, an input image $x$ of size $W\times H$ is divided
into $N$ patches of resolution $(W/S)\times(H/S)$, where $S$ denotes the patch
size in pixels. These patches are then linearly projected to a fixed embedding
size $D$ along with a learnable class token: $f=\\{[\texttt{class}]^{1\times
D},t_{1}^{1\times D},t_{2}^{1\times D},\dots,t_{N}^{1\times D}\\}$ and passed
through $T$ cascaded blocks. Furthermore, the features of a class token from
$T^{th}$ block are fed to the MLP for class prediction, defined as
$p=Softmax_{\tau=1}(MLP(T_{vit}(x))$, where

$Softmax_{\tau}(s)=\frac{\exp(s/\tau)}{\sum_{k=1}^{K}\exp(s_{k}/\tau)}$ (1)

Here, $s\in\mathbb{R}^{K}$ is the MLP output, where $K$ corresponds to the
number of classes. In addition, $T_{vit}$ represents the transformer’s forward
pass, generating features $f$ that are passed through $T$ cascaded blocks. The
MLP predicts the probabilities of concerned classes with cross-entropy
classification loss $-\Sigma_{y}ylog(p_{y})$, and $y$ is the ground truth
label.

Self-Distillation with no labels (DINO) involves harvesting regions of
interest without class-level labels. To achieve this goal, the authors trained
two networks, called student and teacher, to match their probability
distribution. Parameters of the teacher are an exponential moving average of
the student network. The probability $p$ of the representation for these
networks is calculated by normalizing the output of the concerned network
using the softmax function with temperature defined in Eq. 1 ($\tau$ is a
hyper-parameter to be optimized). Furthermore, the input $x$ is augmented in
different ways, $x_{1}$ and $x_{2}$, for the student and teacher networks,
respectively. They represent two set of distorted views of image $x$,
calculated using the strategy defined in [9]. The first set contains several
local views, while the second one contains two global views. Here, local views
contain less than $50\%$ of the original image, and global views cover more
than $50\%$ of the area. Finally, the loss is calculated as cross-entropy
between the output probability of the student $p_{s}$ and the teacher $p_{t}$:

$\min_{\theta_{s}}-p_{s}(x_{1})\log(p_{t}(x_{2})).$ (2)

Figure 3: Architecture of our proposed DiPS method for WSOL. A pertained
transformer is employed along with a classification layer with frozen weights.
After obtaining attention maps, a threshold is applied to produce binary
proposals and generate bounding boxes for each proposal. Based on those
proposals, areas outside the box are blurred, and the top P best bounding
boxes are selected based on the classifier’s score that are greater than a
minimum score. Ultimately, a bounding box is selected among the P best boxes
for sampling from background and foreground regions. Then, the generated
pseudo-pixels are considered as pseudo-labels to training of the localization
network. After training, an input images are processes localization network
without the need of a transformer’s backbone. Here, red box and green box
refers to the training and inference flow, respectively.

###  3.2 Overview of DiPS

As we discussed, SSTs [10] are able to localize all objects in an image. They
decompose the objects of an image into different maps, because they learn to
represent objects in a self-supervised fashion. Each map highlights different
objects or their parts without associating categorical information with them.
If we can successfully identify maps and corresponding regions that contain
the concerned object, then we can use them as pseudo-labels to train a
localization network. Therefore, we propose a new DiPS method that can harvest
efficient pseudo-labels on the target object for accurate WSOL. DIPS utilizes
class tokens of a pretrained ViT to harvest effective pseudo-labels, and a
pretrained CNN classifier to select ROIs proposal to train our localization
network (see Fig.3).

It allows us to directly minimize the loss over the generated map for accurate
localization maps with a similar confidence for all object parts. Given a set
of tokens from a pretrained SST model [10], the associated maps are first
binarized to allow us extracting bounding boxes over potential ROIs. These are
considered proposals candidates that potentially cover object regions with
varying certainty. To measure certainty, we exploit the response of a
pretrained CNN classifier to indicate most relevant ROI. To avoid overfitting
to a single proposal, we consider top-$P$ confident ROI proposals, and
randomly select ones of them for sampling. This final proposal is used to
constrain sampling of pixel-wise pseudo-labels where foreground pixels are
sampled inside the bounding box of the proposals. Sampling is guided by the
activation magnitude, where strong activations are favored to be foreground.
Background pixels are sampled outside the bounding box by favoring low
activations. Sampling of pseudo-labels is done randomly at each Stochastic
Gradient Descent (SGD) iteration, which has been shown to be more effective
and robust against overfitting than static and noisy pseudo-labels [5, 6].
Overall, this sampling allows for the emergence of accurate localization maps,
from few pixels toward segments and the entire object. This can be seen as a
_fill-in-the-gap_ game where only random pixels are selected as
foreground/background and the localization network has to generalize to
similar regions. Then, through iterations, this random sampling is expected to
stimulate convolution filters to have similar response to visually similar
nearby pixels.

The collected pixel-level pseudo-labels are used to train a U-Net style
localization network [37], as illustrated in Fig.3. It takes the image as
input, and yields a foreground and a background CAMs for localization. To
train this model, we consider a composite loss that leverages local and global
constraints. Local constraints aim to guide learning at pixel level. To this
end, standard partial cross-entropy is used to exploit sampled pixel-wise
pseudo-labels. To produce a consistent CAM that is well aligned with object
boundaries, we also include a Conditional Random Field (CRF) loss [42]. To
further ensure that the localized object is well aligned with the image-class
label, we add a global constraint over the foreground CAM. In particular, we
feed the product of the image by the foreground CAM to the pretrained
classifier. Then, the classifier response is maximized with respect to the
true image-class. During training, only the weights of the localization
network are optimized. The SST and classifier models are frozen. At the end of
the training, only the localization network and classifier models are retained
for classification and localization tasks. The rest of this section provides
more details on DiPS components.

###  3.3 Training Architecture

In this paper, two goals are sought: (i) classification of object category in
the image, (ii) localization of the object of interest through an encoder-
decoder localization network that is trained using pseudo-labels. These
pseudo-labels are obtained from class token of the different heads from the
last layer of the transformer. We represent the training set by
$\mathbb{T}=\\{\mathrm{x}_{i},\mathrm{y}_{i}\\}$ and $\mathrm{x}\in R^{W\times
H}$ represents an image with a size of $W\times H$ and its corresponding label
of $K$ classes is represented by $y\in\\{1,\dots,K\\}$. DiPS employs a
mechanism to sample background and foreground regions from activation maps
(class token) to build effective pseudo-labels, as shown in Figure 3. To
achieve these goals, we propose a model that consists of three modules (i)
transformer $\mbox{\hskip 7.22223pt{1 0 .3 1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathrm{T}$}}}_{SST}$ that was trained
in self-supervised fashion (as explained above) for producing an attention map
(ii) localization network $\mbox{\hskip 6.5278pt{1 0 .3 1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathrm{F}$}}}_{\theta}$ for producing
soft-max activation maps represented by $M=\mbox{\hskip 6.5278pt{1 0 .3
1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathrm{F}$}}}_{\theta}(x)\in[0,1]^{n\times
m\times 2}$; here first channel represent the foreground $M_{1}$ and other
represent the background map $M_{2}$ trained using best proposal extracted
from class tokens (iii) a classification layer  1 0 .3 1$\mathrm{C}$ that is
trained on top of transformer’s features.

Selection of Background and Foreground Pixels: In order to train the
localization map predictor, a pixel-level supervisory signal is employed. At
each epoch, new pseudo-labels are generated that consist of few foreground and
background pixels (pseudo-pixels) to increase network certainty of different
regions during training. To select pseudo-pixels, we extract attention maps
from the last layer of the transformer, corresponding to class tokens. From
the maps, we select only the first four maps and an average map of all class
tokens. After this, we apply the Otsu [32] thresholding method for converting
the attention map into a binary map to obtain all connected regions that are
greater than a minimum size222The minimum size is an hyperparameter value. to
extract all $B$ proposal (bounding) boxes enclosing those regions. Then, for
an input image, we produce $B$ images corresponding to each bounding box by
applying the Gaussian-blur filter to hide the background regions outside of
the respective bounding box. Each image is then passed to a classifier  1 0 .3
1$\mathrm{C}$, and the top P best proposals along with their underlying
attention maps are selected based on the classifier’s confidence corresponding
to target class $\mathrm{y}_{i}$. Then, one out of P proposals along with its
respective bounding box is selected for pseudo labeling. The pixels inside and
outside of the bounding box are considered as foreground  1 0 .3
1$\mathbb{Y}_{i}^{+}$ and background regions  1 0 .3 1$\mathbb{Y}_{i}^{-}$
respectively defined as:

$\mbox{\hskip 9.42929pt{1 0 .3 1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathbb{Y}_{i}^{+}$}}}=\varphi^{+}(Y_{i},n^{+})\quad\quad\quad\mbox{\hskip
9.42929pt{1 0 .3 1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathbb{Y}_{i}^{-}$}}}=\varphi^{-}(Y_{i},n^{-})$
(3)

Figure 4: Visualization of test samples from CUB. Results of baseline method
including TS-CAM shows that these method can only cover some parts of the
object and invisible parts are also included in the bounding box due to
extensive search of thresholds. In contrast, our method learns to predict map
that can cover the whole object. Moreover, maps in red box are used to sample
pseudo-labels and our methods learns to localize the object precisely and able
remove noise from them.

where $Y_{i}$ set of all pixel in the generated map,  1 0 .3
1$\mathbb{Y}_{i}^{+}$ represents the top $n\%$ pixels within the bounding box
selected through multinomial sampling and  1 0 .3 1$\mathbb{Y}_{i}^{-}$
represents top $-n\%$ pixels anywhere that are derived using uniform sampling
from the activations of whole image sorted in reverse order. These pixels are
derived from unreliable attention maps and may contain incorrect labels. To
deal with this uncertainty, we select only a few (see experimental details for
exact values) pixels from the foreground and background while rejecting others
[39, 40]. Therefore, the final set of foreground and background pixels for
calculating loss are:

$\varphi(\mbox{\hskip 7.50002pt{1 0 .3 1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathbb{Y}$}}}_{i}^{+,-})=\varphi(\mbox{\hskip
9.42929pt{1 0 .3 1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathbb{Y}_{i}^{+}$}}})\cup\varphi(\mbox{\hskip
9.42929pt{1 0 .3 1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathbb{Y}_{i}^{-}$}}})$ (4)

here $\varphi(\mbox{\hskip 9.42929pt{1 0 .3 1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathbb{Y}_{i}^{+}$}}})$ and
$\varphi(\mbox{\hskip 9.36668pt{1 0 .3 1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathbb{Y}^{-}$}}})$ represent set of
pixels (hyper-parameter) sampled using multinomial distribution. At the end,
we then generated pseudo label $Y\in\\{0,1\\}^{2}$ by assigning 1’s to pixel
in  1 0 .3 1$\mathbb{Y}_{i}^{+}$ and 0’s to pixels in set  1 0 .3
1$\mathbb{Y}_{i}^{-}$.

Overall Training Loss: Our training loss consists of three terms; (i)
classifier’s loss to ensure the integrity of generated map. The generated map
$M_{1}$ is overlaid on the image $x$ and passed it to the classifier to
compute a cross-entropy loss;
$\mathcal{L}_{CLS}=-\sum_{k=1}^{K}y_{k}log(softmax(\mbox{\hskip 7.22223pt{1 0
.3 1}\hbox to0.0pt{\hss\leavevmode\hbox{\set@color$\mathrm{C}$}}}{(x\odot
M_{1})})_{k})$. (ii) Our constrained pixel alignment loss333Loss using few
pixels was also employed by [6]. for learning foreground/background regions.
It aligns the output map $M$ with the selected pixels in $\varphi(\mbox{\hskip
7.50002pt{1 0 .3 1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathbb{Y}$}}}_{i}^{+,-})$ through
partial cross-entropy denoted by $\mathcal{L}_{CPA}(\varphi(\mbox{\hskip
7.50002pt{1 0 .3 1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathbb{Y}$}}}_{i}^{\;r}),M^{\;r})$;
here $r$ represent the selected pixels. (iii) Conditional Random Field (CRF)
to align the localization map with the object boundaries. The detailed
description of the CRF loss $\mathcal{L}_{CRF}$ is presented in supplementary
material. Furthermore, the overall can be formulated as,

$\mathcal{L}_{Total}=\min_{\theta}\lambda_{CLS}\mathcal{L}_{CLS}+\lambda_{CPA}\mathcal{L}_{CPA}+\lambda_{CRF}\mathcal{L}_{CRF}$
(5)

where $\lambda_{CLS}$ and $\lambda_{CPA}$ are hyperparameters between interval
[0, 1], and $\lambda_{CRF}$ is set to $2e^{-9}$ as defined in [42].

##  4 Results and Discussion

###  4.1 Experimental Methodology

Datasets: For validation, we use 3 commonly used WSOL datasets: CUB-200-2011,
OpenImages and ILSVRC. CUB200-2011 consists of 11,788 images divided into 200
categories; 5,794 for testing and 5,994 for training [51]. For validation and
hyperparameter search, we employed an independent validation set collected by
[13] containing 1,000 images. OpenImages [7, 13] contains 37,319 images
divided into 100 classes. From this data set, we considered 29,819 images for
training and 5000 images for testing. The remaining 2,500 images are used as
validation set. ILSVRC consists of 1000 classes with approximately 1.2M
images, from which we extracted 50K images for training and testing, and 10K
images for validation as in [13].

Evaluation measures: We analyze our results based on two evaluation measures,
as suggested in [13] – (i) PxAP refers to the pixel wise precision and recall
at particular threshold, while (ii) MaxBoxAccV2 refers to the proportion of
predicted bounding boxes that have IoU greater than particular threshold
concerning the generated map. It is averaged over three different IoU
thresholds $\delta\in\\{30\%,50\%,70\%\\}$. Additionally, for CUB-200-2011 and
ILSVRC datasets, we also reported three additional localization matrices in
supplementary material.

Implementation details: We follow the protocol in [13] for all experiments. A
batch size of 32 is used for all datasets. For training, images are resized to
$256\times 256$, and then randomly cropped to $224\times 224$ and randomly
flipped horizontally as in [13]. On CUB models are trained for 50 epochs,
while on ImageNet and OpenImages models are trained for 10 epochs. The
learning rate is decayed by a factor of 0.1 after the $15^{th}$ epoch for CUB
and $3^{rd}$ epoch for ImageNet and OpenImages.

Baseline Models: To evaluate the performance of our proposed model, we
compared it with several state-of-the-art methods presented in Tables 1 and 2.
We report results from [13] for CAM [64], HaS [39], ACoL [61], SPG [62], ADL
[14], and CutMix [58]. For all other models, we report the same results as
published in their respective articles. For qualitative evaluation, we
reproduced the results of CAM [64], HaS [39], ACoL [61], SPG [62], ADL [14],
CutMix [58] and TS-CAM [18] for CUB dataset according to the protocols defined
in [13]. For OpenImages, we compare results with VITOL [20].

Methods |  | OpenImages |  | CUB
---|---|---|---|---
|  | (PxAP) |  | (MaxBoxAccV2)
CAM [64] _(cvpr,2016)_ |  | 63.2 |  | 63.7
HaS [39] _(iccv,2017)_ |  | 58.1 |  | 64.7
ACoL [61] _(cvpr,2018)_ |  | 57.3 |  | 66.5
SPG [62] _(eccv,2018)_ |  | 62.3 |  | 60.4
ADL [14] _(cvpr,2019)_ |  | 58.7 |  | 66.3
CutMix [58] _(eccv,2019)_ |  | 62.5 |  | 62.8
PAS [2] _(eccv,2020)_ |  | 63.3 |  | –
ICL [22] _(accv,2020)_ |  | – |  | 63.1
TS-CAM [18] _(iccv,2021)_ |  | – |  | 76.7
CAM-IVR [24] _(iccv,2021)_ |  | 63.6 |  | 66.9
PDM [30] _(tip,2022)_ |  | – |  | 72.4
C2AM [54] _(cvpr,2022)_ |  | – |  | 83.8
ViTOL-GAR [20] _(cvpr,2022)_ |  | – |  | 72.4
ViTOL-LRP [20] _(cvpr,2022)_ |  | – |  | 73.1
TRT [41] _(corr,2022)_ |  | – |  | 82.0
BGC [23] _(cvpr,2022)_ |  | – |  | 80.1
SCM [3] _(eccv,2022)_ |  | – |  | 89.9
BR-CAM [66] _(eccv,2022)_ |  | 67.6 |  | –
CREAM [55] _(cvpr,2022)_ |  | 64.7 |  | 73.5
F-CAM [6] _(wacv,2022)_ |  | 72.2 |  | 80.1
DiPS (ours) |  | 76.3 |  | 90.9

Table 1: PxAP and MaxBoxAccV2 performance on the CUB and OpenImages datasets,
respectively.

###  4.2 Comparison with State-of-Art Methods

Quantitative comparison. Table 1 shows that DiPS achieves state-of-the-art
performance on CUB and OpenImages for MaxBoxAccV2 and PxAP matrices compared
to the other methods in the literature. On ILSVC dataset, DiPS MaxBoxAccV2 is
very competitive. Analyzing the accuracy at different IoU thresholds $\delta$,
we see that our approach outperforms the other methods with high IoU, which
means that our method produces a more accurate localization, as we can see in
the following qualitative evaluation.

|  | ILSVRC (MaxBoxAccV2)
---|---|---
Method |  | $\delta$ |  | Mean
|  | 0.3 | 0.5 | 0.7 |
CAM [64] _(cvpr,2016)_ |  | 83.7 | 65.7 | 41.6 |  | 63.7
HaS [39] _(iccv,2017)_ |  | 83.7 | 65.2 | 41.3 |  | 63.4
SPG [62] _(eccv,2018)_ |  | 83.9 | 65.4 | 40.6 |  | 63.7
ADL [14] _(cvpr,2019)_ |  | 83.6 | 65.6 | 41.8 |  | 63.7
CutMix [58] _(eccv,2019)_ |  | 83.6 | 65.6 | 41.8 |  | 63.7
ICL [22] _(accv,2020)_ |  | 84.3 | 67.6 | 43.6 |  | 65.2
ViTOL-LRP [20] _(cvpr,2022)_ |  | 87.6 | 72.8 | 50.2 |  | 70.4
BGC [23] _(cvpr,2022)_ |  | 86.7 | 71.1 | 48.3 |  | 68.7
DiPS (ours) |  | 83.0 | 69.0 | 51.3 |  | 67.8

Table 2: MaxBoxAccV2 performance on the ILSVRC dataset. The results of some
baseline methods are borrowed from [23]. Figure 5: Our methods sample pseudo-
labels from maps presented in red box and able to localize very well. These
samples are randomly selected from OpenImages test set. Figure 6: Test samples
from ILSVRC. Compare to VITOL, TS-CAM and class token of SST (in red box), our
proposed DiPS provides better coverage of the object of interest. Moreover,
the second and third column shows the results of ViTOL and our proposed
method, respectively. Column from 4-9, represent the attention map (class
token) of SST [10].

Qualitative comparison. The visual results of our method along with the
corresponding baselines are presented in Fig.4, 5 and 6 respectively. The
baseline method tends to focus on discriminative regions and expands to less
discriminative regions because of the blobby nature of these maps. However,
the selection of the optimal threshold allows the less discriminative area to
be included in the localization map because of their texture or color
similarity. Furthermore, the resultant map becomes unreliable and unable to
identify the whole object as it encompasses regions with no concealed
activation over an object. For CUB dataset, we compare the results with the
baseline methods (Fig.4) including TS-CAM and show that our method is capable
of covering the whole object with the same intensities instead of highlighting
different parts of the object. Additionally, the class tokens of SST are also
presented, and DiPS is able to successfully harvest useful information from
them by constructing a reliable pseudo-label. Similarly, on the ILSVRC
dataset, DiPS achieved state-of-the-art visual results by localizing all parts
of the object with the same confidence instead of hot-spotting different
regions. In OpenImages, DiPS is also able to learn object structure from noisy
pseudo-labels, as shown in Fig.5.

More specifically, current state-of-the-art methods (e.g. TS-CAM, VITOL) focus
on some regions of the concerned object by hot-spotting different parts. Due
to this they are able to properly draw a bounding box around the object due to
extensive threshold search that can include low scoring areas to the
localization map. In contrast, our method can produce a map that covers the
whole object with sharper boundaries, which eliminates the need for a precise
threshold value. Additionally, we compare the results of our model with SST’s
activation and show that our method learns to localize well even with noisy
pseudo-labels.

Analysis of distribution shift. We present the effect of varying the threshold
values on the localization performance along with the distribution spread of
foreground and background regions. The change in MaxBoxAcc at different
threshold for CUB test set is presented in Figure 7. We compare our results
with CAM [64], HaS [39], ACoL [61], SPG [62], ADL [14], and CutMix [58]. For
all of these related methods, we can see that the MaxBoxAcc quickly drops to
zero as the threshold increases. Due to this, it is very difficult to find the
optimal threshold for each image and makes these methods unfeasible to deploy
in practical scenarios. In contrast, the output of our method is less
susceptible to the threshold.

Note:  Additional supporting material is presented in supplementary section
including; additional visual results, evaluation matrix for error dissection,
extended error analysis, ablation study, and an overview of CRF loss.

Figure 7: MaxBoxAcc performance at different IoU threshold values for output
map using the CUB dataset.

##  5 Conclusion

In this paper, we proposed a DL method for WSOL, capable of producing fine-
grained object localization maps training by pseudo-labels. These pseudo-
labels are harvested by sampling foreground and background areas from the
class token of SST. Our method is able to generate very good maps compared to
the class token used to build pseudo-labels. The proposed method is capable of
generating reliable maps, thus providing better coverage of the whole object.
To validate the performance of our model, we compare it’s results with the
existing baseline method, and our method achieved competitive performance both
in terms of visual and quantitatively way. Compared to baseline, our method
can identify all parts of the object instead of decomposing and highlighting
different parts of the objects. Furthermore, the proposed method eliminates
the need for extensive threshold search to produce an optimal bounding box
covering the concerned object.

Acknowledgement: This work was funded and supported by Ericsson - Global
Artificial Intelligence Accelerator in Montreal and Mitacs Accelerate. We also
acknowledge Calcul Quebec for the provision of computing resources.

##  Appendix A Supplementary Material

This section contains the following material:

  1. A.1

an evaluation measures and error dissection;

  2. A.2

an overview of CRF loss;

  3. A.3

additional error analyses;

  4. A.4

ablation studies;

  5. A.5

additional visual results;

  6. A.6

details on the TS-CAM method.

###  A.1 Performance Measures and Error Dissection

####  A.1.1 Evaluation Measure for Error Dissection

In this section, we present the evaluation measures that are used in [18] for
error dissection over wrong predictions. These measures are useful for
deciding threshold values for producing bounding boxes from localization maps.
Specifically, localization part error (LPE) and localization more error (LME)
help in deciding whether to increase or decrease the threshold value for
optimal results. More details on error measures are given below:

Localization Part Error (LPE):  This measure identifies that an object
partially detected by the localization map with a large margin has an
intersection over the predicted bounding box (IoP) $>$ 0.5.

Localization More Error (LME):  It indicates that the predicted bounding box
is larger than the actual box and covers other objects or background. This can
be identified if intersection over annotated-bounding-box (IoA) $>$ 0.7.

Multi-instance Error (MIns-Error): It shows that the predicted bounding box
intersects with more than one bounding box with an intersection over ground
truth box (IoG) $>$ 0.3.

####  A.1.2 Additional Performance Measures

Top-1 Localization Accuracy (Top-1 Loc):  A prediction is considered true if
the predicted class is the same as the ground truth and the intersection over
Union (IoU) is greater than 0.5.

Top-5 Localization Accuracy (Top-5 Loc):  A prediction is considered true if
the IoU is greater than 0.5 and the actual class matches at least one of the
top 5 predicted classes.

###  A.2 Overview of CRF loss

Conditional random fields (CRF) loss, aligns the predicted localization map
$\mathrm{M}$ with the boundaries of a concerned object presented in input
$\mathrm{x}$. CRF loss [42] between $\mathrm{x}$ and $\mathrm{M}$ can be
defined as follows:

$\mathcal{L}_{CRF}(\mathrm{A},\mathrm{M})=\sum_{i=0}^{i=1}{M_{i}}^{T}A(1-M_{i})$
(S1)

where $\mathrm{A}$ represents an affinity measure that contains mutual
similarities between pixels, including proximity and color information. For
capturing affinities of pixels, we use a Gaussian kernel [25] and employ
permutohedral lattice [1] to reduce the computation overhead.

| CUB |  | ILSVRC
---|---|---|---
|  LPE $\downarrow$ |  LME $\downarrow$ |  |  MIns-Error $\downarrow$ |  LPE $\downarrow$ |  LME $\downarrow$
VGG16 | 21.91 | 10.53 |  | 10.65 | 3.85 | 9.58
InceptionV3 | 23.09 | 5.52 |  | 10.36 | 3.22 | 9.49
TS-CAM [18] | 6.30 | 2.85 |  | 9.13 | 3.78 | 7.65
DiPS (our) | 0.05 | 0.07 |  | 2.81 | 1.98 | 3.2

Table S1: Extended error analysis on the CUB-200-2011 and ILSVRC datasets.

.

###  A.3 Extended Error Analysis

Further error analysis (according to the error measures defined in Section
A.1.1) on the CUB and ILSVRC datasets are presented in Table S1. Our method
localized the correct region of the concerned object instead of overestimating
or underestimating the region. In contrast to related methods, MIns-Error on
ILSVRC datasets shows that our method is able to localize a particular object
instead of localizing more than one object. It also shows that the maps
generated by DiPS are very robust and have much fewer errors compared to the
baseline methods. The statistics of the baseline methods are from [18].

###  A.4 Additional Ablation Studies

The performance of DiPS with various loss function combinations is shown in
Table S2. It shows that all of the auxiliary losses contribute significantly
towards the final performance. Also, training through arbitrary selection of
pixels (pseudo-labels) rather than the classifier loss or fixed pseudo-labels
allows DL models to explore different regions of an object and can provide
accurate localization. Adding CRF and classification terms at the same time
significantly improves the performance of our model. Furthermore, the
MaxBoxAcc, top-1 and top-5 localization accuracy for CUB and ILSVRC dataset
are presented in Table S3 and S4, respectively. We achieved state-of-the-art
performance on the CUB dataset and maintained competitive performance on the
ILSVRC dataset. As mentioned above, DiPS outperformed baseline methods on
higher IoU thresholds and also accomplished a performance gain on MaxBoxAccV2
(as described in the main paper).

|  | CUB |  | ILSVRC |  | OpenImages
---|---|---|---|---|---|---
|  | (MaxBoxAcc) |  | (MaxBoxAcc) |  | (PxAP)
$\mathcal{L}_{CPA}+\mathcal{L}_{CRF}$ |  | 95.4 |  | 68.178 |  | 65.1
$\mathcal{L}_{CPA}+\mathcal{L}_{CLS}$ |  | 94.6 |  | 68.14 |  | 64.6
$\mathcal{L}_{CPA}+\mathcal{L}_{CRF}+\mathcal{L}_{CLS}$ |  | 97.0 |  | 69.0 |  | 76.3

Table S2: Localization performance of our DiPS method with different losses.

|  | CUB
---|---|---
|  |  | MaxBoxAcc |  top-1 Loc Acc |  top-5 Loc Acc
CAM [64] _(cvpr,2016)_ |  |  | 73.2 | 56.1 | –
HaS [39] _(iccv,2017)_ |  |  | 78.1 | 60.7 | –
ACoL [61] _(cvpr,2018)_ |  |  | 72.7 | 57.8 | –
SPG [62] _(eccv,2018)_ |  |  | 63.7 | 51.5 | –
ADL [14] _(cvpr,2019)_ |  |  | 75.7 | 41.1 | –
CutMix [58] _(eccv,2019)_ |  |  | 71.9 | 54.5 | –
ICL [22] _(accv,2020)_ |  |  | 57.5 | – | –
TS-CAM [18] _(iccv,2021)_ |  |  | 87.7 | 71.3 | 83.8
BR-CAM [66] _(eccv,2022)_ |  |  | – | – | –
CREAM [55] _(cvpr,2022)_ |  |  | 90.9 | 71.7 | 86.3
BGC [23] _(cvpr,2022)_ |  |  | 93.1 | 70.8 | 88.0
F-CAM [6] _(wacv,2022)_ |  |  | 92.4 | 59.3 | 82.7
DiPS (ours) |  |  | 97.0 | 78.8 | 91.3
DiPS (ours) (w/ TransFG classifier [21]) |  |  | 97.0 | 88.2 | 95.6

Table S3: Extended performance analysis on the CUB dataset.

|  | ILSVRC
---|---|---
|  |  | MaxBoxAcc |  top-1 Loc Acc |  top-5 Loc Acc
CAM [64] _(cvpr,2016)_ |  |  | 65.7 | 42.8 | 54.86
ACoL [61] _(cvpr,2018)_ |  |  | 45.8 | 59.4 | 62.9
SPG [62] _(eccv,2018)_ |  |  | 65.4 | 48.6 | –
ADL [14] _(cvpr,2019)_ |  |  | 65.6 | 51.5 | –
CutMix [58] _(eccv,2019)_ |  |  | 65.6 | 51.5 | –
ICL [22] _(accv,2020)_ |  |  | 67.6 | – | –
ViTOL-LRP [20] _(cvpr,2022)_ |  |  | 72.8 | 58.64 | –
TS-CAM [18] _(iccv,2021)_ |  |  | 67.6 | 53.4 | 64.3
BGC [23] _(cvpr,2022)_ |  |  | 49.94 | 65.7 | 69.8
DiPS (ours) |  |  | 69.07 | 55.87 | 66.03

Table S4: Extended performance analysis on the ILSVRC dataset.

###  A.5 Visual Results

Visual representation of our method compared to baseline methods on CUB and
OpenImages are illustrated in Fig.S2 and Fig.S3. Our method generates a very
smooth map instead of hot-spotting different parts of the concerned object.
Ultimately, the map generated by our method does not require an extensive
threshold search to find an optimal bounding box. Compared to the class tokens
of SST (used to harvest pseudo-labels), our method is able to learn an
effective localization map from noisy pseudo-labels.

###  A.6 Details on Baseline Method: TS-CAM

By taking advantage of the attention mechanism, TS-CAM [18] is capable of
capturing long-range dependency among different parts of an image. As a
result, it can efficiently separate background and foreground objects. In
other words, it first divides the images into a set of patches for capturing
long-range dependency information and records its effects in class token. The
attention of class token is then fused with the semantic aware map to produce
the final attention/activation map. The flow diagram of TS-CAM is depicted in
Fig.S1. Lastly, a detailed visualization of the internal representation of the
token TS-CAM class is presented in Fig.S4. It shows that the average of all
maps could potentially include noise and background regions in the final
prediction.

Figure S1: Illustration of the baseline Token Semantic Coupled Attention Map
(TS-CAM) model for WSOL.

Figure S2: Examples of visual results on the OpenImages dataset. Figure S3:
Examples of visual results on the CUB-200-2011 dataset. Figure S4: Attention
map of each transformer head (class token) learned by TS-CAM. However,
different parts of the object are accumulated across the layers/blocks, and it
must include semantic aware CAM to suppress noise and generate final results.

## References

  * [1] Andrew Adams, Jongmin Baek, and Myers Abraham Davis.  Fast high-dimensional filtering using the permutohedral lattice.  In Computer graphics forum, volume 29, pages 753–762. Wiley Online Library, 2010.
  * [2] Wonho Bae, Junhyug Noh, and Gunhee Kim.  Rethinking class activation mapping for weakly supervised object localization.  In European Conference on Computer Vision, pages 618–634. Springer, 2020.
  * [3] Haotian Bai, Ruimao Zhang, Jiong Wang, and Xiang Wan.  Weakly supervised object localization via transformer with implicit spatial calibration.  arXiv preprint arXiv:2207.10447, 2022.
  * [4] David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba.  Network dissection: Quantifying interpretability of deep visual representations.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6541–6549, 2017.
  * [5] S. Belharbi, M Pedersoli, I. Ben Ayed, L. McCaffrey, and E. Granger.  Negative evidence matters in interpretable histology image classification.  In MIDL, 2022.
  * [6] S. Belharbi, A. Sarraf, M. Pedersoli, I. Ben Ayed, L. McCaffrey, and E. Granger.  F-cam: Full resolution class activation maps via guided parametric upscaling.  In WACV, 2022.
  * [7] Rodrigo Benenson, Stefan Popov, and Vittorio Ferrari.  Large-scale interactive object segmentation with human annotators.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11700–11709, 2019.
  * [8] Stephen Boyd and Lieven Vandenberghe.  Convex optimization. cambridge university press, 2004.  Google Scholar Google Scholar Digital Library Digital Library.
  * [9] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  arXiv preprint arXiv:2006.09882, 2020.
  * [10] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin.  Emerging properties in self-supervised vision transformers.  In ICCV, 2021.
  * [11] Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian.  Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks.  In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 839–847. IEEE, 2018.
  * [12] Zhiwei Chen, Changan Wang, Yabiao Wang, Guannan Jiang, Yunhang Shen, Ying Tai, Chengjie Wang, Wei Zhang, and Liujuan Cao.  Lctr: On awakening the local continuity of transformer for weakly supervised object localization.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 410–418, 2022.
  * [13] Junsuk Choe, Seong Joon Oh, Seungho Lee, Sanghyuk Chun, Zeynep Akata, and Hyunjung Shim.  Evaluating weakly supervised object localization methods right.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3133–3142, 2020.
  * [14] Junsuk Choe and Hyunjung Shim.  Attention-based dropout layer for weakly supervised object localization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2219–2228, 2019.
  * [15] J. Dolz, C. Desrosiers, and I. Ben Ayed.  3D fully convolutional networks for subcortical segmentation in MRI: A large-scale study.  NeuroImage, 2018.
  * [16] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [17] Ruigang Fu, Qingyong Hu, Xiaohu Dong, Yulan Guo, Yinghui Gao, and Biao Li.  Axiom-based grad-cam: Towards accurate visualization and explanation of cnns.  arXiv preprint arXiv:2008.02312, 2020.
  * [18] Wei Gao, Fang Wan, Xingjia Pan, Zhiliang Peng, Qi Tian, Zhenjun Han, Bolei Zhou, and Qixiang Ye.  Ts-cam: Token semantic coupled attention map for weakly supervised object localization.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2886–2895, 2021.
  * [19] Guangyu Guo, Junwei Han, Fang Wan, and Dingwen Zhang.  Strengthen learning tolerance for weakly supervised object localization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7403–7412, 2021.
  * [20] Saurav Gupta, Sourav Lakhotia, Abhay Rawat, and Rahul Tallamraju.  Vitol: Vision transformer for weakly supervised object localization.  arXiv preprint arXiv:2204.06772, 2022.
  * [21] Ju He, Jie-Neng Chen, Shuai Liu, Adam Kortylewski, Cheng Yang, Yutong Bai, and Changhu Wang.  Transfg: A transformer architecture for fine-grained recognition.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 852–860, 2022.
  * [22] Minsong Ki, Youngjung Uh, Wonyoung Lee, and Hyeran Byun.  In-sample contrastive learning and consistent attention for weakly supervised object localization, 2020.
  * [23] Eunji Kim, Siwon Kim, Jungbeom Lee, Hyunwoo Kim, and Sungroh Yoon.  Bridging the gap between classification and localization for weakly supervised object localization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14258–14267, 2022.
  * [24] Jeesoo Kim, Junsuk Choe, Sangdoo Yun, and Nojun Kwak.  Normalization matters in weakly supervised object localization.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3427–3436, 2021.
  * [25] Vladlen Koltun et al.  Efficient inference in fully connected crfs with gaussian edge potentials.  Adv. Neural Inf. Process. Syst, 4, 2011.
  * [26] A. Krizhevsky, I. Sutskever, and G. E. Hinton.  Imagenet classification with deep convolutional neural networks.  In NeurIPS. 2012.
  * [27] Jungbeom Lee, Eunji Kim, Sungmin Lee, Jangho Lee, and Sungroh Yoon.  Ficklenet: Weakly and semi-supervised semantic image segmentation using stochastic inference.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5267–5276, 2019.
  * [28] Ming Li.  Caft: Clustering and filter on tokens of transformer for weakly supervised object localization.  arXiv preprint arXiv:2201.00475, 2022.
  * [29] Meng Meng, Tianzhu Zhang, Qi Tian, Yongdong Zhang, and Feng Wu.  Foreground activation maps for weakly supervised object localization.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3385–3395, 2021.
  * [30] Meng Meng, Tianzhu Zhang, Wenfei Yang, Jian Zhao, Yongdong Zhang, and Feng Wu.  Diverse complementary part mining for weakly supervised object localization.  IEEE Transactions on Image Processing, 31:1774–1788, 2022.
  * [31] Rakshit Naidu, Ankita Ghosh, Yash Maurya, Soumya Snigdha Kundu, et al.  Is-cam: Integrated score-cam for axiomatic-based explanations.  arXiv preprint arXiv:2010.03023, 2020.
  * [32] Nobuyuki Otsu.  A threshold selection method from gray-level histograms.  IEEE transactions on systems, man, and cybernetics, 9(1):62–66, 1979.
  * [33] Xingjia Pan, Yingguo Gao, Zhiwen Lin, Fan Tang, Weiming Dong, Haolei Yuan, Feiyue Huang, and Changsheng Xu.  Unveiling the potential of structure preserving for weakly supervised object localization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11642–11651, 2021.
  * [34] Amir Rahimi, Amirreza Shaban, Thalaiyasingam Ajanthan, Richard Hartley, and Byron Boots.  Pairwise similarity knowledge transfer for weakly supervised object localization.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIV 16, pages 395–412. Springer, 2020.
  * [35] Harish Guruprasad Ramaswamy et al.  Ablation-cam: Visual explanations for deep convolutional network via gradient-free localization.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 983–991, 2020.
  * [36] J. Redmon, S. K. Divvala, R. B. Girshick, et al.  You only look once: Unified, real-time object detection.  In CVPR, 2016.
  * [37] O. Ronneberger, P. Fischer, and T. Brox.  U-net: Convolutional networks for biomedical image segmentation.  In MICCAI, pages 234–241, 2015.
  * [38] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In Proceedings of the IEEE international conference on computer vision, pages 618–626, 2017.
  * [39] Krishna Kumar Singh and Yong Jae Lee.  Hide-and-seek: Forcing a network to be meticulous for weakly-supervised object and action localization.  In 2017 IEEE international conference on computer vision (ICCV), pages 3544–3553. IEEE, 2017.
  * [40] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: a simple way to prevent neural networks from overfitting.  The journal of machine learning research, 15(1):1929–1958, 2014\.
  * [41] Hui Su, Yue Ye, Zhiwei Chen, Mingli Song, and Lechao Cheng.  Re-attention transformer for weakly supervised object localization.  arXiv preprint arXiv:2208.01838, 2022.
  * [42] Meng Tang, Federico Perazzi, Abdelaziz Djelouah, Ismail Ben Ayed, Christopher Schroers, and Yuri Boykov.  On regularized losses for weakly-supervised cnn segmentation.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 507–522, 2018.
  * [43] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In International Conference on Machine Learning, pages 10347–10357. PMLR, 2021.
  * [44] Haofan Wang, Rakshit Naidu, Joy Michael, and Soumya Snigdha Kundu.  Ss-cam: Smoothed score-cam for sharper visual feature localization.  arXiv preprint arXiv:2006.14255, 2020.
  * [45] Haofan Wang, Zifan Wang, Mengnan Du, Fan Yang, Zijian Zhang, Sirui Ding, Piotr Mardziel, and Xia Hu.  Score-cam: Score-weighted visual explanations for convolutional neural networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 24–25, 2020.
  * [46] Xiang Wang, Shaodi You, Xi Li, and Huimin Ma.  Weakly-supervised semantic segmentation by iteratively mining common object features.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1354–1362, 2018.
  * [47] Yude Wang, Jie Zhang, Meina Kan, Shiguang Shan, and Xilin Chen.  Self-supervised equivariant attention mechanism for weakly supervised semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12275–12284, 2020.
  * [48] Jun Wei, Qin Wang, Zhen Li, Sheng Wang, S Kevin Zhou, and Shuguang Cui.  Shallow feature matters for weakly supervised object localization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5993–6001, 2021.
  * [49] Yunchao Wei, Jiashi Feng, Xiaodan Liang, Ming-Ming Cheng, Yao Zhao, and Shuicheng Yan.  Object region mining with adversarial erasing: A simple classification to semantic segmentation approach.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1568–1576, 2017.
  * [50] Yunchao Wei, Huaxin Xiao, Honghui Shi, Zequn Jie, Jiashi Feng, and Thomas S Huang.  Revisiting dilated convolution: A simple approach for weakly-and semi-supervised semantic segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7268–7277, 2018.
  * [51] P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona.  Caltech-UCSD Birds 200.  Technical Report CNS-TR-2010-001, California Institute of Technology, 2010\.
  * [52] Pingyu Wu, Wei Zhai, and Yang Cao.  Background activation suppression for weakly supervised object localization.  arXiv preprint arXiv:2112.00580, 2021.
  * [53] Jinheng Xie, Cheng Luo, Xiangping Zhu, Ziqi Jin, Weizeng Lu, and Linlin Shen.  Online refinement of low-level feature based activation map for weakly supervised object localization.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 132–141, 2021.
  * [54] Jinheng Xie, Jianfeng Xiang, Junliang Chen, Xianxu Hou, Xiaodong Zhao, and Linlin Shen.  C2am: Contrastive learning of class-agnostic activation map for weakly supervised object localization and semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 989–998, 2022.
  * [55] Jilan Xu, Junlin Hou, Yuejie Zhang, Rui Feng, Rui-Wei Zhao, Tao Zhang, Xuequan Lu, and Shang Gao.  Cream: Weakly supervised object localization via class re-activation mapping.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9437–9446, 2022.
  * [56] Haolan Xue, Chang Liu, Fang Wan, Jianbin Jiao, Xiangyang Ji, and Qixiang Ye.  Danet: Divergent activation for weakly supervised object localization.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6589–6598, 2019.
  * [57] Seunghan Yang, Yoonhyung Kim, Youngeun Kim, and Changick Kim.  Combinational class activation maps for weakly supervised object localization.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2941–2949, 2020.
  * [58] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6023–6032, 2019.
  * [59] Matthew D Zeiler and Rob Fergus.  Visualizing and understanding convolutional networks.  In European conference on computer vision, pages 818–833. Springer, 2014.
  * [60] Chen-Lin Zhang, Yun-Hao Cao, and Jianxin Wu.  Rethinking the route towards weakly supervised object localization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13460–13469, 2020.
  * [61] Xiaolin Zhang, Yunchao Wei, Jiashi Feng, Yi Yang, and Thomas S Huang.  Adversarial complementary learning for weakly supervised object localization.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1325–1334, 2018.
  * [62] Xiaolin Zhang, Yunchao Wei, Guoliang Kang, Yi Yang, and Thomas Huang.  Self-produced guidance for weakly-supervised object localization.  In Proceedings of the European conference on computer vision (ECCV), pages 597–613, 2018.
  * [63] Xiaolin Zhang, Yunchao Wei, and Yi Yang.  Inter-image communication for weakly supervised localization.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIX 16, pages 271–287. Springer, 2020.
  * [64] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba.  Learning deep features for discriminative localization.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2921–2929, 2016.
  * [65] Z.-H. Zhou.  A brief introduction to weakly supervised learning.  National Science Review, 5(1):44–53, 2017.
  * [66] Lei Zhu, Qian Chen, Lujia Jin, Yunfei You, and Yanye Lu.  Bagging regional classification activation maps for weakly supervised object localization.  arXiv preprint arXiv:2207.07818, 2022.
  * [67] Lei Zhu, Qi She, Qian Chen, Xiangxi Meng, Mufeng Geng, Lujia Jin, Zhe Jiang, Bin Qiu, Yunfei You, Yibao Zhang, et al.  Background-aware classification activation map for weakly supervised object localization.  arXiv preprint arXiv:2112.14379, 2021.