# Learning Common Rationale to Improve Self-Supervised Representation for
Fine-Grained Visual Recognition Problems

Yangyang Shu
Anton van den Hengel
Lingqiao Liu Corresponding author. This work is supported by the Centre for
Augmented Reasoning. School of Computer Science, The University of Adelaide
<EMAIL_ADDRESS>
###### Abstract

Self-supervised learning (SSL) strategies have demonstrated remarkable
performance in various recognition tasks. However, both our preliminary
investigation and recent studies suggest that they may be less effective in
learning representations for fine-grained visual recognition (FGVR) since many
features helpful for optimizing SSL objectives are not suitable for
characterizing the subtle differences in FGVR. To overcome this issue, we
propose learning an additional screening mechanism to identify discriminative
clues commonly seen across instances and classes, dubbed as common rationales
in this paper. Intuitively, common rationales tend to correspond to the
discriminative patterns from the key parts of foreground objects. We show that
a common rationale detector can be learned by simply exploiting the GradCAM
induced from the SSL objective without using any pre-trained object parts or
saliency detectors, making it seamlessly to be integrated with the existing
SSL process. Specifically, we fit the GradCAM with a branch with limited
fitting capacity, which allows the branch to capture the common rationales and
discard the less common discriminative patterns. At the test stage, the branch
generates a set of spatial weights to selectively aggregate features
representing an instance. Extensive experimental results on four visual tasks
demonstrate that the proposed method can lead to a significant improvement in
different evaluation settings.111The source code will be publicly available
at: https://github.com/GANPerf/LCR

##  1 Introduction

Figure 1: GradCAM [30] visualization for MoCo v2 [18], VICReg [4] and our
method on the CUB-200-2011, Stanford Cars and FGVC Aircraft datasets. Compared
with the existing method MoCo v2 and VICReg, which are prone to be distracted
by background patterns, our method can identify features from the foreground
and potentially the key parts of the object.

Recently, self-supervised representations Learning (SSL) has been shown to be
effective for transferring the learned representations to different downstream
tasks [2, 5, 18, 4, 14]. Methods such as contrastive learning [7, 18, 16, 9]
have demonstrated state-of-the-art feature learning capability and have been
intensively studied recently. However, recent studies [11] suggest that
contrastive learning may have a “coarse-grained bias” and could be less
effective for fine-grained classification problems whose goal is to
distinguish visually similar subcategories of objects under the basic-level
category.

This phenomenon is rooted in fine-grained visual recognition (FGVR) properties
and the training objective of SSL. SSL tries to minimize a pretext task, e.g.,
contrastive learning minimizes the distance between same-instance features
while maximizing the distance among different-instance features, and any
visual patterns that could contribute to loss minimization will be learned. On
the other hand, the discriminative patterns for FGVR can be subtle. It is more
likely to reside on key object parts. Thus, the feature learned from SSL may
not necessarily be useful for FGVR. Figure 1 shows our investigation of this
issue. As seen, the existing SSL paradigms, such as MoCo [18] and VICReg [4]
are prone to learn patterns from irrelevant regions. Existing works [42, 31,
38] usually handle this issue by recoursing to pre-trained object part
detectors or saliency detectors to regularize SSL using patterns from valid
regions to achieve the training objective. However, both the part detectors
and saliency detectors could restrict the applicability of SSL for FGVR since
part detectors are trained from human annotations, and the saliency regions
are not always coincident with the discriminative regions.

Therefore, this work aims to directly solve the problem from the target domain
data. Specifically, we propose to learn an additional screening mechanism in
addition to the contrastive learning process. The purpose of the screening
mechanism is to filter out the patterns that might contribute to SSL
objectives but are irrelevant to FGVR. Somehow surprisingly, we find that such
a screening mechanism can be learned from the GradCAM [30] of the SSL loss via
an extremely simple method. The whole process can be described as a “fitting
and masking” procedure: At the training time, we use an additional branch with
limited fitting capacity (see more discussion about it in Section 3.2) to fit
the GradCAM calculated from the SSL objective. At the testing time, we apply
this additional branch to predict an attention mask to perform weighted
average pooling for the final presentation. The motivation for such a design
is that the GradCAM fitting branch tends to characterize the discriminative
patterns that commonly occur across samples due to its limited fitting
capacity, and those common patterns, dubbed common rationale in this paper,
are more likely corresponding to the discriminative clues from key object
parts or at least foreground regions.

We implement our method based on MoCo v2 [8], one of the state-of-the-art
approaches in unsupervised feature learning, which also produces the best
performance on FGVR in our setting. Our implementation uses the training
objective of MoCo v2 to learn feature representations and derive the GradCAM.
Through extensive experiments, we show that our approach can significantly
boost the quality of the learned feature. For example, when evaluating the
learned feature for retrieval tasks, our method achieves 49.69% on the
CUB-200-2011 retrieval task, which is 6.97% higher than our baseline method
(MoCo v2 [8]). In the linear evaluation phase, the proposed method achieves
new state-of-the-art 71.31% Top-1 accuracy, which is 3.01% higher than MoCo
v2.

##  2 Related Work

Self-supervised learning aims to learn feature representation from unlabeled
data. It relies on a pretext task whose supervision could be derived from the
data itself, e.g., image colorization [41], image inpainting [28], and
rotation [24] prediction. Contrastive learning is recently identified as a
promising framework for self-supervised learning and has been extensively
studied [7, 18, 36, 5, 13, 10]. Despite the subtle differences, most
contrastive learning approaches [37, 43, 15] try to minimize the distance
between different views of the same images and push away the views of
different images. The representative methods are SimCLR [7] and MoCo [18].
Besides contrastively learning, consistency-based approaches, such as BYOL
[16], SimSiam [9] and masking-and-prediction-based approaches, such as MAE
[17], BEiT [3], and ADIOS [32] are also proven effective for SSL.

Improving SSL via Better Region Localization. A pipeline to improve the
distinguishing ability is to design better data augmentations of SSL. Three
such methods were recently proposed: DiLo [42], SAGA [39], and ContrastiveCrop
[29]. DiLo uses a copy-and-pasting approach as a kind of augmentation to
create an image with a different background. In such a way, the proposed
method distills localization via learning invariance against backgrounds and
improves the ability of SSL models to localize foreground objects. SAGA adopts
self-augmentation with a guided attention strategy to augment input images
based on predictive attention. In their method, an attention-guided crop is
used to enhance the robustness of feature representation. ContrastiveCrop
shows a better crop as an augmentation to generate better views in SSL and
keep the semantic information of an image. All of these works locate the
object by improving the data augmentations for SSL. In this work, our method
is adaptive to locate the key regions for self-supervised learning without
needing external augmentations. Another family of approaches tries to target
the same problem as ours: making the learned feature capture more of the
foreground region. CVSA [38] proposed a cross-view saliency alignment
framework that first crops and swaps saliency regions of images as a novel
view generation. Then it adopts a cross-view saliency alignment loss to
encourage the model to learn features from foreground objects. CAST [31]
encourages Grad-CAM attention to fit the salient image regions detected by a
saliency detector. Those methods often rely on pre-trained saliency detection.
This implicitly assumes that salient regions are more likely to be
discriminative regions. This assumption, however, does not always hold,
especially for FGVR.

##  3 Method

In this section, we first briefly review self-supervised contrastive learning
[8] and gradient-weighted class activation mapping (GradCAM) [30] as
preliminary knowledge. Then, we introduce the proposed approach.

###  3.1 Preliminary

Self-Supervised Contrastive Learning. Given an image $I$ from a batch of
samples, $x=t(I)$ and $x^{\prime}=t^{\prime}(I)$ are the two augmented views,
where the $t$ and $t^{\prime}$ are two different transformations sampled from
a set of data augmentations $\cal{T}$. Then the views $x$ and $x^{\prime}$ are
used as the input of an encoder $f_{\theta}$ to generate their feature
representations $u=f_{\theta}(x)$ and $u^{\prime}=f_{\theta}(x^{\prime})$.
Finally, the projector heads $g_{\theta}$ are used to map $u$ and $u^{\prime}$
onto the embeddings $z=g_{\theta}(u)$ and $z^{\prime}=g_{\theta}(u^{\prime})$.
Let $(z,z^{\prime})$ be embeddings from the same image and are used as a
positive pair. Let $z_{k}$ be the embedding from a different image, and
$(z,z_{k})$ thus composes a negative pair. SimCLR [7] adopts a contrastive
loss to maximize the agreement of positive pairs over those of negative pairs.
The MoCo [18, 1, 10] family adopts the same contrastive loss but adds a queue
to store the image embeddings to alleviate the memory cost due to the large
batch size. Formally, the contrastive loss takes the following form

$\displaystyle\mathcal{L}_{CL}$ $\displaystyle=-\log\frac{\exp(z\cdot
z^{\prime}/t)}{\sum_{i=0}^{Q}\exp(z\cdot z_{i}/t)},$ (1)

where $t$ denotes a temperature parameter, and $z_{i}$ is the embedding in the
queue.

Gradient-weighted Class Activation Mapping (GradCAM) is a commonly used way to
produce a visual explanation. It identifies the important image regions
contributing to the prediction by using the gradient of the loss function with
respect to the feature map or input images. In this paper, we consider the
gradient calculated with respect to the last convolutional layer feature maps.
Formally, we consider the feature map of the last convolutional layer denoting
as $\phi(I)\in\mathbb{R}^{H\times W\times C}$, $H$, $W$ and $C$ are the
height, width and number of channels of the feature map, respectively. In
standard C-way classification, the GradCAM is calculated by:

$\displaystyle[\mathrm{Grad\text{-}CAM}(\hat{y})]_{i,j}=ReLU\left(\mathbf{\alpha}_{\hat{y}}^{\top}[\phi(I)]_{i,j}\right)$
$\displaystyle
where,~{}\mathbf{\alpha}_{\hat{y}}=\frac{\partial\mathcal{L}_{CE}(P(y),\hat{y})}{\partial[\phi(I)]_{i,j}}\in\mathbb{R}^{C},$
(2)

where $\mathcal{L}_{CE}(P(y),\hat{y})$ is the cross-entropy loss measuring the
compatibility between the posterior probability $P(y)$ and ground-truth class
label $\hat{y}$222 $\mathcal{L}_{CE}(P(y),\hat{y})=\log P(\hat{y})$ for the
multi-classification problem and the gradient of $\log P(y)$ is proportional
to the gradient of the corresponding logit. Those equivalence forms lead to
the different definitions of GradCAM in the literature..
$[\phi(I)]_{i,j}\in\mathbb{R}^{C}$ denotes the feature vector located at the
$(i,j)$-th grid. $\mathrm{Grad\text{-}CAM}(\hat{y})$ denotes GradCAM for the
$\hat{y}$-th class. $[\mathrm{Grad\text{-}CAM}(\hat{y})]_{i,j}$ refers to the
importance value of the $(i,j)$th spatial grid for predicting the $\hat{y}$-th
class.

Note that although GradCAM is commonly used for supervised classification
problems, it can be readily extended to other problems by changing the
corresponding loss function.

###  3.2 Our Method: Learning Common Rationale

Figure 2 gives an overview of the proposed method. Our method is extremely
simple: we merely add one GradCAM fitting branch (GFB) and fit the GradCAM
generated from the CL loss at the training time. At the test time, we let this
GFB predict (normalized) GradCAM and use the prediction as an attention mask
to perform weighted average pooling over the convolutional feature map. The
details about this framework are elaborated as follows.

Figure 2: The overview of our method. At the structure level, it has the
following components: 1)Encoder: MoCo-based contrastive learning is used to
create the image-level representation. 2) GradCAM fitting branch, which is a
convolutional layer with $k$ filters followed by a channel-wise max-out
operator. 3) Inference: at the inference time, the prediction from GFB is used
as a spatial attention mask, and it is used to replace the GAP operation by
weighted average pooling to obtain the image representation.

GradCAM Calculation. For self-supervised learning, we do not have access to
ground-truth class labels. Therefore, we use the contrastive learning loss
$\mathcal{L}_{CL}$ in Eq. 1 to replace $\mathcal{L}_{CE}$ in Eq. 3.1:

$\displaystyle[\mathbf{G}]_{i,j}=ReLU\left(\frac{\partial\mathcal{L}_{CL}(\psi(\phi(I)))}{\partial[\phi(I)]_{i,j}}^{\top}[\phi(I)]_{i,j}\right),$
(3)

where $\psi()$ denotes the feature extractor. Note that in the original
GradCAM, the GradCAM weight indicates how important each region contributes to
classifying the image into the $y$-th class. Similarly, the GradCAM weight
from $\mathcal{L}_{CL}$ indicates the contribution of each region to the
instance discrimination task in the contrastive learning objective. From
Figure 1, we can see that existing Contrastive Learning approaches learn a
diverse set of visual patterns, and not all of them are relevant to the FGVR
task.

Architecture of the GFB. The structure of the GFB plays an important role in
our method. We expect this branch has a limited fitting capability, such that
the branch will not overfit the GradCAM but only capture the commonly
occurring discriminative patterns, i.e., common rationales. Inspired by [33],
we use a convolutional layer with $K$ filter and $1\times 1$ kernel size
followed by the max-out operation [33] as the fitting branch. Formally, such a
branch applies the following operation to the local feature
$[\phi(I)]_{i,j}$at the $(i,j)$-th grid of the feature map:

$\displaystyle A_{i,j}=\max_{k}\\{\mathbf{w}^{\top}_{k}[\phi(I)]_{i,j}\\},$
(4)

where $A_{i,j}$ will be the predicted GradCAM weight at the $(i,j)$-th grid
and $\mathbf{w}_{k}$ is the $k$-th filter (projector) in the convolutional
layer. Intuitively, the convolutional layer can be seen as a collection of $K$
detectors and the above operation can be understood as follows: each
projection vector $\mathbf{w}_{k}$ detects an object part $\mathcal{P}_{k}$;
the max-out operation takes the maximum of $K$ projections at a given
location, which will result in a high value if any of the $K$ object parts are
detected. Varying $K$ could adjust the size of the detector pool and influence
the fitting capability.

Training Loss. In addition to the contrastive learning loss, we require the
GFB to produce a similar attention map as the one produced from the GradCAM of
$\mathcal{L}_{CL}$. We follow [33] to normalize the GradCAM into a probability
distribution and adopt KL divergence as the loss function.

Formally, we normalize the GradCAM $\mathcal{G}$ and $\mathcal{A}$ via the
softmax function:

$\displaystyle[\mathbf{\bar{G}}]_{i,j}=\frac{\exp{([\mathbf{G}]_{i,j}/\tau)}}{\sum_{i=1}^{H}\sum_{j=1}^{W}\exp{{([\mathbf{G}]_{i,j}/\tau)}}},$
$\displaystyle[\mathbf{\bar{A}}]_{i,j}=\frac{\exp{([\mathbf{A}]_{i,j}/\tau)}}{\sum_{i=1}^{H}\sum_{j=1}^{W}\exp{{([\mathbf{A}]_{i,j}/\tau)}}},$
(5)

where $[\cdot]_{i,j}$ denotes the $i,j$-th element of the feature map. $\tau$
is an empirical temperature parameter and we set it to 0.4 here. Thus, the
objective can be expressed as follows:

$\displaystyle\mathcal{L}_{KL}$
$\displaystyle=\sum_{i=1}^{H}\sum_{j=1}^{W}[\mathbf{\bar{A}}]_{i,j}\log\frac{[\mathbf{\bar{A}}]_{i,j}}{[\mathbf{\bar{G}}]_{i,j}},$
(6)

Thus, the final loss function taken on all images over an unlabelled dataset
is shown as followings.

$\displaystyle\mathcal{L}=\lambda\mathcal{L}_{CL}+\nu\mathcal{L}_{KL}.$ (7)

Inference. At the inference time, the GradCAM fitting branch will be firstly
used to produce an attention mask, i.e., prediction of GradCAM. This attention
mask is firstly normalized using
$A^{\prime}_{i,j}=\frac{A_{i,j}-\min(A)}{1e^{-7}+\max(A)}$. Then we use the
normalized attention to perform weighted average pooling of features from the
last-layer convolutional feature map. Formally, it calculates:

$\displaystyle\mathbf{f}=\sum_{i,j}[A^{\prime}]_{i,j}[\phi(I)]_{i,j}\in\mathbb{R}^{C}.$
(8)

$\mathbf{f}$ is used for the downstream tasks.

Discussion. Our approach is inspired by the SAM method in [33]. However, there
are several important differences:

  * •

SAM method was proposed for supervised FGVC under a low-data regime, while our
approach is designed for self-supervised learning.

  * •

Most importantly, our study discovers that for self-supervised feature
learning, the best strategy to interact with the GFB and the feature learning
branch is different from what has been discovered in [33]. Table 1 summarizes
the main differences. As seen, unlike either SAM or SAM-bilinear, we do not
introduce any interaction between the feature learning branch and the GradCAM
fitting branch during training but allow them to perform weighted average
pooling at the test stage. In fact, we find that applying cross-branch
interaction at the training stage will undermine the feature learning process.
This is because applying cross-branch interaction, e.g., using $\mathbf{A}$ to
weighted pool features will prevent CL from exploring discriminative patterns,
especially when $\mathbf{A}$ has not been properly learned.

  * •

For multiple projections, SAM uses bilinear pooling to aggregate $\mathbf{A}$
and the original feature map, resulting in high-dimensional feature
representation. Our work performs max-out on multiple projections, resulting
in a single attention mask for performing weighted average pooling.
Consequently, we could achieve a significant reduction of the feature
dimensionality.

Table 1: Upper part: summary of the major difference between our method and SAM method [33]. Lower part: two variants that are also investigated in this work. Method |  GradCAM Fitting Branch | Cross-branch Interaction |  Feature Dimension |  Loss Function
---|---|---|---|---
train | test
SAM | 1 proj | $\times$ | $\times$ | $C$ | CrossEntropy
SAM-Bilinear | max[$K$ projs] | bilinear pooling | bilinear pooling |  $C$*$K$ | CrossEntropy
Ours | max[$K$ projs] | $\times$ | weighted average pooling | $C$ | Contrastive
Ours-DualPooling | max[$K$ projs] | weighted average pooling | weighted average pooling | $C$ | Contrastive
Ours-MultiTask | max[$K$ projs] | $\times$ | $\times$ | $C$ | Contrastive

To make our study comprehensive, we also explore several variants as baseline
approaches. From the comparison with those methods, the benefit of our design
could become more evident. For comparison, we use the following architectures
as baselines for self-supervised pre-training:

SAM-SSL: this baseline extends SAM by changing its objective function from
cross-entropy to the contrastive loss in MoCo V2. It shares the same
architecture as SAM [33], where a projection is used as the GFB, and GradCAM
fitting is trained as an auxiliary task to contrastive learning.

SAM-SSL-Bilinear: this baseline extends SAM-bilinear by using the contrastive
loss in MoCo V2. The cross-branch interaction of SAM-SSL-Bilinear follows the
original SAM-Bilinear method.

Besides the above two extensions of the SAM methods, we also consider two
variants of our method. The first is called Ours-MultiTask, which does not
perform weighted average pooling at the test stage but merely uses GradCAM
fitting as an auxiliary task. Another is called Ours-DualPooling, which
performs weighted average pooling at both training and testing stages. Those
two variants are summarized in Table 1.

##  4 Experiments

In this section, we will evaluate our proposed method on three widely used
fine-grained visual datasets (Caltech-UCSD Birds (CUB-200-2011) [35], Stanford
Cars [25] and FGVC-Aircraft [27]), and a large-scale fine-grained dataset
(iNaturalist2019 [34]). Our experiments aim to understand the effectiveness
and the components of the proposed algorithm.

###  4.1 Datasets and Settings

Datasets. CUB-200-2011 contains 11,788 images with 200 bird species, where
5994 images are used for training and 5794 images for testing. Stanford Cars
contains 16,185 images with 196 categories, where 8144 images are for training
and 8041 images for testing. FGVC-Aircraft contains 10,000 images with 100
categories, where 6667 images are for training, and 3333 images are for
testing. iNaturalist2019 in its 2019 version contains 1,010 categories, with a
combined training and validation set of 268,243 images. Note that fine-grained
visual task focus on distinguishing similar subcategories within a super-
category, while there are six super-categories in iNaturalist2019.

Implementation Details. We adopt the ResNet-50 [19] as the network backbone,
which is initialized using ImageNet-trained weights, and build our method on
top of MoCo v2 [8]. Therefore the SSL loss term is identical to MoCo v2. The
momentum value and memory size are set similarly to MoCo v2, i.e., 0.999 and
65536, respectively. The projector head $g_{\theta}$ in MoCo v2 is composed of
two fully-connected layers with ReLU and a third linear layer with batch
normalization (BN) [21]. The size of all three layers is 2048
$\times$2048$\times$256\. We set the mini-batch size as 128 and used an SGD
optimizer with a learning rate of 0.03, a momentum of 0.9, and a weight decay
of 0.0001. 100 epochs are used to train the feature extractor. The images from
the four FGVR datasets are resized to 224$\times$224 pixels during training
times. During testing time, images are firstly resized to 256 pixels and then
are center cropped to 224$\times$224 on these four FGVR datasets.

###  4.2 Evaluation Protocols

Figure 3: Comparison of ResNet-50, MoCo v2 and Ours on the retrieval rank-1 on the iNaturalist2019 dataset. “Amphibians”, “Birds”, “Fungi”, “Reptiles”, and “Insects” are the superclasses in iNaturalist2019. Table 2: Classification and retrieval of our method evaluated on the CUB-200-2011, Stanford Cars and FGVC Aircraft datasets. “ResNet-50” represents pre-training in the ImageNet dataset [12] in a supervised manner, then freezing the ResNet-50 backbone and only optimizing the supervised linear classifier in the classification task. We report the Top 1 and Top 5 (in %) on the classification task, rank-1, rank-5, and mAP (in %) on the retrieval task. 100, 50, and 20 are the three different label proportions (in %) in the classification task. Dataset | Method | Classification | Retrieval
---|---|---|---
Top 1/Top 5(100) | Top 1/Top 5(50) | Top 1/Top 5(20) | rank-1 | rank-5 | mAP
CUB-200-2011 | ResNet-50 | 68.17/90.42 | 58.99/85.90 | 46.54/77.09 | 48.13 | 74.12 | 20.11
MoCo v2 | 68.30/90.85 | 60.96/87.00 | 46.91/76.59 | 42.72 | 69.71 | 19.68
Ours |  71.31/92.03 |  66.52/90.06 |  55.33/83.52 | 49.69 | 75.23 | 24.01
Stanford Cars | ResNet-50 | 57.41/83.55 | 46.23/74.31 | 31.19/58.67 | 27.27 | 51.90 | 5.82
MoCo v2 | 58.43/84.85 | 50.17/77.38 | 35.14/64.10 | 32.22 | 58.41 | 8.00
Ours |  60.75/86.44 |  53.87/81.72 |  40.88/69.18 | 34.56 | 60.75 | 8.87
FGVC Aircraft | ResNet-50 | 47.38/74.73 | 37.83/67.12 | 28.20/54.73 | 25.59 | 46.68 | 27.31
MoCo v2 | 52.54/80.74 | 45.52/73.85 | 35.17/65.08 | 32.13 | 58.71 | 12.12
Ours |  55.87/84.73 |  48.22/77.14 |  38.55/68.53 | 34.33 | 61.09 | 15.43

We evaluate the proposed method in two settings: linear probing and image
retrieval. Linear probing is a commonly used evaluation protocol in SSL. In
linear probing, the feature extractor learned from the SSL algorithm will be
fixed and a linear classifier will be trained on top of the learned features.
The classification performance of the linear classifier indicates the quality
of the learned feature.

Besides linear probing, we also use image retrieval (also equivalent to the
nearest neighbor classification task) task to evaluate the learned features,
which was also explored in the literature [6, 22, 23]. This task aims to find
the images with the same category as query images based on the learned
feature. Note that, unlike linear probing, the image retrieval task does not
involve a large amount of labeled data – which could be used to suppress the
less relevant features. In this sense, succeeding in the image retrieval task
imposes higher feature quality requirements. Moreover, image retrieval is a
practically useful task, and unsupervised feature learning is an attractive
solution to image retrieval since the whole retrieval system can be built
without any human annotation and intervention. For the retrieval task, we use
rank-1, rank-5, and mAP as evaluation metrics.

###  4.3 Main Results

The Effectiveness of the Proposed Method. Our method is based on MoCo v2. We
first compare our result against MoCo v2 to examine the performance gain. The
results are reported in Table 2 and Figure 3.

From Table 2, we can see that our method has led to a significant improvement
over MoCo v2. It achieves 71.31% top-1 accuracy and 92.03% top-5 accuracy on
the CUB-200-2011 dataset with 100% label proportions, which is a 3.01%
improvement on top-1 and 1.18% improvement on top-5 over MoCo v2. Similarly,
significant improvement can also be found on the Stanford Cars and FGVC
Aircraft datasets. This demonstrates that our methods improve the quality of
the feature representations learned from the original MoCo v2. Notably, with
the label proportions reduced from 100% to 20%, there is generally a better
improvement in the performance of the proposed method, which demonstrates that
the proposed method learns less noisy features, so that training with fewer
data can generalize better.

The advantage of the proposed method can also be seen when evaluating the
image retrieval task. As seen from Table 2, our method leads to a significant
boost in performance on all datasets. Specifically, the rank-1, rank-5 and mAP
of ours ($K$=32) are 49.69%, 75.23%, and 24.01%, respectively on the
CUB-200-2011 dataset, which is 6.97%, 5.52%, and 4.33% higher than the method
of MoCo v2. Similar improvement can also be observed on the Stanford Cars and
FGVC Aircraft datasets. On the large-scale fine-grained dataset, i.e.,
iNaturalist2019, our method also performs better than MoCo v2 shown in Figure
3. These indicate the proposed method is better for retrieval tasks, which
might be due to its ability to filter out less relevant patterns.

Comparison with Other SSL Frameworks.

Table 3: Compared to other state-of-the-art self-supervised learning
frameworks with Top 1 accuracy on the CUB-200-2011, Stanford Cars and FGVC
Aircraft datasets; running time and peak memory on the CUB-200-2011 dataset.
The training time is measured on 4 Tesla V100 GPUs with 100 epochs, and the
peak memory is calculated on a single GPU. Top 1 accuracy (%) is reported on
linear classification with the frozen representations of their feature
extractor. For fairness, all following models use the ResNet-50 as the network
backbone and initialize the ResNet-50 architecture with ImageNet-trained
weights. blue=best, green=second best.

Method | Batch Size | Top 1(CUB) | Top 1(Cars) | Top 1(Aircraft) | time(CUB) | GPU Memory(CUB)
---|---|---|---|---|---|---
Supervised | - | 81.34 | 91.02 | 87.13 | - | -
DINO [6] | 32/128 | 12.37/16.66 | 9.27/10.51 | 8.52/12.93 | 1.5h/1.5h | 4.9G/8.4G
SimCLR [7]* | 32/128 | 33.49/38.39 | 44.31/49.41 | 40.56/45.22 | 2.5h/2.5h | 7.1G/23.8G
BYOL [16]* | 32/128 | 36.64/39.27 | 43.66/45.21 | 34.90/37.62 | 4.0h/4.0h | 7.4G/24.6G
SimSiam [9] | 32/128 | 35.82/39.97 | 56.87/58.89 | 41.59/43.06 | 2.0h/2.0h | 4.4G/8.9G
MoCo v2 [8] | 32/128 | 68.03/68.30 | 52.61/58.43 | 42.51/52.54 | 2.0h/2.0h | 6.1G/10.3G
BarlowTwins [40] | 32/128 | 28.58/33.45 | 23.34/31.91 | 28.35/34.77 | 1.5h/1.5h | 7.0G/8.9G
VICReg [4] | 32/128 | 30.07/37.78 | 19.29/30.80 | 29.97/36.00 | 1.0h/1.0h | 7.0G/9.0G
Ours | 32/128 |  70.43 /71.31 | 56.90/60.75 | 46.92/55.87 | 2.0h/2.0h | 6.1G/10.3G
BYOL+Ours* | 32/128 | 45.79/51.20 | 48.53 /50.64 | 40.08/45.94 | 4.0h/4.0h | 7.4G/24.6G

  * *

Due to computational constraints, we are unable to evaluate on the batch size
of 4096 as used in the original paper; we leave this for future work.

In addition to the comparison on MoCo v2, we also compare the proposed method
against other commonly used SSL approaches. We report the Top-1 accuracy, the
running time, and the peak memory with two different batch sizes (32 $\&$
128). The results are shown in Table 3. All methods are run on 4 V100 GPUs,
each with 32G memory. We report the training speed, GPU memory usage, and
performance. For classification performance, the blue marks mean the best
classification results, and the green marks mean the second-best
classification results. It is clear to see that our method achieves the best
Top-1 performance on the CUB-200-2011, Stanford Cars, and FGVC Aircraft
datasets. Also, when the GPU resources are limited, e.g., only 1 V100 GPU is
available, the proposed method reduces the batch size but still remains a
competitive classification performance compared to other SSL methods.
Regarding training speed and GPU memory usage, the proposed method has the
same running time as MoCo v2 and SimSiam, slightly more peak memory than
SimSiam, but less peak memory, and quicker running time than SimCLR and BYOL.
Although DINO uses the least training time and GPU memory usage, their
performances are also the worst compared to ours and other self-supervised
learning methods. Barlow Twins and VICReg have a quicker running time and less
GPU memory than our proposed method with a batch size of 128, but their
performances are much worse than ours.

Our method also can be implemented with other self-supervised learning
methods, e.g., BYOL, referring to “BYOL+Ours” presented in Table 3. As we can
see, our method applied to the BYOL objective is consistently superior to the
baseline of BYOL on three fine-grained datasets.

###  4.4 Comparison with Alternative Solutions

Table 4: The linear Top 1 (%) and retrieval rank-1 (%) performance comparisons of recent alternative solutions and Ours on the CUB-200-2011, Stanford Cars and FGVC Aircraft datasets. blue=best, green=second best. Collapse means the model fails to produce meaningful performance. Method | Feature | Classification | Retrieval
---|---|---|---
Dimension | CUB | Cars | Aircraft | CUB | Cars | Aircraft
MoCo v2 [8] | $C$ | 68.30 | 58.43 | 52.54 | 42.72 | 32.22 | 32.13
MoCo v2 -Bilinear |  $C$*$K$ | 68.44 | 58.06 | 53.01 | 41.27 | 30.89 | 30.80
SAM-SSL | $C$ | 68.59 | 58.49 | 52.97 | 42.95 | 32.76 | 32.47
SAM-SSL-Bilinear |  $C$*$K$ | 71.56 | 59.12 | 55.12 | 44.20 | 35.38 | 32.10
DiLo [42, 38] | $C$ | 64.14 | - | - | - | - | -
CVSA [38] | $C$ | 65.02 | - | - | - | - | -
LEWEL [20] |  $C$*$K$ | 69.27 | 59.02 | 54.33 | 43.05 | 33.02 | 32.88
ContrastiveCrop [29] | $C$ | 68.82 | 61.66 | 54.40 | 43.54 | 32.01 | 32.74
Ours-DualPooling | $C$ | collapse | collapse
Ours-MultiTask | $C$ | 68.56 | 58.55 | 52.87 | 42.93 | 32.77 | 32.52
Ours | $C$ | 71.31 | 60.75 | 55.87 | 49.69 | 34.56 | 34.33
Ours+ContrastiveCrop | $C$ | 72.84 | 63.71 | 56.08 | 49.36 | 33.55 | 34.95

Our method is featured by its capability of discovering the key discriminative
regions, which is vital for FGVR. In this section, we compared the proposed
method with nine alternative solutions to take object localization into
account. The first is to simply use a bilinear network [26] for MoCo V2.
Specifically, we follow the similar bilinear structure as in [33], which
implicitly learns $K$ parts and aggregates features from those $K$ parts, and
we set $K=32$ to make a fair comparison to us (since we use 32 projections).
We still use MoCo v2 as the SSL framework and denote this method MoCo v2
-Bilinear. The second and third are the methods of SAM-SSL and SAM-SSL-
Bilinear introduced in Section 3.2 since our method is extended from the self-
boosting attention mechanism (SAM) proposed in [33]. The fourth and fifth are
the two variants considered in Section 3.2. The other four comparing methods
are the very recently localization-based self-supervised methods, DiLo [42],
CVSA [38], LEWEL [20], and ContrastiveCrop [29]. DiLo uses a copy-and-pasting
approach as a kind of augmentation to create an image with a different
background. The work of CVSA targets a similar problem as ours. They exploit
self-supervised fine-grained contrastive learning via cross-view saliency
alignment to crops and swaps saliency regions of images. LEWEL adaptively
aggregates spatial information of features using spatial aggregation operation
between feature map and alignment map to guide the feature learning better.
The work of ContrastiveCrop is based on the idea of using attention to guide
image cropping, which localizes the object and improves the data augmentation
for self-supervised learning.

The comparison to those nine alternatives is shown in Table 4. As seen, by
comparing the SAM-SSL and SAM-SSL-Bilinear, we observe that the proposed
method can lead to overall better performance, achieving a significant boost
on some datasets. Occasionally, SAM-SSL-Bilinear ($K$=32) can achieve
comparable performance as ours, but at the cost of using a much higher feature
dimensionality. This clearly shows the advantage of the proposed scheme over
the scheme in [33]. Furthermore, our method and combined with ContrastiveCrop
(i.e., ours+ContrastiveCrop) with the lowest feature dimensions but achieve
the best Top 1 and rank-1 performance on the CUB-200-2011, Stanford Cars and
FGVC Aircraft datasets, compared to those nine alternatives. Also, compared
with the other localization-aware SSL methods, our method shows a clear
advantage, e.g., ours vs. LEWEL. Finally, we find that the variant of our
method does not produce a good performance. For example, when applying
weighted average pooling at both training and testing, i.e., Ours-DualPooling,
will make the feature learning fail completely, and the representations will
collapse. On the other hand, not performing cross-branch interaction, i.e.,
Ours-MultiTask, does not bring too much improvement over the MoCo v2 baseline.

###  4.5 The Impact of Number of Projections

Figure 4: Comparison of MoCo v2 (the blue plot) baseline with our method (the
red plot) w.r.t. $K$ on the CUB-200-2011.

To explore the impact of the number of linear projections in our method, we
conduct experiments with the different numbers of $K$. Figure D1 shows the
retrieval results of rank-1 w.r.t. eight different projections on the fine-
grained dataset CUB-200-2011. As we can see, with the increase of linear
projections $K$, the rank-1 gradually increases—the final rank-1 peaks at
49.69% with $K$ around 32. After $K$ reaches 64, the performance decreases.
When the number of projections is very large, the combination of $K$-part
detectors becomes spurious and overfits the correlated pattern, thus resulting
in a drop in performance. From the curve, we can also see that choosing any
value between 4 and 64 can lead to similar performance. So our method is not
very sensitive to $K$ once it falls into a reasonable range.

###  4.6 Alternative Structure for GFB

Table 5: Retrieval performance (%) of our methods and using MLP as the alternative GFB branch. The evaluation is on the Stanford Cars dataset. Dataset | Architecture | rank-1 | rank-5 | mAP
---|---|---|---|---
Cars | Ours | 34.56 | 60.75 | 8.87
MLP | 25.57 | 50.91 | 5.92

In this section, we investigate alternative designs for GFB. In particular, we
consider using a two-layer (with 32 as the intermediate feature dimension)
multi-layer perception (MLP) to replace the maximized projections in the
proposed method. Compared with our GFB structure, an MLP has better fitting
capacity due to the extra linear layer. We conduct experiments on the Stanford
Cars dataset, and the result is shown in Table E1. We find that the
performance dramatically decreases when using MLP. This demonstrates the
importance of our GFB module design.

##  5 Conclusion, Further Results (Appendix) , Limitation and Future Work

In this paper, we introduce a simple-but-effective way of learning an
additional screening mechanism to self-supervised feature learning for fine-
grained visual recognition. The idea is to identify discriminative clues
commonly seen across instances and classes by fitting the GradCAM of the SSL
loss with a fitting-capability-limited branch. Our method achieves state-of-
the-art in the classification task and retrieval tasks on fine-grained visual
recognition problems. More experimental results, including adopting our method
for non-fine-grained visual recognition problems, and visualizing each
projection in the GFB, can be found in the Appendix. So far, the proposed
method seems to be most effective for FGVR, which could be a limitation and we
plan to extend the applicability of the proposed method in our future work.

## References

  * [1] Soroush Abbasi Koohpayegani, Ajinkya Tejankar, and Hamed Pirsiavash.  Compress: Self-supervised learning by compressing representations.  Advances in Neural Information Processing Systems, 33:12980–12992, 2020.
  * [2] Philip Bachman, R Devon Hjelm, and William Buchwalter.  Learning representations by maximizing mutual information across views.  Advances in neural information processing systems, 32, 2019.
  * [3] Hangbo Bao, Li Dong, and Furu Wei.  Beit: Bert pre-training of image transformers.  arXiv preprint arXiv:2106.08254, 2021.
  * [4] Adrien Bardes, Jean Ponce, and Yann LeCun.  Vicreg: Variance-invariance-covariance regularization for self-supervised learning.  arXiv preprint arXiv:2105.04906, 2021.
  * [5] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  Advances in Neural Information Processing Systems, 33:9912–9924, 2020.
  * [6] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9650–9660, 2021.
  * [7] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  * [8] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He.  Improved baselines with momentum contrastive learning.  arXiv preprint arXiv:2003.04297, 2020.
  * [9] Xinlei Chen and Kaiming He.  Exploring simple siamese representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15750–15758, 2021.
  * [10] Yuanzheng Ci, Chen Lin, Lei Bai, and Wanli Ouyang.  Fast-moco: Boost momentum-based contrastive learning with combinatorial patches.  arXiv preprint arXiv:2207.08220, 2022.
  * [11] Elijah Cole, Xuan Yang, Kimberly Wilber, Oisin Mac Aodha, and Serge Belongie.  When does contrastive visual representation learning work?  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14755–14764, 2022.
  * [12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [13] Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman.  With a little help from my friends: Nearest-neighbor contrastive learning of visual representations.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9588–9597, 2021.
  * [14] Linus Ericsson, Henry Gouk, and Timothy M Hospedales.  How well do self-supervised models transfer?  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5414–5423, 2021.
  * [15] Yuting Gao, Jia-Xin Zhuang, Shaohui Lin, Hao Cheng, Xing Sun, Ke Li, and Chunhua Shen.  Disco: Remedying self-supervised learning on lightweight models with distilled contrastive learning.  In European Conference on Computer Vision, pages 237–253. Springer, 2022.
  * [16] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent-a new approach to self-supervised learning.  Advances in Neural Information Processing Systems, 33:21271–21284, 2020.
  * [17] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000–16009, 2022.
  * [18] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729–9738, 2020.
  * [19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [20] Lang Huang, Shan You, Mingkai Zheng, Fei Wang, Chen Qian, and Toshihiko Yamasaki.  Learning where to learn in cross-view self-supervised learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14451–14460, 2022.
  * [21] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In International conference on machine learning, pages 448–456. PMLR, 2015.
  * [22] Young Kyun Jang and Nam Ik Cho.  Self-supervised product quantization for deep unsupervised image retrieval.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12085–12094, 2021.
  * [23] Donghyun Kim, Kuniaki Saito, Tae-Hyun Oh, Bryan A Plummer, Stan Sclaroff, and Kate Saenko.  Cds: Cross-domain self-supervised pre-training.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9123–9132, 2021.
  * [24] Nikos Komodakis and Spyros Gidaris.  Unsupervised representation learning by predicting image rotations.  In International Conference on Learning Representations (ICLR), 2018\.
  * [25] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In Proceedings of the IEEE international conference on computer vision workshops, pages 554–561, 2013.
  * [26] Tsung-Yu Lin, Aruni RoyChowdhury, and Subhransu Maji.  Bilinear cnn models for fine-grained visual recognition.  In Proceedings of the IEEE international conference on computer vision, pages 1449–1457, 2015.
  * [27] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi.  Fine-grained visual classification of aircraft.  arXiv preprint arXiv:1306.5151, 2013.
  * [28] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros.  Context encoders: Feature learning by inpainting.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2536–2544, 2016.
  * [29] Xiangyu Peng, Kai Wang, Zheng Zhu, Mang Wang, and Yang You.  Crafting better contrastive views for siamese representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16031–16040, 2022.
  * [30] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In Proceedings of the IEEE international conference on computer vision, pages 618–626, 2017.
  * [31] Ramprasaath R Selvaraju, Karan Desai, Justin Johnson, and Nikhil Naik.  Casting your model: Learning to localize improves self-supervised representations.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11058–11067, 2021.
  * [32] Yuge Shi, N Siddharth, Philip Torr, and Adam R Kosiorek.  Adversarial masking for self-supervised learning.  In International Conference on Machine Learning, pages 20026–20040. PMLR, 2022.
  * [33] Yangyang Shu, Baosheng Yu, Haiming Xu, and Lingqiao Liu.  Improving fine-grained visual recognition in low data regimes via self-boosting attention mechanism.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXV, pages 449–465. Springer, 2022.
  * [34] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie.  The inaturalist species classification and detection dataset.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8769–8778, 2018.
  * [35] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie.  The caltech-ucsd birds-200-2011 dataset.  2011\.
  * [36] Tongzhou Wang and Phillip Isola.  Understanding contrastive representation learning through alignment and uniformity on the hypersphere.  In International Conference on Machine Learning, pages 9929–9939. PMLR, 2020.
  * [37] Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li.  Dense contrastive learning for self-supervised visual pre-training.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3024–3033, 2021.
  * [38] Di Wu, Siyuan Li, Zelin Zang, Kai Wang, Lei Shang, Baigui Sun, Hao Li, and Stan Z Li.  Align yourself: Self-supervised pre-training for fine-grained recognition via saliency alignment.  arXiv preprint arXiv:2106.15788, 2021.
  * [39] Chun-Hsiao Yeh, Cheng-Yao Hong, Yen-Chi Hsu, and Tyng-Luh Liu.  Saga: Self-augmentation with guided attention for representation learning.  In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3463–3467. IEEE, 2022.
  * [40] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny.  Barlow twins: Self-supervised learning via redundancy reduction.  In International Conference on Machine Learning, pages 12310–12320. PMLR, 2021.
  * [41] Richard Zhang, Phillip Isola, and Alexei A Efros.  Colorful image colorization.  In European conference on computer vision, pages 649–666. Springer, 2016.
  * [42] Nanxuan Zhao, Zhirong Wu, Rynson WH Lau, and Stephen Lin.  Distilling localization for self-supervised representation learning.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10990–10998, 2021.
  * [43] Mingkai Zheng, Shan You, Fei Wang, Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu.  Ressl: Relational self-supervised learning with weak augmentation.  Advances in Neural Information Processing Systems, 34:2543–2555, 2021.

##  Appendix A Algorithm

Algorithm A1 Pseudocode of Learning Common Rationale on MoCo, PyTorch-style.

$\\#$ $\psi_{q}$, $\psi_{k}$: encoder networks for query and key

$\\#$ queue: dictionary as a queue of $Q$ keys ($C\times Q$)$\\#$ $m$:
momentum$\\#$ $t$: temperature in contrastive loss$\\#$ $\tau$: temperature in
KL Divergence loss

$\\#$ Initialize key network parameters

$\psi_{k}$.params = $\psi_{q}$.params

$\\#$ Load a minibatch $x$ with $N$ samples

for $x$ in dataloader: do

$\\#$ two different random augmentations

$x_{q}$ = aug($x$)

$x_{k}$ = aug($x$)

$\\#$**************************************

$\\#$ calculate MoCo contrastive loss

$\\#$**************************************

$q$ = $\psi_{q}$.forward($x_{q}$) $\\#$ queries: $N\times C$

$k$ = $\psi_{k}$.forward($x_{k}$)$\\#$ keys: $N\times C$

$k$ = $k$.detach()$\\#$ no gradient to keys

$\\#$ positive and negative logits.

logits$\\_$pos = bmm($q$.view($N$,1,$C$), $k$.view(N,C,1))$\\#$ shape:$N\times
1$

logits$\\_$neg = mm($q$.view($N$,$C$), queue.view(C,Q))$\\#$ shape:$N\times Q$

logits = cat([logits$\\_$pos, logits$\\_$neg], dim=1) $\\#$
shape:$N\times(1+Q)$

$\\#$ MoCo contrastive loss, positives are the 0-th.

labels = zeros($N$)

loss = CrossEntropyLoss(logits/$t$, labels)

$\\#$**************************************

$\\#$calculate KL loss

$\\#$**************************************

$\\#$get feature map in query network.

feat$\\_$map=get$\\_$cov5($\psi_{q}$.params) $\\#$shape:$N\times 7\times
7\times 2048$

$\\#$calculate gradient of feature map w.r.t. logits$\\_$pos.

feat$\\_$grads=autograd.grad(logits$\\_$pos,feat$\\_$map)$\\#$shape:$N\times
7\times 7\times 2048$

$\\#$calculate GradCAM map

gradcam$\\_$map=get$\\_$gradcam(feat$\\_$grads) $\\#$shape:$N\times 7\times 7$

$\\#$calculate attention mask

attention$\\_$mask=projections$\\_$max(feat$\\_$map)$\\#$shape:$N\times
7\times 7$

$\\#$KL loss: attention$\\_$mask and gradcam$\\_$map
KL$\\_$loss=kl$\\_$div((attention$\\_$mask.view($N$,-1)/$\tau$).softmax(dim=-1).log(),(gradcam$\\_$map.view($N$,-1)/$\tau$).softmax(dim=-1))

loss+=$\nu$*KL$\\_$loss

loss.backward()

$\\#$SGD update for query network

update ($\psi_{q}$.params)

$\\#$moment update for key network

$\psi_{k}$.params = $m$*$\psi_{k}$.params + (1-$m$)*$\psi_{q}$.params

$\\#$update dictionary: enqueue and dequeue

enqueue(queue, $k$)

dequeue(queue)

end for

We present the pseudocode of our method in algorithm A1.

##  Appendix B Understanding the Role of Each Projection

Figure B1: Visualization of high-and-low-variance projections of our method.
The first row is the original input images. The 2-4 rows are the visualization
of the top three high-variance linear projections, and the 5-7 rows are the
visualization of the three projections corresponding to the lowest variance.
Note that the colour of the heat map is normalized. It only indicates the
relative strength within images, not the absolute value.  Figure B2: The
rank-1 of retrieval task of our methods with different projections selection
on the FGVC Aircraft datasets. “high-variance $K$” means selecting the first
eight high-variance projections from 32 projections; “random $K$” represents
randomly selecting eight projections from the remaining 24 projections after
exclusive the first eight high-variance projections; “all $K$” represents all
32 projections selected.

In this section, we explore the role of the $K$ linear projections in the
GradCAM Fitting Branch (GFB). To better analyze the importance of each
projection, we consider the variance of the output of each projection. Higher
variance indicates strong signal strength. Figure B1 visualize the output of
the projection with the three highest variances and three lowest variances. It
can be seen that projections with higher variance tend to produce more
meaningful output. For example, the projection with the highest variance seems
to capture clues that humans rely on to judge fine-grained objects, e.g., the
head, body, and back of birds. In contrast, projections with the lowest
variance barely attend the reasonable regions.

Figure B2 shows another experiment to verify the role of different
projections. In Figure B2, the curve shows the rank-1 accuracy of retrieval
task on 100 epochs on the FGVC Aircraft datasets. From 0 to 60 epochs, we use
all 32 projections ($K$=32) and find the rank-1 accuracy increases
significantly. After 60 epochs, we test the three different settings: our
method still with all 32 projections, referring as “Ours (all $K$)” in the red
line; our method with the first eight high-variance projections, referring as
“Ours (high-variance $K$)” in brown line; our method with random eight
projections chosen from the remaining 24 projections after discarding the top-
eight high-variance projections, referring as “Ours (random $K$)” in the green
line. As we can see, the rank-1 greatly increase then gradually becomes stable
in the method of Ours (all $K$) and Ours (high-variance $K$), while the rank-1
drop sharply then keep stable in the method of Ours (random $K$). This
indicates that the top variance projection preserves the most discriminative
rationales. Interestingly, the role of those top-variance projections is akin
to that of the top eigenvectors in Principal Component Analysis.

##  Appendix C Performance on Non-fine-grained Dataset

Table C1: Retrieval performance (%) of our proposed method vs. MoCo v2 on non-fine-grained dataset (CIFAR-100). Method | Metrics
---|---
rank-1 | rank-5 | mAP
MoCo v2 | 24.78 | 48.20 | 6.25
Ours | 27.14 | 51.25 | 6.44

To observe the performance of our proposed method on the non-fine-grained
dataset, we conduct experiments of our method vs. MoCo v2 on CIFAR-100 dataset
shown in Table C1. From Table C1, we can see that our method also works for
non-fine-grained cases.

##  Appendix D The Impact of Number of Projections on Non-fine-grained
Dataset.

Figure D1: Comparison of MoCo v2 (the blue plot) baseline with our method (the
red plot) w.r.t. $K$ on the CIFAR-100 dataset.

To explore the impact of the number of linear projections on the non-fine-
grained dataset CIFAR-100, we conduct experiments with the different numbers
of $K$. Figure D1 shows the retrieval results of rank-1 w.r.t. eight different
projections. As we can see, the rank-1 peaks at 27.14% with $K$ around 8. With
the increase of $K$ from 32, the performance decreases and then gradually
becomes stable.

##  Appendix E The Impact of GFB Structure on Non-fine-grained Dataset.

Table E1: Retrieval performance (%) of our methods and using MLP as the alternative GFB branch. The evaluation is on the CIFAR-100 dataset. Dataset | Architecture | rank-1 | rank-5 | mAP
---|---|---|---|---
CIFAR-100 | Ours($K$=8) | 27.14 | 51.25 | 6.44
Ours($K$=32) | 20.02 | 40.52 | 4.35
MLP | 19.71 | 40.32 | 4.34

In this section, we investigate the impact of GFB structure for the non-fine-
grained case by evaluating on CIFAR-100. We follow a similar experimental
approach as in the main paper by varying the number of projections. As seen
from Figure D1, the best number of projections seems to be 8, but further
increasing the number of projections to 256 does not lead to a significant
drop as the case for fine-grained dataset (refer to Figure 4 in the main
paper).

Also, we consider whether we can use multi-layer perception (MLP) to replace
the maximized projections in the proposed method shown in Table E1. We find
MLP performs similarly to our methods when $K\geq 32$. This is also different
from the case in the fine-grained case. That observation indicates that GFB
might have a different characteristic for non-fine-grained data. We plan to
leave this in our future work.

##  Appendix F Ablation study of weights $\lambda$ and $\nu$.

Table F1: Retrieval performance (%) w.r.t the impact of weights $\lambda$ and $\nu$ on the CUB-200-2011 dataset. Method | $\lambda$ | $\nu$ | rank-1
---|---|---|---
$\mathcal{L}_{CL}$+$\mathcal{L}_{KL}$ (Ours) | 1 | 1 | 47.52
1 | 0.1 | 48.83
1 | 0.01 | 49.69
1 | 0.001 | 46.63

Table F1 reports the retrieval performance for different values of the loss
term coefficients in the proposed method, where $\lambda$ and $\nu$ control
the weight of $\mathcal{L}_{CL}$ and $\mathcal{L}_{KL}$, respectively. We set
the weight of $\mathcal{L}_{CL}$ as 1 using a simple grid search method to
find that the weight of $\mathcal{L}_{KL}$ as 0.01 is the best.

##  Appendix G The Impact of Embedding Dimension.

Table G1: Rank-1 accuracy (%) of our method with different embedding dimensionalities on the retrieval task with 100 pretraining epochs on the CUB-200-2011 dataset. Dimensionality | 64 | 128 | 256 | 512 | 1024
---|---|---|---|---|---
rank-1 | 47.48 | 48.72 | 49.69 | 49.46 | 48.55

Our method is based on the network structure of MoCo v2, and thus we observe
the dependency on the dimensionality of the embedding vector shown in Table
G1. Compared to MoCo v2 using 128 as the dimensionality, the impact of
embedding vector in our method is different. The performance on the
CUB-200-2011 dataset increases firstly from $47.48\%$ rank-1 to $49.69$ rank-1
on retrieval task with the dimensionality from 64 to 256. After 256, the
performance decreased. Thus, our method uses 256 as the dimensionality of the
embedding vector.


* [101] Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li,  “Uformer: A general u-shaped transformer for image restoration,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022, pp. 17683–17693.
  * [102] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  CoRR, vol. abs/1706.03762, 2017.
  * [103] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou,  “Training data-efficient image transformers & distillation through attention,”  CoRR, vol. abs/2012.12877, 2020.
  * [104] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao,  “Pyramid vision transformer: A versatile backbone for dense prediction without convolutions,”  CoRR, vol. abs/2102.12122, 2021.
  * [105] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, José M. Álvarez, and Ping Luo,  “Segformer: Simple and efficient design for semantic segmentation with transformers,”  CoRR, vol. abs/2105.15203, 2021.
  * [106] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip H. S. Torr, and Li Zhang,  “Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers,”  CoRR, vol. abs/2012.15840, 2020.
  * [107] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko,  “End-to-end object detection with transformers,”  CoRR, vol. abs/2005.12872, 2020.
  * [108] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo,  “Swin transformer: Hierarchical vision transformer using shifted windows,”  CoRR, vol. abs/2103.14030, 2021.
  * [109] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai,  “Deformable DETR: deformable transformers for end-to-end object detection,”  CoRR, vol. abs/2010.04159, 2020.
  * [110] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Francis E. H. Tay, Jiashi Feng, and Shuicheng Yan,  “Tokens-to-token vit: Training vision transformers from scratch on imagenet,”  CoRR, vol. abs/2101.11986, 2021.
  * [111] Salman H. Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah,  “Transformers in vision: A survey,”  CoRR, vol. abs/2101.01169, 2021.
  * [112] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby,  “An image is worth 16x16 words: Transformers for image recognition at scale,”  CoRR, vol. abs/2010.11929, 2020.
  * [113] Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang,  “Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network,”  in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 1874–1883.
  * [114] Murat Sazli,  “A brief review of feed-forward neural networks,”  Communications Faculty Of Science University of Ankara, vol. 50, pp. 11–17, 01 2006.
  * [115] François Chollet,  “Xception: Deep learning with depthwise separable convolutions,”  CoRR, vol. abs/1610.02357, 2016.
  * [116] Dan Hendrycks and Kevin Gimpel,  “Bridging nonlinearities and stochastic regularizers with gaussian error linear units,”  CoRR, vol. abs/1606.08415, 2016.
  * [117] Yicheng Wu, Qiurui He, Tianfan Xue, Rahul Garg, Jiawen Chen, Ashok Veeraraghavan, and Jonathan T. Barron,  “How to train neural networks for flare removal,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2021, pp. 2239–2247.
  * [118] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros,  “Image-to-image translation with conditional adversarial networks,”  in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 5967–5976.
  * [119] Syed Waqas Zamir, Aditya Arora, Salman H. Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao,  “Learning enriched features for real image restoration and enhancement,”  CoRR, vol. abs/2003.06792, 2020.
  * [120] Yawei Li, Kai Zhang, Jiezhang Cao, Radu Timofte, and Luc Van Gool,  “Localvit: Bringing locality to vision transformers,” 2021.
  * [121] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang,  “Cvt: Introducing convolutions to vision transformers,” 2021.
  * [122] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani,  “Self-attention with relative position representations,” 2018.
  * [123] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen,  “Mobilenetv2: Inverted residuals and linear bottlenecks,” 2019.
  * [124] Pengwei Liang, Junjun Jiang, Xianming Liu, and Jiayi Ma,  “Image deblurring by exploring in-depth properties of transformer,” 2023.
  * [125] Kangliang Liu, Xiangcheng Du, Sijie Liu, Yingbin Zheng, Xingjiao Wu, and Cheng Jin,  “Ddt: Dual-branch deformable transformer for image denoising,” 2023.
  * [126] Xuaner Zhang, Ren Ng, and Qifeng Chen,  “Single image reflection separation with perceptual losses,”  in IEEE Conference on Computer Vision and Pattern Recognition, 06 2018, pp. 4786–4794.
  * [127] Yuekun Dai, Chongyi Li, Shangchen Zhou, Ruicheng Feng, Yihang Luo, and Chen Change Loy,  “Flare7k++: Mixing synthetic and real datasets for nighttime flare removal and beyond,”  arXiv e-prints, 2023.
  * [128] Yuekun Dai, Chongyi Li, Shangchen Zhou, Ruicheng Feng, and Chen Change Loy,  “Flare7K: A Phenomenological Nighttime Flare Removal Dataset,”  arXiv e-prints, p. arXiv:2210.06570, Oct. 2022.
  * [129] Yuekun Dai, Yihang Luo, Shangchen Zhou, Chongyi Li, and Chen Change Loy,  “Nighttime smartphone reflective flare removal using optical center symmetry prior,” 2023.
  * [130] Simon Niklaus and Feng Liu,  “Context-aware synthesis for video frame interpolation,”  CoRR, vol. abs/1803.10967, 2018.
  * [131] Qilin Sun, Ethan Tseng, Qiang Fu, Wolfgang Heidrich, and Felix Heide,  “Learning rank-1 diffractive optics for single-shot high dynamic range imaging,”  in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [132] Sigma,  “A new solution for mirrorless camera users Combining F1.4 brightness and a compact body Introducing a new high-performance, large-diameter wide-angle lens,” .
  * [133] Wikipedia contributors,  “Sony alpha 7 iii — Wikipedia, the free encyclopedia,” 2023,  [Online; accessed 23-September-2023].
  * [134] Fernando A. Fardo, Victor H. Conforto, Francisco C. de Oliveira, and Paulo S. Rodrigues,  “A formal evaluation of PSNR as quality measurement parameter for image segmentation algorithms,”  CoRR, vol. abs/1605.07116, 2016.
  * [135] Yadolah Dodge,  Mean Squared Error, pp. 337–339,  Springer New York, New York, NY, 2008.
  * [136] Jim Nilsson and Tomas Akenine-Möller,  “Understanding ssim,” 2020.
  * [137] Xiaotian Qiao, Gerhard P. Hancke, and Rynson W. H. Lau,  “Light source guided single-image flare removal from unpaired data,”  in 2021 IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 4157–4165.


whose clusters have asymmetric internal structure and non-uniform densities.
On the other side, our experimental results on real-world data sets show that
the performance of PruneMerge is inferior to Linkage++ and especially to
Average Linkage. We believe that developing more efficient algorithms for
well-clustered graphs is a very meaningful direction for future work.

Finally, our experimental results indicate that the Average Linkage algorithm
performs extremely well on all tested instances, when compared to PruneMerge
and Linkage++. This leads to the open question whether Average Linkage
achieves an $O(1)$-approximation for well-clustered graphs, although it fails
to achieve this for general graphs [CAKMTM19]. In our point of view, the
answer to this question could help us design more efficient algorithms for
hierarchical clustering that not only work in practice, but also have rigorous
theoretical guarantees.

## References

  * [AAV20] Noga Alon, Yossi Azar, and Danny Vainstein.  Hierarchical clustering: a 0.585 revenue approximation.  In 33rd Annual Conference on Learning Theory (COLT’20), pages 153–162, 2020.
  * [ABS15] Sanjeev Arora, Boaz Barak, and David Steurer.  Subexponential algorithms for unique games and related problems.  Journal of the ACM, 62(5), 2015.
  * [ACAH19] Amir Abboud, Vincent Cohen-Addad, and Hussein Houdrouge.  Subquadratic high-dimensional hierarchical clustering.  In 33rd Advances in Neural Information Processing Systems (NeurIPS’19), pages 11576–11586, 2019.
  * [AKK+08] Sanjeev Arora, Subhash Khot, Alexandra Kolla, David Steurer, Madhur Tulsiani, and Nisheeth K. Vishnoi.  Unique games on expanding constraint graphs are easy: extended abstract.  In 40th Annual ACM Symposium on Theory of Computing (STOC’08), pages 21–28, 2008.
  * [Alo86] N. Alon.  Eigenvalues and expanders.  Combinatorica, 6(2):83–96, 1986.
  * [ARV09] Sanjeev Arora, Satish Rao, and Umesh Vazirani.  Expander flows, geometric embeddings and graph partitioning.  Journal of the ACM, 56(2):1–37, 2009.
  * [CAKMT17] Vincent Cohen-Addad, Varun Kanade, and Frederik Mallmann-Trenn.  Hierarchical clustering beyond the worst-case.  In 31st Advances in Neural Information Processing Systems (NeurIPS’17), pages 6201–6209, 2017.
  * [CAKMTM19] Vincent Cohen-Addad, Varun Kanade, Frederik Mallmann-Trenn, and Claire Mathieu.  Hierarchical clustering: Objective functions and algorithms.  Journal of the ACM, 66(4):1–42, 2019.
  * [CC17] Moses Charikar and Vaggos Chatziafratis.  Approximate hierarchical clustering via sparsest cut and spreading metrics.  In 28th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA’17), pages 841–854, 2017.
  * [CCN19] Moses Charikar, Vaggos Chatziafratis, and Rad Niazadeh.  Hierarchical clustering better than average-linkage.  In 30th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA’19), pages 2291–2304, 2019.
  * [CCNY19] Moses Charikar, Vaggos Chatziafratis, Rad Niazadeh, and Grigory Yaroslavtsev.  Hierarchical clustering for euclidean data.  In 22nd International Conference on Artificial Intelligence and Statistics (AISTATS’19), pages 2721–2730, 2019.
  * [Chu97] Fan R.K. Chung.  Spectral graph theory.  1997\.
  * [CYL+20] Vaggos Chatziafratis, Grigory Yaroslavtsev, Euiwoong Lee, Konstantin Makarychev, Sara Ahmadian, Alessandro Epasto, and Mohammad Mahdian.  Bisect and conquer: Hierarchical clustering via max-uncut bisection.  In 23rd International Conference on Artificial Intelligence and Statistics (AISTATS’20), pages 3121–3132, 2020.
  * [Das16] Sanjoy Dasgupta.  A cost function for similarity-based hierarchical clustering.  In 48th Annual ACM Symposium on Theory of Computing (STOC’16), pages 118–127, 2016.
  * [GT14] Shayan Oveis Gharan and Luca Trevisan.  Partitioning into expanders.  In 25th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA’14), pages 1256–1266, 2014.
  * [HLW06] Hoory, Linial, and Wigderson.  Expander graphs and their applications.  BAMS: Bulletin of the American Mathematical Society, 43, 2006.
  * [Kol10] Alexandra Kolla.  Spectral algorithms for unique games.  In 25th Conference on Computational Complexity (CCC’10), pages 122–130, 2010.
  * [KVV04] Ravi Kannan, Santosh S. Vempala, and Adrian Vetta.  On clusterings: Good, bad and spectral.  Journal of the ACM, 51(3):497–515, 2004.
  * [LGT14] James R Lee, Shayan Oveis Gharan, and Luca Trevisan.  Multiway spectral partitioning and higher-order cheeger inequalities.  Journal of the ACM, 61(6):1–30, 2014.
  * [LSZ19] Huan Li, He Sun, and Luca Zanetti.  Hermitian Laplacians and a Cheeger inequality for the Max-2-Lin problem.  In 27th Annual European Symposium on Algorithms (ESA’19), pages 71:1–71:14, 2019.
  * [McS01] Frank McSherry.  Spectral partitioning of random graphs.  In 42nd Annual IEEE Symposium on Foundations of Computer Science (FOCS’01), pages 529–537, 2001.
  * [MRS+19] Aditya Krishna Menon, Anand Rajagopalan, Baris Sumengen, Gui Citovsky, Qin Cao, and Sanjiv Kumar.  Online hierarchical clustering approximations.  arXiv:1909.09667, 2019.
  * [MW17] Benjamin Moseley and Joshua Wang.  Approximation bounds for hierarchical clustering: Average linkage, bisecting $k$-means, and local search.  In 31st Advances in Neural Information Processing Systems (NeurIPS’17), pages 3094–3103, 2017.
  * [NJW01] Andrew Ng, Michael Jordan, and Yair Weiss.  On spectral clustering: Analysis and an algorithm.  In 15th Advances in Neural Information Processing Systems (NeurIPS’01), pages 849–856, 2001.
  * [PSZ17] Richard Peng, He Sun, and Luca Zanetti.  Partitioning well-clustered graphs: Spectral clustering works!  SIAM Journal on Computing, 46(2):710–743, 2017.
  * [PVG+11] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.  Scikit-learn: Machine learning in Python.  Journal of Machine Learning Research, 12:2825–2830, 2011.
  * [RP17] Aurko Roy and Sebastian Pokutta.  Hierarchical clustering via spreading metrics.  The Journal of Machine Learning Research, 18(1):3077–3111, 2017\.
  * [SZ19] He Sun and Luca Zanetti.  Distributed graph clustering and sparsification.  ACM Transactions on Parallel Computing, 6(3):17:1–17:23, 2019\.
  * [UCI] UCI ML Repository.  https://archive.ics.uci.edu/ml/index.php.  Accessed: 2021-05-22.
  * [VCC+21] Danny Vainstein, Vaggos Chatziafratis, Gui Citovsky, Anand Rajagopalan, Mohammad Mahdian, and Yossi Azar.  Hierarchical clustering via sketches and hierarchical correlation clustering.  In 24th International Conference on Artificial Intelligence and Statistics (AISTATS’21), pages 559–567, 2021.
  * [vL07] Ulrike von Luxburg.  A tutorial on spectral clustering.  Statistics and Computing, 17(4):395–416, 2007.
  * [ZLM13] Zeyuan Allen Zhu, Silvio Lattanzi, and Vahab S. Mirrokni.  A local algorithm for finding well-connected clusters.  In 30th International Conference on Machine Learning (ICML’13), pages 396–404, 2013.