# Improving CNN-based Person Re-identification using score Normalization

Ammar Chouchane1, Abdelmalik Ouamane1, Yassine Himeur7, Wathiq Mansoor7, Shadi
Atalla7, Afaf Benzaibak1 and Chahrazed Boudellal1

1 Laboratory of LI3C, University of Biskra, Biskra, Algeria  7College of
Engineering and Information Technology, University of Dubai, Dubai, UAE

###### Abstract

Person re-identification (PRe-ID) is a crucial task in security, surveillance,
and retail analysis, which involves identifying an individual across multiple
cameras and views. However, it is a challenging task due to changes in
illumination, background, and viewpoint. Efficient feature extraction and
metric learning algorithms are essential for a successful PRe-ID system. This
paper proposes a novel approach for PRe-ID, which combines a Convolutional
Neural Network (CNN) based feature extraction method with Cross-view Quadratic
Discriminant Analysis (XQDA) for metric learning. Additionally, a matching
algorithm that employs Mahalanobis distance and a score normalization process
to address inconsistencies between camera scores is implemented. The proposed
approach is tested on four challenging datasets, including VIPeR, GRID,
CUHK01, and PRID450S. The proposed approach has demonstrated its effectiveness
through promising results obtained from the four challenging datasets.

###### Index Terms:

PRe-ID, Score Normalization, XQDA, CNN, feature extraction.

##  I Introduction

Person re-identification, or PRe-ID, involves recognizing an individual across
different images or videos captured in a surveillance system [1]. This is
critical in various real-time applications such as person retrieval, video
monitoring, public safety, long-term human behavior analysis, and cross-camera
tracking [2, 3, 4].

The use of CNN models has become popular in recent deep learning
architectures, either through the development of new models or through the use
of pretrained models known as transfer learning [5, 6]. The process of person
re-identification, which involves matching individuals detected by different
cameras, usually consists of two main steps, as shown in Figure 1. These steps
are: (1) Feature extraction (FE), which involves obtaining more reliable,
robust, and concise features than raw pixel data, and (2) learning the system
with ample data to allow it to perform re-identification automatically during
online testing. The Gaussian of Gaussian (GOG) and Local Maximal Occurrence
(LOMO) descriptors are the two most commonly used FE methods in the field of
person re-identification [7, 8]. GOG descriptor, proposed by Matsukawa et al.
[9], involves dividing the image into k rectangular blocks and representing
each block by 4 Gaussian distributions in different color spaces (RGB, Lab,
HSV, and nRnG). On the other hand, LOMO descriptor, introduced by Liao et al.
[10], extracts two types of features (scale-invariant local ternary pattern
and HSV color histograms) from the imageS by dividing it into horizontal
patches and calculating the occurrence of local geometric features. The aim of
metric learning is to learn a metric that can effectively compare two
pedestrian images. Popular examples of metric learning approaches include
KISSME [10] and Cross-view Quadratic Discriminate Analysis (XQDA) [11].

Figure 1: Person re-identification system outline

The PRe-ID system uses three main approaches to tackle the problem, as shown
in Fig. 2. These approaches include Feature Descriptor Learning, Metric
Learning, and Deep Learning. The Feature Descriptor Learning methods aim to
learn distinctive and representative features from pedestrian images that
distinguish the appearances of individuals in the datasets [13]. Many
effective techniques have been proposed in this category, such as Gaussian of
Gaussian (GOG) [7] and LOMO (Local Maximal Occurrence) descriptors [8] as well
as various other techniques discussed in [14], [15], [16], [17], and [18].

Metric learning is a technique utilized to enhance the precision of machine
learning models. It trains a model to compute the similarity between
pedestrians images captured from different cameras to achieve greater matching
accuracy [19, 11, 20, 21, 22, 12]. On the other hand, deep learning is a
sophisticated task that has gained popularity for enhancing PR systems and
achieving high performance [23, 24, 25, 26].The deep learning approaches can
further be classified into three categories such as CNN, RNN, and GAN-based
methods. Figure 2 summarizes a taxonomy of PRe-ID of approaches.

Figure 2:  PRe-ID Classification approaches

Overall, the main contributions of this work can be summarized as follows:

  * •

Utilizing CNN features as a transfer learning process for effective feature
representation.

  * •

Enhancing the discriminative power in person re-identification by implementing
Cross-view Quadratic Discriminant Analysis (XQDA), a robust metric learning
method that employs the Mahalanobis distance for matching.

  * •

Applying a score normalization technique which greatly improved results on
four benchmark datasets: VIPeR [11], GRID [27], CUHK01 [28], and PRID450s
[29].

  * •

Evaluating and comparing the proposed approach against the state-of-the-art
methods.

The paper is organized as follows: Section 2 outlines the methodology and
details of the proposed approach, including the CNN feature model, the XQDA
metric learning technique, and the score normalization process. Experimental
results are reported in Section 3. Finally, the conclusions and future work
are discussed in Section 4.

##  II Methodology and description of the proposed approach

###  II-A FE based on a pretrained model of CNN

Person Re-identification (Pre-ID) entails the process of recognizing an
individual by correlating the identity of a probe image to a set of images.
This task is fundamentally confronted by two main challenges: Feature
Extraction (FE) and metric learning. In an effort to address these obstacles,
this study leverages a pretrained CNN model derived from the ImageNet dataset
in conjunction with the XQDA metric learning method, which exhibited efficacy
in our experimental trials.

Figure 3: AlexNet architecture model based CNN.

Specifically, the study obtained 4,096-dimensional feature vectors by
extracting the features from the fully connected layer 7 (FC7) of the learned
CNN, as delineated in Figure 3.

The XQDA metric learning method was deployed to bolster the discriminative
capability of the target dataset for the task of person re-identification. The
research utilized the AlexNet architecture, comprising eight layers in total;
the initial five are convolutional, while the last three are fully connected.
In particular, the fully connected seventh layer serves as the feature vector,
representing the unique biometric signature of each individual within the
dataset.

###  II-B Features projection and metric learning using XQDA algorithm

The XQDA metric learning algorithm is a modification of the KISS metric
algorithm [30]. XQDA learns a discriminative feature subspace in a low-
dimensional space through the use of Fisher criterion [19]. It is preferred to
have a lower dimensional space ($\Re^{r}$, where $r<d$) as the initial
dimension of the feature vector $d$ is quite large, leading to better
classification results. The intra-person set difference ($X_{s}$) of $n$
similar pairs is represented as a matrix $\Re^{d\times n}$, while the extra-
person difference set ($X_{D}$) of $n$ dissimilar pairs is represented as a
matrix $\Re^{d\times n}$, where $d$ is the dimension of the feature vectors.
Each column in $X_{s}$ indicates the difference between a similar pair, and
each column in $X_{D}$ represents the difference between a dissimilar pair.
The covariance matrices are calculated using the following equations:

$\Sigma_{s}=\frac{1}{n}X_{s}X^{T}_{s}$ (1)

$\Sigma_{D}=\frac{1}{n}X_{D}X^{T}_{D}$ (2)

XQDA calculates the distance between the training samples ($\mu$ and $\nu$)
from two camera views as follows:

$d(\mu,\nu)=(\mu-\nu)^{T}W(\Sigma^{-1}_{s^{\prime}}-\Sigma^{-1}_{D^{\prime}})W^{T}(\mu-\nu)$
(3)

The new subspace projection matrix ($W$) to be learned by XQDA is represented
in this equation. The covariance matrices, $\Sigma_{s^{\prime}}$ and
$\Sigma_{D^{\prime}}$, are obtained by transforming $\Sigma_{s}$ and
$\Sigma_{D}$ respectively through $W^{T}$. These variances of the subspace are
used to differentiate between intra-person and extra-person differences. As a
result, the projection direction $W$ is estimated through the following
equation.

$J(W)=\frac{W^{T}\Sigma_{s}W}{W^{T}\Sigma_{D}W}$ (4)

The equation can be converted into a generalized eigenvalue decomposition
problem. The $r$ eigenvectors corresponding to the largest eigenvalues of
$\Sigma^{-1}D\Sigma_{s}$ form the projection matrix
$W=[w_{1},w_{2},...,w_{r}]$. The output metric matrix is represented as
$M=\Sigma^{-1}{s^{\prime}}-\Sigma^{-1}_{D^{\prime}}$, which can be used to
calculate the distance between two given feature vectors.

###  II-C Matching based Mahalanobis distance

The Mahalanobis distance is a popular and effective metric for comparing the
similarity between two data points. It is often utilized to improve the
classification process [29, 38, 39]. Given $m$ data points $x_{i}\in\Re^{m}$,
the objective is to find a matrix $M$ such that the distance metric is
expressed as follows.

$d_{M}=(x_{i},x_{j})^{T}M(x_{i},x_{j})$ (5)

Where $x_{i}$ and $x_{j}$ are two vectors (samples), and $M$ is a positive
semidefinite matrix.

###  II-D Score normalization

The Mahalanobis distance is calculated using global samples from different
cameras with varying image resolutions, resulting in heterogeneous scores. To
make these scores comparable, score normalization is performed. A min-max
normalization function was applied in this work, which addresses bias and
variation that could affect the comparison scores [31].Score normalization aim
to mitigate the variations in similarity scores that arise from different
camera viewpoints. By normalizing the scores, we can effectively reduce the
influence of these factors and enhance the discriminative power of the system.
This normalization step improved the results and is defined as follows.

$N=\frac{x-x_{min}}{x_{max}-x_{min}}$ (6)

where $x_{min}$ is the minimum score vector and where $x_{max}$ is the maximum
score vector.

##  III Implementations and results

###  III-A Datasets and protocols

The proposed approach utilizing CNN features has been tested on four demanding
datasets: PRID450s, VIPeR, GRID and CUHK01. The traits of these datasets are
detailed in Table I. Examples from the utilized datasets are depicted in
Figure 4.

TABLE I: Datasets and its characteristics. Dataset | Release | Identities | Images | Cameras | Size
---|---|---|---|---|---
| Time | Identities |  |  |
PRID450s | 2011 | 450 | 900 | 2 | 128$\times 48$
VIPeR | 2014 | 632 | 1264 | 2 | 128$\times 48$
GRID | 2009 | 1025 | 1275 | 8 | Vary
CUHK01 | 2012 | 971 | 3884 | 2 | $160\times 60$
Figure 4: Sample images from the datasets considered in this study: (A)
PRID450s, (B) VIPeR, (C) GRID, (D) CUHK01.

###  III-B Evaluation metrics

To assess the effectiveness of our proposed approach in PRe-ID systems, we
employed a 10-fold cross-validation methodology. This involved randomly
dividing the image set into ten subsets, with nine sets used for training and
one set for testing in each fold. To evaluate the performance, we used the
Cumulative Matching Characteristic (CMC) metric, which is a popular method for
re-identification performance evaluation. The CMC curve displays the rank
matching rates, where a rank "r" matching rate refers to the percentage of
probe images that have correct matches within the top "r" ranks among the
gallery images. To conduct our experiments, we split the datasets randomly
into training and test sets, with an equal number of individuals in each set.
It is worth noting that for the GRID dataset, the gallery collection now
includes an additional 775 pictures. The number of probe images in all
datasets is equivalent to the number of gallery images.

To evaluate the effectiveness of the proposed strategy, we employed a 10-fold
cross-validation,in which, with CMC curve averages being reported. The CMC
score represents the success rate of accurately identifying the matched image
at each rank. Specifically, we focused on the accuracy at rank-1 since it
indicates the probability of the correct image being displayed as the top
result. In addition, we also considered the accuracy at ranks 5, 10, and 20
during the evaluation process.

###  III-C Discussions

To address the challenges of person re-identification, we investigated the
effectiveness of the XQDA metric learning technique, along with the
Mahalanobis distance metric, and hence helps in enhancing the classification
outcomes. We also employed score normalization to eliminate any potential
biases or discrepancies in the scores, which could lead to unfair comparisons.
Our results showed that the proposed method was effective in achieving
improved performance on the PRID450s, VIPeR, GRID, and CUHK01 datasets, as
demonstrated by the CMC curves. In particular, the CMC curves with score
normalization were significantly higher than those without it, which
highlights the importance of this step in achieving fair comparisons. These
findings suggest that the proposed XQDA metric learning approach, along with
the Mahalanobis distance metric and score normalization, can significantly
improve the accuracy of person re-identification. The application of this
technique may have significant implications for the development of robust and
effective surveillance systems in the future. However, it is important to note
that further studies are needed to assess the generalizability of these
findings to other datasets and scenarios.

Figure 5: CMC curves of all datasets, Samples images from the used datasets,
(A) CUHK01, (B) GRID, (C), VIPeR (D) PRID450s  Figure 6: Performance of the
proposed scheme on different datasets (with and without normalization).

Furthermore, to provide a comprehensive overview of the re-identification
accuracy for each dataset, we included a table (Table II) that reports the
performance metrics at various ranks, including rank-1, rank-5, rank-10, and
rank-20. To better understand the impact of score normalization on the re-
identification accuracy, Figure 5 specifically highlights the rank-1
performance in terms of the CMC curves of each dataset without score
normalization. Additionally, Figure 6 portrays the accuracy performance of the
proposed scheme on the four datasets.The results of our experiments suggest
that the proposed XQDA metric learning approach, in combination with the
Mahalanobis distance metric and score normalization, can effectively improve
the performance of person re-identification on the four challenging datasets.
As indicated in Table II, the score normalization improved the rank-1 rate on
all datasets. Without normalization, the rank-1 rates were 42.16%, 25.20%,
41.51% and 64.22% for VIPeR, GRID, CUHK01, and PRID450s respectively. With
score normalization, the rates improved to 43.16%, 35.68%, 45.24% and 64.32%
respectively. The GRID dataset showed a particularly significant improvement
of 10.48% in rank-1 after score normalization. We attribute the improvement in
the re-identification system to the use of score normalization on our
datasets.

TABLE II: Accuracy on the different datasets with and without score normalization. Without/with | Ranks
---|---
Normalization | Rank-1 | Rank-5 | Rank-10 | Rank-15 | Rank-20
| CUHK01
Without | 41.51% | 65.61% | 75.11% | 80.39% | 83.90%
With | 45.24% | 74.35% | 82.78% | 86.95% | 89.30%
| GRID
Without | 25.20% | 45.52% | 54.08% | 60.84% | 61.92%
With | 35.68% | 46.96% | 53.04% | 56.88% | 64.64%
| PRID450S
Without | 64.22% | 86.44% | 91.78% | 94.53% | 96.22%
With | 64.32% | 91.11% | 96.18% | 97.78% | 98.76%
| VIPeR
Without | 42.16% | 71.99% | 83.01% | 88.04% | 92.03%
With | 43.16% | 74.11% | 85.13% | 90.13% | 92.78%

###  III-D comparison with the state-of-the-art

In this particular section, we assess the effectiveness of the proposed
approach by contrasting it with several established re-identification
approaches based on Rank-1 rate.The summarized findings of this comparison can
be observed in table III. From the previous table, we can observe the strength
of our approach, as we achieved almost the best results in all four databases.
This confirms the robustness of our proposed approach despite the variations
in the data. The impressive performance exhibited by our method across
multiple datasets highlights its capability to handle different scenarios and
reinforces its potential for real-world applications.

TABLE III: Comparison with the state-of-the-art of rank-1 identification rates(%)  |  Dataset
---|---
Methods |  VIPeR |  CUHK01 |  PRID450S |  GRID
KEPLER [32], 2015 |  42.40 |  - |  - |  -
FT-CNN+XQDA [9], 2016 |  42.50 |  46.80 |  58.20 |  25.20
MKSSL+LOMO [33],2017 |  31.20 |  - |  - |  24.60
OneShot [34], 2017 |  34.30 |  - |  41.40 |
DeepRank [35], 2018 |  38.37 |  - |  - |  -
EML [36], 2020 |  44.37 |  - |  63.58 |  19.47
VISUAL-DAC [37], 2022 |  39.70 |  - |  - |  34.60
CNN+XQDA with score normalization (Our) 2023 |  43.16 |  45.24 |  64.22 |  35.68

##  IV Conclusion

This research work addresses the challenging task of PRe-ID in security,
surveillance, and retail analysis, which involves identifying an individual
across multiple cameras and views. The proposed approach combines a CNN-based
FE method with Cross-view XQDA for metric learning. In addition, a matching
algorithm that employs Mahalanobis distance and a score normalization process
to address inconsistencies between camera scores are also implemented. The
evaluation results on four challenging datasets, including VIPeR, GRID,
CUHK01, and PRID450S, demonstrate the effectiveness of the proposed approach.
The implementation of score normalization shows significant improvement in the
different rank rate accuracies of the datasets.

## References

  * [1] Y. Himeur, S. Al-Maadeed, H. Kheddar, N. Al-Maadeed, K. Abualsaud, A. Mohamed, and T. Khattab, “Video surveillance using deep transfer learning and deep domain adaptation: Towards better generalization,” _Engineering Applications of Artificial Intelligence_ , vol. 119, p. 105698, 2023.
  * [2] H. Liu, Z. Xiao, B. Fan, H. Zeng, Y. Zhang, and G. Jiang, “Prgcn: Probability prediction with graph convolutional network for person re-identification,” _Neurocomputing_ , vol. 423, pp. 57–70, 2021.
  * [3] Y. Himeur, S. Al-Maadeed, N. Almadeed, K. Abualsaud, A. Mohamed, T. Khattab, and O. Elharrouss, “Deep visual social distancing monitoring to combat covid-19: A comprehensive survey,” _Sustainable cities and society_ , p. 104064, 2022.
  * [4] A. Chouchane, M. Bessaoudi, E. Boutellaa, and A. Ouamane, "A new multidimensional discriminant representation for robust person re-identification," _Pattern Analysis and Applications_ , pp. 1-14, 2023.
  * [5] O. Elharrouss, S. Al-Maadeed, N. Subramanian, N. Ottakath, N. Almaadeed, and Y. Himeur, “Panoptic segmentation: A review,” _arXiv preprint arXiv:2111.10250_ , 2021.
  * [6] Y. Himeur, S. Al-Maadeed, I. Varlamis, N. Al-Maadeed, K. Abualsaud, and A. Mohamed, “Face mask detection in smart cities using deep and transfer learning: lessons learned from the covid-19 pandemic,” _Systems_ , vol. 11, no. 2, p. 107, 2023.
  * [7] R. Prates and W. R. Schwartz, “Kernel cross-view collaborative representation based classification for person re-identification,” _Journal of Visual Communication and Image Representation_ , vol. 58, pp. 304–315, 2019.
  * [8] M. Gou, Z. Wu, A. Rates-Borras, O. Camps, R. J. Radke _et al._ , “A systematic evaluation and benchmark for person re-identification: Features, metrics, and datasets,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 41, no. 3, pp. 523–536, 2018.
  * [9] T. Matsukawa, T. Okabe, E. Suzuki, and Y. Sato, “Hierarchical gaussian descriptor for person re-identification,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 1363–1372.
  * [10] S. Liao, Y. Hu, X. Zhu, and S. Z. Li, “Person re-identification by local maximal occurrence representation and metric learning,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 2197–2206.
  * [11] D. Gray and H. Tao, “Viewpoint invariant pedestrian recognition with an ensemble of localized features,” in _Computer Vision–ECCV 2008: 10th European Conference on Computer Vision, Marseille, France, October 12-18, 2008, Proceedings, Part I 10_. Springer, 2008, pp. 262–275.
  * [12] A. Chouchane, M. Bessaoudi, A. Ouamane, and O. Laouadi, “Face Kinship Verification Based VGG16 and new Gabor Wavelet Features,” in _2022 5th International Symposium on Informatics and its Applications (ISIA)_ , IEEE, 2022, pp. 1–6.
  * [13] Y. Himeur and K. A. Sadi, “Robust video copy detection based on ring decomposition based binarized statistical image features and invariant color descriptor (rbsif-icd),” _Multimedia Tools and Applications_ , vol. 77, pp. 17 309–17 331, 2018.
  * [14] B. J. Prosser, S. Gong, and T. Xiang, “Multi-camera matching using bi-directional cumulative brightness transfer functions.” in _BMVC_ , vol. 8. Citeseer, 2008, pp. 164–1.
  * [15] C. Su, S. Zhang, F. Yang, G. Zhang, Q. Tian, W. Gao, and L. S. Davis, “Attributes driven tracklet-to-tracklet person re-identification using latent prototypes space mapping,” _Pattern Recognition_ , vol. 66, pp. 4–15, 2017.
  * [16] C. Su, F. Yang, S. Zhang, Q. Tian, L. S. Davis, and W. Gao, “Multi-task learning with low rank attribute embedding for multi-camera person re-identification,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 40, no. 5, pp. 1167–1181, 2017.
  * [17] L. Wei, S. Zhang, W. Gao, and Q. Tian, “Person transfer gan to bridge domain gap for person re-identification,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 79–88.
  * [18] G. Chen, T. Gu, J. Lu, J.-A. Bao, and J. Zhou, “Person re-identification via attention pyramid,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 7663–7676, 2021.
  * [19] Z. Yang, Y. Wu, J. Cheng, S. Peng, L. Wang, and D. Tao, “Incremental xqda metric learning for person reidentification,” in _2018 IEEE International Conference on Information and Automation (ICIA)_. IEEE, 2018, pp. 433–438.
  * [20] O. Javed, K. Shafique, Z. Rasheed, and M. Shah, “Modeling inter-camera space–time and appearance relationships for tracking across non-overlapping views,” _Computer Vision and Image Understanding_ , vol. 109, no. 2, pp. 146–162, 2008.
  * [21] L. Zhang, T. Xiang, and S. Gong, “Learning a discriminative null space for person re-identification,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 1239–1248.
  * [22] Y. Gavini, A. Agarwal, and B. Mehtre, “Thermal to visual person re-identification using collaborative metric learning based on maximum margin matrix factorization,” _Pattern Recognition_ , vol. 134, p. 109069, 2023\.
  * [23] X. Chang, T. M. Hospedales, and T. Xiang, “Multi-level factorisation net for person re-identification,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 2109–2118.
  * [24] J. Song, Y. Yang, Y.-Z. Song, T. Xiang, and T. M. Hospedales, “Generalizable person re-identification by domain-invariant mapping network,” in _Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition_ , 2019, pp. 719–728.
  * [25] Y. Liu, W. Zhou, J. Liu, G.-J. Qi, Q. Tian, and H. Li, “An end-to-end foreground-aware network for person re-identification,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 2060–2071, 2021.
  * [26] Z. Ming, M. Zhu, X. Wang, J. Zhu, J. Cheng, C. Gao, Y. Yang, and X. Wei, “Deep learning-based person re-identification methods: A survey and outlook of recent works,” _Image and Vision Computing_ , vol. 119, p. 104394, 2022.
  * [27] C. C. Loy, T. Xiang, and S. Gong, “Time-delayed correlation analysis for multi-camera activity understanding,” _International Journal of Computer Vision_ , vol. 90, pp. 106–129, 2010.
  * [28] W. Li, R. Zhao, and X. Wang, “Human reidentification with transferred metric learning,” in _Computer Vision–ACCV 2012: 11th Asian Conference on Computer Vision, Daejeon, Korea, November 5-9, 2012, Revised Selected Papers, Part I 11_. Springer, 2013, pp. 31–44.
  * [29] P. M. Roth, M. Hirzer, M. Köstinger, C. Beleznai, and H. Bischof, “Mahalanobis distance learning for person re-identification,” _Person re-identification_ , pp. 247–267, 2014.
  * [30] M. Koestinger, M. Hirzer, P. Wohlhart, P. M. Roth, and H. Bischof, “Large scale metric learning from equivalence constraints,” in _2012 IEEE conference on computer vision and pattern recognition_. IEEE, 2012, pp. 2288–2295.
  * [31] A. Nautsch, J. Patino, A. Treiber, T. Stafylakis, P. Mizera, M. Todisco, T. Schneider, and N. Evans, “Privacy-preserving speaker recognition with cohort score normalisation,” _arXiv preprint arXiv:1907.03454_ , 2019.
  * [32] Martinel, N., Micheloni, C., & Foresti, G. L. (2015). Kernelized saliency-based person re-identification through multiple metric learning. IEEE Transactions on Image Processing, 24(12), 5645-5658.
  * [33] Sun, C., Wang, D., & Lu, H. (2016). Person re-identification via distance metric learning with latent variables. IEEE Transactions on Image Processing, 26(1), 23-34.
  * [34] Bak, S., & Carr, P. (2017). One-shot metric learning for person re-identification. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2990-2999).
  * [35] Wang, J., Zhu, X., Gong, S., & Li, W. (2018). Transferable joint attribute-identity deep learning for unsupervised person re-identification. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2275-2284).
  * [36] Ma, W., Han, H., Kong, Y., & Zhang, Y. (2020). A new date-balanced method based on adaptive asymmetric and diversity regularization in person re-identification. International Journal of Pattern Recognition and Artificial Intelligence, 34(09).
  * [37] Prasad, M. V., & Balakrishnan, R. (2022). Spatio-Temporal association rule based deep annotation-free clustering (STAR-DAC) for unsupervised person re-identification. Pattern Recognition, 122.
  * [38] A. Ouamane, A. Chouchane, E. Boutellaa, M. Belahcene, S. Bourennane, and A. Hadid, "Efficient tensor-based 2D+3D face verification," _IEEE Transactions on Information Forensics and Security_ , vol. 12, no. 11, pp. 2751-2762, 2017.
  * [39] M. Bessaoudi, A. Chouchane, A. Ouamane, and E. Boutellaa, "Multilinear subspace learning using handcrafted and deep features for face kinship verification in the wild," _Applied Intelligence_ , vol. 51, pp. 3534-3547, 2021.


* [67] N. Cabibbo and L. Maiani, The Lifetime of Charmed Particles, Phys. Lett. B 79 (1978) 109–111.
  * [68] Y. Nir, The Mass Ratio $m_{c}/m_{b}$ in Semileptonic $B$ Decays, Phys. Lett. B 221 (1989) 184–190.
  * [69] M. E. Luke and A. V. Manohar, Reparametrization invariance constraints on heavy particle effective field theories, Phys. Lett. B 286 (1992) 348–354, [hep-ph/9205228].
  * [70] E. Bagan, P. Ball, B. Fiol, and P. Gosdzinsky, Next-to-leading order radiative corrections to the decay $b\to c\bar{c}s$, Phys. Lett. B 351 (1995) 546–554, [hep-ph/9502338].
  * [71] M. Neubert and C. T. Sachrajda, Spectator effects in inclusive decays of beauty hadrons, Nucl. Phys. B 483 (1997) 339–370, [hep-ph/9603202].
  * [72] S. Gershtein, V. Kiselev, A. Likhoded, and A. Tkabladze, $B_{c}$ spectroscopy, Phys. Rev. D 51 (1995) 3613–3627, [hep-ph/9406339].
  * [73] Fermilab Lattice, MILC, TUMQCD Collaboration, A. Bazavov et al., Up-, down-, strange-, charm-, and bottom-quark masses from four-flavor lattice QCD, Phys. Rev. D 98 (2018), no. 5 054517, [arXiv:1802.04248].
  * [74] B. Colquhoun, R. J. Dowdall, C. T. H. Davies, K. Hornbostel, and G. P. Lepage, $\Upsilon$ and $\Upsilon^{\prime}$ Leptonic Widths, $a_{\mu}^{b}$ and $m_{b}$ from full lattice QCD, Phys. Rev. D 91 (2015), no. 7 074514, [arXiv:1408.5768].
  * [75] HPQCD Collaboration, A. T. Lytle, C. T. H. Davies, D. Hatton, G. P. Lepage, and C. Sturm, Determination of quark masses from $\mathbf{n_{f}=4}$ lattice QCD and the RI-SMOM intermediate scheme, Phys. Rev. D 98 (2018), no. 1 014513, [arXiv:1805.06225].
  * [76] Particle Data Group Collaboration, M. Tanabashi et al., Review of Particle Physics, Phys. Rev. D 98 (2018), no. 3 030001.
  * [77] C. McNeile, C. Davies, E. Follana, K. Hornbostel, and G. Lepage, Heavy meson masses and decay constants from relativistic heavy quarks in full lattice QCD, Phys. Rev. D 86 (2012) 074503, [arXiv:1207.0994].
  * [78] R. Dowdall, C. Davies, T. Hammant, and R. Horgan, Precise heavy-light meson masses and hyperfine splittings from lattice QCD including charm quarks in the sea, Phys. Rev. D 86 (2012) 094510, [arXiv:1207.5149].


# Task Discrepancy Maximization for Fine-grained Few-Shot Classification

SuBeen Lee, WonJun Moon, Jae-Pil Heo
Sungkyunkwan University
{leesb7426, wjun0830<EMAIL_ADDRESS>Corresponding author

###### Abstract

Recognizing discriminative details such as eyes and beaks is important for
distinguishing fine-grained classes since they have similar overall
appearances. In this regard, we introduce Task Discrepancy Maximization (TDM),
a simple module for fine-grained few-shot classification. Our objective is to
localize the class-wise discriminative regions by highlighting channels
encoding distinct information of the class. Specifically, TDM learns task-
specific channel weights based on two novel components: Support Attention
Module (SAM) and Query Attention Module (QAM). SAM produces a support weight
to represent channel-wise discriminative power for each class. Still, since
the SAM is basically only based on the labeled support sets, it can be
vulnerable to bias toward such support set. Therefore, we propose QAM which
complements SAM by yielding a query weight that grants more weight to object-
relevant channels for a given query image. By combining these two weights, a
class-wise task-specific channel weight is defined. The weights are then
applied to produce task-adaptive feature maps more focusing on the
discriminative details. Our experiments validate the effectiveness of TDM and
its complementary benefits with prior methods in fine-grained few-shot
classification. 111Our code is available at
https://github.com/leesb7426/CVPR2022-Task-Discrepancy-Maximization-for-Fine-
grained-Few-Shot-Classification.

##  1 Introduction

Figure 1: Effect of the channel weight in the CUB dataset. (a) Existing
methods treat channels of feature maps equally. In such case, high variance
channels within a class highly likely to disturb the classification task,
where channel variance represents the channel-wise variances of feature maps
of the same class – intuitively, the instances of the same class having
similar features at a channel lead a low channel variance for the
corresponding channel. This is mainly because it is hard to make a consensus
among features for classification criteria. (b) However, in fine-grained
datasets, simply removing high variance channels for each class shows marginal
improvements. It is because classes share similar features, e.g., feather, and
wings in CUB dataset, and thus channels with a low variance may not be
discriminative. Therefore, in fine-grained datasets, we should grant different
weights to channels depending on whether each channel reflects distinct
characteristics. (c) TDM produces per-class channel weight by discovering
discriminative channels for each class in the episode. Note that, the numbers
in boxes are classification accuracies.

With the advancement of deep learning, it has achieved remarkable performance
beyond humans in various downstream tasks [5, 10]. However, there is a strong
assumption that numerous labeled images should exist to achieve such
performance. If the number of labeled images is insufficient, it shows drastic
degradation in the performance[38, 8, 3]. To resolve such degradation from a
shortage of labeled images and reduce the cost of labeling, the computer
vision community recently paid more attention to few-shot classification[8,
34, 38]. Briefly, the goal of few-shot classification is to train a model with
high adaptability to novel classes. To achieve this goal, the episodic
learning strategy is mainly used, where each episode consists of sampled
categories from the dataset. Furthermore, each class has a support set for
training and a query set for evaluation.

The stream of metric-based learning is a promising direction for the few-shot
classification. These methods[38, 34, 16, 35] learn a deep representation with
a predefined metric or online-trained metric. Specifically, the inference for
a query is performed based on the distances among support and query sets under
such metric.

However, the features of a novel class extracted by a model trained on the
base classes hardly form a tight cluster, since the feature extractor is
highly sensitive and activates the semantically discriminative variations in
the distribution of base classes[46, 32]. To alleviate this, recent methods
utilize primitive knowledge[46, 20] or propose task-dynamic feature alignment
strategies[14, 44, 7, 33, 42, 45, 12]. Among two strategies, task-dynamic
feature alignment methods are being spotlighted. The task-dynamic feature
alignment methods can be further divided into two main streams: spatial
alignment and channel alignment. The spatial alignment methods[12, 7, 44, 14,
42, 42] aim to resolve the spatial mismatch between key features on the
feature maps of different instances. On the other hand, the channel alignment
methods[44, 14, 45, 33] modify feature maps to better represent the semantic
features for novel classes.

Although these alignment methods are shown to be effective on the general few-
shot classification task, they achieved insignificant gains for fine-grained
datasets. This is mainly because they only focus on exploiting features that
describe novel objects, which may not be discriminative in such tasks. Indeed,
localizing discriminative details is important in fine-grained classification,
since categories share similar overall appearances [9, 27, 6, 50]. Therefore,
distinct clues for each category also should be discovered for fine-grained
few-shot classification. In Fig. 1 (c), we verify that localizing
discriminatory details of the object through channel weights is effective for
fine-grained few-shot task.

In this context, we introduce novel Task Discrepancy Maximization (TDM), a
module that localizes discriminative regions by weighting channels per class.
TDM highlights the channels that represent discriminative regions and
restrains the contributions of other channels based on class-wise channel
weight. Specifically, TDM is composed of two components: Support Attention
Module (SAM) and Query Attention Module (QAM). Given a support set, SAM
outputs a support weight per class that presents high activations on
discriminative channels. On the other hand, QAM is fed with the query set to
produce a query weight per instance. The query weight is to highlight the
object-relevant channels. To infer these weights, the relation between each
feature map and the average channel pooled features are considered. Note that,
the channel pooled average feature map has the spatial information of the
object[43, 22] as described in Fig. 2. Therefore, channels are highly likely
to represent objects when they are similar to spatially averaged feature map.
By combining two weights computed from our sub-modules, a task-specific weight
is finally defined. Consequently, the task-specific weight is utilized to
produce task-adaptive feature maps.

Our main contributions are summarized as follows:

  * •

We propose a novel feature alignment method, TDM, to define the class-wise
channel importance, for fine-grained few-shot classification.

  * •

Our proposed TDM is highly applicable to prior metric-based few-shot
classification models.

  * •

When combined with the recent few-shot classification models, the TDM achieves
the state-of-the-art performance in fine-grained few-shot classification task.

Figure 2:  Visualization of pooling results. Each column shows the locations
where each pooling method focuses on the image. The second and third columns
visualize the results of average pooling and the max pooling, respectively.
GAP tends to concentrate on the object parts in the images, while GMP is often
out of focus.

##  2 Related Works

###  2.1 Few-Shot Classification

Figure 3:  Overview of our approach. The Task Discrepancy Maximization
consists of two sub-modules. Each sub-module takes feature maps $F$ and
generates channel weights $w$. The support attention module utilizes feature
maps of the support instances as input and finds discriminative channels for
each class. Then, it produces a support weight $w^{S}_{i}$ for $i$-th category
where the weight holds high values in those channels. On the other hand, the
query attention module is fed the query instance and discovers the object-
relevant channels of the query. Then, a query weight $w^{Q}$ from the query
attention module emphasizes specific channels with object information about
the query. These weights from two sub-modules are integrated by a linear
combination to define a task weight $w^{T}_{i}$ for each $i$-th category.
Finally, task-adaptive feature maps which concentrate on the discriminative
regions are obtained by multiplying the task weights to original feature maps.

The methods of few-shot classification can be divided into two main streams:
optimization- and metric-based. The concept of optimization-based methods was
introduced in MAML [8] to learn good initial conditions that can be easily
adapted. Meta-LSTM [31] adopts an LSTM-based meta-learner which is not only
for the general initial point but also for effective fine-tuning. MetaOptNet
[19] employs convex base learners, and provides a differentiation process for
end-to-end learning. The optimization-based methods show comparable
performance, but they need online updates for novel classes.

The metric-based methods learn deep representations by utilizing a
predefined[38, 34, 16] or online-trained metric[35]. MatchNet [38] uses an
external memory module which augments neural networks and infers categories of
the query set by the cosine similarity. ProtoNet [34] forms prototypes with a
mean feature of each class in support set, and exploits them for computing the
distance between a query to each class. RelationNet [35] utilizes the distance
metric learned by a model instead of the predefined metric.

Metric-based methods generally learn to reduce distances among instances
within a class, and we have the same goal since TDM is a module for them.
However, TDM enables to compute the distances based on adaptive channel
weights by identifying discriminative channels dynamically, while prior
techniques treat all the channels equally.

###  2.2 Feature Alignment

Feature alignment methods can be categorized into spatial and channel
alignments. The spatial alignment methods [12, 7, 44, 14, 42, 47] assert that
object location differences in the support and query set cause the performance
degradation. CAN [12] computes cross attention maps by calculating correlation
for each pair of the classes and query feature maps to highlight the common
regions to locate the object. CTX [7] finds a coarse spatial correspondence
between the query instance and the support set by the attention[2] to produce
a query-aligned prototype per each class. FRN [42] reconstructs the feature
maps of the support set to the query instance by exploiting a closed-form
solution of the ridge regression.

The channel alignment methods [44, 14, 45, 33] manipulate feature maps to be
capable to represent novel classes. FEAT [45] increases the distances among
classes of the support set by adopting the transformer [25, 37]. DMF [44]
aligns feature maps of the query instance by the dynamic meta-filter which has
both position and channel-specific support knowledge. RENet [14] transforms a
feature map with self-correlation capturing structural patterns of each image.

TDM also deals with the feature alignment. Unlike existing methods that
typically consider a pairwise relationship between the support image and the
query image, TDM considers the entire task.

##  3 Method

The overall architecture of our method is illustrated in Fig. 3. Given an
episode consisting of the support and query instances, feature maps are first
computed by the feature extractor. However, the feature maps are not optimal
for each episode since the feature extractor is trained to find discriminative
features for classifying base classes [45, 46, 32]. TDM transforms the feature
maps by exploiting task-specific weights representing channel-wise
discriminative power for a specific task. As a result, we aim to focus on the
discriminatory details by refining the original feature maps into task-
adaptive feature maps. In this section, we introduce the components of TDM and
their purpose. First, we formulate the problem in Sec. 3.1. In Sec. 3.2, we
define two representative scores to produce channel weights. Then, with these
scores, we describe two modules of the TDM: SAM and QAM in Sec. 3.3 and Sec.
3.4, respectively. Finally, TDM is described in Sec. 3.5 with the discussion
in Sec. 3.6.

###  3.1 Problem Formulation

In standard few-shot classification, we are given two datasets: meta-train set
$D_{base}=\left\\{\left(x_{i},y_{i}\right),y_{i}\in{C_{base}}\right\\}$ and
meta-test set
$D_{novel}=\left\\{\left(x_{i},y_{i}\right),y_{i}\in{C_{novel}}\right\\}$.
$C_{base}$ and $C_{novel}$ represent base classes and novel classes,
respectively, where they do not overlap (${C_{base}}\cap{C_{novel}}=\phi$).
Generally, training and testing of few-shot classification are composed of
episodes. Each episode consists of randomly sampled $N$ classes and each class
is composed of $K$ labeled images and $U$ unlabeled images, i.e., $N$-way
$K$-shot episode. The labeled images are called the support set
$S=\left\\{\left(x_{j},y_{j}\right)\right\\}^{{N}\times{K}}_{j=1}$, and the
unlabeled images are named the query set
$Q=\left\\{\left(x_{j},y_{j}\right)\right\\}_{j=1}^{{N}\times{U}}$, while two
sets are disjoint (${S}\cap{Q}=\phi$). The support and query sets are utilized
for learning and testing, respectively.

###  3.2 Channel-wise Representativeness Scores

For each pair of $i$-th class and $c$-th channel, we define two channel-wise
representativeness scores; intra score $R^{\text{intra}}_{i,c}$, and inter
score $R^{\text{inter}}_{i,c}$. Prior to explaining scores, we first define
feature maps $F$ of the support and query instance as follows:

$\begin{split}F^{S}_{i,j}=g_{\theta}(x^{S}_{i,j})\\\
F^{Q}=g_{\theta}(x^{Q}),\end{split}$ (1)

where $x^{S}_{i,j}$ is $j$-th instance of $i$-th class in the support set,
$x^{Q}$ is the query instance, and $g_{\theta}$ is our feature extractor
parameterized by $\theta$. Each feature map $F$ $\in$
$\mathbb{R}^{{C}\times{H}\times{W}}$ where $C,H,W$ denote the number of
channels, height, and width, respectively. Additionally, we utilize a
prototype[34] as the representative of each class. The prototype $F^{P}_{i}$
for $i$-th class is computed as follows:

$\begin{split}F^{P}_{i}=\frac{1}{K}\sum^{K}_{j=1}F^{S}_{i,j}.\end{split}$ (2)

For each class, we first compute a mean spatial feature to represent a salient
object regions. When the $c$-th channel of the prototype for $i$-th class is
denoted as ${f^{P}_{i,c}}\in{\mathbb{R}^{{H}\times{W}}}$, the corresponding
mean spatial feature $M^{P}_{i}$ is computed as follows:

$\begin{split}M^{P}_{i}=\frac{1}{C}\sum^{C}_{j=1}f^{P}_{i,j}.\end{split}$ (3)

Based on this, we further compute the channel-wise representativeness score
defined within a class, $R^{\text{intra}}_{i,c}$, for $c$-th channel of $i$-th
class as follows:

$\begin{split}R^{\text{intra}}_{i,c}=\frac{1}{H\times W}\parallel
f^{P}_{i,c}-M^{P}_{i}\parallel^{2}.\end{split}$ (4)

The score indicates how well the highly activated regions on the channel are
matched with the class-wise salient area represented by the mean spatial
feature. On the other hand, the channel-wise representativeness score across
classes, $R^{\text{inter}}_{i,c}$, for $c$-th channel of $i$-th class is
defined as follows:

$\begin{split}R^{\text{inter}}_{i,c}=\frac{1}{H\times W}\min_{1\leq j\leq
N,j\neq{i}}\parallel f^{P}_{i,c}-M^{P}_{j}\parallel^{2}.\end{split}$ (5)

It represents how much $c$-th channel contains the distinct information of
each class. Intuitively, the channel is more discriminative when it has the
smaller intra score and the larger inter score. We utilize both scores to
define channel weights.

###  3.3 Support Attention Module (SAM)

Figure 4:  Schemetic illustration of Support Attention Module.

For each class, our support attention module (SAM) takes the class prototypes
as input, and first compute two channel-wise representativeness scores based
on Eq. 4 and Eq. 5. We transform those two scores to two weights,
$w^{\text{intra}}_{i}$ and $w^{\text{inter}}_{i}$, for $i$-th class as
follows:

$\begin{split}w^{\text{intra}}_{i}&=b^{\text{intra}}\left(R^{\text{intra}}_{i}\right)\\\
w^{\text{inter}}_{i}&=b^{\text{inter}}\left(R^{\text{inter}}_{i}\right),\end{split}$
(6)

where $b^{\text{intra}}$ and $b^{\text{inter}}$ are different fully-connected
blocks. The architectures of the blocks are reported in Tab. 1.

The support weight vector $w^{S}_{i}$ for $i$-th class is defined by a linear
combination of the corresponding two weights, $w^{\text{intra}}_{i}$ and
$w^{\text{inter}}_{i}$, with a balancing hyperparameter $\alpha$ as follows:

$\begin{split}w^{S}_{i}={\alpha}{w^{\text{intra}}_{i}}+{\left(1-\alpha\right)}{w^{\text{inter}}_{i}},\>{\alpha}\in{\left[0,1\right]}.\end{split}$
(7)

The support weight vector for $i$-th class emphasizes discriminative channels
of $i$-th class while suppressing channels that corresponds to the common
information shared throughout classes in the episode. When we multiply the
support weight vector for $i$-th class, $w^{S}_{i}$, to the feature maps, the
instances of $i$-th class should be gathered, while other class instances
become separated from the $i$-th class.

###  3.4 Query Attention Module (QAM)

Figure 5: Schemetic illustration of Query Attention Module

Although the support weight vectors are useful to distinguish the class-wise
informative channels, we are also encouraged to utilize the query set to
overcome the data scarcity in the few-shot learning. To utilize the query set
for the complementary benefits with SAM, we propose the query attention module
(QAM). Since we do not have label information for a query instance, QAM only
exploits the relationship among channels within an instance. By utilizing the
mean spatial feature $M^{Q}$ defined by the channel-wise mean of query’s
feature map $F^{Q}$, we compute the channel-wise representativeness score of
query instance, $R^{\text{intra}}_{Q}$, for $c$-th channel as follows:

$\begin{split}R^{\text{intra}}_{Q}=\frac{1}{H\times W}\parallel
f^{Q}_{c}-M^{Q}\parallel^{2},\end{split}$ (8)

where $f^{Q}_{c}$ denotes $c$-th channel of query’s feature map. Then, the
query weight $w^{Q}$ is produced by passing the intra score
$R^{\text{intra}}_{Q}$ to the fully connected block $b^{Q}$ as described in
Tab. 1.

$\begin{split}w^{Q}=b^{\text{Q}}\left(R^{\text{intra}}_{Q}\right).\end{split}$
(9)

The query weight vector highlights object-relevant channels of the query
instance while restraining others. Therefore, query weight vector assists the
model to focus on object-related information.

Fully Connected Block
---
Layer | Output Size
Input |  B $\times$ C
Fully Connected Layer |  B $\times$ 2C
Batch Normalization |  B $\times$ 2C
ReLU |  B $\times$ 2C
Fully Connected Layer |  B $\times$ C
1 + Tanh |  B $\times$ C
Table 1:  The architecture of the fully connected blocks. When used in the
support attention module, the batch size B is the number of categories which
is comprising an episode. In the query attention module, B is the number of
queries.

###  3.5 Task Discrepancy Maximization (TDM)

As the support and query weight vectors computed by SAM and QAM are
complementary in their purposes, we use them to produce a task weight vector.
Specifically, the task weight vector $w^{T}_{i}$ for $i$-th class is defined
by a linear combination of the corresponding support and query weight vectors,
$w^{S}_{i}$ and $w^{Q}$, with a hyperparameter $\beta$ as follows:

$\begin{split}w^{T}_{i}={\beta}{w^{S}_{i}}+{\left(1-\beta\right)}{w^{Q}},\quad{\beta}\in{\left[0,1\right]}.\end{split}$
(10)

Then, the feature maps of all the support and query instances are transformed
with the task weight vector. Specifically, each feature map
$F\in\mathbb{R}^{{C}\times{H}\times{W}}$ is transformed to a task-adaptive
feature map $A$ with channel-wise scaling by the task weight vector
$w^{T}_{i}\in\mathbb{R}^{{C}}$ as follows:

$\small\begin{split}A=w^{T}_{i}\odot
F=\left[{w^{T}_{i,1}}{f_{1}},{w^{T}_{i,2}}{f_{2}},...,{w^{T}_{i,C}}{f_{C}}\right],\end{split}$
(11)

where $w^{T}_{i,j}$ is a scalar value at $j$-th dimension of the vector
$w^{T}_{i}$, and the $c$-th channel of the feature map $F$ is denoted by
${f_{c}}\in{\mathbb{R}^{{H}\times{W}}}$.

The feature maps of the support instances of $i$-th class are transformed by
its corresponding task weight vector $w^{T}_{i}$. However, since the label of
the query is not available, we apply the task weight vector for $i$-th class,
$w^{T}_{i}$, to the query, when we are testing the query for the $i$-th class.
As a result, based on Eq. 11, we compute task adaptive feature maps of the
support instances $A^{S}_{i,j}$ and the query instance $A^{Q}_{i}$ transformed
by the task weight vector $w^{T}_{i}$ as follows:

$\begin{split}A^{S}_{i,j}&={w^{T}_{i}}\odot{F^{S}_{i,j}}\\\
A^{Q}_{i}&={w^{T}_{i}}\odot{F^{Q}},\end{split}$ (12)

where $i$, $j$ denote the class index and instance index within the class,
respectively.

For instance, when TDM is applied to the ProtoNet[34], the inference process
is done by the following criteria:

$\begin{split}p_{\theta}(y=i|x)=\frac{{\exp}(-d(A^{S}_{i},A^{Q}_{i}))}{\sum_{j=1}^{N}{\exp}(-d(A^{S}_{j},A^{Q}_{j}))},\end{split}$
(13)

where $d$ indicates the distance metric, and $A^{S}_{i}$ is the prototype
computed by the adaptive feature maps of support instances in $i$-th class.

###  3.6 Discussion

Figure 6:  2D-aggregated feature activation on 2-way 1-shot. (Case 1) If beaks
and wings are not similar between species, TDM regards both beaks and wings to
be discriminative. (Case 2) However, when birds have similar beaks, TDM
considers only wings as a discriminative part.

Commonly, it is known that the feature map containing various information
about the object is beneficial for a general dataset[13, 23, 21]. However, in
a fine-grained dataset, those feature maps are often harmful to infer the
categories. Instead, it is advantageous to focus only on the discriminative
parts since the categories share common overall apperance [9, 27, 6, 50].
Moreover, unlike general fine-grained classification where the discriminative
region of each category is almost constant, the distinct region of each class
in few-shot setting may vary depending on the contents of the episode.
Therefore, the key point of fine-grained few-shot classification is
dynamically discovering the discriminative regions of each class based on very
small number of instances. In Fig. 6, we observe that the baseline model
treats all characteristics equally regardless of the composition of each
episode. However, TDM adaptively allows the model to highlight regions
expected as discriminative parts and suppress other regions. Thus, TDM is a
specialized module for fine-grained few-shot classification.

##  4 Experiments

In this section, we evaluate TDM on standard fine-grained classification
benchmarks. To verify high adaptability of TDM, we apply it to various
existing methods: ProtoNet[34], DSN[33], CTX[7], and FRN[42]. For a fair
comparison, we reproduce each baseline model with the same hyperparameter and
implementation details regardless of whether TDM is attached or not. In each
table, $\dagger$ indicates the reproduced version of the baseline model. While
TDM generally exploits the prototype[34] defined in Eq. 2 for computing the
intra and inter score, in CTX, it utilizes a query-aligned prototype[7].

###  4.1 Datasets

2.5 Dataset | $C_{all}$ | $C_{train}$ | $C_{val}$ | $C_{test}$
---|---|---|---|---
2.5 CUB-200-2011 | 200 | 100 | 50 | 50
Aircraft | 100 | 50 | 25 | 25
meta-iNat | 1135 | 908 | - | 227
tiered meta-iNat | 1135 | 781 | - | 354
Stanford-Cars | 196 | 130 | 17 | 49
Stanford-Dogs | 120 | 70 | 20 | 30
Oxford-Pets | 37 | 20 | 7 | 10
2.5 |  |  |  |

Table 2:  The splits of datasets. While $C_{all}$ is the number of total
classes, $C_{train}$, $C_{val}$, $C_{test}$ are the number of training,
validation, and test classes, respectively. The classes of subsets are
disjoint.

We utilize seven benchmarks for few-shot classification: CUB-200-2011,
Aircraft, meta-iNat, tiered meta-iNat, Stanford-Cars, Stanford-Dogs, and
Oxford-Pets. The data split for each dataset is provided in Sec. 4.1.

CUB-200-2011[39] is an image dataset with 11,788 photos of 200 bird species.
This benchmark can be used in two ways: raw form[3] or preprocessed form by a
human-annotated bounding box[45, 47]. For a fair comparison, we utilizes both
settings. Following [3], we split this benchmark and our split is the same
with [42].

Aircraft[28] contains 10,000 airplane images of 100 models. The main challenge
of this benchmark is the similarity by the symbol of the airline. Although the
types of aircraft are different, the symbol can be equivalent when they belong
to the same airline, making it more difficult. Following [42], we preprocess
all images of this benchmark based on the bounding box and divide the dataset.

meta-iNat[41, 36] is a realistic, heavy-tailed benchmark for few-shot
classification. It contains 1,135 animal species spanning 13 super categories,
and the number of images in each class is imbalanced with a range between 50
and 1000 images. We follow the dataset split introduced in [41]. While a full
227-way evaluation scheme, that each episode consists of all testing
categories at once, is adopted in [41], we employ a standard 5-way few-shot
evaluation scheme following [42].

tiered meta-iNat[41] is comprised of the same images with meta-iNat. However,
this benchmark divides the dataset by super categories. The super categories
of test dataset are Insects and Arachnids, while the super categories of train
dataset are Plant, Bird, Mammal, and etc. Therefore, a large domain gap exists
between the train and the test classes. As following [42], we also employ a
standard 5-way few-shot evaluation scheme in this benchmark.

Stanford Cars[18] contains 16,185 images of 196 classes of cars. Classes are
typically at the level of Year, Brand and Model name, e.g., 2012 Tesla Model S
and 2012 BMW M3 coupe. This dataset was introduced by [24] for the few-shot
classification. Likewise, we adopt the same data split with [24].

Stanford Dogs[15] is also introduced by [24] for fine-grained few-shot
classification. This dataset contains 20,580 images of 120 breeds of dogs
around the world. Our split is the same with [24].

Oxford Pets[30] is comprised of 37 pet categories with roughly 200 images for
each class. Since the lack of training images generally causes overfitting,
the generalization capability is essential to attain high accuracy in this
benchmark. To our knowledge, this benchmark dataset has never been used for
few-shot classification. Therefore, we randomly divide this dataset by
referring to the split ratio of the other datasets. We report the split
information in the supplementary material.

2.5 Model | Conv-4 | ResNet-12
---|---|---
| 1-shot | 5-shot | 1-shot | 5-shot
2.5 MatchNet[38, 45, 47] | 67.73 | 79.00 | 71.87 | 85.08
ProtoNet[34, 45, 47] | 63.73 | 81.50 | 66.09 | 82.50
FEAT∗[45] | 68.87 | 82.90 | 73.27 | 85.77
DeepEMD[47] | - | - | 75.65 | 88.69
RENet[14] | - | - | 79.49 | 91.11
1.0 ProtoNet†[34] | 62.90 | 84.13 | 78.99 | 90.74
\+ TDM | 69.94 | 86.96 | 79.58 | 91.28
1\. DSN†[33] | 72.09 | 85.03 | 80.51 | 90.23
\+ TDM | 73.38 | 86.07 | 81.33 | 90.65
1\. CTX†[7] | 72.14 | 87.23 | 80.67 | 91.55
\+ TDM | 74.68 | 88.36 | 83.28 | 92.74
1\. FRN†[42] | 73.24 | 88.33 | 83.16 | 92.42
\+ TDM | 74.39 | 88.89 | 83.36 | 92.80
2.5 |  |  |  |

Table 3: Performance on CUB using bounding-box cropped images as input.
“$\ast$” denotes reproduced one in RENet. Confidence intervals for our
implemented model are all below 0.23.

###  4.2 Implementation Details

2.5 Model | Backbone | 1-shot | 5-shot
---|---|---|---
2.5 Baseline♭[3] | ResNet-18 |  65.51$\pm$0.87 |  82.85$\pm$0.55
Baseline++♭[3] | ResNet-18 |  67.02$\pm$0.90 |  83.58$\pm$0.54
MatchNet♭[3, 38] | ResNet-18 |  73.42$\pm$0.89 |  84.45$\pm$0.58
ProtoNet♭[3, 34] | ResNet-18 |  72.99$\pm$0.88 |  86.65$\pm$0.51
MAML♭[3, 8] | ResNet-18 |  68.42$\pm$1.07 |  83.47$\pm$0.62
RelatioNet♭[3, 35] | ResNet-18 |  68.58$\pm$0.94 |  84.05$\pm$0.56
S2M2♭[29] | ResNet-18 |  71.43$\pm$0.28 |  85.55$\pm$0.52
Neg-Cosine♭[26] | ResNet-18 |  72.66$\pm$0.85 |  89.40$\pm$0.43
Afrasiyabi et al.♭[1] | ResNet-18 |  74.22$\pm$1.09 |  88.65$\pm$0.55
1\. ProtoNet†[34] | ResNet-12 |  78.58$\pm$0.22 |  89.83$\pm$0.12
\+ TDM | ResNet-12 |  79.11$\pm$0.22 |  90.83$\pm$0.11
1\. DSN†[33] | ResNet-12 |  80.47$\pm$0.20 |  89.92$\pm$0.12
\+ TDM | ResNet-12 |  80.58$\pm$0.20 |  89.95$\pm$0.12
1\. CTX†[7] | ResNet-12 |  80.95$\pm$0.21 |  91.54$\pm$0.11
\+ TDM | ResNet-12 |  83.45$\pm$0.19 |  92.49$\pm$0.11
1\. FRN†[42] | ResNet-12 |  83.54$\pm$0.19 |  92.96$\pm$0.10
\+ TDM | ResNet-12 | 84.36$\pm$0.19 | 93.37$\pm$0.10
2.5 |  |  |

Table 4: Performance on CUB using raw images as input. “$\flat$” denotes
larger backbones than ResNet-12.

Architecture. We adopt common protocols from recent few-shot classification
works[17, 4, 49, 11, 48]; we employ Conv-4 and ResNet-12. While both backbone
networks accept an image of size 84$\times$84, the size of feature maps is
different according to the backbone network. ResNet-12 yields a feature map
with size 640$\times$5$\times$5 while Conv-4 offers 64$\times$5$\times$5 shape
of the feature map. For our proposed TDM, we additionally utilize fully-
connected layer blocks where the size of blocks are proportional to the
dimension of channels of the feature maps as described in Tab. 1.
Training Details. Following existing methods[3, 40, 45, 47, 42], we use
standard data augmentation techniques including random crop, horizontal flip,
and color jitter. The $\alpha,\beta$ in Eq. 7, Eq. 10 are fixed to 0.5 and
other parameters are adopted from [42] which is our baseline model. More
details are described in the supplementary material. To prevent overfitting,
we add random noise between -0.2 and 0.2 to the task weight for each category
from TDM.
Evaluation Details. For the N-way K-shot, we conduct few-shot classification
on 10,000 randomly sampled episodes where it contains 16 queries per class. We
report average classification accuracy with 95% confidence intervals.

(a) Stanford Cars

(b) Stanford Dogs

(c) Oxford Pets

Figure 7:  Accuracies on additional datasets. The left and right graphs for
each dataset show 1-shot accuracies and 5-shot accuracies, respectively. The
left side of the graph reports the performance of baseline models, while the
right side shows performance with TDM. The baseline methods are differentiated
with colors.

###  4.3 Comparison to Existing Methods

CUB-200-2011 results. Sec. 4.1 and Sec. 4.2 reports the results of TDM and
baseline few-shot classification methods. Although some cases do not surpass
confidence intervals, our TDM consistently improves the performance of
baselines models in all cases, and achieves state-of-the-art scores regardless
the depth of the backbone network. Especially, TDM improves more than 7% on
ProtoNet in the 1-shot scenario of CUB with Conv-4.
Aircraft results. As shown in Sec. 4.3, TDM improves the performances of the
baseline models in all cases. The increases are beyond the confidence
intervals regardless of the type of baseline models and the number of labeled
images. As a result, we attain top accuracy scores for all benchmarks by large
margin. Specifically, TDM boosts the performance of CTX[7] up to 7% on the
Conv-4 network. These results demonstrates the effectiveness of TDM.
meta-iNat results. We validate the generalization ability of TDM in this
benchmark. Meta-iNat is vulnerable to overfitting since the validation set
does not exist. However, as shown in Sec. 4.3, TDM is not only robust to
overfitting issues but also powerful in generalization capability.
Consequently, TDM shows consistent improvements over the baselines,
particularly on ProtoNet where TDM assists them to retain competitive results
compared to state-of-the-art methods.
tiered meta-iNat results. We further validate the generalization capability of
TDM in a more difficult configuration where the super categories of train set
and test set do not overlap. Specifically, TDM enhances the performance in
most configurations and also accomplishes the best performance in the 1-shot
scenario with FRN. For the slight decrease in a 5-shot scenario, $\lambda$,
the learnable parameter in FRN, is responsible. In general, large $\lambda$
shows good performance when a domain gap exists but TDM restrains the
$\lambda$ to be relatively small. We think that this is because TDM assists
the classifier to focus on discriminative channels[42].
Stanford Cars, Stanford Dogs and Oxford Pets results. Unlike previous
benchmarks, these datasets were not evaluated in our baseline models[34, 33,
7, 42]. To further validate the effectiveness of TDM, we additional conduct
experiments on those fine-grained datasets with Conv-4. As reported in Fig. 7,
TDM is capable to improve the performance that the confidence interval does
not overlap in all cases regardless of the baseline models. In detail, TDM
shows performance boosts in which their accuracies are 4.44 and 3.27 points
higher than the baseline at the 1-shot and 5-shot scenarios, respectively.

Throughout the extensive experiments on seven benchmark datasets, we clearly
validate the strength of TDM in fine-grained few-shot classification. To
summarize, we improve the performance of the baseline models in all benchmarks
datasets, achieving the state-of-the-art results except the one case: FRN on
the tiered meta-iNat 5-shot scenario.

2.5 Model | Conv-4 | ResNet-12
---|---|---
| 1-shot | 5-shot | 1-shot | 5-shot
2.5 ProtoNet†[34] | 47.37 | 68.96 | 67.28 | 83.21
\+ TDM | 50.55 | 71.12 | 69.12 | 84.77
1\. DSN†[33] | 52.22 | 68.75 | 70.23 | 83.05
\+ TDM | 53.77 | 69.56 | 71.57 | 83.65
1\. CTX†[7] | 51.58 | 68.12 | 65.53 | 79.31
\+ TDM | 55.15 | 70.45 | 69.42 | 83.25
1\. FRN†[42] | 53.12 | 70.84 | 69.58 | 82.98
\+ TDM | 54.21 | 71.37 | 70.89 | 84.54
2.5 |  |  |  |

Table 5: Performance on Aircraft. Confidence intervals for our implemented
model are all below 0.25.

2.5 Model | meta-iNat | tiered meta-iNat
---|---|---
| 1-shot | 5-shot | 1-shot | 5-shot
2.5 ProtoNet†[34] | 55.37 | 76.30 | 34.41 | 57.60
\+ TDM | 61.82 | 79.95 | 38.30 | 61.18
1\. DSN†[33] | 60.06 | 76.15 | 40.83 | 58.34
\+ TDM | 61.87 | 78.07 | 41.00 | 58.66
1\. CTX†[7] | 60.80 | 78.57 | 42.24 | 60.54
\+ TDM | 63.26 | 80.75 | 43.90 | 62.29
1\. FRN†[42] | 61.98 | 80.04 | 43.95 | 63.45
\+ TDM | 63.97 | 81.60 | 44.05 | 62.91
2.5 |  |  |  |

Table 6:  Performance on meta-iNat and tiered meta-iNat using Conv-4
backbones. Confidence intervals for our implemented model are all below 0.23.

##  5 Ablation Study

We conduct ablation study with ProtoNet[34] on the Conv-4 backbone using
CUB_cropped and Aircraft datasets.

###  5.1 Effect of Submodules

Sec. 5.1 reports the effects of sub-modules of TDM. We observe that both SAM
and QAM consistently improve the classification accuracies. As the second and
fourth rows show, SAM improves the baseline by a large gain up to 11%. This
large gain confirms that recognizing discriminative channels for each category
is crucial for fine-grained few shot classification. Furthermore, although the
improvement of QAM is slightly lower than SAM, QAM is shown to be effective
for all scenarios. We think that this is because the object-relevant channels
not always represent discriminate channels. More importantly, the performances
can be further boosted when combined. This validates that two sub-modules are
complementary to one another.

2.5 SAM | QAM | CUB_cropped | Aircraft
---|---|---|---
1-shot | 5-shot | 1-shot | 5-shot
2.5 - | - | 62.90 | 84.13 | 47.37 | 68.96
✓ | - | 68.53 | 85.95 | 49.45 | 69.33
- | ✓ | 65.11 | 84.82 | 48.96 | 70.85
✓ | ✓ | 69.94 | 86.96 | 50.55 | 71.12
2.5 |  |  |  |  |

Table 7: Effects of SAM and QAM.

###  5.2 Choice of Pooling Function

We also study the effects of pooling functions, as described in Sec. 6. As
reported in the second and third rows, both pooling methods boost the
performance since they are capable of representing the details of the objects.
However, as shown in Fig. 2, max-pooling function has its limitation in that
it is vulnerable to noise. So thus, we adopt the average-pooling function to
predict discriminative channels.

###  5.3 Metric Compatibility

Following our baselines [34, 33, 7, 42], we employ the Euclidean distance for
TDM. Since TDM is compatible with other metrics, we evaluate it with the
cosine distance. As reported in Sec. 6, TDM improves the baseline. This
validates that TDM’s flexibility to commonly used distance metrics: the
Euclidean and cosine distances.

##  6 Limitation

Since representing the whole objects could be rather harmful for classifying
fine-grained categories unlike the general classification task, TDM is
developed for highlighting features discriminative for fine-grained details.
Thus, the benefits of TDM could be limited in coarse-grained tasks.

2.5 TDM | Pooling | CUB_cropped | Aircraft
---|---|---|---
1-shot | 5-shot | 1-shot | 5-shot
2.5 - | - | 62.90 | 84.13 | 47.37 | 68.96
✓ | Max | 67.23 | 86.73 | 50.16 | 71.32
✓ | Avg | 69.94 | 86.96 | 50.55 | 71.12
2.5 |  |  |  |  |

Table 8: Effects of Pooling functions. Top row is the baseline result.

2.5 TDM | CUB_cropped | Aircraft
---|---|---
1-shot | 5-shot | 1-shot | 5-shot
2.5 - | 68.69 | 82.89 | 48.36 | 63.45
✓ | 70.47 | 84.34 | 49.21 | 66.26
2.5 |  |  |  |

Table 9: Effectiveness of TDM with the cosine distance

##  7 Conclusion

In this paper, we introduced Task Discrepancy Maximization (TDM), a tailored
module for fine-grained few-shot classification. TDM produces channel weights
that emphasize features of fine discriminative details to distinguish similar
classes with two submodules: Support Attention Module (SAM) and Query
Attention Module (QAM). Our extensive experiments on several fine-grained
benchmarks validated the merits of our proposed TDM in terms of its
effectiveness and high applicability with the prior few-shot classification
methods. As a future direction, we will investigate how the significance for
each channel vary in other computer vision tasks and extend this module to
those tasks.

Acknowledgements. This work was supported in part by MCST/KOCCA (No.
R2020070002), MSIT/IITP (No. 2020-0-00973, 2019-0-00421, 2020-0-01821,
2021-0-02003, and 2021-0-02068), and MSIT&KNPA/KIPoT (Police Lab 2.0, No.
210121M06).

## References

  * [1] Arman Afrasiyabi, Jean-François Lalonde, and Christian Gagné.  Associative alignment for few-shot image classification.  In European Conference on Computer Vision, pages 18–35. Springer, 2020.
  * [2] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  Neural machine translation by jointly learning to align and translate.  In International Conference on Learning Representations, 2015.
  * [3] Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang.  A closer look at few-shot classification.  In International Conference on Learning Representations, 2019.
  * [4] Zhengyu Chen, Jixie Ge, Heshen Zhan, Siteng Huang, and Donglin Wang.  Pareto self-supervised training for few-shot learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13663–13672, 2021.
  * [5] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [6] Yao Ding, Yanzhao Zhou, Yi Zhu, Qixiang Ye, and Jianbin Jiao.  Selective sparse sampling for fine-grained image recognition.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6599–6608, 2019.
  * [7] Carl Doersch, Ankush Gupta, and Andrew Zisserman.  Crosstransformers: spatially-aware few-shot transfer.  In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
  * [8] Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In International Conference on Machine Learning, pages 1126–1135. PMLR, 2017.
  * [9] Weifeng Ge, Xiangru Lin, and Yizhou Yu.  Weakly supervised complementary parts models for fine-grained image classification from the bottom up.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3034–3043, 2019.
  * [10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [11] Jie Hong, Pengfei Fang, Weihao Li, Tong Zhang, Christian Simon, Mehrtash Harandi, and Lars Petersson.  Reinforced attention for few-shot learning and beyond.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 913–923, 2021.
  * [12] Ruibing Hou, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen.  Cross attention network for few-shot classification.  Advances in Neural Information Processing Systems, 32, 2019.
  * [13] Zeyi Huang, Haohan Wang, Eric P Xing, and Dong Huang.  Self-challenging improves cross-domain generalization.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 124–140. Springer, 2020.
  * [14] Dahyun Kang, Heeseung Kwon, Juhong Min, and Minsu Cho.  Relational embedding for few-shot classification.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8822–8833, 2021.
  * [15] Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li.  Novel dataset for fine-grained image categorization: Stanford dogs.  In Proc. CVPR Workshop on Fine-Grained Visual Categorization (FGVC), volume 2. Citeseer, 2011.
  * [16] Valentin Khrulkov, Leyla Mirvakhabova, Evgeniya Ustinova, Ivan Oseledets, and Victor Lempitsky.  Hyperbolic image embeddings.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6418–6428, 2020.
  * [17] Jongmin Kim, Taesup Kim, Sungwoong Kim, and Chang D Yoo.  Edge-labeling graph neural network for few-shot learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11–20, 2019.
  * [18] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In Proceedings of the IEEE international conference on computer vision workshops, pages 554–561, 2013.
  * [19] Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto.  Meta-learning with differentiable convex optimization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10657–10665, 2019.
  * [20] Aoxue Li, Weiran Huang, Xu Lan, Jiashi Feng, Zhenguo Li, and Liwei Wang.  Boosting few-shot learning with adaptive margin loss.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12576–12584, 2020.
  * [21] Bohao Li, Boyu Yang, Chang Liu, Feng Liu, Rongrong Ji, and Qixiang Ye.  Beyond max-margin: Class margin equilibrium for few-shot object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7363–7372, 2021.
  * [22] Congcong Li, Dawei Du, Libo Zhang, Longyin Wen, Tiejian Luo, Yanjun Wu, and Pengfei Zhu.  Spatial attention pyramid network for unsupervised domain adaptation.  In European Conference on Computer Vision, pages 481–497. Springer, 2020.
  * [23] Junjie Li, Zilei Wang, and Xiaoming Hu.  Learning intact features by erasing-inpainting for few-shot classification.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 8401–8409, 2021.
  * [24] Wenbin Li, Lei Wang, Jinglin Xu, Jing Huo, Yang Gao, and Jiebo Luo.  Revisiting local descriptor based image-to-class measure for few-shot learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7260–7268, 2019.
  * [25] Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio.  A structured self-attentive sentence embedding.  In International Conference on Learning Representations, 2017.
  * [26] Bin Liu, Yue Cao, Yutong Lin, Qi Li, Zheng Zhang, Mingsheng Long, and Han Hu.  Negative margin matters: Understanding margin in few-shot classification.  In European Conference on Computer Vision, pages 438–455. Springer, 2020.
  * [27] Chuanbin Liu, Hongtao Xie, Zheng-Jun Zha, Lingfeng Ma, Lingyun Yu, and Yongdong Zhang.  Filtration and distillation: Enhancing region attention for fine-grained visual categorization.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11555–11562, 2020.
  * [28] S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi.  Fine-grained visual classification of aircraft.  Technical report, 2013.
  * [29] Puneet Mangla, Nupur Kumari, Abhishek Sinha, Mayank Singh, Balaji Krishnamurthy, and Vineeth N Balasubramanian.  Charting the right manifold: Manifold mixup for few-shot learning.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2218–2227, 2020.
  * [30] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar.  Cats and dogs.  In 2012 IEEE conference on computer vision and pattern recognition, pages 3498–3505. IEEE, 2012.
  * [31] Sachin Ravi and Hugo Larochelle.  Optimization as a model for few-shot learning.  In International Conference on Learning Representations, 2017.
  * [32] Ryne Roady, Tyler L Hayes, Ronald Kemker, Ayesha Gonzales, and Christopher Kanan.  Are open set classification methods effective on large-scale datasets?  Plos one, 15(9):e0238302, 2020.
  * [33] Christian Simon, Piotr Koniusz, Richard Nock, and Mehrtash Harandi.  Adaptive subspaces for few-shot learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4136–4145, 2020.
  * [34] Jake Snell, Kevin Swersky, and Richard Zemel.  Prototypical networks for few-shot learning.  Advances in neural information processing systems, 30, 2017.
  * [35] Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales.  Learning to compare: Relation network for few-shot learning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1199–1208, 2018.
  * [36] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie.  The inaturalist species classification and detection dataset.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8769–8778, 2018.
  * [37] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008, 2017.
  * [38] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al.  Matching networks for one shot learning.  Advances in neural information processing systems, 29:3630–3638, 2016.
  * [39] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie.  The Caltech-UCSD Birds-200-2011 Dataset.  Technical report, 2011.
  * [40] Yan Wang, Wei-Lun Chao, Kilian Q Weinberger, and Laurens van der Maaten.  Simpleshot: Revisiting nearest-neighbor classification for few-shot learning.  arXiv preprint arXiv:1911.04623, 2019.
  * [41] Davis Wertheimer and Bharath Hariharan.  Few-shot learning with localization in realistic settings.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6558–6567, 2019.
  * [42] Davis Wertheimer, Luming Tang, and Bharath Hariharan.  Few-shot classification with feature map reconstruction networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8012–8021, 2021.
  * [43] Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon.  Cbam: Convolutional block attention module.  In Proceedings of the European conference on computer vision (ECCV), pages 3–19, 2018.
  * [44] Chengming Xu, Yanwei Fu, Chen Liu, Chengjie Wang, Jilin Li, Feiyue Huang, Li Zhang, and Xiangyang Xue.  Learning dynamic alignment via meta-filter for few-shot learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5182–5191, 2021.
  * [45] Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha.  Few-shot learning via embedding adaptation with set-to-set functions.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8808–8817, 2020.
  * [46] Baoquan Zhang, Xutao Li, Yunming Ye, Zhichao Huang, and Lisai Zhang.  Prototype completion with primitive knowledge for few-shot learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3754–3762, 2021.
  * [47] Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen.  Deepemd: Few-shot image classification with differentiable earth mover’s distance and structured classifiers.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12203–12213, 2020.
  * [48] Hongguang Zhang, Piotr Koniusz, Songlei Jian, Hongdong Li, and Philip HS Torr.  Rethinking class relations: Absolute-relative supervised and unsupervised few-shot learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9432–9441, 2021.
  * [49] Jiabao Zhao, Yifan Yang, Xin Lin, Jing Yang, and Liang He.  Looking wider for better adaptive representation in few-shot learning.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10981–10989, 2021.
  * [50] Heliang Zheng, Jianlong Fu, Zheng-Jun Zha, and Jiebo Luo.  Looking for the devil in the details: Learning trilinear attention sampling network for fine-grained image recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5012–5021, 2019.