Task-Oriented Channel Attention for
Fine-Grained Few-Shot Classification

SuBeen Lee, WonJun Moon, Hyun Seok Seong, and Jae-Pil Heo, Member, IEEE,

SuBeen Lee, WonJun Moon, and Hyun Seok Seong are with the Department of Artificial Intelligence, Sungkyunkwan University, South Korea, 16419.

E-mail: {leesb7426, wjun0830<EMAIL_ADDRESS>Jae-Pil Heo is with the Department of Computer Science and Engineering and Department of Artificial Intelligence, Sungkyunkwan University, South Korea, 16419.

E-mail<EMAIL_ADDRESS>
Corresponding author: Jae-Pil Heo

Journal of  Class Files, Vol. 14, No. 8, August 2015
Shell et al.: Bare Demo of IEEEtran.cls for Computer Society Journals

The difficulty of the fine-grained image classification mainly comes from a shared overall appearance across classes. Thus, recognizing discriminative details, such as eyes and beaks for birds, is a key in the task. It is especially challenging when only a few training data is available. In this regard, we propose a simple module specialized for the fine-grained few-shot classification, Task Discrepancy Maximization (TDM). Our objective is to localize the class-wise discriminative regions by highlighting channels encoding distinct information about the class.
To be specific, TDM produces task-specific channel weights based on two novel components, Support Attention Module (SAM) and Query Attention Module (QAM).
SAM yields a support weight vector per class to represent channel-wise discriminative power for each category.
Still, since SAM only considers the label support sets, it can be vulnerable to bias toward such support sets.
Therefore, we also introduce QAM which complements SAM by producing a query weight vector that gives higher weights to object-relevant channels for the query image.
By combining these two weight vectors, a class-wise task-specific channel weight vector for each category is defined.
Then, the weight vectors are applied to produce task-adaptive feature maps more focusing on the discriminative details.
Although TDM can solely boost the performance in fine-grained few-shot classification with simplicity, it mostly influences to relatively high-level features.
To further realize our motivation to focus on object-relevant information even for the feature map themselves, we introduce Instance Attention Module (IAM) by extending the QAM.
Specifically, for all instances regardless of support- and query-set, IAM operates in intermediate layers of the feature extractor to highlight instance-wise informative channels.
Our extensive experiments validate the effectiveness of TDM and IAM, which possess complementary benefits, in the fine-grained few-shot classification.
Moreover, we further show that IAM can also be utilized in coarse-grained and cross-domain few-shot classification tasks.

The difficulty of the fine-grained image classification mainly comes from a shared overall appearance across classes. Thus, recognizing discriminative details, such as eyes and beaks for birds, is a key in the task. However, this is particularly challenging when training data is limited. To address this, we propose Task Discrepancy Maximization (TDM), a task-oriented channel attention method tailored for fine-grained few-shot classification with two novel modules Support Attention Module (SAM) and Query Attention Module (QAM). SAM highlights channels encoding class-wise discriminative features, while QAM assigns higher weights to object-relevant channels of the query. Based on these submodules, TDM produces task-adaptive features by focusing on channels encoding class-discriminative details and possessed by the query at the same time, for accurate class-sensitive similarity measure between support and query instances. While TDM influences high-level feature maps by task-adaptive calibration of channel-wise importance, we further introduce Instance Attention Module (IAM) operating in intermediate layers of feature extractors to instance-wisely highlight object-relevant channels, by extending QAM. The merits of TDM and IAM and their complementary benefits are experimentally validated in fine-grained few-shot classification tasks. Moreover, IAM is also shown to be effective in coarse-grained and cross-domain few-shot classifications.

Fine-grained few-shot classification is a challenging task since only a few data are not sufficient to recognize discriminative details between classes sharing overall appearances.
In this regard, we propose Task Discrepancy Maximization (TDM) for fine-grained few-shot classification.
TDM localizes class-wise discriminative regions by highlighting distinct informative channels about the class with Support Attention Module (SAM) and Query Attention Module (QAM).
SAM yields a support weight vector per class to represent channel-wise discriminative power for each category.
QAM produces a query weight vector giving higher weights object-relevant channels for the query image.
By combining these two vectors, TDM defines a class-wise task-specific channel weight vector for each category and applies it to produce task-adaptive feature maps focusing on the discriminative details.
Although TDM can boost the performance in fine-grained few-shot classification, it highly depends on the quality of the backbone since it only exploits high-level features.
To focus on object-relevant information even for computing the feature map, we introduce Instance Attention Module (IAM) by extending QAM.
IAM operates in intermediate layers of the backbone to highlight instance-wise informative channels.
Experiments show that TDM and IAM complementarily improve fine-grained few-shot classification, and IAM is also effective in coarse-grained and cross-domain few-shot classification.

Discriminative details such as eyes and beaks are the key components in distinguishing fine-grained classes since these classes share general appearances.
In this regard, we suggest a simple module for fine-grained few-shot classification, Task Discrepancy Maximization (TDM).
Our objective is to localize the class-wise discriminative regions by highlighting channels encoding distinct information about the class.
To be specific, TDM produces task-specific channel weights based on two novel components: Support Attention Module (SAM) and Query Attention Module (QAM).
SAM yields a support weight vector per class to represent channel-wise discriminative power for each category.
Still, since SAM only considers the label support sets, it can be vulnerable to bias toward such support sets.
Therefore, we also suggest QAM which complements SAM by producing a query weight vector that gives more weight to object-relevant channels for a given query image.
By combining these two weight vectors, a class-wise task-specific channel weight vector for each category is defined.
Then, the weight vectors are applied to produce task-adaptive feature maps more focusing on the discriminative details.

Although TDM shows superior performance in fine-grained few-shot classification, it does not contain a method to improve the quality of the feature map influencing the output of TDM.
To induce the feature extractor to compute feature maps by focusing on object-relevant information, we also introduce Instance Attention Module (IAM), the extended version of QAM.
Specifically, for every instance including support- and query-set, IAM operates in the intermediate layer of the feature extractor and highlights instance-wise informative channels.
Therefore, since the feature maps are highly influenced by such channels, they contain more object-relevant information than before.
Our experiments validate the effectiveness of TDM and IAM, which possess complementary benefits, in the fine-grained few-shot classification.
Moreover, we further show that IAM also can be utilized in coarse-grained and cross-domain few-shot classification.

Few-Shot Classification, Fine-grained Classification, Feature Alignment, Attention Module.

§ INTRODUCTION

Deep learning has made great strides in various vision tasks, even achieving remarkable performance beyond humans in many downstream tasks <cit.>.

However, such performance is achieved under the presence of numerous labeled images, which require huge labeling costs.

In other words, the performance can be significantly degraded if the number of labeled images is insufficient <cit.>.

Therefore, such limited condition from a shortage of labeled images and the high cost of labeling promotes the growth of few-shot classification <cit.>, which is to train a model highly adaptable to novel classes.

To achieve this goal, the training of the few-shot classification is mainly based on the episodic learning strategy, where each episode comprises a few sampled categories from the dataset.

In addition, images of each class are split into a support set and a query set for the training and evaluation, respectively.

The metric-based learning is a mainstream of the few-shot classification <cit.>.

These methods learn a deep representation with a predefined or online-trained metric,

and the inference for a query is performed based on the distances among support and query sets under such metric.

However, since the feature extractor is only trained with base classes, the feature maps for novel classes computed by the learned extractor, hardly form a tight cluster <cit.>.
To alleviate this, recent methods utilize primitive knowledge <cit.> or propose task-dynamic feature alignment strategies <cit.>.

Among two strategies, task-dynamic feature alignment approaches are being spotlighted and can be further divided into two main streams; spatial alignment and channel alignment.
The spatial alignment methods <cit.> aim to resolve the spatial mismatch between key features on the feature maps of different instances.

On the other hand, since the semantic feature maps of novel classes are not optimized for each episode, the channel alignment methods try to adapt those feature maps to the target classification task by considering the composition of the episode.

Although aforementioned alignment methods accomplish huge improvements on the coarse-grained few-shot classification task, they provide insignificant gains for fine-grained datasets.

This is mainly because they only focus on exploit channel or spatial information which may not be discriminative for the episode.
Indeed, localizing discriminative details is important in fine-grained classification, since categories are highly likely to share similar overall appearances <cit.>.

Therefore, distinct clues for each category, which have only subtle differences from other categories, should also be captured for fine-grained few-shot classification.

In this context, we introduce a novel module, Task Discrepancy Maximization (TDM), that localizes discriminative regions by weighting channels per class.
TDM highlights the channels representing discriminative regions and restrains the contributions of other channels based on a class-wise channel weight vector.
Specifically, TDM is composed of two components: Support Attention Module (SAM) and Query Attention Module (QAM).
Given a support set, SAM produces a support weight vector per class that presents high activations on discriminative channels.
On the other hand, QAM is fed with the query set to output a query weight vector per instance, where such query weight vector highlights the object-relevant channels.

To compute these weight vectors, the relation between each feature map and the corresponding channel-wisely average pooled feature is considered.

Since the channel-wisely average pooled feature map has the spatial information of the object <cit.>,

channels are highly likely to represent salient regions when they are similar to spatially averaged feature maps.
By combining two weight vectors computed from our sub-modules, a task-specific weight vector is finally defined.

Consequently, the task-specific weight vector is utilized to produce task-adaptive feature maps which replace the original feature maps.

Although TDM is a tailored module for the fine-grained few-shot classification task, its performance can be highly dependent on the quality of given feature maps produced by the feature extractor since the TDM is designed to work with high-level feature maps. Therefore, we further introduce IAM, Instance Attention Module as an extended version of QAM, to implement our main idea even for the feature extraction. Unlike QAM, IAM operates in the intermediate layers of the feature extractor and computes a channel weight vector per instance to enhance the quality of the feature representation like existing attention methods <cit.>.
Since IAM induces the feature extractor to focus on instance-wise informative channels, the resulting feature map contains more object-relevant information and less background information. As mentioned, the IAM is designed to complement the TDM in the feature extraction stage, interestingly, however, it also helps to boost the performances of the general few-shot classification task.

Although TDM is a well-tailored module for fine-grained few-shot classification, the quality of its output can be dependent on the feature map.
Therefore, we further introduce Instance Attention Module (IAM), an extended version of QAM.
Unlike QAM, IAM operates in the intermediate layer of the feature extractor and computes a channel weight vector per instance to enhance the quality of the feature representation like existing attention methods <cit.>.
Since IAM induces the feature extractor to focus on instance-wise information channels, the feature map contains more object-relevant information and less background-relevant information.
Note that, IAM is initially designed to complement TDM, but it can be also adopted in the general few-shot classification.

Our main contributions are summarized as follows:

* We propose a novel feature alignment method, TDM, to define the class-wise channel importance based on identifying class-discriminative and query-relevant channels, tailored for the fine-grained few-shot classification task.

* We further extend QAM to introduce IAM by reflecting the concepts of TDM to the feature extractor, which not only complements TDM in the fine-grained tasks but also benefits for more general scenarios including coarse-grained and cross-domain few-shot classification.

* We experimentally validate the high applicability of proposed TDM and IAM to the prior few-shot classification models and strength of them by achieving new state-of-the-art performances in standard benchmarks.

Our main contributions are summarized as follows:

* We propose a novel feature alignment method, TDM, to define the class-wise channel importance, for fine-grained few-shot classification.

* Our proposed TDM is highly applicable to prior metric-based fine-grained few-shot classification models.

* We extend QAM to introduce IAM which not only complements TDM in fine-grained datasets but is also utilized in more general scenarios like coarse-grained and cross-domain.

* Our experiments with popular few-shot classification models verify the strength of TDM and IAM with new state-of-the-art performances.

§ RELATED WORK

§.§ Few-Shot Classification

There are two main streams in the few-shot image classification research, the optimization- and metric-based approaches. At early stage, MAML introduced the concept of optimization-based methods where it learns good initial conditions for adaptation to the novel tasks. Then, Meta-LSTM <cit.> used an LSTM-based meta-learner for general initial point and effective fine-tuning.
Moreover, MetaOptNet <cit.> provides a differentiation process for end-to-end learning by utilizing convex base learners.
Although these optimization-based methods show promising results, they need online updates for novel classes.

On the other hand, the metric-based methods aim to learn deep representations by adopting a predefined <cit.> or online-trained metric <cit.>.
Its concept is introduced in MatchNet <cit.> which infers categories of the query set by the cosine similarity.
ProtoNet <cit.> further employs a mean feature of each class as a prototype and utilizes them for computing the distance between a query and each class.
Instead of the predefined metrics, RelationNet <cit.> exploits a learnable distance metric.

As aforementioned, metric-based methods generally try to reduce distances among instances belonging to the same category. TDM is an applicable module for those metric-based methods to boost thier performance. Specifically, TDM enables the distances to be measured based on adaptive channel weights where it identifies and emphasizes discriminative channels dynamically, while prior techniques treat all the channels equally.

Optimization- and metric-based methods are the two main streams of few-shot classification.
Initially, MAML introduced the concept of optimization-based methods where it learns good initial conditions for adaptation.
Then, Meta-LSTM <cit.> used an LSTM-based meta-learner for general initial point and effective fine-tuning.
And, MetaOptNet <cit.> provides a differentiation process for end-to-end learning by utilizing convex base learners.
Although the optimization-based methods show comparable performance, they need online updates for novel classes.

On the other hand, the metric-based methods aim to learn deep representations by adopting a predefined<cit.> or online-trained metric<cit.>.
The concept of it was introduced by MatchNet <cit.> which infers categories of the query set by the cosine similarity.
ProtoNet <cit.> employs a mean feature of each class as a prototype and exploits them for computing the distance between a query and each class.
Instead of the predefined metrics, RelationNet <cit.> exploits a distance metric that is learned by a model.

As aforementioned, metric-based methods generally try to reduce distances between instances which belong to the same category, and we have the same goal since TDM is a module for them.
However, TDM enables the distances to be measured based on adaptive channel weights where it identifies and emphasizes discriminative channels dynamically, while prior techniques treat all the channels equally.

§.§ Feature Alignment Methods

In the metric-based classification, feature alignment methods are developed for classification-friendly distance computations. These feature alignment approaches can be classified into spatial and channel alignment methods.
The spatial alignment methods <cit.> aim to align the features of the support and query sets to match object regions. For example, CAN <cit.> computes a correlation map for each pair of the classes and query feature maps to emphasize the common regions where the object likely to exist.
CTX <cit.> measures a coarse spatial correspondence between the query instance and the support set by the attention <cit.>, then it produces a query-aligned prototype per each class based on the correspondence.
FRN <cit.> reconstructs the feature maps of the support set in accordance with the feature map of the query instance by exploiting a closed-form solution of the ridge regression.

On the other hand, the channel alignment methods <cit.> alter feature maps so that the novel classes are well distinguished.
Specifically, FEAT manipulates the feature maps of support sets to increase the distance among classes by utilizing the transformer <cit.>.
DMF <cit.> aligns each feature map of the query instances by the dynamic meta-filter produced in the consideration of the support and query pair.
And, RENet <cit.> transforms feature maps of the support and query pair with self- and cross-correlation which capture the structural patterns of each image and encode semantically relevant contents, respectively.

Although TDM is basically a channel alignment method,

unlike existing methods that typically consider a pairwise relationship between the support set for each category and the query instance, TDM utilizes the entire task to adapt the feature maps.

Feature alignment methods can be classified into spatial and channel alignment methods.
The spatial alignment methods <cit.> assert that the object location of the support and query set should be aligned.
For example, CAN <cit.> computes a correlation map for each pair of the classes and query feature maps to emphasize the common regions where the object is presumed to exist.
CAN <cit.> measures a coarse spatial correspondence between the query instance and the support set by the attention <cit.>, then it produces a query-aligned prototype per each class based on the correspondence.
FRN <cit.> reconstructs the feature maps of the support set in accordance with the feature map of the query instance by exploiting a closed-form solution of the ridge regression.

On the other hand, the channel alignment methods <cit.> modify the feature maps so that the novel classes are well distinguished.
Specifically, FEAT manipulates the feature maps of support sets to increase the distance among classes of the support set by exploiting the transformer <cit.>.
DMF <cit.> aligns each feature map of the query instances by the dynamic meta-filter which is produced in consideration of the support and query pair.
And, RENet <cit.> transforms feature maps of the support and query pair with self- and cross-correlation which capture the structural patterns of each image and encode semantically relevant contents, respectively.

TDM also deals with feature alignment, especially channel alignment.
However, unlike existing methods that typically consider a pairwise relationship between the support set for each category and the query instance, TDM utilizes the entire task to adapt the feature maps.

§.§ Attention Modules

In various downstream tasks, attention modules yield remarkable performance gain <cit.>.
The existing attention methods can be divided into spatial attentions <cit.> and channel attentions <cit.>.

Specifically, to resolve poor scaling properties of CNN, SA <cit.> proposes a spatial attention module to capture long-range dependencies in an image by representing each grid of the feature map with other regions and itself based on similarity.
Based on the success of SA, ViT <cit.> proposes the network architecture, only comprised of fully-connected layers and multi-head attention, and shows a powerful performance of spatial attention.

On the other hand, SENet <cit.> proposes a channel attention module that produces channel weights highlighting more informative channels from spatially averaged features via a simple fully-connected block.
CBAM <cit.> is another notable channel attention module and it used not only averaged feature but also max-pooled features.

Among the two attention approaches, TDM and IAM belong to channel attention methods.
However, unlike the existing attention methods that only consider the information of an instance to produce a result of attention, TDM computes a channel weight vector for each class and query instance with the entire task.
It is to estimate the category of the query instance by focusing on query-relevant features among the distinct characteristics of each class.
Furthermore, IAM induces the feature map of each instance to consist of object-relevant features for minimizing the impact of background in TDM.
Therefore, we claim that our modules are specialized in the few-shot classification.

§ PRELIMINARY

§.§ Problem Formulation

As a standard formulation of the few-shot classification problem,

we are given two datasets: meta-train set $D_{base}=\left\{\left(x_i,y_i\right),y_i\in{C_{base}}\right\}$ for training a model and meta-test set $D_{novel}=\left\{\left(x_i,y_i\right),y_i\in{C_{novel}}\right\}$ for evaluating a learned model.
$C_{base}$ and $C_{novel}$ indicate base classes and novel classes, respectively, where they do not overlap (i.e., ${C_{base}}\cap {C_{novel}}=\phi$).
Generally, training and testing of few-shot classification are composed of episodes.
Each episode consists of randomly sampled $N$ classes and each class is composed of $K$ labeled images and $U$ unlabeled images, i.e., $N$-way $K$-shot episode.
The labeled images are called the support set $S=\left\{\left(x_j, y_j\right)\right\}^{{N}\times{K}}_{j=1}$, and the unlabeled images are named the query set $Q=\left\{\left(x_j, y_j\right)\right\}_{j=1}^{{N}\times{U}}$, while two sets are disjoint (i.e., ${S}\cap{Q}=\phi$).
The support and query sets are utilized for learning and validation, respectively.
Commonly, the category of the query instance is predicted by utilizing feature maps of the support and query instances.
If we define $x^S_{i,j}$ as $j$-th instance of $i$-th class in the support set and $x^Q$ as the query instance, their corresponding feature maps $F^S_{i,j}$ and $F^Q$ are expressed as follows:

\begin{equation}
\begin{split}
\label{feature_maps}
\\
\end{split}
\end{equation}

where $g_\theta$ is the feature extractor parameterized by $\theta$.
The shape of each feature map is $\mathbb{R}^{{C}\times{H}\times{W}}$ where $C,H,$ and $W$ denote the number of channels, height, and width, respectively.

§.§ Motivation

In metric-based few-shot learning <cit.>, the classification is generally performed based on the distances.
Suppose that such distances are defined for $C$-dimensional vectors $s \in \mathbb{R}^C$ computed by channel-wise spatial average of the feature map $F$ as follows:

\begin{equation}
\begin{split}
\label{scalar}
s_{c}&=\frac{1}{H \times W}\sum^{H}_{h=1}\sum^{W}_{w=1}f_{c,h,w},
\end{split}
\end{equation}

where $f_{c,h,w}$ the value spatially located at $(h,w)$ in $c$-th channel of $F$ which is the feature map of an instance.
Then, we compute the average value for $c$-th dimension of support- and query-set which belong to the $i$-th class, as follows:

\begin{equation}
\begin{split}
\label{bar_scalar}
\bar{s}_{i,c}&=\frac{1}{J_i}\sum^{J_i}_{j=1}s_{i,j,c},
\end{split}
\end{equation}

where $J_i$ is the number of instances that belongs to the $i$-th class and $s_{i,j,c}$ denotes $s_c$ of $j$-th instance in $i$-th class.
Based on those, we define the channel-wise variance $V_{i,c}$ of $c$-th channel within $i$-th class as follows:

\begin{equation}
\begin{split}
\label{variance_within}
V_{i,c}=\frac{1}{J_i}\sum^{J_i}_{j=1} {\left( s_{i,j,c} - \bar{s}_{i,c}\right)^2}.
\end{split}
\end{equation}

Since the values at the same dimension are compared for distance computation among vectors, the dimensions with small variance less contribute to the distance.
Thus, metric-based few-shot classification methods try to reduce $V_{i,c}$ in the training phase.
However, even though the categories used in the training phase form low $V_{i,c}$, the same is not guaranteed for the novel classes in the validation phase <cit.>.

Is this about an episode or whole dataset?
In metric-based few-shot learning <cit.>, the classification is generally performed based on the distances.
Suppose that such distances are defined for $C$-dimensional vectors $s_{i,j} \in \mathbb{R}^C$ computed by channel-wise spatial average of the feature map $F_{i,j}$ as follows:

\begin{equation}
\begin{split}
\label{scalar}
s_{i,j,c}&=\frac{1}{H \times W}\sum^{H}_{h=1}\sum^{W}_{w=1}f_{i,j,c,h,w},
\end{split}
\end{equation}

where $f_{i,j,c,h,w}$ the value spatially located at $(h,w)$ in $c$-th channel of $F_{i,j}$ which is the feature map of $j$-th instance in $i$-th class.
Simply, $s_{i,j,c}$ is the average activation of $c$-th channel of $F_{i,j}$.
We then compute the average value for $c$-th dimension of $s_{i,j,c}$ within $i$-th class as follows:

\begin{equation}
\begin{split}
\label{bar_scalar}
\bar{s}_{i,c}&=\frac{1}{J_i}\sum^{J_i}_{j=1}s_{i,j,c},
\end{split}
\end{equation}

where $J_i$ is the number of instances that belongs to the $i$-th class. Based on those, we define the channel-wise variance $V_{i,c}$ of $c$-th channel within $i$-th class as follows:

\begin{equation}
\begin{split}
\label{variance_within}
V_{i,c}=\frac{1}{J_i}\sum^{J_i}_{j=1} {\left( s_{i,j,c} - \bar{s}_{i,c}\right)^2}.
\end{split}
\end{equation}

Since the values at the same dimension are involved in the distance computation among vectors, the dimensions with small variances less contribute to the distance. Thus, in metric-based few-shot classification method, small channel-wise variance within a class is preferred to have small distances among the same class instances.
So it is tried to reduce $V_{i,c}$ in the training phase.
However, even though the categories used in the training phase form low $V_{i,c}$, the same is not guaranteed for the novel classes in the validation phase <cit.>.

Therefore, as described in <ref> (a), it is not a proper solution that utilizes all channels equally.
This is because there are channels with high $V_{i,c}$ in novel classes since the feature extractor is trained with base classes, as aforementioned.
Thus, as described in <ref> (b), it is effective to utilize only channels with low $V_{i,c}$ by eliminating channels with high $V_{i,c}$ in novel classes, just as base classes which consist of channels with low $V_{i,c}$.

However, in fine-grained datasets, categories belong to the same super-class and share common features, then, even if channels possess low $V_{i,c}$, they may not contain distinct information from other classes.
Accordingly, we should consider whether the information of each channel is discriminative with other classes as described in <ref> (c).
To achieve it, we introduce our two novel channel attention modules in <ref>.
Further, as described in <ref>, we extend one of them to capture the instance-descriptive information.

Within an episode,

\begin{equation}
\begin{split}
\label{scalar}
s_{i,j,c}&=\frac{1}{H \times W}\sum^{H}_{h=1}\sum^{W}_{w=1}f_{i,j,c,h,w},
\end{split}
\end{equation}

\begin{equation}
\begin{split}
\label{bar_scalar}
\bar{s}_{i,j,c}&=\frac{1}{J}\sum^{J}_{j=1}s_{i,j,c},
\end{split}
\end{equation}

where $f_{i,j,c,h,w}$ is the value located spatially $h,w$ in $c$-th channel of $j$-th instance's feature map which is affiliated $i$-th class and $J$ denotes the number of instances belonging to the $i$-th category.
Then, the channel-wise variance of $c$-th channel within $i$-th class $V_{i,c}$ is defined as follows:

\begin{equation}
\begin{split}
\label{variance_within}
V_{i,c}=\frac{1}{J}\sum^{J}_{j=1} {\left( s_{i,j,c} - \bar{s}_{i,j,c}\right)^2}.
\end{split}
\end{equation}

In metric-based methods <cit.>, since the category of the query instance is predicted by the distance to the support instances, they try to reduce $V_{i,c}$ in the training phase.
However, even though the categories used in the training phase form low $V_{i,c}$, the novel categories used in the validation phase may not <cit.>.
Therefore, as described in <ref> (a), utilizing all channels equally is not a proper solution since there are channels with high $V_{i,c}$.
However, even if we eliminate some channels with high $V_{i,c}$, it only shows a slight improvement as described in <ref> (b).
This is because categories in fine-grained datasets belong to the same super-class and share common features, then, even if channels possess low $V_{i,c}$, they may not contain distinct information from other classes.

Accordingly, we should consider whether the information of each channel is discriminative with other classes as described in <ref> (c).
To achieve it, we introduce our two novel channel attention modules in <ref>.
Further, as described in <ref>, we extend one of them to capture the instance-descriptive information.

§ TASK DISCREPANCY MAXIMIZATION :
ALIGNMENT METHOD FOR FINE-GRAINED DATASETS

The overall architecture of TDM is illustrated in <ref>.
Given an episode that consists of the support and query instances, feature maps are first computed by the feature extractor.
Since the feature extractor is trained to find discriminative features for distinguishing base classes <cit.>, the feature maps are not optimal for each episode.
To produce optimized feature maps for each episode, TDM transforms the feature maps by computing task-specific weight vectors representing channel-wise discriminative power for a specific task.
As a result, TDM aims to refine the original feature maps into task-adaptive feature maps focusing on the discriminatory details.
In this section, we describe the components of TDM and their purposes.
First, we define two channel-wise representativeness scores based on the estimated salient regions in  <ref>.
Then, with these scores, we introduce two sub-modules of the TDM: SAM and QAM in <ref> and <ref>, respectively.
Finally, TDM is described in <ref> with the discussion in <ref>.

§.§ Channel-wise Representativeness Scores

Given feature maps $F^S_{i,j}$ of the support set, for each pair of $i$-th class and $c$-th channel, we define two channel-wise representativeness scores; intra score $R^{\text{intra}}_{i,c}$, and inter score $R^{\text{inter}}_{i,c}$.
Since there may be multiple instances for each category, we utilize a prototype<cit.> as the representative of each class.
The prototype $F^P_i$ for $i$-th class is computed as follows:

\begin{equation}
\begin{split}
\label{prototype}
\end{split}
\end{equation}

where $K$ and $F^S_{i,j}$ are the number of instances for each class and feature map of $j$-th instance for $i$-th class, respectively.
Then, for each prototype, we define a mean spatial feature to represent salient object regions.
When the $c$-th channel of the prototype $F^P_i$ for $i$-th class is indicated as ${f^P_{i,c}}\in{\mathbb{R}^{{H}\times{W}}}$, the corresponding mean spatial feature $M^P_i$ is computed as follows:

\begin{equation}
\begin{split}
\label{mean_spatial_feature}
\end{split}
\end{equation}

Based on this, we further define the channel-wise representativeness score within a class,  $R^{\text{intra}}_{i,c}$, for $c$-th channel of $i$-th class as follows:

\begin{equation}
\begin{split}
\label{in_class_distance}
R^{\text{intra}}_{i,c} = \frac{1}{H\times W}\parallel f^P_{i,c} - M^P_i \parallel^2.
\end{split}
\end{equation}

It represents how well the highly activated regions on the $c$-th channel match the class-wise salient areas represented by the mean spatial feature.
On the other hand, the channel-wise representativeness score across classes, $R^{\text{inter}}_{i,c}$, for $c$-th channel of $i$-th class is computed as follows:

\begin{equation}
\begin{split}
\label{out_class_distance}
R^{\text{inter}}_{i,c} = \frac{1}{H \times W}\min_{ 1 \leq j \leq N, j \neq{i} } \parallel f^P_{i,c} - M^P_j \parallel^2.
\end{split}
\end{equation}

Since the score is large when $f^P_{i,c}$ is different from $M^P_j$, it denotes how much the channel contains the discriminative information of each category.

Intuitively, a channel is more distinct when it has a small $R^{\text{intra}}_{i,c}$ and a large  $R^{\text{inter}}_{i,c}$ as illustrated in <ref>.

§.§ Support Attention Module (SAM)

Support attention module (SAM) receives the class prototypes as input and computes two channel-wise representativeness scores for each class based on <ref> and <ref>.
Then, to reflect the importance of each channel by considering the distribution of those scores, we transform two score vectors, $R^\text{intra}_i$ and $R^\text{inter}_i$, into two weight vectors, $w^{\text{intra}}_i$ and $w^{\text{inter}}_i$ for $i$-th class, as follows:

\begin{equation}
\begin{split}
\label{in_out_weight_vector}
\\
\end{split}
\end{equation}

where $b^{\text{intra}}$ and $b^{\text{inter}}$ denote fully-connected blocks for producing two weight vectors.
The architecture of them are described in <ref>

The support weight vector $w^{S}_i$ for $i$-th class is obtained by linear interpolation of the corresponding two weight vectors, $w^{\text{intra}}_i$ and $w^{\text{inter}}_i$, as follows:

\begin{equation}
\begin{split}
\label{support_weight_vector}
w^{S}_i={\alpha}{w^{\text{intra}}_i}+{\left(1-\alpha\right)}{w^{\text{inter}}_i},\: {\alpha}\in{\left[0,1\right]},
\end{split}
\end{equation}

where $\alpha$ is a balancing hyperparameter for the support weight vector.
The vector for $i$-th class highlights distinct channels of $i$-th class while suppressing channels that include common information throughout classes in the episode.
Therefore, when the support weight vector $w^{S}_i$ for $i$-th class is multiplied to the feature maps, instances of $i$-th class are gathered, while others become separated from the $i$-th class.

§.§ Query Attention Module (QAM)

Although the support weight vector $w^S_i$ for $i$-th class is develop to emphasize the distinct channels for $i$-th class, the query instance which belongs to the $i$-th category may not possess those features. Specifically, the emphasized discriminative channels of the support set become useless or they even disturb the distance-based class prediction for the query, if the query does not have such features corresponding to the highlighted channels. This problem motivates us to propose query attention module (QAM) to focus on the channels which are class-discriminative and taken by the query at the same time.
Since we do not have any label information for the query instance unlike the support set, QAM only utilizes the relationship among channels within the query instance.
Specifically, QAM computes the channel-wise representativeness score within the query instance, $R^\text{intra}_{c}$, for $c$-th channel, as follows:

\begin{equation}
\begin{split}
\label{qwer}
R^{\text{intra}}_{c} = \frac{1}{H\times W}\parallel f^{Q}_{c} - M^Q \parallel^2,
\end{split}
\end{equation}

where $f^Q_{u}$ denotes $c$-th channel of the feature map $F^Q$, and $M^Q$ is the mean spatial feature which is defined by the channel-wise average of $F^Q$.
Then, the query weight vector $w^Q$ is produced by passing the intra score vector $R^\text{intra}$ to the fully-connected block $b^Q$, as follows:

\begin{equation}
\begin{split}
\label{query_weight_vector}
\end{split}
\end{equation}

where the architecture of $b^Q$ is described in <ref>.
The query weight vector emphasizes object-relevant channels of the query instance while restraining other channels.
Therefore, the query weight vector guides the model to focus on information related to the object of the query.

§.§ Task Discrepancy Maximization (TDM)

Since two weight vectors $w^{S}_i$ and $w^{Q}$ respectively produced by the SAM and QAM are complementary in their purposes, we utilize them to define a task weight vector.
Specifically, the task weight vector $w^T_i$ for $i$-th class is defined by linear interpolation of the corresponding support and query weight vectors, $w^S_i$ and $w^Q$, as follows:

\begin{equation}
\begin{split}
\label{task_weight_vector}
w^{T}_i={\beta} {w^{S}_i}+{\left(1-\beta\right)} {w^{Q}},\quad {\beta}\in{\left[0,1\right]},
\end{split}
\end{equation}

where $\beta$ is a balancing hyperparameter.

Based on the above task weight vectors, the feature maps of all the support and query instances are transformed into task-adaptive feature maps.
Specifically, the feature maps $F^S_{i,j} \in \mathbb{R}^{{C}\times{H}\times{W}}$ of the support instances for $i$-th class are converted to the task-adaptive feature map $A^S_{i,j}$ by its corresponding task weight vector $w^T_i \in \mathbb{R}^{{C}}$, as follows:

\begin{equation}
A^S_{i,j} = \left[{w^T_{i,1}}{f^S_{i,j,1}},{w^T_{i,2}}{f^S_{i,j,2}}, ... , {w^T_{i,C}}{f^S_{i,j,C}} \right],
\end{equation}

where $w^T_{i,c}$ and $f^S_{i,j,c}$ are a scalar value at $c$-th dimension of $w^T_i$ and the $c$-th channel of $F^S_{i,j}$, respectively.
On the other hand, since the label of the query is not available, it is not possible to specify which task weight vector should be multiplied by the feature map of the query.
Therefore, we apply all the task weight vectors $w^T$ to the feature map $F^Q\in \mathbb{R}^{{C}\times{H}\times{W}}$ of the query to produce task-adaptive feature maps $A^Q$ about all categories, as follows:

\begin{equation}
A^Q_{i} = \left[{w^T_{i,1}}{f^Q_{1}},{w^T_{i,2}}{f^Q_{2}}, ... , {w^T_{i,C}}{f^Q_{C}} \right],
\end{equation}

where $i$ indicates class index, and $f^Q_c$ is the $c$-th channel of $F^Q$.
When we are testing the query for $i$-th class, the corresponding adaptive feature map $A^Q_i$ of the query is utilized.

For instance, when TDM is applied to the ProtoNet<cit.>, the probability that the query instance belongs to $i$-th class is computed by the following criteria:

\begin{equation}
\begin{split}
\label{Inferring_class}
\end{split}
\end{equation}

where $d$ is the distance metric, and $A^P_i$ is the prototype computed by the average of the adaptive feature maps of support instances for $i$-th class.

§.§ Discussion

For a general dataset, it is widely known that the feature map, which contains various information about the object, is beneficial <cit.>.
On the other hand, in a fine-grained dataset, it is advantageous to focus only on the discriminative regions since the categories share a common overall apperance <cit.>.
Moreover, in fine-grained few-shot classification, the distinct parts of each class may be variable depending on the contents of the episode, unlike general fine-grained classification where the discriminative regions of each category is almost constant.

Thus, dynamically discovering the distinct parts of each class in the episode is the key point in the fine-grained few-shot classification. As described in <ref>, the baseline model estimates the category of the query by treating all characteristics equally regardless of the composition of each episode. In contrast, TDM predicts the class of the query by concentrating on discriminative parts which are discovered with consideration for the episode.
This is why TDM is a tailored module for the fine-grained few-shot classification.

§ INSTANCE ATTENTION MODULE:
GENERALIZED QUERY ATTENTION MODULE

Since the TDM is developed for high-level feature maps such as the last convolution layer, its performance can be dependent on the quality of given features produced by the feature extractor. In this section, we further introduce the instance attention module (IAM) by extending the QAM to reflect our motivation even for the intermediate feature representations. Specifically, the IAM is designed to highlight the object-relevant channels for each instance regardless of the support or query sets.

Although TDM is a well-tailored module for fine-grained few-shot classification and shows superior performance as described in <ref>, it can be affected by the quality of the feature map extracted by the feature extractor.
In this section, thus, we further extend QAM to produce the feature map with more object-relevant information for each instance.
Since SAM highly assumes the existence of the support set, it is not proper to handle a unit of instance.
Similarly, QAM is also devised for the query set, but it computes every weight vector only with a single query instance.
Therefore, QAM can be extended in an instance-wise manner to enhance the quality of feature representation.

The architecture of the fully-connected blocks, $b^{\text{intra}}$ and  $b^{\text{inter}}$ in <ref>, and $\overline{b}$ in <ref>.
The batch size B and the size of input C are different across SAM, QAM, and IAM.
For the SAM, B is the number of categories comprising an episode, and C is the number of channels of the feature map $F$. On the other hand, for the QAM, C is the same with the SAM, while B is the number of queries. In the IAM, B and C are the numbers of images and channels of intermediate feature map $\overline{F}$, respectively.

3|c|Fully Connected Block

Layer     Input Shape     Output Shape

Fully Connected Layer     B $\times$ C     B $\times$ 2C

Batch Normalization     B $\times$ 2C     B $\times$ 2C

ReLU     B $\times$ 2C     B $\times$ 2C

Fully Connected Layer     B $\times$ 2C     B $\times$ C

1 + Tanh     B $\times$ C     B $\times$ C

The overall architecture of the IAM is illustrated in <ref>.

IAM operates in intermediate layers of the feature extractor for each instance separately and aims to emphasize the channels encoding object-relevant features. Specifically, IAM receives an intermediate feature map $\overline{F} \in \mathbb{R}^{\overline{C}{\times}\overline{H}{\times}\overline{W}}$ as input, where $\overline{C}$, $\overline{H}$, and $\overline{W}$ denote the number of channels, height, and width of the feature map, respectively.

Then, the channel-wise representativeness score is then defined within the feature map, $\overline{R}^{\text{intra}}_{c}$, for $c$-th channel, as follows:

\begin{equation}
\label{eq:iam_intra}
\overline{R}^{\text{intra}}_{c} = \frac{1}{\overline{H} \times \overline{W}} \parallel \overline{f}_{c} - \overline{M} \parallel^2,
\end{equation}

where $\overline{f}_c$ and $\overline{M}$ are the $c$-th channel of $\overline{F}$ and the mean spatial feature computed by the channel-wise average of $\overline{F}$, respectively.
Based on the score vector $\overline{R}^\text{intra} \in \mathbb{R}^{\overline{C}}$, IAM infers a channel weight vector

$\overline{w} \in \mathbb{R}^{\overline{C}} $
in a similar way to QAM described in <ref> as follows:

\begin{equation}
\label{eq_iam_w_b}
\overline{w}=\overline{b} \left(\overline{R}^{\text{intra}}\right),
\end{equation}

where $\overline{b}$ is a fully-connected block as described in <ref>.
Subsequently, an intermediate feature map $\overline{F}$ is then transformed into an attentive feature map $\overline{A}$ based on the channel weight vector $\overline{w}$ as follows:

\begin{equation}
\label{IAMapplication}
\overline{A} =  \left[{\overline{w}_1}{\overline{f}_1},{\overline{w}_2}{\overline{f}_2}, ... , {\overline{w}_{\overline{C}}}{\overline{f}_{\overline{C}}} \right],
\end{equation}

where $\overline{w}_{i}$ is a scalar at $i$-th dimension of $\overline{w}$, and $\overline{f}_i$ represents $i$-th channel of $\overline{F}$. Finally, the transformed feature map $\overline{A}$ is fed to the next layer in the feature extractor.

In IAM, the computation and application of the channel weight vectors are only performed within each instance; there is no consideration of the support and query sets in the episodic training. Thus, the IAM is applied instance-wisely to all the support and query instances to improve the quality of the feature map. Furthermore, the operations involved in the IAM is conducted only within an instance, the additional computational and memory overhead is small, thereby allowing its usage in the intermediate blocks of the backbone.

The overall architecture of Instance Attention Module (IAM), the extended version of QAM, is illustrated in <ref>.
IAM operates in the intermediate layer of the feature extractor with a unit of instance and aims that the feature map of each instance consists of more object-relevant information and less background-relevant information.
To accomplish the above goal, IAM induces the feature extractor to produce the feature map by concentrating on channels encoding object-relevant information of each instance.
Specifically, IAM receives an intermediate feature map $\overline{F} \in \mathbf{R}^{\overline{C}{\times}\overline{H}{\times}\overline{W}}$ as input, where $\overline{C}$, $\overline{H}$, and $\overline{W}$ denote the number of channels, height, and width of the feature map, respectively.

Then, IAM first defines the channel-wise representativeness score within the feature map, $\overline{R}^{\text{intra}}_{c}$, for $c$-th channel, as follows:

\begin{equation}
\overline{R}^{\text{intra}}_{c} = \frac{1}{\overline{H} \times \overline{W}} \parallel \overline{f}_{c} - \overline{M} \parallel^2,
\end{equation}

where $\overline{f}_c$ and $\overline{M}$ are the $c$-th channel of $\overline{F}$ and mean spatial feature which is computed by the channel-wise average of $\overline{F}$, individually.
Based on the score vector $\overline{R}^\text{intra}$, IAM infers a channel weight vector $\overline{w}^\text{intra}$ in a similar way to QAM described in <ref> as follows:

\begin{equation}
\label{eq_iam_w_b}
\overline{w}=\overline{b} \left(\overline{R}^{\text{intra}}\right),
\end{equation}

where $\overline{b}$ is the fully-connected block as described in <ref>.
Consequently, each intermediate feature map $\overline{F}$ is rescaled by its importance which is represented by $\overline{w}$ as follows:

\begin{equation}
\label{IAMapplication}
\overline{A} =  \left[{\overline{w}_1}{\overline{f}_1},{\overline{w}_2}{\overline{f}_2}, ... , {\overline{w}_{\overline{C}}}{\overline{f}_{\overline{C}}} \right],
\end{equation}

where $\overline{w}_{i}$ is a element at $i$-th dimension of $\overline{w}$, and $\overline{f}_i$ denotes by $i$-th channel of $\overline{F}$.

Simply put, the main difference comes from Eq. <ref> as the computation and application of the weight vector are only within each instance; there is no consideration of the support and query set in episodic training.

In other words, IAM is applied to all support and query instances to improve the quality of the feature map of each instance.
Furthermore, as the operation of IAM is conducted only within an instance, not much additional memory is required, thereby allowing its usage in the intermediate blocks of the backbone.

§ EXPERIMENTS

In this section, we evaluate the proposed TDM on fine-grained classification benchmarks, and further verify the generalization capability of IAM on the both fine- and coarse-grained benchmarks.
Throughout the tables in this section, we use $\dagger$ to denote a reproduced version of the baselines.

§.§ Implementation Details

To verify the effectiveness and adaptability of TDM and IAM in fine-grained classification problem, we apply it to various existing methods including ProtoNet <cit.>, DSN<cit.>, CTX<cit.>, and FRN<cit.>.
On the other hand, for coarse-grained classification problem, we attach IAM to the ProtoNet, FRN, and DeepBDC <cit.>.
For a fair comparison, we reproduce each baseline model with the same hyperparameter described in FRN and DeepBDC.
And, the same training and evaluation scheme is utilized whether TDM or IAM is applied or not.
While TDM generally exploits the prototype<cit.> defined in <ref> for computing the intra and inter scores, it instead utilizes a query-aligned prototype proposed in the CTX <cit.> when combining with the CTX.

We adopt model architectures commonly utilized in the recent few-shot classification literature <cit.>; we employ Conv-4 and ResNet-12.
While both backbone networks accept an image of size 84$\times$84, the size of feature maps is different according to the backbone network. Specifically,
ResNet-12 yields a feature map with dimensions of 640$\times$5$\times$5, while Conv-4 produces 64$\times$5$\times$5 shape..
For our proposed TDM and IAM, we additionally utilize fully-connected layer blocks where the size of blocks are proportional to the number of channels of the feature maps as described in <ref>.

We attach IAM to the first and second blocks.
The $\alpha, \beta$ in <ref>, <ref> are fixed to 0.5.

Training Details.
Following the baseline methods <cit.>, we use standard data augmentation techniques including random crop, horizontal flip, and color jitter.
To prevent overfitting, we add random noise between $-0.2$ and $0.2$ to each output of TDM and IAM. We also regulate the each output of our modules to be in a range of $[0,2]$.
The hyperparameter and training details are followed our baselines for a fair comparison regardless of use of TDM or IAM.

Evaluation Details.
For the 5-way $K$-shot experiments, we conduct the evaluation with 10,000 randomly sampled episodes which contain 16 queries per class.
We report average classification accuracy with 95% confidence intervals.
The 1-shot performances of DSN and CTX are measured by models trained by 5-shot episodes since it shows better performance like FRN.

The splits of datasets. While $C_{\text{all}}$ is the number of total classes, $C_{\text{train}}$, $C_{\text{val}}$, $C_{\text{test}}$ are the number of training, validation, and test classes, respectively.
The classes of these subsets are disjoint.

1lDataset     $C_{\text{all}}$     $C_{\text{train}}$     $C_{\text{val}}$     $C_{\text{test}}$

1lCUB-200-2011     200     100     50     50

1lAircraft     100     50     25     25

1lmeta-iNat     1135     908     -     227

1ltiered meta-iNat     1135     781     -     354

1lStanford-Cars     196     130     17     49

1lStanford-Dogs     120     70     20     30

1lOxford-Pets     37     20     7     10

1lmini-ImageNet     100     64     20     16

1ltiered-ImageNet     608     351     97     160

§.§ Datasets

We use seven benchmarks for fine-grained few-shot classification: CUB-200-2011, Aircraft, meta-iNat, tiered meta-iNat, Stanford-Cars, Stanford-Dogs, and Oxford-Pets.
For the evaluation in coarse-grained scenarios, mini-ImageNet and tiered-ImageNet are used.
The split information of each dataset is reported in <ref>.

CUB-200-2011<cit.> comprises 11,788 photos of 200 bird species.
This dataset can be utilized in two types: raw form<cit.> or preprocessed form by a human-annotated bounding box<cit.>.
In our work, experiments are conducted with both forms as did in <cit.>.

Aircraft<cit.> is a dataset with 10,000 images of 100 airplane classes.
The main challenge of this dataset arises from airline symbols. Specifically, although the aircraft models are different, their airline symbol can be the same.
It makes the recognition task more difficult.
Our protocols in splitting the train/test data and image preprocessing with the bounding box are following a way of our baseline model, FRN.

is a long-tailed dataset.
It contains 1,135 animal species, and the number of images for each category is non-uniform and ranging between 50 and 1000.
For train and test data split, we adopt the way introduced in <cit.> which initially proposed this benchmark for the few-shot classification.
However, unlike <cit.> where a 227-way evaluation scheme is employed, we adopt a standard 5-way evaluation scheme following our baseline model, FRN.

tiered meta-iNat<cit.> has the same images with meta-iNat.
However, the difference comes from how the train and test data are organized; unlike meta-iNat, the tiered version divides the split by super categories.
Therefore, a bigger domain gap exists between train and test classes.

Stanford Cars<cit.> consists of 16,185 images of 196 car classes.
We employ the same data split protocol with <cit.> that first introduced this dataset for the few-shot classification task.

Performance on CUB using bounding-box cropped images as input.
“$\ast$" denotes reproduced one in RENet.
Confidence intervals for our implemented model are all below 0.23.

1l2*Model     2cConv-4     2cResNet-12

1c    1-shot     5-shot     1-shot     5-shot

1lMatchNet <cit.>     67.73     79.00     71.87     85.08

1lFEAT$^{\ast}$ <cit.>     68.87     82.90     73.27     85.77

1lDeepEMD <cit.>     -     -     75.65     88.69

1lRENet <cit.>     -     -     79.49     91.11

1lProtoNet$^{\dagger}$ <cit.>     62.90     84.13     78.99     90.74

1l   + TDM     69.94     86.96     79.58     91.28

1l   + IAM     68.18     85.96     79.65     91.20

1l   + TDM + IAM     72.96     88.02     80.93     91.80

1lDSN$^{\dagger}$ <cit.>     72.09     85.03     80.51     90.23

1l   + TDM     73.38     86.07     81.33     90.65

1l   + IAM     75.10     86.66     82.03     90.67

1l   + TDM + IAM     74.75     86.89     82.85     91.47

1lCTX$^{\dagger}$ <cit.>     72.14     87.23     80.67     91.55

1l   + TDM     74.68     88.36     83.28     92.74

1l   + IAM     75.65     89.07     82.87     92.49

1l   + TDM + IAM     77.17     89.90     83.76     92.85

1lFRN$^{\dagger}$ <cit.>     73.24     88.33     83.16     92.42

1l   + TDM     74.39     88.89     83.36     92.80

1l   + IAM     76.29     89.66     83.63     92.59

1l   + TDM + IAM     75.49     89.72     84.17     93.30

Stanford Dogs<cit.> contains 20,580 images belonging to one of 120 breeds of dogs around the world.
Similar to Stanford Cars, it is also introduced by <cit.> for fine-grained few-shot classification.
Thus, we follow <cit.> in the way of splitting this dataset.

Oxford Pets<cit.> is another fine-grained image dataset that has 37 pet classes with approximately 200 images per category.
To the best of our knowledge, this dataset has never been used for the few-shot classification task before the previous conference version of this paper <cit.>. Thus, we randomly divide classes to define the train/test split as did in <cit.>.

mini-ImageNet <cit.> is one of the representative benchmarks for the few-shot classification.
It is a subset of ImageNet and comprises 100 classes where 600 different images exist per category.
Our dataset split is adopted from <cit.>.
Unlike the aforementioned datasets which are utilized to validate the effectiveness of TDM and IAM for fine-grained classification, we use mini-ImageNet to evaluate the generalization capability of IAM.

tiered-ImageNet <cit.> is also a subset of ImageNet, but it is the largest dataset for the the few-shot classification.
It contains 601 categories which is much greater than the number of classes of mini-ImageNet.
Moreover, unlike mini-ImageNet, this benchmark separates train, evaluation, and test classes by super categories, therefore, a large domain gap exists like tiered meta-iNat.
We utilize this benchmark to validate the IAM, since it is a coarse-grained classification dataset.

Performance on CUB using raw images as input.

1lModel     Backbone     1-shot     5-shot

1lBaseline <cit.>     ResNet-18     65.51$\pm$0.87     82.85$\pm$0.55

1lBaseline++ <cit.>     ResNet-18     67.02$\pm$0.90     83.58$\pm$0.54

1lMatchNet <cit.>     ResNet-18     73.42$\pm$0.89     84.45$\pm$0.58

1lMAML <cit.>     ResNet-18     68.42$\pm$1.07     83.47$\pm$0.62

1lRelatioNet <cit.>     ResNet-18     68.58$\pm$0.94     84.05$\pm$0.56

1lS2M2 <cit.>     ResNet-18     71.43$\pm$0.28     85.55$\pm$0.52

1lNeg-Cosine <cit.>     ResNet-18     72.66$\pm$0.85     89.40$\pm$0.43

1lAfrasiyabi et al. <cit.>     ResNet-18     74.22$\pm$1.09     88.65$\pm$0.55

1lProtoNet$^{\dagger}$ <cit.>     ResNet-12     78.58$\pm$0.22     89.83$\pm$0.12

1l   + TDM     ResNet-12     79.11$\pm$0.22     90.83$\pm$0.11

1l   + IAM     ResNet-12     78.28$\pm$0.22     90.72$\pm$0.12

1l   + TDM + IAM     ResNet-12     78.89$\pm$0.22     90.86$\pm$0.12

1lDSN$^{\dagger}$ <cit.>     ResNet-12     80.47$\pm$0.20     89.92$\pm$0.12

1l   + TDM     ResNet-12     80.58$\pm$0.20     89.95$\pm$0.12

1l   + IAM     ResNet-12     81.33$\pm$0.20     89.87$\pm$0.12

1l   + TDM + IAM     ResNet-12     81.96$\pm$0.20     90.54$\pm$0.12

1lCTX$^{\dagger}$ <cit.>     ResNet-12     80.95$\pm$0.21     91.54$\pm$0.11

1l   + TDM     ResNet-12     83.45$\pm$0.19     92.49$\pm$0.11

1l   + IAM     ResNet-12     81.97$\pm$0.20     92.04$\pm$0.11

1l   + TDM + IAM     ResNet-12     83.82$\pm$0.19     92.79$\pm$0.10

1lFRN$^{\dagger}$<cit.>     ResNet-12     83.54$\pm$0.19     92.96$\pm$0.10

1l   + TDM     ResNet-12     84.36$\pm$0.19     93.37$\pm$0.10

1l   + IAM     ResNet-12     84.50$\pm$0.19     93.21$\pm$0.10

1l   + TDM + IAM     ResNet-12     84.84$\pm$0.18     93.60$\pm$0.10

§.§ Fine-grained Few-Shot Classification

Performance on Aircraft. Confidence intervals for our implemented model are all below 0.25.

1l2*Model     2cConv-4     2cResNet-12

1c     1-shot     5-shot     1-shot     5-shot

1lProtoNet$^{\dagger}$<cit.>     47.37     68.96     67.28     83.21

1l   + TDM     50.55     71.12     69.12     84.77

1l   + IAM     49.67     68.57     69.10     84.04

1l   + TDM + IAM     52.88     72.81     69.80     85.41

1lDSN$^{\dagger}$<cit.>     52.22     68.75     70.23     83.05

1l   + TDM     53.77     69.56     71.57     83.65

1l   + IAM     54.62     68.87     72.01     83.36

1l   + TDM + IAM     54.64     70.34     73.83     85.11

1lCTX$^{\dagger}$<cit.>     51.58     68.12     65.53     79.31

1l   + TDM     55.15     70.45     69.42     83.25

1l   + IAM     54.70     70.61     70.93     82.38

1l   + TDM + IAM     57.04     72.46     71.40     84.12

1lFRN$^{\dagger}$<cit.>     53.12     70.84     69.58     82.98

1l   + TDM     54.21     71.37     70.89     84.54

1l   + IAM     54.98     72.12     71.23     83.66

1l   + TDM + IAM     56.08     72.62     72.36     85.05

CUB-200-2011 results.

<ref> and <ref> report the results of our baselines and their performances when our proposed modules, TDM and IAM, are combined.
With cropped images in <ref>, TDM and IAM consistently improve the performance of baseline models in all cases and achieve the state-of-the-art scores when they are applied together.
Despite a few settings when IAM is not much effective in <ref>, TDM and IAM together, still show superior performances.

Aircraft results.
As reported in <ref>, TDM and IAM show a consistent tendency by boosting performances of all the baselines regardless of the model size and the size of support sets, except for one case.
Although there is a slight performance decrease when IAM is used in the 5-shot scenario of ProtoNet with Conv-4 backbone, more outstanding performance is achieved when IAM and TDM are adopted together, compared to utilizing TDM only.

meta-iNat and tiered meta-iNat results.
These datasets are suitable for evaluation of model's generalization capability, since it is widely known that models trained on those datasets are vulnerable to overfitting due to the absence of validation set <cit.>.
Moreover, the tiered meta-iNat makes the task more difficult since super categories of train and test set do not overlap.
As reported in <ref>, TDM and IAM are robust to the overfitting and the large domain gap as they enhance the results in most configurations.
For a slight performance decrease of TDM combined with FRN in a 5-shot scenario on tiered meta-iNat, we believe that this is mainly due to the learnable parameter $\lambda$ in FRN.
In general, large $\lambda$ shows good performances when a domain gap exists.
Yet, we found that TDM restrains the $\lambda$ to be relatively small since TDM assists to focus on discriminative channels.
Therefore, except for the above case, we accomplish the state-of-the-art performances when TDM and IAM are utilized together.

Performance on meta-iNat and tiered meta-iNat with Conv-4 backbones.
Confidence intervals for our implemented model are all below 0.23.

1l2*Model     2cmeta-iNat     2ctiered meta-iNat

1l     1-shot     5-shot     1-shot     5-shot

1lProtoNet$^{\dagger}$<cit.>     55.37     76.30     34.41     57.60

1l   + TDM     61.82     79.95     38.30     61.18

1l   + IAM     59.12     79.83     37.94     63.47

1l   + TDM + IAM    65.10     81.93     41.87     64.32

1lDSN$^{\dagger}$<cit.>     60.06     76.15     40.83     58.34

1l   + TDM     61.87     78.07     41.00     58.66

1l   + IAM     63.41     77.76     44.05     61.45

1l   + TDM + IAM     62.99     78.84     43.39     61.69

1lCTX$^{\dagger}$<cit.>     60.80     78.57     42.24     60.54

1l   + TDM     63.26     80.75     43.90     62.29

1l   + IAM     63.80     80.97     45.87     64.92

1l   + TDM + IAM     64.96     81.89     47.40     66.12

1lFRN$^{\dagger}$<cit.>     61.98     80.04     43.95     63.45

1l   + TDM     63.97     81.60     44.05     62.91

1l   + IAM     65.11     82.43     47.33     67.48

1l   + TDM + IAM     65.95     83.30     46.45     66.55

Performances of IAM on mini-ImageNet and tiered-ImageNet.
FRN-EMD denotes the FRN implemented in the DeepEMD.

1lMatchNet <cit.>     ResNet-12     64.64$\pm$0.20     78.72$\pm$0.15     68.50$\pm$0.92     80.60$\pm$0.71

1lBaseline++ <cit.>     ResNet-12     60.56$\pm$0.45     77.40$\pm$0.34     -     -

1lCTM <cit.>     ResNet-18     64.14$\pm$0.82     80.51$\pm$0.13     68.41$\pm$0.39     84.28$\pm$1.73

1lTADAM <cit.>     ResNet-12     58.50$\pm$0.30     76.70$\pm$0.38     -     -

1lS2M2 <cit.>     ResNet-12     64.06$\pm$0.18     80.58$\pm$0.12     -     -

1lNeg-Cosine <cit.>     ResNet-12     63.85$\pm$0.81     81.57$\pm$0.56     -     -

1lAfrasiyabi et al. <cit.>     ResNet-12     59.88$\pm$0.67     80.35$\pm$0.73     69.29$\pm$0.56     85.97$\pm$0.49

1lFEAT <cit.>     ResNet-12     66.78$\pm$0.20     82.05$\pm$0.14     70.80$\pm$0.23     84.79$\pm$0.16

1lDeepEMD <cit.>     ResNet-12     65.91$\pm$0.82     82.41$\pm$0.56     71.16$\pm$0.87     86.03$\pm$0.58

1lProtoNet$^\dagger$ <cit.>     ResNet-12     61.72$\pm$0.20     78.75$\pm$0.14     -     -

1l + IAM     ResNet-12     62.25$\pm$0.20     79.44$\pm$0.15     -     -

1lFRN$^\dagger$ <cit.>     ResNet-12     66.69$\pm$0.19     82.89$\pm$0.13     71.13$\pm$0.22     86.13$\pm$0.15

1l + IAM     ResNet-12     66.96$\pm$0.19     83.19$\pm$0.13     71.85$\pm$0.22     86.55$\pm$0.15

1lFRN-EMD$^{{\dagger}}$ <cit.>     ResNet-12     -     -     72.15$\pm$0.22     86.49$\pm$0.15

1l + IAM     ResNet-12     -     -     72.84$\pm$0.22     87.04$\pm$0.14

1lMeta DeepBDC$^\dagger$ <cit.>     ResNet-12     65.74$\pm$0.20     83.23$\pm$0.13     -     -

1l + IAM     ResNet-12     66.21$\pm$0.20     83.78$\pm$0.13     -     -

1lSTL DeepBDC$^\dagger$ <cit.>     ResNet-12     67.62$\pm$0.20     84.65$\pm$0.13     -	    -

1l + IAM     ResNet-12     67.95$\pm$0.19     84.86$\pm$0.13     -     -

Cross-domain few-shot classification performance of a scenario where models are trained with mini-ImageNet and tested on CUB.

1lModel     Backbone     1-shot     5-shot

1lBaseline <cit.>     ResNet-18     -     51.34$\pm$0.72

1lBaseline++ <cit.>     ResNet-18     -     62.02$\pm$0.70

1lMAML <cit.>     ResNet-18     -     51.34$\pm$0.72

1lAfrasiyabi et al. <cit.>     ResNet-18     46.85$\pm$0.75     70.37$\pm$1.02

1lProtoNet$^{\dagger}$ <cit.>     ResNet-12     46.58$\pm$0.19     66.19$\pm$0.17

1l   + IAM     ResNet-12     47.67$\pm$0.19     68.70$\pm$0.17

1lFRN$^{\dagger}$<cit.>     ResNet-12     52.80$\pm$0.21     73.75$\pm$0.18

1l   + IAM     ResNet-12     54.94$\pm$0.22     75.76$\pm$0.18

1lMeta DeepBDC$^{\dagger}$<cit.>     ResNet-12     42.83$\pm$0.20     74.11$\pm$0.16

1l   + IAM     ResNet-12     45.80$\pm$0.21     77.71$\pm$0.15

1lSTL DeepBDC$^{\dagger}$<cit.>     ResNet-12     55.01$\pm$0.21     75.47$\pm$0.16

1l   + IAM     ResNet-12     55.92$\pm$0.20     76.43$\pm$0.16

Stanford Cars, Stanford Dogs, and Oxford Pets results.
Although these datasets have fine-grained classes, they were not utilized for evaluation of our baselines in the literature.
To further validate the effectiveness of TDM and IAM, we additionally conduct experiments on those datasets with Conv-4.
As shown in <ref>, TDM and IAM generally improve the performances except for a few cases when IAM is solely used.
In detail, TDM improves the accuracy scores by 4.44 and 3.27%p compared to the baselines at 1-shot and 5-shot scenarios, respectively.
On the other hand, IAM provides 1.69 and 1.46%p performance improvements, respectively.
Furthermore, as our modules are compatible with one another, we observe significant boosts when they are both adopted; there are 4.79 and 3.81%p boosts in accuracy scores in 1-shot and 5-shot cases, respectively.

Throughout the extensive experiments on seven benchmark datasets, we validated the merits of TDM and IAM in the fine-grained classification task.
To summarize the experimental results, TDM has shown its superiority regardless of the datasets and baseline methods. For IAM, although it encourages to highlight object-relevant channels within the feature map, sometimes it is not beneficial for the fine-grained classification since the objects in the fine-grained datasets could have excessive common features across classes.
However, when IAM and TDM are utilized together, this is no longer a problem since TDM highlights class-discriminative features among object-relevant ones identified by the IAM.
On the other side, IAM helps TDM to discover more discriminative features, since IAM provides more object-focused feature maps to the TDM. Therefore, we claim that TDM and IAM have complementary benefits.

Cross-domain few-shot classification performance of a scenario where models are trained with mini-ImageNet and tested on Aircraft. Unlike the results on <ref>, STL DeepBDC in this table does not perform distillation stages since skipping these phases shows better performances.

1lModel     Backbone     1-shot     5-shot

1lProtoNet$^{\dagger}$ <cit.>     ResNet-12     33.48$\pm$0.15     49.55$\pm$0.18

1l + IAM     ResNet-12     34.65$\pm$0.15     51.00$\pm$0.18

1lFRN$^{\dagger}$<cit.>     ResNet-12     38.71$\pm$0.16     62.10$\pm$0.18

1l + IAM     ResNet-12     39.77$\pm$0.17     63.61$\pm$0.18

1lMeta DeepBDC$^{\dagger}$<cit.>     ResNet-12     36.11$\pm$0.16     59.52$\pm$0.19

1l + IAM     ResNet-12     37.46$\pm$0.17     60.66$\pm$0.19

1lSTL DeepBDC$^{\dagger}$<cit.>     ResNet-12     38.18$\pm$0.17     57.61$\pm$0.19

1l + IAM     ResNet-12     38.82$\pm$0.17     58.64$\pm$0.19

§.§ Coarse-grained Few-Shot Classification

mini-ImageNet and tiered-ImageNet results.
As discussed in <ref>, TDM is could be not a proper module for the coarse-grained few-shot classification task since it restrains utilizing various features of the object. On the other hand, since IAM encourages the feature extractor to produce various object-relevant features for each instance, we think that IAM is also beneficial for the coarse-grained few-shot classification.
To validate the effectiveness of IAM in coarse-grained benchmarks, we perform experiments on mini-ImageNet and tiered-ImageNet, and the results are reported in <ref>.
As can be seen, IAM improves all the baselines regardless of the training scheme.
Besides its effectiveness, we also emphasize the high applicability of IAM, because these results are obtained without any extensive hyperparameter searching or optimizing processes.

§.§ Cross-domain Few-Shot Classification

mini-ImageNet $\rightarrow$ CUB-200-2011 results.
To evaluate the cross-domain generalization ability of the few-shot classification algorithms, we validate each model when its train and test datasets are different, following the protocol of <cit.>.
Since images of fine-grained category are typically collected by professionals in each field, we argue that this setting is deeply related to reducing the cost of labeling.
Specifically, we train each model with mini-ImageNet and validate them with CUB (raw form), as did in <cit.>.
As reported in <ref>, IAM consistently improves the performances of all the baselines and achieves the state-of-the-art without any adaptation process.

mini-ImageNet $\rightarrow$ Aircraft results.
Although there is a big domain gap between mini-ImageNet and CUB, the train categories of mini-ImageNet still include two bird species as different classes.
Therefore, each model trained with mini-ImageNet, could be already learned to distinguish bird species.
On the other hand, since there are no airplane images in the train set of mini-ImageNet, classifying airplane types is a more proper setting for evaluating the cross-domain generalization capability of models.
Specifically, we evaluate each model trained with mini-ImageNet on the test set of Airplane dataset. As reported in <ref>, our IAM shows its effectiveness even in categories that have never been seen in training stages.

§ ABLATION STUDY

In this section, we conduct ablation experiments.

Most experiments for the ablation studies are performed based on  ProtoNet<cit.> with the Conv-4 backbone using CUB_cropped and Aircraft datasets.

§.§ Varying N and K for N-way K-shot

In <ref>, we extensively validate the merits of our proposed modules in various scenarios. However, the number of categories of each episode for those experiments was fixed to 5 by following the protocol of existing work <cit.>.
In a real-world scenario, the number of classes can be varying depending on the circumstance. Therefore, to verify the effectiveness of TDM and IAM in the such scenario, we first evaluate our modules with varying number of classes $N$ comprising an episode.
As reported in <ref>, TDM and IAM provide consistent improvements compared to the baseline, except for one case of 5-shot in Aircraft benchmark.
Moreover, the relative performance improvements are in proportion to the number of categories. It clearly demonstrates that our modules are more effective in more difficult settings (i.e., more classes).

On the other hand, the number of labeled images $K$ of each category was in a range of [1,5] following the existing methods <cit.>.
Similar to the experiments with respect to the number of classes, we perform experiments with varying numbers of labeled images and the results are provided in  <ref>.
As reported, the benefits of our modules are especially highlighted at low-shots in terms of relative performance improvements. It validates that our modules are more suitable for the few-shot scenarios, the main task of this study, while showing their effectiveness in many-shot.

Ablation study on SAM, QAM, and IAM.

1-shot     5-shot     1-shot     5-shot

-     -     -     62.90     84.13     47.37     68.96

-     -     68.53     85.95     49.45     69.33

-         -     65.11     84.82     48.96     70.85

-     -         68.18     85.96     49.67     68.57

-     69.94     86.96     50.55     71.12

-         71.97     88.00     51.98     70.67

-             68.87     86.68     51.69     69.60

72.96     88.02     52.88     72.81

§.§ Ablation Study on SAM, QAM, and IAM

Since our method consists of three sub-modules, SAM, QAM, and IAM, we perform experiments with various combinations of these sub-modules to evaluate the contribution of each component and confirm their complementary benefits. As reported in <ref>, each sub-module consistently improves the classification accuracies across the datasets except for one case of IAM.
The large gains by SAM confirm that identifying and focusing on discriminative channels for each category are crucial for fine-grained few-shot classification. Furthermore, although the improvements by QAM is slightly lower compared to SAM, QAM is also shown to be effective for all tested configurations. It confirms the benefits of applying more importance to the support set features that are possessed by the query instance. On the other hand, the effect of IAM varies from time to time which may degrade the performance as in the 5-shot scenario on Aircraft.
This is because the object-relevant feature maps induced by IAM may hinder accurate predictions when classes share many characteristics. However, as can be observed in the sixth row, SAM is able to resolve the limitation of IAM by restraining common features and discovering discriminative features.
Most importantly, the the best performances are achieved when all three components are utilized together. These results validate the merit of each sub-module and their complementary benefits.

Effects of Pooling functions.

1c2*Method     1c2*Pooling     2cCUB_cropped     2cAircraft

1c     1c     1-shot     5-shot     1-shot     5-shot

1lProtoNet$^{\dagger}$ <cit.>     1c-     62.90     84.13     47.37     68.96

1l + TDM     1cAvg     69.94     86.96     50.55     71.12

1l + IAM     1cAvg     68.18     85.96     49.67     68.57

1l + TDM + IAM     1cAvg     72.96     88.02     52.88     72.81

1l + TDM     1cMax     67.23     86.73     50.16     71.32

1l + IAM     1cMax     67.87     86.41     49.16     69.55

1l + TDM + IAM     1cMax     69.75     87.66      51.18     72.68

§.§ Pooling Functions

Do we have to investigate pooling functions? I think that the max pooling does not fit to our motivation.

To construct salient regions that are used to compute channel-wise representativeness scores, the choice of pooling function may affect the performance.
As described in <ref>, max- and average-pooling functions reflect spatial information of the object in the image.
Thus, in <ref>, we explore which pooling function fits better with TDM and IAM.
As reported, we can notice that both methods achieve significant boosts as either pooling function is capable of maintaining the general descriptions of the object.
Despite that there is only an insignificant gap between the two, our final model constructs the salient regions with the average pooling function since the max-pooling has its limitation in vulnerability to the outliers.

To construct salient regions that are used to compute channel-wise representativeness scores, the choice of pooling function may affect the performance.
As described in <ref>, max- and average-pooling functions reflect spatial information of the object in the image.
Thus, in <ref>, we explore which pooling function fits better with TDM and IAM.
As reported, we can notice that both methods achieve significant boosts as either pooling function is capable of maintaining the general descriptions of the object.
Despite that there is only an insignificant gap between the two, our final model constructs the salient regions with the average pooling function since the max-pooling has its limitation in vulnerability to the outliers.

Metric Compatibility. ED and CD denote the Euclidean and Cosine distance are utilized as the metric, respectively.

1c2*Method     1c2*Metric     2cCUB_cropped     2cAircraft

1c     1c     1-shot     5-shot     1-shot     5-shot

1lProtoNet$^{\dagger}$ <cit.>     ED     62.90     84.13     47.37     68.96

1l + TDM     ED     69.94     86.96     50.55     71.12

1l + IAM     ED     68.18     85.96     49.67     68.57

1l + TDM + IAM     ED     72.96     88.02     52.88     72.81

1lProtoNet$^{\dagger}$ <cit.>     CD     68.69     82.89     48.36     63.45

1l + TDM     CD     70.47     84.34     49.21     66.26

1l + IAM     CD      68.42     84.39     49.65     63.12

1l + TDM + IAM     CD     70.75     84.68     51.77     68.51

Compatibility with the Cosine Distance.
L2-Norm denotes that the L2-normalization along the channel axis is applied to the feature map, the input of each module.

1c2*Method     1c2*L2-Norm     2cCUB_cropped     2cAircraft

1c         1-shot     5-shot     1-shot     5-shot

1lProtoNet$^{\dagger}$ <cit.>     -     68.69     82.89     48.36     63.45

1l + TDM     -     69.90     84.95     51.51     68.35

1l + IAM     -     69.72     84.28     50.69     66.13

1l + TDM + IAM     -     71.17     85.15     52.62     69.62

1l + TDM         70.47     84.34     49.21     66.26

1l + IAM         68.42     84.39     49.65     63.12

1l + TDM + IAM         70.75     84.68     51.77     68.51

§.§ Metric Compatibility

In this work, we followed our baselines <cit.> to employ the Euclidean distance as the metric to compute the probability that the query instance belongs to each class.
Yet, the Cosine distance is another decent metric that has been widely adopted in other works <cit.>.
Therefore, we also test the compatibility of TDM and IAM to Cosine distance in <ref>.
Except for a few cases where IAM is not effective, there is a general tendency that adopting IAM or TDM leads to large gains in performance.
Similarly, when we simultaneously utilize IAM and TDM, the improvements get bigger regardless of the metric.

§.§ Compatibility with the Cosine Distance

In this paper, we mostly evaluate our method by employing the Euclidean distance when computing the similarity among instances as did in our baselines <cit.>. Meanwhile, the cosine distance is another popular metric adopted in other techniques <cit.>. Therefore, we also validate the compatibility of our method with the cosine distance, and results are reported in <ref>. The consistent tendency that adopting IAM or TDM leads to significant performance gains confirms the compatibility of our method with the cosine distance metric.

In this work, we followed our baselines <cit.> to employ the Euclidean distance as the metric to compute the probability that describes the class information of the query instance.
Yet, the Cosine distance is another decent metric that has been widely adopted in other works <cit.>.
Therefore, we also test whether our method works well with the Cosine distance.

Since L2-normalization is applied when measuring the Cosine distance between two vectors, we have two choices for computing channel-wise representativeness scores.
The first is computing those scores in the same way as before.
And, the other is utilizing the Cosine distance for producing those scores.
If the Cosine distance is adopted, the intra and inter scores of <ref>, <ref>, <ref>, and <ref> are redefined as follows:

\begin{equation}
\begin{split}
R^{\text{intra}}_{i,c} = 1-\frac{{f^P_{i,c}}\cdot{M^P_i}}{\parallel{f^P_{i,c}\parallel_2}\cdot{\parallel{M^P_i}\parallel_2}},
\end{split}
\end{equation}

\begin{equation}
\begin{split}
R^{\text{inter}}_{i,c} = 1-\min_{ 1 \leq j \leq N, j \neq{i} } \frac{{f^P_{i,c}}\cdot{M^P_j}}{\parallel{f^P_{i,c}\parallel_2}\cdot{\parallel{M^P_j}\parallel_2}},
\end{split}
\end{equation}

\begin{equation}
\begin{split}
R^{\text{intra}}_{c} = 1-\frac{{f^{Q}_{c}}\cdot{M^Q}}{\parallel{f^{Q}_{c}\parallel_2}\cdot{\parallel{M^Q}\parallel_2}},
\end{split}
\end{equation}

\begin{equation}
\begin{split}
\overline{R}^{\text{intra}}_{c} = 1-\frac{{\overline{f}_{c}}\cdot{\overline{M}}}{\parallel{\overline{f}_{c}\parallel_2}\cdot{\parallel{\overline{M}}\parallel_2}},
\end{split}
\end{equation}

where $\left[a^1_1,a^1_2,\dots,a^1_N\right]\cdot\left[a^2_1,a^2_2,\dots,a^2_N\right]=a^1_1a^2_1+a^1_1a^2_2+\dots+a^1_Na^2_N$.
As reported in <ref>, we proceed with the test in both cases.
Regardless of the methods for computing the scores, TDM or IAM leads to large grains in performance.
Therefore, we argue that they also can be compatible with the Cosine distance.

In <ref>, we report the results of ProtoNet <cit.> with and without our modules when the distance metric is replaced with the Cosine distance.

As can be found, we point out the consistent tendency that adopting IAM or TDM leads to significant gains in performance.
Therefore, we argue that our modules are also compatible with the Cosine distance.

Comparison with general attention methods.
CBAM-S denotes utilizing only spatial attention from CBAM.

And, C and S of the Type indicate that the method belongs to spatial or channel attention, respectively.

The performances of the existing attention methods are measured with a version applied to ProtoNet.

1c2*Method     1c2*Type     2cCUB_cropped     2cAircraft

1-shot     5-shot     1-shot     5-shot

ProtoNet$^{\dagger}$<cit.>     -     62.90     84.13     47.37     68.96

+ SENet$^{\dagger}$<cit.>     C     69.62     85.90     48.58     67.84

+ CBAM-S$^{\dagger}$<cit.>     S     66.45     85.57     50.13     69.94

+ CBAM$^{\dagger}$<cit.>     C + S     69.21     85.37     48.10     70.03

+ SA$^{\dagger}$<cit.>     S     69.23     87.49     50.07     70.41

+ TDM + IAM     C     72.96     88.02     52.88     72.81

+ TDM + IAM + CBAM-S
C + S

+ TDM + IAM + SA
C + S

§.§ Comparison to Existing Attention Methods

To further verify the benefits of our method over existing attention modules in the fine-grained few-shot classification task, we compare our method with SENet <cit.>, CBAM <cit.>, and Self-attention <cit.>. As reported in <ref>, TDM+IAM outperforms the existing attention methods by large margins. Note that, the main difference between our method and the existing modules is that we explicitly measure and leverage the channel-wise importance based on their representativeness scores in our attention modules, while existing methods are relied on the learnable parameters less suitable for the few-shot and fine-grained scenarios. Consequently, these results confirm that our modules has clear benefits over the existing attention modules in the fine-grained few-shot classification task.

As elaborated throughout the paper, our work focuses on designing attention modules for fine-grained few-shot classification.
In this subsection, we verify our specialty by comparing our proposed modules with other attention methods <cit.>.
In <ref>, it is shown that TDM+IAM outperforms the existing attention methods, i.e., SENet, CBAM, and Self-Attention (SA), by large margins.
Consequently, we claim that our modules clearly benefit the fine-grained few-shot classification task.

Performance when IAM is applied to each block. IAM-# indicates that IAM is employed to #-th block.

1c2*Method     2cCUB_cropped     2cAircraft

1-shot     5-shot     1-shot     5-shot

ProtoNet<cit.>     62.90     84.13     47.37     68.96

+ IAM-1     66.54     85.36     49.04     70.44

+ IAM-2     66.28     86.03     49.11     69.45

+ IAM-3     66.86     84.52     50.24     66.10

+ IAM-4     69.13     84.85     49.55     67.06

+ TDM + IAM-1     71.28     87.68     52.08     72.46

+ TDM + IAM-2     71.17     87.73     52.60     72.47

+ TDM + IAM-3     72.22     87.96     52.42     69.42

+ TDM + IAM-4     70.14     85.77     50.29     68.58

§.§ Block Selection of IAM

<ref> compares performances when IAM is added to different stages of the backbone network.
In the 1-shot scenarios, IAM shows its effectiveness regardless of the level of the stage.
On the other hand, in the 5-shot scenarios, adopting IAM in the latter blocks does not improve the performances as much as utilizing IAM in the early stage.

Therefore, we apply IAM on the first two blocks of the backbone where IAM consistently enhances the results.

Correlation test between the channel-wise representativeness scores and channel weight vectors of each module.
The Coefficient is the Pearson Correlation Coefficient.
The values in this table are computed in ProtoNet 1-shot scenario with CUB_cropped dataset.

1c2*     3cInter Score     1cIntra Score

SAM     QAM     IAM     SAM

§.§ Channel-wise Representativeness Scores and Channel Weight Vetors

SAM, QAM, and IAM receive the channel-wise representativeness scores (intra and inter scores) and produce the channel weight vectors.
Therefore, we should verify whether the value of the vectors is actually related to the scores and is computed based on the scores.
As described in <ref>, we progress a correlation test between the value and the score.

Compatibility with the cosine distance.

2cCUB_cropped     2cAircraft

1c     1-shot     5-shot     1-shot     5-shot

1lProtoNet$^{\dagger}$ <cit.>     68.69     82.89     48.36     63.45

1l + TDM     69.90     84.95     51.51     68.35

1l + IAM     69.72     84.28     50.69     66.13

1l + TDM + IAM     71.17     85.15     52.62     69.62

Since representing the whole objects could be rather harmful for classifying fine-grained categories unlike the general classification task, TDM is developed for highlighting features discriminative for fine-grained details. Thus, the benefits of TDM could be limited in coarse-grained tasks.

1c2*TDM     1c2*Pooling     2cCUB_cropped     2cAircraft

1c     1c     1-shot     5-shot     1-shot     5-shot

1c-     1c-     62.90     84.13     47.37     68.96

1c     1cMax     67.23     86.73     50.16     71.32

1c     1cAvg     69.94     86.96     50.55     71.12

Effects of Pooling functions. Top row is the baseline result.

1c2*TDM     2cCUB_cropped     2cAircraft

1c     1-shot     5-shot     1-shot     5-shot

1c-     68.69     82.89     48.36     63.45

1c     70.47     84.34     49.21     66.26

Effectiveness of TDM with the cosine distance

§ CONCLUSION

In this paper, we first introduced channel attention modules tailored for the fine-grained few-shot image classification, Task Discrepancy Maximization (TDM) with two submodules, Support Attention Module (SAM) and Query Attention Module (QAM). The core principle of the SAM is to emphasize feature map channels encoding class-discriminative information, while one of the QAM is to concentrate object-relevant channels for the query image. These channel attention modules enable to produce task-adaptive feature maps more focusing on the discriminative details to distinguish among fine-grained categories. To further improve the representation capability for both fine- and coarse-grained few-shot classification, we extended the QAM to present the Instance Attention Module (IAM). Specifically, the IAM operates in the intermediate layers to highlight object-relevant channels for each instance regardless of support or query image unlike the QAM which works for high-level feature maps of the query instance. We extensively evaluated the proposed modules on several fine- and coarse-grained image classification benchmarks to validate their unique merits in terms of effectiveness and applicability to the prior few-shot classification methods.

In this paper, we first introduced Task Discrepancy Maximization (TDM), a tailored module for fine-grained few-shot classification.
To distinguish similar classes, TDM computes channel weight vectors that emphasize features of fine discriminative detail with two submodules: Support Attention Module (SAM) and Query Attention Module (QAM).
Then, to further improve the representation quality for both fine- and coarse-grained few shot task, we extended QAM to present IAM.
Unlike QAM, IAM operates in the intermediate layers by producing and highlighting the object features with an instance-wise channel weight vector.
Our extensive experiments on several fine- and coarse-grained benchmarks validated the merits of our proposed TDM and IAM in terms of their effectiveness and high applicability with the prior few-shot classification methods.

Comparison with existing attention methods.

1c2*Method     2cCUB_cropped     2cAircraft

1-shot     5-shot     1-shot     5-shot

ProtoNet$^{\dagger}$<cit.>     62.90     84.13     47.37     68.96

+ SENet$^{\dagger}$<cit.>     69.62     85.90     48.58     67.84

+ CBAM$^{\dagger}$<cit.>     69.21     85.37     48.10     70.03

+ SA$^{\dagger}$<cit.>     69.23     87.49     50.07     70.41

+ TDM + IAM     72.96     88.02     52.88     72.81

§ ACKNOWLEDGMENTS

§ ACKNOWLEDGMENT

This work was supported in part by MSIT&KNPA/KIPoT (Police Lab 2.0, No. 210121M06) and MSIT/IITP (No. 2019-0-00421, 2020-0-01821).

SuBeen Lee
receives the B.S. degree in computer science from the Jeonbuk National University (JBNU), South Korea, in 2020, and the M.S. degree in artificial intelligence from Sungkyunkwan University (SKKU), South Korea, in 2022, where he is currently pursuing the Ph.D. degree in artificial intelligence.
His research interests include few-shot classification, attention, and deep learning.

WonJun Moon
is a Ph.D. student at the Department of Artificial Intelligence, Sungkyunkwan University (SKKU), South Korea.
He received B.S., and M.S. degrees in computer science from Sungkyunkwan University (SKKU) in 2021 and 2022, respectively. Currently, his research areas include computer vision and deep learning.

Hyun Seok Seong
received the B.S. degree in electronic and electrical engineering from Sungkyunkwan University (SKKU), South Korea, in 2019, where he is currently pursuing the combined M.S. and Ph.D. degrees in artificial intelligence. His research interests include metric learning for image categorization, machine learning, and deep learning.

Jae-Pil Heo
(Member, IEEE) received the B.S., M.S., and Ph.D. degrees in computer science from the Korea Advanced Institute of Science and Technology (KAIST), in 2008, 2010, and 2015, respectively.
He is currently an Associate Professor at Sungkyunkwan University (SKKU), South Korea.
Before joining SKKU, he was a Researcher at the Electronics and Telecommunications Research Institute (ETRI). His research interests include computer vision and machine learning.