# Sparse Spatial Transformers for Few-Shot Learning

Haoxing Chen, Huaxiong Li, Yaohui Li, Chunlin Chen
Department of Control and Systems Engineering, Nanjing University, Nanjing,
China
$\\{$haoxingchen<EMAIL_ADDRESS>$\\{$huaxiongli,
<EMAIL_ADDRESS>
###### Abstract

Learning from limited data is a challenging task since the scarcity of data
leads to a poor generalization of the trained model. The classical global
pooled representation is likely to lose useful local information. Recently,
many few shot learning methods address this challenge by using deep
descriptors and learning a pixel-level metric. However, using deep descriptors
as feature representations may lose the contextual information of the image.
And most of these methods deal with each class in the support set
independently, which cannot sufficiently utilize discriminative information
and task-specific embeddings. In this paper, we propose a novel Transformer
based neural network architecture called Sparse Spatial Transformers
(SSFormers), which can find task-relevant features and suppress task-
irrelevant features. Specifically, we first divide each input image into
several image patches of different sizes to obtain dense local features. These
features retain contextual information while expressing local information.
Then, a sparse spatial transformer layer is proposed to find spatial
correspondence between the query image and the entire support set to select
task-relevant image patches and suppress task-irrelevant image patches.
Finally, we propose an image patch matching module to calculate the distance
between dense local representations to determine which category the query
image belongs to in the support set. Extensive experiments on popular few-shot
learning benchmarks show that our method achieves the state-of-the-art
performance. Our code is available at
https://github.com/chenhaoxing/SSFormers.

##  1 Introduction

With the availability of large-scale labeled data, visual understanding
technology has made great progress in many tasks [1, 15, 28]. However,
collecting and labeling such a large amount of data is time-consuming and
laborious. Few-shot learning is committed to solving this problem, which
enables deep models to have better generalization ability even on a small
number of samples.

Recently, many few-shot learning methods have been proposed, which can be
roughly divided into two categories: meta-learning [7, 18, 5] and metric-
learning [25, 22, 23, 13, 2, 3]. The goal of meta-learning is to learn how to
deal with new tasks in the process of learning multiple tasks. Metric learning
focuses on learning a good feature representation or relation measure.

For feature representations, most of the existing metric-learning based
methods [25, 22, 12] adopt global features for recognition, which may cause
useful local information to be lost and overwhelmed. Recently, DN4 [13],
MATANet [2] and DeepEMD [27] adopt dense feature representations (i.e., deep
descriptors) for few-shot learning tasks, which have been verified to be more
expressive and effective than using global features. And another branch that
enhances image representation uses the attention mechanism to align the query
image with the support set. For example, Cross Attention Network (CAN) [10]
and SAML [9] use the semantic correlation between the support set and query
image to highlight the target object.

For the relation measure, existing dense feature based methods usually adopt a
pixel-level metric and the query image is taken as a set of deep descriptors.
For example, in DN4 [13], for each query deep descriptor, they find its
nearest neighbor descriptors in each support class. Also, CovaMNet [14]
calculates a local similarity between each query deep descriptor and a support
class by a covariance metric.

However, most existing methods use global features or deep descriptors, and
they are not effective for few-shot image classification. Due to the global
feature lose local information, and deep descriptors lose the contextual
information of the image. Moreover, the above methods all process each support
class independently, and cannot use the context information of the entire task
to generate task-specific features.

In this paper, we propose a novel sparse spatial transformer (SSFormers),
which extracts the spatial correlation between the query image and the current
task (the entire support set), aiming to align task-relevant image patches and
suppress task-irrelevant image patches. First, we divide each input image into
several patches and get dense local features. Second, we select task-relevant
query patches by a two-way selection function, i.e., mutual nearest neighbour
[8, 16]. And use selected query patches to align support classes. Finally, a
patch matching module is proposed to measure the similarity between query
images and aligned support classes. For each patch from a query image, the
patch matching module calculates its similarity scores to the nearest neighbor
patch in each aligned class prototype. Then, similarity scores from all query
patches are accumulated as a patch-to-class similarity.

The main contributions of this work are summarized as follows:

  * 1)

We propose a novel _sparse spatial transformers_ for few-shot learning, which
can select task-relevant patches and generate a task-specific prototype.

  * 2)

We propose a _patch matching module_ to get similarity between query images
and task-specific prototypes. Experiments prove that it is more suitable for
image patch-based feature representation than directly using the cosine
similarity.

  * 3)

We conduct extensive experiments on popular few-shot learning benchmarks and
show that the proposed model achieving competitive results compared to other
state-of-the-art methods.

##  2 Related Work

Global Feature based Methods. The traditional metric-learning based few-shot
learning methods use an additional global average pooling (GAP) layer to
obtain the global feature representation at the end of the backbone and
utilize different metrics for classification. MatchingNet [25] utilizes the
cosine distance to measure the similarity between the query image and each
support class. ProtoNet [22] takes the empirical mean as the prototype
representation of each category and uses Euclidean distance as the distance
metric. RelationNet [23] proposed a non-linear learnable distance metric.
These methods based on global features will lose a lot of useful local
information, which is harmful to classification tasks under few-shot learning
settings.

Dense Feature based Methods. Another branch of metric-learning based methods
uses pixel-level deep descriptors as feature representations. DN4 [13] uses
the $k$-nearest neighbor algorithm to obtain the pixel-level similarity
between images. MATANet [2] proposes a multi-scale task adaptive network to
select task-relevant deep descriptors at multiple scales. DeepEMD [27]
proposes a differentiable earth mover’s distance to calculate the similarity
between image patches. Our SSFormers also belong to this method based on dense
features. A major difference in our method is that we divide input images into
several patches of different sizes and extract features. Compared with global
features, the features extracted by our method can express local information.
And compared with deep descriptors, the extracted features contain context
information.

Attention based Methods. CAN [10] proposes a cross-attention algorithm to
highlight the common objects in the image pair. SAML [9] proposes a collect
and select strategy to align the main objects in the image pair.
CrossTransformers [6] proposes to use a self-supervised learning algorithm to
enhance the feature representation ability of the pre-trained backbone, and
use a transformer to achieve alignment. SSFormers also be treated among the
family of attention-based methods as they also aligned query images and
support sets. Differently, our SSFormers select task-relevant patches in the
query image to align support set to query image by a sparse spatial cross
attention algorithm.

Figure 1: Illustration of the proposed SSFormers. We propose to generate dense
local features and finding task-relevant features by a sparse spatial
transformers layer.

##  3 Preliminary

Before presenting our method in detail, we first introduce the problem
definition of few-shot learning. Few-shot learning is dedicated to learning
transferable knowledge between tasks and using the learned knowledge to solve
new tasks. In the few-shot learning scenario, the task is usually set in the
form of _N_ -way _M_ -shot, where _N_ is the number of categories and _M_ is
the number of labeled samples in each category. Under this setting, the model
is trained on a training set $\mathcal{D}_{train}$ with a large amount of
labeled data. To learn transferable knowledge, we use episodic training
mechanisms to train our model. The episodic training mechanism samples batched
tasks from $\mathcal{D}_{train}$ for training. In each episode, we first
construct query set $\mathcal{D}_{\rm
Q}=\\{(x^{q}_{i},y^{q}_{i})\\}_{i=1}^{N\times B}$ and support set
$\mathcal{D}_{S}=\\{(x^{s}_{i},y^{s}_{i})\\}_{i=1}^{N\times M}$, where $B$ is
a hyperparameter that we need to fix in our experiments. Then our model
predicts which support set category each sample in the query set belongs to.
When the model training is complete, we sample tasks from unlabeled test sets
$\mathcal{D}_{test}$ to verify the performance of the model.

##  4 Our Method

In this section, we first introduce our method for generating dense local
representations. Then we describe our sparse spatial transformers layer, which
spatial aligns query images and support classes. Finally, we describe the
patch matching module (PMM), which is used to calculate the final
similarities. The overview of our framework is shown in Figure 1.

###  4.1 Dense Local Feature Extractor

The metric-learning based few-shot learning method aims to find an effective
feature representation and a good distance metric to calculate the similarity
between images. Different from the methods that use global features, local
representation based methods have achieved better results because the local
representation contains richer and more transferable semantic information. The
difference from previous work is that our SSFormers aim to establish
hierarchical local representations for spatial comparison.

As illustrated in Figure 1, dense local representations extractor $F_{\theta}$
evenly divides the image into $H\times W$ patches, and each image patch is
individually encoded by the backbone network to generate a feature vector. The
feature vectors generated by all patches constitute the dense local
representations set of each image. To generate hierarchical local
representations, we adopt a pyramid structure in the experiments. Thus the
feature representation of an input image $x$ can be denoted as
$F_{\theta}(x)\in\mathbb{R}^{K\times C}$, where $C$ is the number of channels,
and $K$ is the number of all local patch representations. Specifically, we
adopt two image patch division strategies of size 2$\times$2 and 3$\times$3 to
obtain 13 dense local representations.

In each _N_ -way _M_ -shot few-shot image recognition task, for each support
class, we have $M$ samples and get _M_ feature representations. Instead of
using empirical mean of _M_ feature representations [22] to obtain the class
representation, we utilize all the patches in each support class, i.e.,
$S_{n}\in\mathbb{R}^{MK\times C}$, where $S_{n}$ is the class representation
of the $n$-th support class. The entire support set representation can be
denoted as $S\in\mathbb{R}^{N\times MK\times C}$. Similarly, for a query image
$x_{q}$, through $F_{\theta}$, we can get feature representation
$q=F_{\theta}(x_{q})\in\mathbb{R}^{K\times C}$.

###  4.2 Sparse Spatial Transformers Layer

Sparse spatial transformers aim to enhance the discriminant ability of local
feature representations by modeling the interdependencies between different
patches in the query image and the entire support set. In a _N_ -way _M_ -shot
task, key $k_{S}$ and value $v_{S}$ are generated for support set feature $S$
using two independent linear projection: the key projection head $h_{k}$:
$\mathbb{R}^{C}\mapsto\mathbb{R}^{C^{\prime}}$ and the value projection head
$h_{v}$: $\mathbb{R}^{C}\mapsto\mathbb{R}^{C^{\prime}}$. Similarly, the query
image feature $q$ is embedded using the value projection head $h_{v}$ and
another the query projection head $h_{q}$:
$\mathbb{R}^{C}\mapsto\mathbb{R}^{C^{\prime}}$ to obtain value $v_{q}$ and
query $q_{q}$.

We argue that if the patch with the closest semantic distance to patch $q_{i}$
is $S_{j}$, and the patch with the closest semantic distance to patch $S_{j}$
is $q_{i}$, then they are likely to have the same local feature, where
$q_{i}\in q_{q},i\in\\{1,...,K\\}$ and $S_{j}\in k_{S},j\in\\{1,...,NMK\\}$.
On the other hand, if the closest patch of patch $S_{j}$ is not $q_{i}$, then
even if the closest descriptor of $q_{i}$ is $S_{j}$, the actual relationship
between them is relatively weak. In other words, the correlation between two
patches is a function of mutual perception, not a function of one-way
perception. Therefore, we can use this bidirectionality to select task-
relevant patches in the current task.

We first calculated semantic relation matrix between query image and each
support class $n$, and get $\mathbf{R}_{n}$:

$\mathbf{R}_{n}=\frac{q_{q}\times
k_{S_{n}}^{\top}}{\sqrt{C^{\prime}}}\in\mathbb{R}^{K\times MK}$ (1)

To finding task-relevant patches, we concatenate all semantic relation
matrixes $\mathbf{R}_{n},n=\\{1,...,N\\}$ to get
$\mathbf{R}\in\mathbb{R}^{K\times NMK}$. Each row in $\mathbf{R}$ represents
the semantic similarity of each patch in the query image to all patches of all
images in the support set.

Specifically, we propose a novel sparse spatial cross attention algorithm to
find task-relevant patches in the query image. For each patch $q_{i}\in
q_{q}$, we find its nearest neighbor $n_{q}^{i}$ in $k_{S}$, and then find the
nearest neighbor $n_{S}^{i}$ of $n_{q}^{i}$ in $q_{q}$. If $i=n_{S}^{i}$, then
we consider $q_{i}$ to be a task-relevant patch. After collecting all task-
relevant patches in $q_{q}$, we can get the mask $m=[m_{1};...;m_{K}]$, which
can be computed as:

$\displaystyle n_{q}^{i}=\operatorname*{arg\,max}_{j}\mathbf{R}_{i,j}$ (2)
$\displaystyle n_{S}^{i}=\operatorname*{arg\,max}_{k}\mathbf{R}_{k,n_{q}^{i}}$
(3) $\displaystyle m^{i}=\mathbbm{1}(i=n_{S}^{i})$ (4)

where $\mathbbm{1}$ is the indicator function, when $i=n_{S}^{i}$,
$\mathbbm{1}$ is equal to 1, otherwise it is 0. Using mask $m$ and semantic
relation matrix $\mathbf{R}_{n}$, we can get sparse attention map $a^{n}$ and
use it to align each support class $n$ to query image $q$ and get task-
specific prototype $v_{S_{n}|q}$, which can be computed as:

$\displaystyle a_{n}=m*\mathbf{R_{n}}\in\mathbb{R}^{K\times MK}$ (5)
$\displaystyle v_{S_{n}|q}=a_{n}\times v_{S_{n}}^{\top}\
\in\mathbb{R}^{K\times C^{\prime}}$ (6)

Algorithm 1 Pseudo code of sparse spatial transformers layer in a PyTorch-like
style

# key projection head $h_{k}$
# query projection head $h_{q}$
# value projection head $h_{v}$

Input: query feature $q$, support feature $S$
Output: aligned prototype $v_{S|q}$

support key $\in\mathbb{R}^{N\times C^{\prime}\times MK}$

$k_{S}$ = $h_{k}(S)$

# query query $\in\mathbb{R}^{C^{\prime}\times K}$

$q_{q}$ = $h_{q}(q)$

# query value $\in\mathbb{R}^{C^{\prime}\times K}$, support value
$\in\mathbb{R}^{N\times C^{\prime}\times MK}$

$v_{q}$, $v_{S}$ = $h_{v}(q)$, $h_{v}(S)$

# similarity matrix $\in\mathbb{R}^{N\times K\times MK}$

$\mathbf{R}$ = $q_{q}$.unsqueeze(0).transpose(-1, -2) $\times$ $k_{S}$

# max index $q_{q}^{max}\in\mathbb{R}^{K},k_{S}^{max}\in\mathbb{R}^{NMK}$

$q_{q}^{max}$ = $\mathbf{R}$.permute(1, 0, 2).view($K,-1$).max(-1)[1]

$k_{S}^{max}$ = $\mathbf{R}$.permute(1, 0, 2).view($K,-1$).max(-2)[1]

# mask $m\in\mathbf{R}^{K}$

$m$= torch.gather($k_{S}^{max}$, 0, $q_{q}^{max}$)

$m$ = ($m_{q|S}$ == torch.arange($K$))

# attention map $a\in\mathbb{R}^{N\times K\times MK}$

$a$ = $\mathbf{R}$ * $m$.unsqueeze(0).unsqueeze(-1)

$a$ = dropout(nn.Softmax(dim=-1)($a$))

aligned prototype feature $v_{S|q}\in\mathbb{R}^{N\times K\times C^{\prime}}$

$v_{S|q}$ = $a\times v_{S}$.transpose(-1, -2)

return $v_{S|q}$

Table 1: Average classification accuracy of 5-way 1-shot and 5-way 5-shot tasks with $95\%$ confidence intervals on _mini_ ImageNet and _tiered_ ImageNet. † Results re-implemented by [16]. (Red/blue is best/second best performances)  Method | Venue | Backbone | Type |  _mini_ ImageNet
---|---|---|---|---
5-way 1-shot | 5-way 5-shot
ProtoNet [22] | NeurIPS’17 | Conv-64F | Metric | 49.42$\pm$0.78 | 68.20$\pm$0.66
CovaMNet [14] | AAAI’19 | Conv-64F | Metric | 51.19$\pm$0.76 | 67.65$\pm$0.63
DN4 [13] | CVPR’19 | Conv-64F | Metric | 51.24$\pm$0.74 | 71.02$\pm$0.64
DSN [21] | CVPR’20 | Conv-64F | Metric | 51.78$\pm$0.96 | 68.99$\pm$0.69
DeepEMD† [27] | CVPR’20 | Conv-64F | Metric | 52.15$\pm$0.28 | 65.52$\pm$0.72
SSFormers | Ours | Conv-64F | Metric | 55.00$\pm$0.22 | 70.55$\pm$0.17
ProtoNet [22] | NeurIPS’17 | ResNet12 | Metric | 62.59$\pm$0.85 | 78.60$\pm$0.16
CAN [10] | NeurIPS’19 | ResNet12 | Metric | 63.85$\pm$0.48 | 79.44$\pm$0.34
DSN [21] | CVPR’20 | ResNet12 | Metric | 62.64$\pm$0.66 | 78.83$\pm$0.45
DeepEMD [27] | CVPR’20 | ResNet12 | Metric | 65.91$\pm$0.82 | 82.41$\pm$0.56
FEAT [26] | CVPR’20 | ResNet12 | Metric | 66.78$\pm$0.20 | 82.05$\pm$0.14
RENet [11] | ICCV’21 | ResNet12 | Metric | 67.60$\pm$0.44 | 82.58$\pm$0.30
PSST [4] | CVPR’21 | ResNet12 | Optimization | 64.05$\pm$0.49 | 80.24$\pm$0.45
GLoFA [17] | AAAI’21 | ResNet12 | Optimization | 66.12$\pm$0.42 | 81.37$\pm$0.33
SSFormers | Ours | ResNet12 | Metric | 67.25$\pm$0.24 | 82.75$\pm$0.20
Method | Venue | Backbone | Type |  _tiered_ ImageNet
5-way 1-shot | 5-way 5-shot
ProtoNet [22] | NeurIPS’17 | Conv-64F | Metric | 53.31$\pm$0.89 | 72.69$\pm$0.74
CovaMNet [14] | AAAI’19 | Conv-64F | Metric | 54.98$\pm$0.90 | 71.51$\pm$0.75
DN4 [13] | CVPR’19 | Conv-64F | Metric | 53.37$\pm$0.86 | 74.45$\pm$0.70
DSN [21] | CVPR’20 | Conv-64F | Metric | 53.22$\pm$0.66 | 71.06$\pm$0.55
DeepEMD† [27] | CVPR’20 | Conv-64F | Metric | 50.89$\pm$0.30 | 66.12$\pm$0.78
SSFormers | Ours | Conv-64F | Metric | 55.54$\pm$0.19 | 73.72$\pm$0.21
ProtoNet [22] | NeurIPS’17 | ResNet12 | Metric | 68.37$\pm$0.23 | 83.43$\pm$0.16
CAN [10] | NeurIPS’19 | ResNet12 | Metric | 69.89$\pm$0.51 | 84.23$\pm$0.37
DSN [21] | CVPR’20 | ResNet12 | Metric | 67.39$\pm$0.82 | 82.85$\pm$0.56
FEAT [26] | CVPR’20 | ResNet12 | Metric | 70.80$\pm$0.23 | 84.79$\pm$0.16
DeepEMD [27] | CVPR’20 | ResNet12 | Metric | 71.16$\pm$0.87 | 83.95$\pm$0.58
RENet [11] | ICCV’21 | ResNet12 | Metric | 71.61$\pm$0.51 | 85.28$\pm$0.35
GLoFA [17] | AAAI’21 | ResNet12 | Optimization | 69.75$\pm$0.33 | 83.58$\pm$0.42
SSFormers | Ours | ResNet12 | Metric | 72.52$\pm$0.25 | 86.61$\pm$0.18

A pseudo code of sparse spatial transformers layer in a PyTorch-like style is
shown in Algorithm 1.

###  4.3 Patch Matching Module

Patching maching module is built as a similarity metric, which does not have
any parameter to train. Given query value $v_{q}\in\mathbb{R}^{K\times
C^{\prime}}$ and the aligned prototype of class $n$
$v_{S_{n}|q}\in\mathbb{R}^{K\times C^{\prime}}$, we can get their patch-to-
patch similarity matrix by:

$\displaystyle\mathbf{D}^{n}=\frac{v_{q}\times
v_{S_{n}|q}}{||v_{q}||\cdot||v_{S_{n}|q}||}\in\mathbb{R}^{K\times K}$ (7)

Them, for each patch in $v_{q}$, we select the most similar patch in all
patches from prototype $v_{S_{n}|q}$. Then, we sum $K$ selected patches as the
similarity between the query image and support class $n$:

$\displaystyle
P^{n}=\sum_{i=1}^{K}\max\limits_{j\in\\{1,...,K\\}}\mathbf{D}^{n}_{i,j}$ (8)

Under the _N_ -way _M_ -shot few-shot learning setting, we can get semantic
similarity vectors $P\in\mathbb{R}^{N}$.

Table 2: Ablation study on our model, we can find that each part of our model has important contribution. The experiments are conducted with _ResNet12_ on _mini_ ImageNet. (SSTL: Sparse Spatial Transformer Layer and PMM: Patch Matching Module) Dense Local Feature | SSTL | PMM | Cosine Classifier | 5-way 1-shot | 5-way 5-shot
---|---|---|---|---|---
| ✔ | ✔ |  | 63.15$\pm$0.20 | 79.15$\pm$0.25
✔ |  | ✔ |  | 66.84$\pm$0.47 | 79.72$\pm$0.50
✔ | ✔ |  | ✔ | 64.35$\pm$0.22 | 80.17$\pm$0.17
✔ | ✔ | ✔ |  | 67.25$\pm$0.24 | 82.75$\pm$0.20
Figure 2: 5-way semi-supervised few-shot learning results on _mini_ ImageNet. We show the results with (w/ D) and without _distractors_(w/o D). And we compare our methods with PN, Non-Masked [19], PN, Masked [19] and DSN-semi [21]. Table 3: The 5-way, 1-shot and 5-shot classification accuracy (%) with different number of image patches on _mini_ ImageNet. Embedding | 5-way 1-shot | 5-way 5-shot
---|---|---
$5\times 5$ | 65.20$\pm$0.24 | 80.07$\pm$0.26
$4\times 4$ | 66.37$\pm$0.23 | 81.06$\pm$0.25
$3\times 3$ | 65.17$\pm$0.22 | 81.69$\pm$0.20
$2\times 2$ | 65.18$\pm$0.22 | 80.10$\pm$0.16
$4\times 4+3\times 3$ | 66.08$\pm$0.24 | 81.50$\pm$0.26
$4\times 4+2\times 2$ | 67.25$\pm$0.24 | 82.75$\pm$0.20
$3\times 3+2\times 2$ | 67.05$\pm$0.22 | 82.53$\pm$0.18
Table 4: The 5-way, 1-shot and 5-shot classification accuracy (%) with different attention methods on _tiered_ ImageNet. Method | Self | Cross | 1-shot | 5-shot
---|---|---|---|---
ProtoNet |  |  | 68.37$\pm$0.23 | 83.43$\pm$0.16
CAN |  | ✔ | 69.89$\pm$0.51 | 84.23$\pm$0.37
FEAT |  | ✔ | 70.80$\pm$0.23 | 84.79$\pm$0.16
RENet | ✔ | ✔ | 71.61$\pm$0.51 | 85.28$\pm$0.35
SSFormers |  | ✔ | 72.52$\pm$0.25 | 86.61$\pm$0.18
Table 5: Stability evaluation on _mini_ ImageNet. Comparison with Rethink-Distill [24]. |  $+$ GaussianBlur ($\delta\in\left[0.1,2\right]$)
---|---
Rethink-Distill | 82.14$\rightarrow$49.30
SSFormers | 82.75$\rightarrow$77.17
|  $+$ pepperNoise ($r=0.01$)
Rethink-Distill | 82.14$\rightarrow$63.97
SSFormers | 82.75$\rightarrow$66.74
|  $+$ ColorJitter ($B=0.8$)
Rethink-Distill | 82.14$\rightarrow$81.05
SSFormers | 82.75$\rightarrow$81.21
Figure 3: Grad-class activation mapping (Grad-cam) visualization on two 2-way
1-shot tasks. In contrast to ProtoNet, our sparse spatial transformer layer
(SSTL) can automatically highlight task-relevant areas.

##  5 Experiments

To evaluate the effectiveness of our method, we conduct extensive experiments
on several common-used benchmarks for few-shot image recognition. In this
section, we first present details about datasets and experimental settings in
our network design. Then, we compare our method with the state-of-the-art
methods on various few-shot learning tasks, i.e., standard few-shot learning,
cross-domain few-shot learning. Finally, we conduct comprehensive ablation
studies to validate each component in our network.

###  5.1 Datasets

We conduct few-shot image recognition problems on two popular benchmarks,
i.e., _mini_ ImageNet, _tiered_ ImageNet.

_mini_ ImageNet. _mini_ ImageNet is a subset random sampled from ImageNet and
is an important benchmark in few-shot learning community. _mini_ ImageNet
consists of 60,000 images in 100 categories. We follow the standard partition
settings [22], where there are 64/16/20 categories for training, validation
and evaluation.

_tiered_ ImageNet. _tiered_ ImageNet is also a subset random sampled from
ImageNet, which consists of 779,165 images in 608 categories. All 608
categories are grouped into 34 broader categories. Following the same
partition settings [10], we use 20/6/8 broader categories for training,
validation, and evaluation respectively.

###  5.2 Implementation Details

Backbone networks. For fair comparison, following [21], we employ _Conv-64F_
and _ResNet12_ as our model backbone. To generate pyramid dense features, we
add a global average pooling layer at the end of the backbone, such that the
backbone generates a vector for each input image patch. And we slightly expand
the area of the local patches in the grid by 2 times to merge the context
information, which is helpful to generate the local representations.

Training details. The training process can be divided into two stages: pre-
training and meta-training. Following [27], we apply a pre-train strategy. The
backbone networks are trained on training categories with a softmax layer. In
this stage, we apply data argumentation methods to increase the generalization
ability of the model, i.e., color jitter, random crop, and random horizontal
flip. The backbone network in our model is initialized with pre-trained
weights, which are then fine-tuned along with other components in our model.
In the meta-training stage, we conduct _N_ -way _M_ -shot tasks on all
benchmarks, i.e., 5-way 1-shot and 5-way 5-shot. _Conv-64F_ is optimized by
Adam, and the initial learning rate is set to 0.1 and decay 0.1 every 10
epochs. And _ResNet12_ is optimized by SGD, and the initial learning rate is
set to 5e-4 and decay 0.5 every 10 epochs.

Evaluation. During the test stage, we random sample 10,000 tasks from the
meta-testing set, and take averaged top-1 classification accuracy as the
performance of methods.

###  5.3 Standard Few-shot Image Classification

To verify the effectiveness of our proposed SSFormers for few-shot image
classification task, we conduct comprehensive experiments and compare our
methods with other state-of-the-art methods. The experimental results are
shown in Table 1 and Table 2. The results show that our method achieves the
best results in almost all settings. For example, compared with the classic
attention-based method CAN [10], our model is around 4.4%/4.0% better than CAN
on _mini_ ImageNet with _ResNet12_. And compare with the previous
transformers-based method FEAT [26], which uses transformers to find task-
specific features in the support set, our method achieves 1.4%/2.0%
improvements on _tiered_ ImageNet with _ResNet12_.

The reason why we can achieve this improvement is that SSFormers can find
task-relevant patches in the current task and perform sparse spatial cross
attention algorithms based on hierarchical dense representations.

###  5.4 Semi-supervised Few-Shot Learning

We further verify the effectiveness of our model on more challenging semi-
supervised few-shot learning tasks. Under semi-supervised few-shot learning
settings, we can select image patches from unlabeled samples that meet the
mutual perception function (Eq. (2)-(4)) with the current support set and add
them to the support set to provide more support features. Specifically, the
workflow of SSFormers-semi is as follows: for the support set $n$, we first
search for all patches that satisfy the mutual perception function in
unlabeled sets and put them into the set $U_{n}$.Then we use $U_{n}$ to extend
$S_{n}$: $S_{n}=\\{S_{n}^{1},...,S_{n}^{MK}\\}\bigcup U_{n}$. Then, we use the
original SSFormers to calculate the similarity.

We use the same experiment setting in [19]. We use _Conv-64F_ as our backbone
and train SSFormers-semi on 300,000 tasks on _mini_ ImageNet. The results are
shown in Figure 3, where SSFormers-semi shows competitive results with
classical baseline methods.

###  5.5 Ablation Study

Analysis of Our Method. Our model consists of different components: dense
local feature extractor, sparse spatial transformer layer, and patch matching
module. As shown in Table 2, we studied the effect of each component on the
results on the _mini_ ImageNet. Note that when replacing the patch matching
module, we calculate the cosine similarity spatially. The results show that
every component in SSFormers has a significant contribution. For example,
without our sparse spatial transformer layer, the performance of the model
will drop by 3.73% on 5-shot tasks.

Influence of the number of patches. During dividing input images into patches,
we have to define the grid for patches. We select various grids and their
combinations and conduct analysis experiments on _mini_ ImageNet. As shown in
Table 4, it is better to use a combination of grids of different sizes. A
possible explanation is that the size of the main object in different images
is different, and the use of a single size may lose context information, and
make it difficult to generate high-level semantic representations.

Comparison with other attention methods. As shown in Table 4, our model has
achieved state-of-the-art in all attention-based methods. Specifically,
compared with RENet [11], our SSFormers only utilize cross attention, but
still leads them in both 1-shot and 5-shot tasks.

Analyze of Stability. A good model should have good robustness and be able to
adapt to various environments. For this reason, we tested the stability of our
model under three different attacks. As shown in Table 5, the performance of
our SSFormers is relatively stable under various attacks.

Qualitative visualizations of the sparse spatial transformer layer. To
qualitatively evaluate the proposed sparse spatial transformer layer (SSTL),
we provide some visualization cases as shown in Figure 3. We fixed two 2-way
1-shot tasks and used grad-cam [20] to visualize ProtoNet and ProtoNet+SSTL
respectively. Specifically, in task 1, ProtoNet highlighted people by mistake,
and in task 2, ProtoNet highlighted background information by mistake. On the
contrary, ProtoNet+SSTL can correctly highlight the task-relevant areas, which
once again verifies the effectiveness of our model.

##  6 Conclusion

In this work, we proposed a novel sparse spatial transformers (SSFormers) for
few-shot learning, which customizes task-specific prototypes via a
transformer-based architecture. Specifically, we use image patches to
represent images and propose a sparse spatial transformer layer to select
task-relevant patches and generate task-specific prototypes. Then we use the
patch matching module to calculate similarity scores. Experimental results
demonstrate SSFormers can achieve competitive results with other state-of-the-
art few-shot learning methods.

## References

  * [1] Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh.  Openpose: Realtime multi-person 2d pose estimation using part affinity fields.  TPAMI, 43(1):172–186, 2021.
  * [2] Haoxing Chen, Huaxiong Li, Yaohui Li, and Chunlin Chen.  Multi-scale adaptive task attention network for few-shot learning, 2020\.
  * [3] Haoxing Chen, Huaxiong Li, Yaohui Li, and Chunlin Chen.  Multi-level metric learning for few-shot image recognition, 2021.
  * [4] Zhengyu Chen, Jixie Ge, Heshen Zhan, Siteng Huang, and Donglin Wang.  Pareto self-supervised training for few-shot learning, 2021.
  * [5] Wen-Hsuan Chu, Yu-Jhe Li, Jing-Cheng Chang, and Yu-Chiang Frank Wang.  Spot and learn: A maximum-entropy patch sampler for few-shot image classification.  In CVPR, pages 6251–6260, 2019.
  * [6] Carl Doersch, Ankush Gupta, and Andrew Zisserman.  Crosstransformers: spatially-aware few-shot transfer.  In NeurIPS, 2020.
  * [7] Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In ICML, volume 70, pages 1126–1135, 2017.
  * [8] K. Chidananda Gowda and G. Krishna.  The condensed nearest neighbor rule using the concept of mutual nearest neighborhood (corresp.).  TIT, 25(4):488–490, 1979.
  * [9] Fusheng Hao, Fengxiang He, Jun Cheng, Lei Wang, Jianzhong Cao, and Dacheng Tao.  Collect and select: Semantic alignment metric learning for few-shot learning.  In ICCV, pages 8459–8468, 2019.
  * [10] Ruibing Hou, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen.  Cross attention network for few-shot classification.  In NeurIPS, pages 4005–4016, 2019.
  * [11] Dahyun Kang, Heeseung Kwon, Juhong Min, and Minsu Cho.  Relational embedding for few-shot classification, 2021.
  * [12] Aoxue Li, Tiange Luo, Tao Xiang, Weiran Huang, and Liwei Wang.  Few-shot learning with global class representations.  In ICCV, pages 9714–9723, 2019.
  * [13] Wenbin Li, Lei Wang, Jinglin Xu, Jing Huo, Yang Gao, and Jiebo Luo.  Revisiting local descriptor based image-to-class measure for few-shot learning.  In CVPR, pages 7260–7268, 2019.
  * [14] Wenbin Li, Jinglin Xu, Jing Huo, Lei Wang, Yang Gao, and Jiebo Luo.  Distribution consistency based covariance metric networks for few-shot learning.  In AAAI, pages 8642–8649, 2019.
  * [15] Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  IEEE Trans. Pattern Anal. Mach. Intell., 42(2):318–327, 2020\.
  * [16] Yang Liu, Tu Zheng, Jie Song, Deng Cai, and Xiaofei He.  DMN4: few-shot learning via discriminative mutual nearest neighbor neural network, 2021.
  * [17] Su Lu, Han-Jia Ye, and De-Chuan Zhan.  Tailoring embedding function to heterogeneous few-shot tasks by global and local feature adaptors.  In AAAI, pages 8776–8783, 2021.
  * [18] Sachin Ravi and Hugo Larochelle.  Optimization as a model for few-shot learning.  In ICLR, 2017.
  * [19] Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B. Tenenbaum, Hugo Larochelle, and Richard S. Zemel.  Meta-learning for semi-supervised few-shot classification.  In ICLR, 2018.
  * [20] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In ICCV, pages 618–626, 2017.
  * [21] Christian Simon, Piotr Koniusz, Richard Nock, and Mehrtash Harandi.  Adaptive subspaces for few-shot learning.  In CVPR, pages 4135–4144, 2020.
  * [22] Jake Snell, Kevin Swersky, and Richard S. Zemel.  Prototypical networks for few-shot learning.  In NeurIPS, pages 4077–4087, 2017.
  * [23] Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip H. S. Torr, and Timothy M. Hospedales.  Learning to compare: Relation network for few-shot learning.  In CVPR, pages 1199–1208, 2018.
  * [24] Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B. Tenenbaum, and Phillip Isola.  Rethinking few-shot image classification: A good embedding is all you need?  In ECCV, volume 12359, pages 266–282, 2020.
  * [25] Oriol Vinyals, Charles Blundell, Tim Lillicrap, Koray Kavukcuoglu, and Daan Wierstra.  Matching networks for one shot learning.  In NeurIPS, pages 3630–3638, 2016.
  * [26] Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha.  Few-shot learning via embedding adaptation with set-to-set functions.  In CVPR, pages 8808–8817, 2020.
  * [27] Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen.  Deepemd: Few-shot image classification with differentiable earth mover’s distance and structured classifiers.  In CVPR, pages 12200–12210, 2020.
  * [28] C. Zhang, H. Li, C. Chen, Y. Qian, and X. Zhou.  Enhanced group sparse regularized nonconvex regression for face recognition.  TPAMI, 2020.  to be published, doi:10.1109/TPAMI.2020.3033994.