# Trainable Class Prototypes for Few-Shot Learning

Jianyi Li  School of Information and Communications
Xi’an Jiaotong University
Xi’an, P.R. China
<EMAIL_ADDRESS>Guizhong Liu  School of Information and
Communications
Xi’an Jiaotong University
Xi’an, P.R. China
<EMAIL_ADDRESS>
###### Abstract

Metric learning is a widely used method for few shot learning in which the
quality of prototypes plays a key role in the algorithm. In this paper we
propose the trainable prototypes for distance measure instead of the
artificial ones within the meta-training and task-training framework. Also to
avoid the disadvantages that the episodic meta-training brought, we adopt non-
episodic meta-training based on self-supervised learning. Overall we solve the
few-shot tasks in two phases: meta-training a transferable feature extractor
via self-supervised learning and training the prototypes for metric
classification. In addition, the simple attention mechanism is used in both
meta-training and task-training. Our method achieves state-of-the-art
performance in a variety of established few-shot tasks on the standard few-
shot visual classiﬁcation dataset, with about 20% increase compared to the
available unsupervised few-shot learning methods.

##  I Introduction

In recent years deep learning has made major advances in computer vision areas
such as image recognition, video object detection and tracking. A deep neural
network needs a large amount of labeled data to fit its parameters whereas it
is laborious to label so many examples by human annotators. Thus the problem
of learning with few labeled samples called few-shot learning has been paid
more and more attention. Few-shot learning is described as a classification
task set in N -way and k -shot, which means to distinguish N categories, each
of which has k (quite small) labeled samples. The model predict classes for
new examples only depending on k labeled data. The annotated data is called
the support set, and the new data belonging to the N categories is called
query set.

People have proposed varieties of few-shot methods, all of which rely on meta-
training assisted with base classes. The universal approach is to use the base
classes to construct fake few-shot tasks for training the network first, with
the purpose of enabling the network an ability to accomplish real few-shot
tasks through simulating the process of carrying out the fake tasks. This is
called the meta-training stage with tasks as samples. Next, use the trained
network to complete real few-shot tasks of novel classes, and calculate the
classification accuracy on the query set in the tasks to evaluate the
algorithm, which is usually called the meta-testing. The whole procedure is
shown in Fig.1.

Figure 1: The universal method used in supervised few-shot learning, which
consists of meta-training and meta-testing. In the meta-training, the training
sample is actually a mimic few-shot task comprised of some labeled data chosen
from base classes. And in the meta-testing the model will solve a real task
with few labeled data and an unlabeled query set chosen from novel classes. We
show a model trained for solving 3-way 1-shot tasks in this figure.

Few-shot learning algorithms could be classified into three categories. The
first [1, 2, 3, 4] is based on metric learning, which consists of three steps
of feature extraction, distance measure, and prediction, relying on effective
metric design and reducing the cross entropy loss in meta training to improve
classification accuracy. The second are the teacher-student network based
methods including [5, 6, 7, 8]. The teacher network guides the student network
to solve the few-shot tasks in terms of parameter initialization, parameter
update and other aspects. The algorithm enables the teacher network to obtain
the ability to instruct the student network via meta-training. The third
category such as [9] and [10] is based on the transduction, which propagates
the label of the support data to the queries through a specific graph, thereby
obtaining the predicted class of the query set. The algorithm optimizes the
accuracy of propagation label through meta-training.

The meta-training determines the model’s performance in the few-shot learning
algorithm. However it brings two obvious drawbacks. First, the meta-training
requires a large number of labeled auxiliary examples (base classes). Those
algorithms can not work without adequate labeled samples. Second, the meta-
training phase uses tasks as training samples. Therefore, a task type decided
by the values of N and k needs to be certain before meta-training to ensure
that the number of images contained by each mimic few shot task (i.e. a meta-
training sample) is consistent during training. The meta-trained network can
only be used to solve few-shot tasks with the same type as the meta-training
samples, and it performs worse in other types of tasks. However, in reality we
need to solve various types of few-shot tasks, and it is unreasonable to meta-
train the network from scratch in order to solve a few shot task with new
type.

In order to solve these two problems, we solve the few shot classification
based on self-supervised learning according to [11]. Specifically, our method
called TrainPro-FSL abandons the episodic meta-training phase, which takes the
fake few-shot tasks as samples, and uses instead two new phases: the non-
episodic meta-training via self-supervised learning directly using a single
image as a training sample, and the task-training to solve a real few shot
task. In the first phase, a discriminative self-supervised learning method is
used to obtain a feature extractor with good generalization ability using
unlabeled images. In the second phase, we use metric learning to solve real
few-shot tasks. The meta-trained feature extractor is used to extract features
from all the images in the current task, and SGC is carried out, based on a
specific graph defined by the current task as the feature aggregation method
described in [11]. We use the aggregated support set features to obtain the
class prototypes which are trained via a simple neural network. Finally we use
the class prototypes to implement distance measure and further classify the
query images.

Furthermore, we demonstrate that the attention mechanism are helpful for both
meta-training and task-training. We add attention structure to the neural
network in the meta-training phase. In the task-training phase, an attention
mask that does not increase network parameters is introduced to correct the
query image features. Both of these methods can improve the classification
accuracy of our algorithm.

Our key contributions can be summarized as follows:

  * •

We propose TrainPro-FSL, a unsupervised few-shot learning algorithm with
trainable prototypes. By adopting the methodology of self-supervised learning,
the two problems intrinsic in the existing episodic meta-training paradigm are
solved simultaneously. Thus our method does not require a large number of
labeled samples for training. In addition, the meta-trained model can carry
out different types of real few-shot tasks.

  * •

We propose to use trainable prototypes to implement metric classification,
which is better than the manual design in [12].

  * •

We propose to use a simple attentive mechanism in the metric learning to
further correct the feature of examples in the query set.

  * •

Adequate experiments demonstrate that our method reaches state-of-the-art
accuracy on miniImageNet , a standardized benchmark in few-shot learning.

The paper is organized as follows. In II, we introduce the related works. Our
methodology is described in III. In IV, experimental results on the standard
vision dataset are shown in comparison with the proposed works. Finally, a
conclusion is drawn in V.

##  II Related Works

In this section we aim to show the metric learning used in few-shot learning
algorithms proposed in previous years. In addition, we introduce some
unsupervised few-shot learning methods presented recently and popular self-
supervised algorithms.

###  II-A Metric Learning

The core of metric learning is to extract features from the support set and
query set, then obtain the class prototypes using the support set, and predict
classes of queries via the nearest neighbor algorithm and attention mechanism.
Therefore the quality of class prototypes obtained in algorithm are vital for
the performance of the metric learning. Through episodic meta-training, a
metric based method obtains a feature extractor that facilitates completing
the classification task based on distance measurement. Matching Networks [1]
used LSTM to extract full context embeddings from images and applied attention
mechanism to classify. Prototypical Networks [2] proposed to use Euclidean
distance to better measure the similarity between features, and use prototypes
of each class to classify queries. Relation Network [3] used a neural network
to replace the traditional distance metric, and directly output the queries’
categories via an end-to-end network. DC-IMP [13] introduced dense
classification and leverage implanting to bring metric learning the task
dependency.

###  II-B Unsupervised Few Shot Learning

The base classes in unsupervised methods has no labels. Some existing methods
use unsupervised learning or data enhancement methods to leverage these
unlabeled base classes to artificially construct fake support set and query
set for meta training. They are able to combine with the few-shot learning
methods as mentioned above (such as MAML [5] and Prototypical Net [2]) to
fulfil few-shot tasks. UFLST [14] and CACTU [15] use clustering to make
pseudo-labels for unlabeled examples, then use the pseudo-labeled data as
ordinary labeled data to construct fake few-shot tasks to complete meta-
training. AAL [16] and UMTRA [17] took each instance as one class and randomly
sample multiple examples to construct a fake support set, then generate a
corresponding query set according to the support set by data augmentation
techniques. ULDA [18] developed a new simple data augmentation method to
enhance the difference between the support set distribution and query set
distribution when constructing the fake few-shot tasks for meta-training.

###  II-C Self-Supervised Learning

The popular self-supervised algorithms can be divided into three categories,
namely contrastive self-supervised learning, generative self-supervised
learning and discriminative self-supervised learning. The core of contrastive
self-supervised learning is contrastive learning. The main idea of contrastive
learning is to make the features of positive sample pairs close to each other
in the feature space, and keep the features of negative sample pairs away from
each other, such as BYOL[19], SimCLR[20], and their performance can even be
comparable to supervised algorithms. Generative self-supervised learning
algorithms such as image colorization[21] and VQ-VAE[22], usually take the
original image as the groundtruth, train the network and make the network
output close to the groundtruth. In generative self-supervised learning,
images are generated at the pixel level, so the features extracted by the
network are partial to image details, rather than semantic features.

Discriminant self-supervised learning can train the classification algorithm
without the category label. The category here is set by the algorithm itself,
and the label of the training sample is automatically generated according to
the algorithm setting. In the discriminative self-supervised algorithm, the
design of pretext tasks is diverse. Commonly used algorithms include jigsaw
puzzle[23], rotation prediction[24], and deep clustering [25].

##  III Methodology

The notations and problem formulation of self-supervised few-shot learning are
introduced in III-A, and our paradigm is presented in III-B. Finally, the loss
design for trainable prototypes and attention mechanism are described in III-C
and III-D respectively.

###  III-A Problem Formulation

Given two datasets, namely $D_{base}$ and $D_{task}$ with disjoint classes.
$D_{base}$ consists of a large number of unlabeled examples from the base
classes. $D_{task}$ has a small number of labeled examples called the support
set $D_{s}$, along with some unlabeled ones called the query set $D_{q}$, all
from the new classes. They stand for the total data in a few-shot learning
task. The number of classes in the novel dataset $D_{task}$, the number of
support samples and the number of query inputs for each of these classes are
denoted $N$, $k$ and $q$ respectively. So there are totally $N\times(k+q)$
examples in a few-shot learning task. Our aim is to predict the classes of the
query set of $D_{task}$. Different from the previous works like [8], our
$D_{base}$ does not have any labels. So we train the classification network
with only a few labeled examples namely $D_{s}$ in a real sense.

###  III-B Proposed Paradigm of Solution

We first train a backbone deep neural network able to extract useful and
compact features from inputs, which will be used as a generic feature
extractor. In this non-episodic meta-training phase, we train the network with
$D_{base}=\left\\{x_{1}^{{}^{\prime}},x_{2}^{{}^{\prime}},......,x_{n}^{{}^{\prime}}\right\\}~{}$
where $x^{{}^{\prime}}\in R^{w\times h\times 3}$ via ODC in [26], a kind of
discrimitive self-supervised learning method, which promises a transferable
feature extractor. Thus we obtain the extractor $\left.f_{\varphi}:R^{w\times
h\times 3}\rightarrow R^{e}\right.$.

We then use $f_{\varphi}$ to obtain the features of the total data in
$D_{task}$ (both $D_{s}$ and $D_{q}$) namely
$f_{\varphi}\left(D_{task}\right)=\left\\{f_{\varphi}\left(x\right)\middle|x\in
D_{task}\right\\}$. Then we step into the second phase namely the task-
training phase. First we build a nearest neighbor graph using the cosine
similarity according to [27] :

$cos\left({f_{\varphi}\left(x_{1}\right),f_{\varphi}\left(x_{2}\right)}\right)=\frac{f_{\varphi}\left(x_{1}\right)^{T}f_{\varphi}\left(x_{2}\right)}{\left\|{f_{\varphi}\left\|x_{1}\right\|}\right\|_{2}\left\|{f_{\varphi}\left\|x_{2}\right\|}\right\|_{2}}$
(1)

The base graph denoted $G_{task}\left(V,E\right)$ uses $f_{\varphi}(D_{task})$
to construct vertices. In details, its vertices matrix $V\in
R^{[N\times(k+q)]\times e}$ is the stacked representations of support set and
query set i.e. each vertex represents an image’s feature. We make the values
of graph edges represent the similarity between vertices-that is, similar
vertices have larger adjacency values. To get the adjacency matrix $E\in
R^{[N\times(k+q)]\times[N\times(k+q)]}$, we first define a similarity matrix
$S$ with the same dimension computed as follows:

$S_{i,j}=\begin{cases}\left.{\mathit{\cos}(}V_{i,:},V_{j,:}\right)&{i\neq
j}\\\ 0&{i=j}\\\ \end{cases}$ (2)

where $V_{i,:}$ denotes the $i$ -th row in $V$. Then we just save the $m$
largest values on each row and on the corresponding column in S to obtain a
more sparse matrix helpful to reduce the interference. Finally, we normalize
the resulting matrix to get the adjacency matrix:

$E=D^{-\frac{1}{2}}SD^{-\frac{1}{2}}$ (3)

where $D$ is the degree diagonal matrix computed by $D_{i,i}=\sum_{j}S_{i,j}$.
We can consider $E$ as the Laplacian matrix in GCN[28] used to aggregate
information among vertices.

We aggregate features for each vertex via the graph structure to get $V^{new}$
according to [11]. Then we use the metric learning to classify the query
images based on $V^{new}$. In metric learning, the quality of class prototypes
has a great influence on classification accuracy. Therefore, we abandon the
traditional method of artificially constructing class prototypes and use
neural networks to obtain good prototypes via training to implement the metric
classification. In order to improve the performance, we adopt the attention
mechanism in training phase in the meanwhile.

Figure 2: The overall architecture of the proposed paradigm. The left shows
the meta-training phase using ODC resulting in a global feature extractor. The
right is the task-training phase comprised of graph-aggregation, classifier
training with the support set and the prototypes training.

Input: A $N$ -way $k$ -shot task with the dataset
$D_{task}=\left\\{D_{s},D_{q}\right\\}$; The meta-trained feature extractor
$f_{\varphi}$

Output: Trained prototypes

1 Obtain features of all inputs including labeled and unlabeled ones,
$f_{\varphi}\left(D_{task}\right)$;

2 Build the graph $G_{task}(V,E)$ based on $f_{\varphi}(D_{task})$;

3 Aggregate vertices features $V$ of the graph to get $V^{new}$;

4 randomly initialize $\theta$; Classifier training;

5 Use manifold augmentation to extend labeled data in semantic level according
to [11], and get the augmented feature set $V_{aug}^{new}$;

6 Train $Cls_{\theta}$ using $V_{aug}^{new}$ and cross entropy loss to obtain
$Cls_{\theta^{*}}$; Prototypes training;

7 Initialization the prototypes $proto^{0}$ randomly;

8 Use $Cls_{\theta^{*}}$ to calculate the loss of the prototypes
classification and the metric classification of the support set according to
Eq. (8);

9 Update $proto^{0}$ based on the loss function and finally obtain
$proto^{*}$;

10 Return $proto^{*}$;

Algorithm 1 The process of task-training phase with prototypes training.

Our paradigm is illustrated in Fig.2. In general TrainPro-FSL has two phases:
(1) Meta-training phase: training a generic feature extractor via ODC
algorithm. (2) Task-training phase: training the prototypes using the support
set data after the feature aggregation through graph. Once the latter is
finished, the performance of this model is evaluated by measuring the distance
between features of $D_{q}$ and prototypes. The process of task-training phase
is formalized in Algorithm 1.

The details of the two phases are provided in the following, ﬁrst the meta-
training phase then the task-training phase.

Meta-training phase: We follow the methodology called ODC, an effective
discrimitive self-supervised learning method, proposed in [29]. The ODC is an
online update deep clustering algorithm, which improves the DeepCluster [25]
in the way of pseudo-label update. It discards the clustering after each epoch
of training, and chooses to update the image features and pseudo-labels at the
same time during the training process. ODC adds sample memory and centroids
memory to DeepCluster to achieve the online update of pseudo-labels.

In a training process of ODC, there are three main steps: update network
parameters, update sample memory and update centroids memory. During training
we adopt the simple cross entropy loss as the loss function. Through the self-
supervised learning we obtain a feature embedding, which can get features with
semantic information.

Task-training phase: In this phase we aim to obtain the well trained
prototypes for the distance measure. Therefore we need to train a classifier
firstly as the base neural network to train the prototypes. Specifically we ﬁx
the meta-trained parameters in the backbone and train a task-dependent
classiﬁer $Cls_{\theta}$ on the transferred representations of the few-shot
task’s dataset namely $D_{task}$. Before training a linear classiﬁer with
$D_{s}$ having few labeled examples, a method similar to simpliﬁed graph
convolution [12], namely the graph aggregation is used according to [11]. We
construct the graph $G_{task}(V,E)$ for the current few-shot task through the
steps introduced before.

Then we propagate feature ([12]) to obtain new features for each vertex:

$V^{new}=\left(\alpha I+E)^{\gamma}V\right.$ (4)

where $I$ is the identity matrix and $\gamma$ is a hyperparameter denoting the
number of times to aggregate feature. At the same time, $\alpha$ is also a key
value to balance between the neighbors representations and the self-ones.

After aggregation, we use the labeled part of the vertices to train the task-
dependent classifier $Cls_{\theta}$, a simple fully connected network. Then we
use the trained network $Cls_{\theta}$ with loss function specially designed
for prototypes training to do the backpropagation and further obtain the
trained prototypes. The design for loss function is introduced in details in
III-C. After finishing the task-training, we have the helpful class prototypes
to do distance measure. As done in the method of usual metric learning
algorithms, we classify the query image into the class corresponding to the
prototype with the closest distance to the feature of this query example. To
improve the classification accuracy we design a simple attention mask add to
the query examples’ features before the calculation of distance which is
described in III-D.

###  III-C Loss Design for Trainable Prototypes

In the metric learning algorithm, the quality of the prototypes determine the
classification accuracy of the distance metric, but the method of directly
averaging the support examples features as in the [2] cannot achieve the best
results. We proposes trainable prototypes and consider it as a network
parameter to participate in the training, so that you can get the prototypes
that cannot design artificially. During the prototypes training, the design of
loss function is very important. There are two requirements for the good
prototypes: one is that it can be correctly classified by a trained
classification head, and the other is that it can effectively improve the
accuracy of metric classification. According to these two aspects, the
corresponding loss function can be designed respectively.

The first term of the total loss is the cross-entropy loss of prototypes
classification. In the calculation loss, the groundtruth of the class
prototype is the one-hot form of its corresponding category which is a vector
of length N, each category corresponds to a dimension in the vector, the value
is 1 in the dimension representing the category to which it belongs, and the
value is 0 in other dimensions. The classification loss can be described as
follows:

$L_{class}=\frac{1}{N}{\sum\limits_{i=1}^{N}{-\frac{1}{N}\left({\sum\limits_{c=1}^{N}{y_{i,c}log\left({\hat{y}}_{i,c}\right)}}\right)}}$
(5)

where ${\hat{y}}_{i,c}$ is the value of the prediction vector of the $i$-th
class prototype in the $c$-th dimension, $y_{i,c}$ is the value on the $c$-th
dimension of the one-hot label of the $i$-th class prototype.

In order to make the training process converge, it is not enough to use only
the cross entropy loss, and an entropy loss of the prototypes classification
result must be designed. Because the last layer of the classification head is
the softmax layer, so that the classification result is a probability
distribution. The value of each dimension represents the probability that the
prototypes belongs to the corresponding class. We believe that the higher
credibility of the classification result of the prototype is better.
Therefore, the value of the classification result in the corresponding
dimension of the correct category should be close to 1, and the value of other
positions should be close to 0. In order to strengthen this trend, Entropy
loss can be designed as follows:

$L_{entropy}=\frac{1}{N}{\sum\limits_{i=1}^{N}{-\frac{1}{N}\left({\sum\limits_{c=1}^{N}{{\hat{y}}_{i,c}log\left({\hat{y}}_{i,c}\right)}}\right)}}$
(6)

Entropy loss hopes that the smaller the entropy of the prediction result, the
better, which ensures that the prediction result tends to the one-hot vector
and strengthens the training tendency. In addition to the above two losses,
cross-entropy loss needs to be calculated for the metric classification
results of the support examples, which is called metric loss. This loss is to
ensure that the class prototypes have a good distance metric classification
ability. Since the query examples has no label, the support examples are used
instead in calculating the metric loss during training. We write the metric
loss as:

$\begin{split}L_{metric}=\frac{1}{N\times S}{\sum\limits_{i=1}^{N\times S}}\\\
{{-\frac{1}{N}\left({\sum\limits_{c=1}^{N}{y_{i,c}log\left(softmax\left({\cos\left({V_{i,:}^{new},~{}{proto}_{c}}\right)}\right)\right)}}\right)}}\end{split}$
(7)

where the ${proto}_{c}$ represents the prototype of the $c$-th category. The
total loss is the weighted summation of these three items expressed as
follows:

$L_{total}=\lambda L_{entropy}+\delta L_{class}+L_{metric}$ (8)

###  III-D Attention Mechanism

We aim to pay more attention to important features while ignoring other
content that are not vital for distance measure by adding attention mechanism
to metric learning. Therefore, we introduce the attention mask used to correct
the query examples features. Because the number of labeled support examples at
task-training phase is small, the network parameters involved in this process
cannot be increased, otherwise it will cause serious overfitting. The purpose
of adding an attention mechanism in the metric classification is to highlight
the features channel related to a certain category in both the query example
feature and the corresponding prototype, so as to obtain the corrected query
feature that is easier to classify. For this purpose, it is found that the
class prototype trained in task-training is a natural attention mask.

The channel corresponding to a larger value in a prototype represents an
important feature that distinguishes this class from other classes.
Multiplying the prototype as the attention mask with the query example feature
can highlight the channel related to this class in the query feature. If the
query image belongs to this category, multiply the mask and then do the
distance measure to get a larger similarity value, which can be better
distinguished from other categories; if the query image does not belong to
this category, then the similarity value will be smaller, and a better
classification accuracy can also be achieved.

Therefore the attention mask are proposed as follows:

${atten\\_mask}_{n}=Softmax\left(\mu*abs\left({proto}_{n}\right)\right)$ (9)

where ${atten\\_mask}_{n}$ represents the attention mask corresponding to the
$n$-th category and $\mu$ is a scale factor. Then we use the attention mask to
correct the query example features:

${query}_{n}=\varepsilon*query*{atten\\_mask}_{n}+query$ (10)

where ${query}_{n}$ is the query example features after correcting by the
attention mask of the $n$-th category. $query$ is the aggregated query
features. $\varepsilon$ is a weighting factor set as 10000 in our experiments.

In addition to adding the attention mask in the task-training phase, we also
add the attention module SENet[30] to the network in the meta-training phase,
and verify its effectiveness through experiments.

##  IV Experiments

We conduct experiments on the widely used few-shot image classification
benchmark: miniImageNet [1], which is a derivative of ImageNet.

###  IV-A Models and implementation details

Architecture. In the meta-training phase we use ResNet50[31] as the structure
of feature extractors $f_{\varphi}$. This backbone has 50 convolutional layers
grouped into 16 blocks. And in every block we add a SE block. We set the input
size as $224\times 224$ and flatten the outputs as inputs to the graph
aggregation, so that $e=2048$ in $V\in R^{[N\times(k+q)]\times e}$.

In consideration of the extreme few labeled examples we take only one fully
connected layer and a following softmax layer as the structure of the
classifier $Cls$ to avoid overfitting. And in the metric classification we
measure the feature distance by calculating the cosine between two vectors.

Optimization and hyper-parameters setup. For the meta-training phase, we train
the backbone in a total of 400 epochs from scratch using the SGD optimizer
[32] and the cross-entropy loss. In order to get a better model we also adopt
early stopping. We set the total clusters number as 1680 while the unlabled
samples used in ODC composed of 168 categories. For the task-training phase,
we train the classifier and the prototypes in 11 epochs 1000 epochs
respectively, using the Adam optimizer [33] and the loss function shown in the
previous section.

TABLE I: Performance of TrainProto-FSL in comparison to the previous works on miniImageNet on 5-way 1-shot and 5-way 5-shot tasks. Average accuracies are reported with 95% conﬁdence intervals. |  5-way Accuracy
---|---
miniImageNet |  1-shot |  5-shot
|  CACTUs-MAML [15] |  39.90$\pm$0.74% |  53.97$\pm$0.70%
|  CACTUs-ProtoNets [15] |  39.18$\pm$0.71% |  53.36$\pm$0.70%
|  UFLST [14] |  33.77$\pm$0.70% |  45.03$\pm$0.73%
|  UMTRA [17] |  39.93$\pm-$% |  50.73${\pm-}$%
unsupervised |  AAL-ProtoNets [16] |  37.67$\pm$0.39% |  40.29$\pm$0.68%
AAL-MAML++ [16] |  34.57$\pm$0.74% |  49.18$\pm$0.47%
|  ULDA-ProtoNets [18] |  40.63$\pm$0.61% |  55.41$\pm$0.57%
|  ULDA-MetaOptNet [18] |  40.71$\pm$0.62% |  54.49$\pm$0.58%
|  CSSL-FSL_Image168[11] |  54.17$\pm$1.31% |  68.91$\pm$0.90%
|  TrainProto-FSL(ours) |  58.92$\pm$0.91% |  73.94$\pm$0.63%
supervised |  MAML |  46.60$\pm$0.74% |  60.00$\pm$0.71%
ProtoNets |  47.01$\pm$0.72% |  67.90$\pm$0.76%

###  IV-B Results on miniImageNet

The miniImageNet dataset consists of 100 classes randomly sampled from the
ImageNet and each class contains 600 images of size $84\times 84$. It is
usually divided into three parts [8]: training set with 64 base classes,
validation set with 16 classes, and testing set with 20 novel classes. In the
meta-training phase we use 168 classes randomly chosen from ImageNet according
to [11] to get a well feature extractor, having no labels. In the task-
training phase we sample novel classes to design few-shot tasks as inputs. We
ensure that the novel classes have never been seen in the meta-training phase.

We evaluate our method on 1000 randomly sampled tasks and report their mean
accuracy in TABLE I. We compare our method in both 5-way 1-shot and 5-way
5-shot setting with some classical supervised few-shot learning methods and
novel unsupervised methods proposed recently. It can be found that our method
is much better than previous unsupervised few-shot learning methods( [14]
etc.), improving them by more than 18%. The CSSL-FSL[11] also use non-episodic
meta-training to train a feature extractor via a contrastive self-supervised
algorithm. Our method improves it by more than 4%. Even compared with
supervised methods([5] and [2]), our method still has improvement by 2-11%
both on 5-way 1-shot and 5-way 5-shot tasks.

###  IV-C Ablation experiments

In this section, we conduct ablation experiments to analyze how trainable
prototypes and attention mechanism affects the few-shot image classification
performance. TABLE II shows the results of the ablation studies on
miniImageNet in 5-way 1-shot, 5-way 5-shot, 5-way 10-shot and 5-way 30-shot
setting. In the table, without TrainProto means that the mean value of support
features is directly calculated as the prototype to participate in the
distance measure and without AttenMask means don’t use attention mask to
correct the query features. From the table we can see the original model
performs best. Attention mask before the distance measure improves accuracy by
0.1%. When the number of shots is 1 or 5, the improvement in accuracy brought
by the AttenMask is more obvious, indicating that the attention mask can
achieve greater advantages when there are few label samples.

The SENet used in feature extractor can provide about 1% extra gain. And
without TrainProto, the result will decrease by 2-6%. The reason is that
TrainProto is actually equivalent to obtaining the class prototypes through a
very complicated function, which cannot be designed manually. Therefore, the
prototypes obtained by training must be better than those obtained by directly
averaging.

TABLE II: Results of ablation studies on miniImageNet. The meta-training dataset consists of 168 classes from ImageNet for all the four models. SENet | AttenMask | TrainProto | 1-shot | 5-shot | 10-shot | 30-shot
---|---|---|---|---|---|---
|  | $\surd$ | 57.90$\pm$0.86% | 72.55$\pm$0.62% | 77.22$\pm$0.52% | 82.96$\pm$0.41%
$\surd$ |  | $\surd$ | 58.82$\pm$0.91% | 73.83$\pm$0.63% | 77.13$\pm$0.55% | 83.09$\pm$0.43%
$\surd$ | $\surd$ |  | 59.00$\pm$0.70% | 71.77$\pm$0.52% | 74,54$\pm$0.47% | 77.02$\pm$0.43%
$\surd$ | $\surd$ | $\surd$ | 58.92$\pm$0.91% | 73.94$\pm$0.63% | 77.22$\pm$0.55% | 83.10$\pm$0.43%

###  IV-D Other Experiments

In this section, we compare the effects of two different attention modules on
meta-training. We added Non-Local(NL) [34] and SENet modules(SE) to the ODC
backbone network respectively, and compared their effects on the accuracy of
few shot classification after training. The results are shown in TABLE III. We
also compares the algorithm without attention module(w/o Atten) in meta-
training with the two improved algorithms mentioned above.

Generally speaking, the addition of SENet and Non-local modules can improve
the performance of the algorithm, and the improvement of SENet is more than
that of Non-local. On the few-sample classification tasks with shot=1 and 5,
the two attention modules brought the higher improvement in classification
accuracy, reaching a maximum of 1.2%. But when shot=10, the SENet module
brings a 0.06% decrease in accuracy, and when shot=30, the Non-local module
brings a 0.38% decrease in accuracy. In general, these two attentions have
played a positive role in the training of the feature extractor, and the
improvement of SEnet is more obvious especially when shot is extreme small.

TABLE III: Performance of TrainProto-FSL with different attention modules on miniImageNet on different types of tasks. For each type of task, the best-performing method is in bold. 5-way Accuracy | 1shot | 5shot | 10shot | 20shot | 30shot
---|---|---|---|---|---
Ours_w/o Atten | 57.94% | 72.67% | 77.28% | 81.17% | 82.97%
Ours_NL | 58.29% | 73.23% | 77.42% | 81.37% | 82.59%
Ours_SE | 58.92% | 73.94% | 77.22% | 81.34% | 83.10%

##  V Conclusion

A novel method of obtaining prototypes for metric classification called
TrainProto-FSL is proposed in this paper, which contains classifier training
and prototypes training. And the whole framework designed as [11] helps our
algorithm avoid the shortcomings related with the traditional episodic meta-
training.

Experiments show a state-of-the-art performance on a standard vision dataset
miniImageNet. It proves that the trained prototypes get from neural network
are more efficacious than the hand-designed ones, and the attention mechanism
is also useful for the performance improvement. In the following study we will
explore the effective self-supervised learning algorithm specially for few
shot learning.

## References

  * [1] O. Vinyals, C. Blundell, T. Lillicrap, D. Wierstra _et al._ , “Matching networks for one shot learning,” in _Advances in neural information processing systems_ , 2016, pp. 3630–3638.
  * [2] J. Snell, K. Swersky, and R. Zemel, “Prototypical networks for few-shot learning,” in _Advances in neural information processing systems_ , 2017, pp. 4077–4087.
  * [3] F. Sung, Y. Yang, L. Zhang, T. Xiang, P. H. Torr, and T. M. Hospedales, “Learning to compare: Relation network for few-shot learning,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 1199–1208.
  * [4] B. Oreshkin, P. R. López, and A. Lacoste, “Tadam: Task dependent adaptive metric for improved few-shot learning,” in _Advances in Neural Information Processing Systems_ , 2018, pp. 721–731.
  * [5] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in _Proceedings of the 34th International Conference on Machine Learning-Volume 70_. JMLR. org, 2017, pp. 1126–1135.
  * [6] A. A. Rusu, D. Rao, J. Sygnowski, O. Vinyals, R. Pascanu, S. Osindero, and R. Hadsell, “Meta-learning with latent embedding optimization,” _arXiv preprint arXiv:1807.05960_ , 2018.
  * [7] H. Li, W. Dong, X. Mei, C. Ma, F. Huang, and B.-G. Hu, “Lgm-net: Learning to generate matching networks for few-shot learning,” _arXiv preprint arXiv:1905.06331_ , 2019.
  * [8] S. Ravi and H. Larochelle, “Optimization as a model for few-shot learning,” _International Conference on Learning Representations_ , 2016.
  * [9] Y. Liu, J. Lee, M. Park, S. Kim, E. Yang, S. J. Hwang, and Y. Yang, “Learning to propagate labels: Transductive propagation network for few-shot learning,” _arXiv preprint arXiv:1805.10002_ , 2018.
  * [10] J. Kim, T. Kim, S. Kim, and C. D. Yoo, “Edge-labeling graph neural network for few-shot learning,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 11–20.
  * [11] J. Li and G. Liu, “Few-shot image classification via contrastive self-supervised learning,” _arXiv preprint arXiv:2008.09942_ , 2020.
  * [12] F. Wu, T. Zhang, A. H. d. Souza Jr, C. Fifty, T. Yu, and K. Q. Weinberger, “Simplifying graph convolutional networks,” _arXiv preprint arXiv:1902.07153_ , 2019.
  * [13] Y. Lifchitz, Y. Avrithis, S. Picard, and A. Bursuc, “Dense classification and implanting for few-shot learning,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 9258–9267.
  * [14] Z. Ji, X. Zou, T. Huang, and S. Wu, “Unsupervised few-shot learning via self-supervised training,” _arXiv preprint arXiv:1912.12178_ , 2019.
  * [15] K. Hsu, S. Levine, and C. Finn, “Unsupervised learning via meta-learning,” _arXiv preprint arXiv:1810.02334_ , 2018.
  * [16] A. Antoniou and A. Storkey, “Assume, augment and learn: Unsupervised few-shot meta-learning via random labels and data augmentation,” _arXiv preprint arXiv:1902.09884_ , 2019.
  * [17] S. Khodadadeh, L. Boloni, and M. Shah, “Unsupervised meta-learning for few-shot image classification,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 10 132–10 142.
  * [18] T. Qin, W. Li, Y. Shi, and Y. Gao, “Unsupervised few-shot learning via distribution shift-based augmentation,” _arXiv preprint arXiv:2004.05805_ , 2020.
  * [19] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. A. Pires, Z. D. Guo, M. G. Azar _et al._ , “Bootstrap your own latent: A new approach to self-supervised learning,” _arXiv preprint arXiv:2006.07733_ , 2020.
  * [20] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in _International conference on machine learning_. PMLR, 2020, pp. 1597–1607.
  * [21] R. Zhang, P. Isola, and A. A. Efros, “Colorful image colorization,” in _European conference on computer vision_. Springer, 2016, pp. 649–666.
  * [22] A. v. d. Oord, O. Vinyals, and K. Kavukcuoglu, “Neural discrete representation learning,” _arXiv preprint arXiv:1711.00937_ , 2017.
  * [23] M. Noroozi and P. Favaro, “Unsupervised learning of visual representations by solving jigsaw puzzles,” in _European conference on computer vision_. Springer, 2016, pp. 69–84.
  * [24] S. Gidaris, P. Singh, and N. Komodakis, “Unsupervised representation learning by predicting image rotations,” _arXiv preprint arXiv:1803.07728_ , 2018\.
  * [25] K. Tian, S. Zhou, and J. Guan, “Deepcluster: A general clustering framework based on deep learning,” in _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_. Springer, 2017, pp. 809–825.
  * [26] Y. Tian, D. Krishnan, and P. Isola, “Contrastive multiview coding,” _arXiv preprint arXiv:1906.05849_ , 2019.
  * [27] Y. Hu, V. Gripon, and S. Pateux, “Exploiting unsupervised inputs for accurate few-shot classification,” _arXiv preprint arXiv:2001.09849_ , 2020.
  * [28] T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” _arXiv preprint arXiv:1609.02907_ , 2016.
  * [29] X. Zhan, J. Xie, Z. Liu, Y.-S. Ong, and C. C. Loy, “Online deep clustering for unsupervised representation learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 6688–6697.
  * [30] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7132–7141.
  * [31] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [32] L. Bottou, “Large-scale machine learning with stochastic gradient descent,” in _Proceedings of COMPSTAT’2010_. Springer, 2010, pp. 177–186.
  * [33] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [34] X. Wang, R. Girshick, A. Gupta, and K. He, “Non-local neural networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7794–7803.