# Beyond Simple Meta-Learning:
Multi-Purpose Models for Multi-Domain, Active and Continual Few-Shot Learning

Peyman Bateni1,9,∗, Jarred Barber7, Raghav Goyal1,4, Vaden Masrani1,
Jan-Willem van de Meent5,6, Leonid Sigal1,4,8,9, Frank Wood1,2,3,8
∗Correspondence to<EMAIL_ADDRESS>of British Columbia,
2Inverted AI, 3Mila,
4Vector Institute, 5University of Amsterdam, 6Northeastern University,
7Amazon, 8CIFAR AI Chair, 9Beam AI

###### Abstract

Modern deep learning requires large-scale extensively labelled datasets for
training. Few-shot learning aims to alleviate this issue by learning
effectively from few labelled examples. In previously proposed few-shot visual
classifiers, it is assumed that the feature manifold, where classifier
decisions are made, has uncorrelated feature dimensions and uniform feature
variance. In this work, we focus on addressing the limitations arising from
this assumption by proposing a variance-sensitive class of models that operate
in a low-label regime. The first method, Simple CNAPS, employs a
hierarchically regularized Mahalanobis-distance based classifier combined with
a state of the art neural adaptive feature extractor to achieve strong
performance on Meta-Dataset, mini-ImageNet and tiered-ImageNet benchmarks. We
further extend this approach to a transductive learning setting, proposing
Transductive CNAPS. This transductive method combines a soft k-means parameter
refinement procedure with a two-step task encoder to achieve improved test-
time classification accuracy using unlabelled data. Transductive CNAPS
achieves state of the art performance on Meta-Dataset. Finally, we explore the
use of our methods (Simple and Transductive) for “out of the box” continual
and active learning. Extensive experiments on large scale benchmarks
illustrate robustness and versatility of this, relatively speaking, simple
class of models. All trained model checkpoints and corresponding source codes
have been made publicly available9. ††thanks: 10Code available at
github.com/plai-group/simple-cnaps

###### Index Terms:

Few-Shot Image Classification, Meta-Learning, Learning with Limited Labels,
Metric Learning

##  1 Introduction

Deep neural networks have facilitated transformative advances in machine
learning [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]. However, much of their success
depends on training using large-scale extensively-labelled datasets, and when
an exhaustive set of training examples is not available, performance degrades
significantly. Few-shot learning [11, 12, 13, 14] seeks to address this issue
by developing architectures for learning effectively from few labelled
instances. For a novel task, a few-shot visual classifier is derived using a
small number of labelled “support” images per category, and is then evaluated
on a set of unlabelled “query” images.

(a) Euclidean (b) Mahalanobis (c) Transductive

(d) Active Learning

(e) Continual Learning

Figure 1: Two-dimensional illustration of task-adapted support image features
are shown in the top row. When using Euclidean distance (a) as the metric,
each cluster is assumed to have identity covariance, leading to the incorrect
classification of query examples. Mahalanobis distance (b) resolves this
issue. In the transductive setting (c), all query examples are labelled all at
once, permitting semi-supervised refinement of clusters. During active
learning (d), methods can iteratively request labels from a pool of unlabelled
examples. In continual learning (e), models see a new task at each iteration,
aiming to perform well without forgetting previous tasks.

Existing few-shot classification approaches can be divided into two
categories. First, there are approaches based on nearest-neighbour
classification [15], either directly in the feature space [16, 17, 18] or the
mapping of the feature vectors onto the natural language space [19]. The
second group consists of methods that distill examples into class-wise vector
prototypes that are either learned [20, 21] or mathematically derived by mean-
pooling the examples [22]. The prototypes are usually defined in the feature
or the natural language space (e.g. word2vec [23]). Most research in this area
has focused on learning non-linear mappings (typically in the form of deep
multi-layer neural networks) from the images to a high-dimensional vector
space, namely the embedding space. A pre-specified metric within the embedding
space is then used for final nearest-class classification (such as the cosine
similarity between the feature embedding of the query image and the class
embedding). Significant work has focused on effective task-conditioned
adaptation of these non-linear mappings at different levels of granularity
[24, 21, 25, 26]. Recently, Conditional Neural Adaptive Processes (CNAPS) [21]
achieved high few-shot visual classification accuracy through the use of
sparse FiLM [27] layers for partial network adaptation to prevent over-fitting
problems that arise from adapting the entire embedding neural network using
few support examples.

Overall, far less attention has been given to the metric used to compute
distances for classification within the embedding space. Snell et al. [22]
study the underlying distance function in order to justify the use of sample
means as prototypes. They argue that Bregman divergences [28] are the
theoretically sound family of metrics to be used in the few-shot setting, but
only utilize a single instance within this family — the squared Euclidean
distance, which they find to perform better than the more traditional cosine
metric. However, the choice of the Euclidean metric involves making two
assumptions: 1) that feature dimensions are un-correlated and 2) that they
have uniform variance. In addition, the Euclidean distance is insensitive to
the distribution of within-class samples with respect to their prototype and
recent results [29, 22] suggest that this is problematic. Modelling this
distribution (in the case of [28] using extreme value theory) is, as we
observe, a key to achieving better accuracy.

Starting from this intuition, we develop our first method, the “Simple CNAPS”
architecture that achieves a 6.1% improvement over CNAPS [21] while removing
788,485 parameters (3.2% of the total) from the original CNAPS architecture,
replacing them with fixed, closed-form, deterministic covariance estimation
and Mahalanobis distance computations. We find that surprisingly, we are able
to generate useful high-dimensional estimates of covariance even in the few-
shot classification setting, where the number of available support examples
per class is in theory far too small to estimate the required class-specific
covariances.

In a standard few-shot learning setting, the classifier is adapted using
labelled examples in the support set. Performance can be improved further by
exploiting additional unlabelled support data (semi-supervised few-shot
learning) [30], or examples in the query set (transductive few-shot learning)
[31, 16]. Existing transductive few-shot methods reason about unlabelled
examples by performing k-means clustering with Euclidean distance [30] or
message passing in graph convolutional networks [31, 16]. These methods, while
improving performance, lack the expressibility of the Mahalanobis-distance
based classification framework used in Simple CNAPS. Furthermore, they
primarily focus on tranductive adaptation within the classification space,
disregarding tranductive adaptation of the feature extractor all together.

Figure 2: Overview of research on few-shot image classification, organized by
image feature extractor adaptation scheme (_vertical axis_) versus final
classification methodology (_horizontal axis_).

Motivated by these observations, we develop a transductive variant of the
Simple CNAPS architecture. In this variant, we infer labels for query set
examples, which allows us to make use of these examples in class parameter
estimates. The resulting architecture, namely “Transductive CNAPS”, extends
Simple CNAPS with a transductive two-step task-encoder, and an iterative soft
k-means procedure for refining class parameter estimates (mean and
covariance). We demonstrate that this approach achieves state of the art
performance on Meta-Dataset [32]. In addition, Transductive CNAPS achieves
notable performance on mini-ImageNet [22] and tiered-ImageNet [30] benchmarks.

Furthermore, Requeima et al. [21] recently proposed to evaluate pre-trained
few-shot classifiers for active learning and continual learning “out of the
box” without any additional problem specific fine-tuning. Following their
work, we also explore Simple and Transductive CNAPS [33, 34, 35] in the
context of “out of the box” active learning and continual learning. In the
active learning setting, a meta-trained few-shot classifier is presented with
a set of unlabelled examples, from which it can acquire the label for a single
example at each iteration. The goal is to select the examples to maximally
improve performance on a separate set of test examples. We evaluate both
methods without any additional training specific to active learning, and
demonstrate that uncertainty-driven selection methodologies outperform random
selection in our models, thus demonstrating their abilities in producing
effective measures of uncertainty.

In the continual learning problem domain, a few-shot learning method is
presented with a new classification task at each iteration. The goal is to
continuously learn to classify accurately on these new tasks while maintaining
good accuracy on the previous tasks. Continual learning requires this to be
achieved without explicitly saving every data point to memory. In our work, we
propose and explore three continual learning strategies within the framework
of our methods.

Figure 3: Overview of neural adaptive feature extraction in Simple,
Transductive and ”original” CNAPS. Figure adapted from [34].

Our contributions: (1) We describe “Simple CNAPS”, a neural adaptive few-shot
classifier with a regularized Mahalanobis based classifier that achieves
strong performance on Meta-Dataset. (2) We expand our work to transductive
few-shot learning by developing “Transductive CNAPS”, which extends our first
method with a two-step transductive set-encoder and a soft-k means iterative
procedure for refinement of class parameters. This method achieves state of
the art (SoTA) performance on Meta-Dataset. (3) In addition to grounding our
work in probabilistic mixture models, we provide an extensive discussion and
interpretation of both architectures as Riemannian metric learners. This
analysis is consistent with our empirical results. (4) We evaluate both
methods on “out of the box” active learning using three active label
acquisition strategies. We also modify Simple and Transductive CNAPS for “out
of the box” continual learning, proposing three approaches to continual
estimation of the task-encoding that achieve competitive performance. (5)
Finally, through extensive experiments / ablations we study importance of our
design choices.

##  2 Related Work

###  2.1 Few-Shot Learning with Labelled Data

Past research on few-shot classification [12] can be differentiated along two
major axes: 1) how images are transformed into vectorized embeddings, and 2)
how “distances” are computed between vectors in order to assign labels. This
is illustrated in Figure 2.

Siamese networks [17], an early approach to few-shot learning, employed a
shared feature extractor to produce vector embeddings for both the support and
the query images. Classification was then done by picking the smallest
weighted L1 distance between query and labelled image embeddings. Relation
networks [36], and recent GCNN variants [16, 18], extended this by
parameterizing and learning the classification metric using a Multi-Layer
Perceptron (MLP). Matching networks [15] learned distinct feature extractors
for support and query images which were then used to compute cosine
similarities for classification.

The feature extractors used by these models were, notably, not adapted to
test-time classification tasks. It has become established that adaptation of
the feature extractor to new tasks at test time is generally a good thing to
do. Fine tuning transfer-learned networks [37] did this by fine-tuning the
feature extraction network using the task-specific support images but found
limited success due to problems related to overfitting to, the generally very
few, support examples. MAML [24] (and its various extensions [38, 26, 25])
mitigated this problem by learning a set of meta-parameters that specifically
enabled feature extractors to be adapted to new tasks given few support
examples using few gradient descent steps.

The two supervised few-shot learning approaches most directly related to this
work are CNAPS [21] (and the related TADAM [29]) and Prototypical Networks
[22]. CNAPS is a few-shot adaptive visual classifier based on conditional
neural processes (CNPs) [39]. It is a high-performing approach for few-shot
image classification [21] that uses a pre-trained feature extractor augmented
with FiLM layers [27] that are adapted for each task using the support images
specific to that task. CNAPS uses a dot-product distance in a final linear
classifier; the parameters of which are also adapted at test-time to each new
task (see Section 4.1).

Prototypical Networks [22] employ a non-adaptive feature extractor with a
simple mean-pooling operation to form class “prototypes” from support
instances in a meta-learned feature space. They employ episodic training to
learn said feature extractor, where at each training iteration, a few-shot
task is provided with a “support” set for adaptation and a “query” set for
evaluation and calculating a classification loss that is then used for
gradient descent. Squared Euclidean distances to these prototypes are
calculated and subsequently used for nearest-class classification. Their
choice of the distance metric was motivated by the theoretical properties of
Bregman divergences [28], a family of functions of which the squared Euclidean
distance is a member of. These properties allow for a correspondence between
the use of the squared Euclidean distance in a Softmax classifier and
performing density estimation. Expanding on the work of Snell et al.[22], we
also exploit similar properties of the squared Mahalanobis distance as a
Bregman divergence [28] to draw theoretical connections to Bregman soft-
clustering. We additionally provide an extensive alternative theoretical
grounding of both methods in Riemannian metric learning.

Our work differs from CNAPS [21] and Prototypical Networks [22] in the
following ways. First, while CNAPS has demonstrated the importance of adapting
the feature extractor to a specific task, we show that adapting the classifier
is actually unnecessary to obtain good performance. Second, we demonstrate
that an improved choice of Bregman divergence can significantly impact
accuracy. Specifically, we demonstrate that regularized class-specific
covariance estimation from task-specific adapted feature vectors allows for
use of the Mahalanobis distance for classification, achieving significant
improvements in performance and establishing state of the art in the case of
Transductive CNAPS.

Our methods also relate to recent works [40, 41] in modelling per-class
clusters with Gaussian means and covariance estimates. Fort et al. [40]
extended Prototypical Networks to use the Mahalanobis distance by
incorporating class-wise diagonal covariance estimates generated by a learned
network. Unlike their work, our methods produce per-class closed-form full
covariance estimates within a conditional neural adapted feature space, all
trained end-to-end, making them especially effective empirically. Yang et al.
[41] produce class-wise Gaussian distributions within a fixed pre-trained
feature space. These distributions are first fine-tuned using mixtures of
Gaussian statistics from visually similar classes. They are then used to
sample new support examples within the feature space, resulting in an
augmented support set that is then used to train a logistic regression
classifier for the task at hand. Although we similarly regularize the class-
wise covariance estimates using the task-level covariance, our methods do not
retain any Gaussian statistics about previous tasks/classes, relying solely on
the support examples provided. Furthermore, we adapt the underlying feature
space through an end-to-end learned adaptation procedure, resulting in
empirically useful higher-dimensional mean and covariance estimates from the
very few available support examples. Lastly, we use the Mahalanobis distance
within the distribution space itself to perform classification, incorporating
inter/intra-class variances within the decision boundaries.

###  2.2 Few-Shot Learning with Unlabelled Data

The use of unlabelled instances for transductive few-shot learning has also
been the subject of study by several methods [16, 31, 30]. EGNN [16] employs a
graph convolutional edge-labelling network for iterative propagation of labels
from support to query instances. Similarly, TPN [31] learns a graph
construction module for neural propagation of soft labels between elements of
the query set. These methods rely on neural parameterizations of distance in
the feature space. TEAM [42] uses an episode-wise transductively adaptable
metric for performing inference on query examples using a task-specific
metric. Song et al. [43] use a cross attention network with a transductive
iterative approach for augmenting the support set using the query examples.

Figure 4: Transductive CNAPS extends the Mahalanobis-distance based classifier
in Simple CNAPS through transductive soft k-means clustering of the visual
space.

The closest approach to our transductive work, namely Transductive CNAPS, is
that of Ren et al. [30]. Their method extends Prototypical Networks [22]
through incorporating a single further soft-labelled weighted estimation of
class prototypes. Transductive CNAPS, on the other hand, differs in three
major ways. First, we produce soft-label estimates of both the mean and
covariance. Second, we use an expectation-maximization (EM) inspired algorithm
that performs a dynamic number of soft-label updates, depending on the task at
hand. Lastly, we employ a neural-adaptive procedure for feature extraction
that is conditioned on a two-step learned transductive task representation, as
opposed to a fixed feature-extractor. This novel task-representation encoder
is responsible for significant performance gains on out-of-domain tasks (Sec.
VI).

###  2.3 Active Learning

Active learning is a major research paradigm in machine learning that focuses
on requesting labels for unlabelled examples such that performance gain is
maximized. Specifically, active learning aims to make data-labelling part of
the learning process itself, such that samples are chosen for labelling by the
model.

Existing methods focus on 3 major label selection strategies. First, there are
uncertainty-based methods [44, 45, 46, 47, 48, 49] where sample selection is
performed using class-wise probabilities as measure for model uncertainty on
said samples. Second are diversity-based methods [50, 51, 52, 53] where
selection is performed such that diversity among categories and their labelled
examples is maximized. Third, approaches have also been proposed [54, 55, 56]
to use expected model change as the criterion for sample selection, where
unlabelled examples are ranked on the basis of expected parametric change to
the model. Dataset-independent sample selection based on uncertainty, where
the uncertainty of any unlabelled example is not dependent on the rest of the
set, can lead to sampling bias. However, focusing on strategies that promote
diversity may result in limited performance gain relative to the number of
labels obtained. Motivated by these observations, hybrid instance selection
strategies [57, 58, 59] have been proposed that use mixtures of uncertainty-
based, diversity-based and expected-update criteria for active learning,
leveraging each of their strengths while minimizing their weaknesses.

Although active learning has been the subject of much study, “out of the box”
use of few-shot classifiers for active learning has only been explored
recently [21, 60]. Requeima et al. demonstrate that CNAPS [21], when used for
“out of the box” active learning, is able to outperform Prototypical Networks
[22]. The margins of improvement is even greater when uncertainty-based
approaches for label acquisition is employed as oppose to random selection.
Pezeshkpour et al. [60] study the effects of various active few-shot learning
strategies within the context of Simple CNAPS and Prototypical Networks,
ultimately concluding that better strategies need to be developed for
effective active few-shot learning.

###  2.4 Continual Learning

Continual learning focuses on developing methods that can learn to perform new
tasks without “forgetting” previous ones. In particular, it studies the
problem of learning from an infinite stream of tasks, where the goal is to
gradually acquire knowledge, using it for future learning without loss of
existing knowledge. Most previous work propose different criterions and
strategies for gradually updating sections of the learned network as new
tasks, and by consequence, labelled examples are seen. Zenke et al. [61]
propose Synaptic Intelligence (SI), a network of synapses with complex three-
dimensional state-spaces, tracking the past and the current parameter values.
Online estimates of each synapse’s “importance” is employed to consolidate
important synapses by preventing them from future learning, while allocating
unimportant synapses to learn from future tasks. EWC [62] slows down learning
on select parameters based on how important they are to previously seen tasks,
thus preserving the learned knowledge. VCL [63] fuses online variational
inference (VI) with Monte Carlo VI to train deep discriminative and generative
models in complex continual learning settings. Chaudhry et al. propose RWalk
[64], a generalized and more efficient EWC with a theoretically grounded KL-
divergence based perspective that achieves superior accuracy on a range of
continual learning image classification benchmarks.

CNAPS [21] presents the first instance of “out of the box” use of few-shot
image classifiers for continual learning. Here, the few-shot classifier is
first trained on Meta-Dataset [32] and then used for continual learning
without any additional training. They employ context-weighted estimates of
class mean for computing class means from old/new tasks. This is performed
with current-task only adaptation of the feature space, thus resulting in
feature manifolds that differ from each other. We refer to this strategy as
the “Moving Encoding”, and explain it in detail when discussing continual
learning in Simple and Transductive CNAPS.

##  3 Problem Definition

Figure 5: Overview of the transductive task-encoder,
$g_{\theta}(\mathcal{S},\mathcal{Q})$, used for task-adaptation in
Transductive CNAPS.

Following previous work [22, 33, 21, 24], we focus on a few-shot learning
setting where a distribution $D$ over image classification tasks
$(\mathcal{S},\mathcal{Q})$ is available for episodic training. Each task
$(\mathcal{S},\mathcal{Q})\sim D$ consists of a support set
$\mathcal{S}=\\{(\mathbf{x}_{i},y_{i})\\}_{i=1}^{n}$ of labelled images and a
query set $\mathcal{Q}=\\{\mathbf{x}_{i}^{*}\\}_{i=1}^{m}$ of unlabelled
images; the objective is to predict labels for these query examples, given the
(typically small) support set. Each query image
$\mathbf{x}_{i}^{*}\in\mathcal{Q}$ has a corresponding ground truth label
$y_{i}^{*}$ available at training time. A model will be trained by minimizing,
over some parameters $\theta$ (which are shared across tasks), the expected
query set classification loss over tasks. We minimize
$\mathbb{E}_{(\mathcal{S},\mathcal{Q})\sim
D}[-\sum_{\mathbf{x}_{i}^{*}\in\mathcal{Q}}\log
p_{\theta}(y^{*}_{i}|\mathbf{x}^{*}_{i},\mathcal{S})]$ for Simple CNAPS and
$\mathbb{E}_{(\mathcal{S},\mathcal{Q})\sim
D}[-\sum_{\mathbf{x}_{i}^{*}\in\mathcal{Q}}\log
p_{\theta}(y^{*}_{i}|\mathbf{x}^{*}_{i},\mathcal{S},\mathcal{Q})]$ to train
Transductive CNAPS. Note that the inclusion of the dependence on all of
$\mathcal{Q}$ in the transductive case allows for joint prediction of labels
for the query set, all at once. At test time, a separate distribution of tasks
generated from previously unseen images and classes is used to evaluate
performance. Let us also define shot as the number of support examples per
class, and way as the number of classes within the task.

##  4 Methods

###  4.1 CNAPS

Conditional Neural Adaptive Processes (CNAPS) [21] consist of two modules: a
feature extractor and a linear classifier, which are each task-dependent.
Adaptation is performed by meta-trained network adaptation modules that
condition on the support set.

The method uses a ResNet18 [6] architecture (Figure 3) as the feature
extractor. This ResNet18 is trained separately prior to episodic training of
the adaptation networks. Within each residual block, Feature-wise Linear
Modulation (FiLM) layers are inserted to compute a scale factor $\gamma$ and
shift $\beta$ for each output channel, using block-specific adaptation
networks $\psi_{\theta}$ that are conditioned on a task encoding. The task
encoding $g_{\theta}(\mathcal{S})$ consists of the mean-pooled feature vectors
of support examples produced by $d_{\theta}$, a separate but end-to-end
learned Convolution Neural Network (CNN). This produces a task-adapted feature
extractor $f_{\theta}$ (which implicitly depends on the support set
$\mathcal{S}$) that maps support/query images onto the corresponding adapted
feature space. We will use $\mathcal{S}_{\theta},\mathcal{Q}_{\theta}$ to
denote versions of the support/query sets where each image $\mathbf{x}$ is
mapped into its 512-dimensional feature vector representation
$\mathbf{z}=f_{\theta}(\mathbf{x})$.

Classification in CNAPS is performed by a task-adapted linear classifier where
the class probabilities for a query image $\mathbf{x}_{i}^{*}$ are computed as
$\text{softmax}(\mathbf{W}\mathbf{z}_{i}^{*}+\mathbf{b})$. The classification
weights $\mathbf{W}$ and biases $\mathbf{b}$ are produced by the classifier
adaptation network $\psi_{\theta}^{c}$ forming
$[\mathbf{W},\mathbf{b}]=[\psi_{\theta}^{c}(\boldsymbol{\mu}_{1})\;\psi_{\theta}^{c}(\boldsymbol{\mu}_{2})\;\ldots\;\psi_{\theta}^{c}(\boldsymbol{\mu}_{K})]^{T}$
where for each class $k$ in the task, the corresponding row of classification
weights is produced by $\psi_{\theta}^{c}$ from the class mean
$\boldsymbol{\mu}_{k}$. The class mean $\boldsymbol{\mu}_{k}$ is obtained by
mean-pooling the feature vectors of the support examples for class $k$
extracted by the adapted feature extractor $f_{\theta}$.

###  4.2 Simple CNAPS

Figure 6: Transductive CNAPS’ soft k-means Mahalanobis-distance based
clustering procedure. First, cluster parameters are initialized using the
support examples. Then, during cluster update iterations, the query examples
are assigned class probabilities as soft labels and subsequently, both soft-
labelled query examples and labelled support examples are used to estimate new
cluster parameters.

In Simple CNAPS, we employ the same ResNet18 for feature extraction with the
same adaptation module $\psi_{\theta}$. However, because of the novel
classification architecture we use, it becomes trained to do something
different than it does in CNAPS. This choice, like for CNAPS, allows for a
task-dependent adaptation of the feature extractor. Unlike CNAPS, we undertake
the classification step in Simple CNAPS by computing softmax of a Mahalanobis
distance111Note that we explicitly do not use the $\frac{1}{2}$ coefficient as
we experimentally verified doing so to decrease performance.,

$p(y^{*}=k\mid\mathbf{z}^{*})\propto\exp\big{(}-(\mathbf{z}^{*}-\boldsymbol{\mu}_{k})^{T}\mathbf{Q}_{k}^{-1}(\mathbf{z}^{*}-\boldsymbol{\mu}_{k})\big{)},$
(1)

of feature vector $\mathbf{z}^{*}_{i}$ relative to each class $k$ by
estimating a mean $\boldsymbol{\mu}_{k}$ and regularized covariance
$\mathbf{Q}_{k}$ in the adapted feature space, using the support instances,

$\displaystyle\boldsymbol{\mu}_{k}$
$\displaystyle=\frac{1}{n_{k}}\sum_{i}\>\mathbb{I}[y_{i}=k]\>\mathbf{z}_{i},$
(2) $\displaystyle\mathbf{Q}_{k}$
$\displaystyle=\lambda_{k}\,\boldsymbol{\Sigma}_{k}+(1-\lambda_{k})\,\boldsymbol{\Sigma}+\beta
I,$ $\displaystyle\lambda_{k}$ $\displaystyle=\frac{n_{k}}{n_{k}+1}.$ (3)

Here $\mathbb{I}[y_{i}=k]$ is the indicator function and
$n_{k}=\sum_{i}\mathbb{I}[y_{i}=k]$ is the number of examples with class $k$
in the support set $\mathcal{S}$. The ratio $\lambda_{k}$ balances a task-
conditional sample covariance $\boldsymbol{\Sigma}$ and a class-conditional
sample covariance $\boldsymbol{\Sigma}_{k}$,

$\displaystyle\boldsymbol{\Sigma}$
$\displaystyle=\frac{1}{n}\sum_{i}\\!\big{(}\mathbf{z}_{i}\\!-\\!\boldsymbol{\mu}\big{)}\big{(}\mathbf{z}_{i}\\!-\\!\boldsymbol{\mu}\big{)}^{T},$
(4) $\displaystyle\boldsymbol{\Sigma}_{k}$
$\displaystyle=\frac{1}{n_{k}}\sum_{i}\mathbb{I}[y_{i}=k]\>\big{(}\mathbf{z}_{i}\\!-\\!\boldsymbol{\mu}_{k}\big{)}\big{(}\mathbf{z}_{i}\\!-\\!\boldsymbol{\mu}_{k}\big{)}^{T}.$
(5)

where $\boldsymbol{\mu}=\frac{1}{n}\sum_{i}\mathbf{z}_{i}$ is the task-level
mean. When few support examples are available for a particular class,
$\lambda_{k}$ is small, and the estimate is regularized towards the task-level
covariance $\boldsymbol{\Sigma}$. As the number of support examples for the
class increases, the estimate tends towards the class-conditional covariance
$\boldsymbol{\Sigma}_{k}$. Additionally, a regularizer $\beta I$ (we set
$\beta=1$ in our experiments) is added to ensure invertibility.

###  4.3 Transductive CNAPS

Transductive CNAPS extends Simple CNAPS by making use of query set, both for
feature adaptation and classification. First, the task encoder $g_{\theta}$ is
extended to incorporate both a support-set embedding $\mathbf{e}_{s}$ and a
query-set embedding $\mathbf{e}_{q}$ such that,

$\displaystyle\mathbf{e}_{s}$
$\displaystyle=\frac{1}{K}\sum_{k}\frac{1}{n_{k}}\sum_{i}\mathbb{I}[y_{i}=k]\>d_{\theta}(\mathbf{x}_{i}),$
(6) $\displaystyle\mathbf{e}_{q}$
$\displaystyle=\frac{1}{n_{q}}\sum_{i*}d_{\theta}(\mathbf{x}_{i}^{*}),$ (7)

where $d_{\theta}$ is a learned CNN. The support embedding $\mathbf{e}_{s}$ is
formed by an average of the (encoded) support examples, with weighting
inversely proportional to their class counts to prevent bias from the class
imbalance. The query embedding $\mathbf{e}_{q}$ uses simple mean-pooling; both
embeddings $\mathbf{e}_{s}$ and $\mathbf{e}_{q}$ are invariant to permutations
of the corresponding support/query instances. We then process $\mathbf{e}_{s}$
and $\mathbf{e}_{q}$ through two steps of a Long Short Term Memory (LSTM)
network in the same order to generate a final transductive task representation
$g_{\theta}(\mathcal{S},\mathcal{Q})$ to be used for adaptation. This process
is visualized in Figure 5.

Second, we can interpret Simple CNAPS as a form of “supervised clustering” in
feature space; each cluster (corresponding to a class $k$) is parameterized
with a centroid $\bm{\mu}_{k}$ and a metric $\mathbf{Q}_{k}^{-1}$, and we
interpret (1) as class assignment probabilities based on the distance to each
centroid. With this viewpoint in mind, a natural extension to consider is to
use the estimates of the class assignment probabilities on unlabelled
instances to refine the class parameters $\bm{\mu}_{k}$ and $\mathbf{Q}_{k}$
in a soft $k$-means framework based on per-cluster Mahalanobis distances [65].
In this framework, as shown in Figure 6, we alternate between computing
updated assignment probabilities using (1) on the query set and using those
assignment probabilities to compute updated class parameters.

We define
$\mathcal{R}_{\theta}=\mathcal{S}_{\theta}\sqcup\mathcal{Q}_{\theta}$ as the
disjoint union of the support set and the query set. For each element of
$\mathcal{R}_{\theta}$, which we index by $j$, we define responsibilities
$w_{jk}$ in terms of their class predictions when it is part of the query set
and in terms of the label when it is part of the support set,

$w_{jk}=\begin{cases}p\big{(}y_{j}^{\prime}=k\mid\mathbf{z}_{j}^{\prime}\big{)}&\mathbf{z}_{j}^{\prime}\in\mathcal{Q}_{\theta},\\\
\mathbb{I}[y^{\prime}_{j}=k]&(\mathbf{z}_{j}^{\prime},y_{j}^{\prime})\in\mathcal{S}_{\theta}.\end{cases}$
(8)

Using these responsibilities we can incorporate unlabelled samples from the
support set by defining weighted estimates $\bm{\mu}^{\prime}_{k}$ and
$\mathbf{Q}^{\prime}_{k}$:

$\displaystyle\bm{\mu}^{\prime}_{k}$
$\displaystyle=\frac{1}{n^{\prime}_{k}}\sum_{j}w_{jk}\>\mathbf{z}_{j}^{\prime},$
(9) $\displaystyle\mathbf{Q}^{\prime}_{k}$
$\displaystyle=\lambda^{\prime}_{k}\bm{\Sigma}^{\prime}_{k}+(1-\lambda^{\prime}_{k})\bm{\Sigma}^{\prime}+\beta
I,$ (10)

where $n^{\prime}_{k}=\sum_{j}w_{jk}$ defines
$\lambda^{\prime}_{k}=n^{\prime}_{k}/(n^{\prime}_{k}+1)$. The covariance
estimates $\bm{\Sigma}^{\prime}$ and $\bm{\Sigma}_{k}^{\prime}$ are

$\displaystyle\bm{\Sigma}^{\prime}$
$\displaystyle=\frac{1}{\sum_{k}n^{\prime}_{k}}\sum_{jk}\\!w_{jk}\big{(}\mathbf{z}_{j}^{\prime}\\!-\\!\bm{\mu}^{\prime}\big{)}\big{(}\mathbf{z}_{j}^{\prime}\\!-\\!\bm{\mu}^{\prime}\big{)}^{T},$
(11) $\displaystyle\bm{\Sigma}^{\prime}_{k}$
$\displaystyle=\frac{1}{n^{\prime}_{k}}\sum_{j}\\!w_{jk}\big{(}\mathbf{z}_{j}^{\prime}\\!-\\!\bm{\mu}^{\prime}_{k}\big{)}\big{(}\mathbf{z}_{j}^{\prime}\\!-\\!\bm{\mu}^{\prime}_{k}\big{)}^{T}.$
(12)

where
$\bm{\mu}^{\prime}=\left(\sum_{k}n_{k}^{\prime}\right)^{-1}\sum_{jk}w_{jk}\mathbf{z}_{j}^{\prime}$
is the task-level mean.

Algorithm 1 Iterative Refinement in Transductive-CNAPS

1:procedure
compute_query_labels($\mathcal{S}_{\theta},\mathcal{Q}_{\theta},N_{\text{iter}}$)

2: For $j$ ranging over support and query sets,
$w_{jk}\leftarrow\begin{cases}1&\text{if}~{}(\mathbf{z}^{\prime}_{j},y^{\prime}_{j})\in\mathcal{S}_{\theta}\text{~{}and~{}}y_{j}=k\\\
0&\text{otherwise}\end{cases}$

3: for iter = $0\cdots N_{\text{iter}}$ do$\triangleright$ The first iteration
is equivalent to Simple CNAPS;

4: Compute class parameters $\bm{\mu}_{k},\mathbf{Q}_{k}$ according to update
equations (9)-(11)

5: Compute class weights using class parameters according to (8)

6: break if the most probable class for each query example hasn’t changed

7: end for

8: return class probabilities $w_{jk}$ for $j$ corresponding to
$\mathcal{Q}_{\theta}$

9:end procedure

These update equations are weighted versions of the original Simple CNAPS
estimators from Section 4.2, and reduce to them exactly in the case of an
empty query set.

Algorithm 1 summarizes the soft k-means procedure based on these updates. We
initialize class weights using only the labelled support data. We use those
weights to compute class parameters, then compute updated weights using both
the support and query sets. At this point, the weights associated with the
query set $\mathcal{Q}$ are the same class probabilities as estimated by
Simple CNAPS. However, in Transductive CNAPS, we repeat this procedure
iteratively until we reach either reach a maximum number of iterations, or
until class assignments $\text{argmax}_{k}\,w_{jk}$ stop changing.

Unlike the transductive task-encoder, this second extension in Transductive
CNAPS, namely the soft k-mean iterative estimation of class means and
covariance estimates, is used at test time only. During training, a single
estimation is produced for both mean and covariance using only the support
examples. This, as we discuss more in Section VI, was shown to empirically
perform better. See Figure 4 for a high-level comparison of classification in
Simple CNAPS vs. Transductive CNAPS.

###  4.4 Active Learning

Following past work [66], our main objective is to evaluate if Simple and
Transductive CNAPS perform better with uncertainty-based active label
aquisition methods as oppose to random selection when used in “out of the box”
active learning.

Here, both models are first trained on a large few-shot learning benchmark,
namely Meta-Dataset [32]. Then, without any further training, they are
evaluated for active learning, where given a set of unlabelled images, a label
can be acquired at each iteration. We consider three standard label
acquisitions methods: random selection, and two uncertainty-based approaches,
specifically predictive entropy [67] and variation ratios [67].

For uncertainty-based label acquisition, unlabelled examples are first
processed by the adapted network where class probabilities are produced. These
class probabilities are then used to rank examples. In predictive entropy, the
entropy of class probabilities are calculated and examples with maximal
predictive entropy are selected for label acquisition. In variation ratios,
$\text{MaxProb}(\mathbf{y}_{i})$ is calculated as defined by the maximum
probability assigned to any of the classes. Examples are then ranked based on
this score, where the lowest scoring examples are selected for label
acquisition.

###  4.5 Continual Learning

To perform “out of the box” continual learning, we modify Simple and
Transductive CNAPS in two major ways. After both methods have been
episodically meta-trained on a large few-shot learning benchmark (e.g. Meta-
Dataset [32]), each architecture is deployed in a continuous learning problem
setting where at each time $t$, a new task consisting of
$(\mathcal{S}^{t},\mathcal{Q}^{t})$ is presented for undertaking.

Figure 7: “Out of the box” active learning performance of Simple CNAPS and
Transductive CNAPS on CIFAR 10 and select OMNIGLOT languages. For each method,
we use 3 label acquisition methods. Results on other OMNIGLOT languages can be
found in the Appendix.

Each task at time $t$ can consist of both previously seen classes and entirely
novel categories. For novel classes, we produce and save class-wise estimates
of mean and covariance using equations 1 and 3 for Simple CNAPS and their
corresponding transductive extensions for Transductive CNAPS. For previously
seen classes, weighted estimates are produced for both means and covariances
based on the number of class examples present in the task and the class
instances previously seen,

$\displaystyle\bm{\mu}_{k}$
$\displaystyle=\frac{n_{k}^{t}}{n_{k}^{t}+n_{k}^{1:t-1}}\times\bm{\mu}_{k}^{t}+\frac{n_{k}^{1:t-1}}{n_{k}^{t}+n_{k}^{1:t-1}}\times\bm{\mu}_{k}^{\prime},$
(13) $\displaystyle\mathbf{Q}_{k}$
$\displaystyle=\frac{n_{k}^{t}}{n_{k}^{t}+n_{k}^{1:t-1}}\times\mathbf{Q}_{k}^{t}+\frac{n_{k}^{1:t-1}}{n_{k}^{t}+n_{k}^{1:t-1}}\times\mathbf{Q}_{k}^{\prime}.$
(14)

where $n_{k}^{t}$ indicates the number of support examples for class $k$
within the new task seen at time $t$ and $n_{k}^{1:t-1}$ is the number of
support instances seen for class $k$ prior to the task.
$\bm{\mu}_{k}^{\prime}$ and $\mathbf{Q}_{k}^{\prime}$ specify saved mean and
covariance estimates prior to the current task whereas $\bm{\mu}_{k}^{t}$ and
$\mathbf{Q}_{k}^{t}$ are task-parameters estimated using the task at time $t$.

The second modification to both methods for continual learning involves the
continuous adaptation of the feature extractor $f_{\theta}$. This is primarily
accomplished through developing strategies for continual updating of the task-
encoding. Let us denote this task-encoding at time $t$ as $\mathbf{d}^{t}$. We
consider three update strategies.

First, we use the “Moving Encoding”, where
$\mathbf{d}^{t}=g_{\theta}(\mathcal{S}^{t})$ for Simple CNAPS and
$\mathbf{d}^{t}=g_{\theta}(\mathcal{S}^{t},\mathcal{Q}^{t})$ for Transductive
CNAPS. That is, the encoding is set to the task-encoding of the most recent
task. This naturally results in different feature space manifolds at different
time steps $t$ while saved class representations are based on manifolds at the
time of their respective tasks.

Second, we focus on the case of fixing the feature space to the task-encoding
of the first task, $t=1$. We refer to this variation at “First Encoding”,
where $\mathbf{d}^{t}=g_{\theta}(\mathcal{S}^{1})$ for Simple CNAPS and
$\mathbf{d}^{t}=g_{\theta}(\mathcal{S}^{1},\mathcal{Q}^{1})$ for Transductive
CNAPS. This assures that all class-wise and query embeddings are generated
within the same feature manifold. However, that manifold has been task-adapted
for maximal performance on only the first task observed during continual
learning.

Third, we consider the “Averaging Encoding” strategy, with

$\displaystyle\mathbf{d}^{t}$
$\displaystyle=\frac{1}{t}g_{\theta}(\mathcal{S}^{t})+\frac{t-1}{t}d^{t-1}\text{
(Simple CNAPS)},$ (15) $\displaystyle\mathbf{d}^{t}$
$\displaystyle=\frac{1}{t}g_{\theta}(\mathcal{S}^{t},\mathcal{Q}^{t})+\frac{t-1}{t}d^{t-1}\text{
(Transductive CNAPS)}.$ (16)

where a $t$-scaled weighted convex combination of the current task-encoding
$g_{\theta}(\mathcal{S}^{t})$/$g_{\theta}(\mathcal{S}^{t},\mathcal{Q}^{t})$
and the task-encoding up to the previous time step $d^{t-1}$. This update
strategy strikes a hybrid balance between maintaining feature space manifolds
that don’t vary considerably between time-steps, while partially adapting the
feature space to new tasks as they come.

##  5 Theoretical Motivation

(a) Multi-H Simple CNAPS + Moving Encoding (b) Multi-H Simple CNAPS + First
Encoding (c) Multi-H Simple CNAPS + Averaging Encoding

(d) Single-H Simple CNAPS + Moving Encoding (e) Single-H Simple CNAPS + First
Encoding (f) Single-H Simple CNAPS + Averaging Encoding

Figure 8: Task-wise classification accuracy of “out of the box” continual
learning on MNIST. Here, consecutive pairs of digits are grouped together one
task and iteratively shown to the models. X-axis specifies the iteration in
the continual learning process while the coloured bars signify classification
accuracy on the query test set of each task.

###  5.1 Relationship to Bregman Soft Clustering

The procedure in Algorithm 1 resembles the Bregman clustering algorithms of
Banerjee et al. [28]. Specifically, the updates to soft assignments $w_{jk}$
in Equation 1 are the semi-supervised equivalent of those in Bregman soft
clustering, in which the divergence is based on the Mahalanobis distance
$F(\textbf{z})=\textbf{z}^{T}\textbf{Q}^{-1}\textbf{z}$,

$\displaystyle
D_{F}(\textbf{z},\textbf{z}^{\prime})=F(\textbf{z})-F(\textbf{z}^{\prime})-\nabla
F(\textbf{z}^{\prime})^{T}(\textbf{z}-\textbf{z}^{\prime}).$ (17)

However, Algorithm 1 differs in that it updates both $\bm{\mu}^{\prime}_{k}$
and $\textbf{Q}^{\prime}_{k}$ at each iteration, rather than just
$\bm{\mu}^{\prime}_{k}$.

In general, any (regular) exponential family can be associated with a Bregman
divergence and vice versa, which gives rise to a correspondence between EM-
based clustering and Bregman soft clustering algorithms [28]. Standard Bregman
soft clustering corresponds to EM in which the likelihood is a Gaussian with
unknown mean and a known covariance Q that is shared across clusters. The case
where the covariance is unknown corresponds to Gaussian mixture models (GMMs),
but the function $F(\textbf{z})$ is not simply the Mahalanobis distance in
this case.

The updates for $\bm{\mu}^{\prime}_{k}$ and $\textbf{Q}^{\prime}_{k}$ in
Algorithm 1 are equivalent to those in a GMM that incorporates regularization
for the covariances. However, GMM clustering differs in the calculation of the
assignment probabilities

$\displaystyle p(y^{*}$ $\displaystyle=k\mid\mathbf{z}^{*})\propto\pi_{k}$
(18)
$\displaystyle\exp\left(-\frac{1}{2}(\mathbf{z}-\bm{\mu_{k}})^{T}\mathbf{Q}_{k}^{-1}(\mathbf{z}-\bm{\mu}_{k})-\frac{1}{2}\log|\mathbf{Q}_{k}|\right).$
(19)

These probabilities incorporate a term $\pi_{k}=p(y^{*}=k)$, which defines a
prior probability of assignments to a cluster, and a term
$\exp(-\log|\textbf{Q}_{k}|)$, which reflects the fact that GMMs employ a
likelihood with unknown covariance.

In short, our clustering procedure employs an update to soft assignments
$w_{jk}$ that is similar to that of soft Bregman clustering, but employs an
updates to $\bm{\mu}^{\prime}_{k}$ and $\textbf{Q}^{\prime}_{k}$ that are
similar to those in a (regularized) Gaussian Mixture Model (GMM). In Section
VI we demonstrate through ablations that this combination of updates improves
empirical performance relative to baselines that perform GMM-based clustering.

###  5.2 Connections with Riemannian Metric Learning

By dropping the log-determinant of the class covariances $\mathbf{Q}_{k}$, we
lose the ability to interpret the model as a Gaussian mixture model, since
each mixture component is no longer a normalized conditional density
$p(\bm{z}^{*}\mid\bm{\mu}_{k},\bm{Q}_{k},\bm{y}^{*}\\!=\\!k)$. We show that by
postulating that the feature space is best described as a Riemannian manifold,
our relative class scores (1) approximate the squared _geodesic_ distance
between a test point $\mathbf{z}^{*}$ and the centroid $\bm{\mu}_{k}$.

(a) Transductive (b) Non-Transductive

Figure 9: Class recall averaged between classes across Meta-Dataset.

The geometry of a Riemannian manifold is defined by a local _metric tensor_
($d\times d$ positive-definite matrix defined for each point in
$\mathbb{R}^{d}$), $\mathbf{g}(\mathbf{x})$, in a data or feature space
$\mathbb{R}^{d}$ (more general topologies are not considered in this
discussion). The metric tensor defines the geometry of the underlying space;
particularly, we can employ it to define a notion of length. The distance
along a path $\gamma:[0,1]\rightarrow\mathbb{R}^{d}$ is computed in terms of
this metric tensor via the arclength functional:

$\mathcal{L}[\gamma]=\int_{0}^{1}\sqrt{\dot{\gamma}(\lambda)^{T}\mathbf{g}(\gamma(\lambda))\dot{\gamma}(\lambda)}d\lambda$
(20)

From this, we can derive a global distance (the geodesic distance) between
points (at least in the $\mathbb{R}^{d}$ case) as the length of the shortest
path between $\mathbf{x}$ and $\mathbf{y}$:

$d(\mathbf{x},\mathbf{y})=\inf_{\gamma}\mathcal{L}[\gamma]~{}\text{for
$\gamma$ where}~{}\gamma(0)=\mathbf{x},\gamma(1)=\mathbf{y}$ (21)

The arclength functional is difficult to analyze, but we can instead analyze
the related _energy functional_ [68],

$E[\gamma]=\int_{0}^{1}\dot{\gamma}(\lambda)^{T}\mathbf{g}(\gamma)\dot{\gamma}(\lambda)d\lambda$
(22)

Both $E$ and $\mathcal{L}$ yield the same local minimizers; these are called
geodesics, and are equivalent to straight lines within the geometry defined by
$\mathbf{g}(\mathbf{x})$.

In metric learning, our goal is to estimate the metric tensor
$\mathbf{g}(\mathbf{x})$ from data. This is an underdetermined task, since its
only constraints are smoothness and positive definiteness. To reduce the space
of metric tensors under consideration, we treat the class centroids
$\bm{\mu}_{k}$ as local inducing points for a metric that is locally (near
$\mu_{k}$) the Mahalanobis distance defined by $\mathbf{Q}_{k}$, and model the
global metric tensor as some smooth interpolation of these local Mahalanobis
metrics:

$\mathbf{g}(\mathbf{x})=\sum_{k}w_{k}(\mathbf{x}-\mu_{k})\mathbf{Q}_{k}^{-1}$
(23)

Here, $\\{w_{k}(\mathbf{r})\\}_{k}$ is a smooth partition of unity, which
satisfies

$\displaystyle w_{k}(\mathbf{0})$ $\displaystyle=1$ $\displaystyle
w_{k}(\mathbf{x})$ $\displaystyle\geq 0$
$\displaystyle\sum_{k}w_{k}(\mathbf{x})$ $\displaystyle=1$ (24)

The existence of such functions is guaranteed [69].

Even with these simplifying assumptions, the global geodesic distance is
extremely challenging to compute. Since the geodesic distance is the minimum
path length over all paths, we can upper bound it by a specific path [70]:

$d(\mathbf{x},\mathbf{y})\leq\int_{0}^{1}\sqrt{\dot{\mathbf{c}}(\lambda)^{T}\mathbf{g}(\mathbf{c}(\lambda))\dot{\mathbf{c}}(\lambda)}d\lambda$
(25)

where $\mathbf{c}(\lambda)=(1-\lambda)\mathbf{x}+\lambda\mathbf{y}$ is a
straight line (in the coordinate space) interpolation. The time derivative of
$\mathbf{c}$ is easily computed to be
$\dot{\mathbf{c}}=\mathbf{y}-\mathbf{x}$. The corresponding energy functional
upper bounds the squared distance,

$d(\mathbf{x},\mathbf{y})^{2}\leq\tilde{E}(\mathbf{x},\mathbf{y})=\int_{0}^{1}\dot{\mathbf{c}}(\lambda)^{T}\mathbf{g}(\mathbf{c}(\lambda))\dot{\mathbf{c}}(\lambda)d\lambda.$
(26)

For purposes of classification, computing the exact distance to each centroid
is not necessary; we only need to reason about the _difference_ in distance
between a test point $\mathbf{x}$ and two class centroids
$\bm{\mu}_{i},\bm{\mu}_{j}$, which corresponds to their relative (log)
probabilities. Substituting the energy along the straight-line paths yields

$\displaystyle\Delta\tilde{E}_{ij}$
$\displaystyle=\tilde{E}(\mathbf{x},\mu_{i})-\tilde{E}(\mathbf{x},\mu_{j}),$
(27)
$\displaystyle=\int_{0}^{1}\Delta_{i}^{T}\mathbf{g}_{i}(\lambda)\Delta_{i}-\Delta_{j}^{T}\mathbf{g}_{j}(\lambda)\Delta_{j}d\lambda,$
(28)

where $\Delta_{k}$ and and $g_{k}(\lambda)$ denote,

$\displaystyle\Delta_{k}$ $\displaystyle=(\mathbf{x}-\mu_{k}),$ (29)
$\displaystyle\mathbf{g}_{k}(\lambda)$
$\displaystyle=\mathbf{g}(\lambda\mu_{k}+(1-\lambda)\mathbf{x}).$ (30)

We can write $\Delta E_{ij}$ in terms of a path parameter
$\tau\in[0,\frac{1}{2}]$ as

$\displaystyle\Delta\tilde{E}_{ij}(\tau)=$
$\displaystyle\int_{0}^{\tau}\Delta_{i}^{T}\mathbf{g}_{i}(\lambda)\Delta_{i}-\Delta_{j}^{T}\mathbf{g}_{j}(\lambda)\Delta_{j}d\lambda$
$\displaystyle+\int_{1-\tau}^{1}\Delta_{i}^{T}\mathbf{g}_{i}(\lambda)\Delta_{i}-\Delta_{j}^{T}\mathbf{g}_{j}(\lambda)\Delta_{j}d\lambda.$

As $T$ increases, we simultaneously grow the path inward from each class
centroid towards $\mathbf{x}$ and outward from $\mathbf{x}$ to the centroids.

A first-order Taylor expansion of this around $T=0$ causes the first integral
to drop out (since $g_{i}(0)=g_{j}(0)$), and the second integral yields (when
evaluated at $T=\frac{1}{2}$):

$\displaystyle\Delta\tilde{E}_{ij}\approx\frac{1}{2}[$
$\displaystyle(\mathbf{x}-\mu_{i})^{T}\mathbf{Q}_{i}^{-1}(\mathbf{x}-\mu_{i})$
$\displaystyle-(\mathbf{x}-\mu_{j})^{T}\mathbf{Q}_{j}^{-1}(\mathbf{x}-\mu_{j})].$

The higher order terms can be controlled to some extent by forcing the
partition functions $w_{k}(\mathbf{x})$ to be flat near the class centroids.

This allows us to think of the Simple CNAPS classifier logit function (the
squared Mahalanobis distance to the class centroids with per-class metrics) as
a coarse approximation of the (squared) geodesic distance between a query test
point and a class centroid; this coarse approximation is improved slightly by
noting that other low-order terms that could be considered drop out when
examining the difference in squared geodesic distance between a test point and
two class centroids.

There are a few implications of this viewpoint, which point to directions for
future work. First, the performance gains that we see from using per-class
Mahalanobis metrics suggests that modeling the local geometry of the adapted
features is important. By taking the manifold viewpoint more seriously, we
could consider more principled geometric algorithms such as manifold-
regularized SVMs [71] to further exploit the local geometry of the adapted
feature space. Second, while the inverse weighted sample covariance works in
practice as a local metric, and makes intuitive sense, we could consider
techniques specifically designed for estimating local Riemannian metrics [72]
to provide better (or at least more principled) estimates of our local metrics
$\mathbf{Q}_{k}^{-1}$.

##  6 Experiments

###  6.1 Few-Shot Learning Benchmarks

Meta-Dataset [32] is a few-shot visual classification benchmark consisting of
10 widely used datasets: ILSVRC-2012 (ImageNet) [73], Omniglot [74], FGVC-
Aircraft (Aircraft) [75], CUB-200-2011 (Birds) [76], Describable Textures
(DTD) [77], QuickDraw [78], FGVCx Fungi (Fungi) [79], VGG Flower (Flower)
[80], Traffic Signs (Signs) [81] and MSCOCO [82]. Consistent with past work
[21, 33], we train our model on the official training splits of the first 8
datasets and use the test splits to evaluate in-domain performance. We use the
remaining two datasets as well as three external benchmarks, namely MNIST
[83], CIFAR10 [84] and CIFAR100 [84], for out-of-domain evaluation.

Task generation in Meta-Dataset follows a complex procedure. The task is
defined as an instance of classification problem that can be of different ways
(number of classes) and individual classes can be of varying shots (number of
samples per class) even within the same task. For example, an instance of
MNIST classification task, may require classification of digits ‘1’, ‘2’ and
‘4’, each of which contains 7, 2 and 9 training support instances
respectively. This would constitute a 3-way task with 7-, 2- and 9-shots. In
task formation, the task way is first sampled uniformly between 5 and 50 and
way classes are selected at random from the corresponding class/dataset split.
Then, for each class, 10 instances are sampled at random and used as query
examples for the class, while of the remaining images for the class, a shot is
sampled uniformly from [1, 100] and shot number of images are selected at
random as support examples with total support set size of 500.

Additional dataset-specific constraints are enforced, as discussed in Section
3.2 of [32], and since some datasets have fewer than 50 classes and fewer than
100 images per class, the overall way and shot distributions resemble Poisson
distributions where most tasks have fewer than 10 classes and most classes
have fewer than 10 support examples (see Appendix-A.1). Following [33] and
[21], we first train our ResNet18 feature extractor on the Meta-Dataset
defined training split of ImageNet following the procedure in Appendix-A.3.
The ResNet18 parameters are then kept fixed while we train the adaptation
network for a total of sampled 110K tasks using Episodic Training [22, 24]
(see Appendix-A.3).

Mini/tiered-ImageNet [15, 30] are two benchmarks for few-shot learning. Both
datasets employ subsets of ImageNet [73] with a total of 100 classes and 60K
images in mini-ImageNet and 608 classes and 779K images in tiered-ImageNet.
Unlike Meta-Dataset, tasks across these datasets have pre-defined shots and
ways that are uniform across every task in the specified setting.

Following [26, 31, 22], we report performance on the 1/5-shot 5/10-way
settings across both datasets with 10 query examples per class. We first train
the ResNet18 on the training set of the corresponding benchmark at hand
following the procedure noted in Appendix-A.4. We also consider a more
feature-rich ResNet18 trained on the larger ImageNet dataset. However, we
exclude classes and examples from test sets of mini/tiered-ImageNet to address
potential class/example overlap issues, resulting in 825 classes and 1,055,494
images remaining. Then, with the ResNet18 parameters fixed, we train
episodically for 20K tasks (see Appendix-A.2 for details).

###  6.2 Results

| In-Domain Accuracy (%) | Out-of-Domain Accuracy (%) | Avg Rank
---|---|---|---
Model |  ImageNet |  Omniglot |  Aircraft |  Birds |  DTD |  QuickDraw |  Fungi |  Flower |  Signs |  MSCOCO |  MNIST |  CIFAR10 |  CIFAR100 | In | Out | All
RelationNet [85] | 30.9±0.9 | 86.6±0.8 | 69.7±0.8 | 54.1±1.0 | 56.6±0.7 | 61.8±1.0 | 32.6±1.1 | 76.1±0.8 | 37.5±0.9 | 27.4±0.9 | - | - | - | 10.5 | 11.0 | 10.6
MatchingNet [86] | 36.1±1.0 | 78.3±1.0 | 69.2±1.0 | 56.4±1.0 | 61.8±0.7 | 60.8±1.0 | 33.7±1.0 | 81.9±0.7 | 55.6±1.1 | 28.8±1.0 | - | - | - | 10.1 | 8.5 | 9.8
MAML [87] | 37.8±1.0 | 83.9±1.0 | 76.4±0.7 | 62.4±1.1 | 64.1±0.8 | 59.7±1.1 | 33.5±1.1 | 79.9±0.8 | 42.9±1.3 | 29.4±1.1 | - | - | - | 9.2 | 10.5 | 9.5
ProtoNet [22] | 44.5±1.1 | 79.6±1.1 | 71.1±0.9 | 67.0±1.0 | 65.2±0.8 | 64.9±0.9 | 40.3±1.1 | 86.9±0.7 | 46.5±1.0 | 39.9±1.1 | - | - | - | 8.2 | 9.5 | 8.5
ProtoMAML [32] | 46.5±1.1 | 82.7±1.0 | 75.2±0.8 | 69.9±1.0 | 68.3±0.8 | 66.8±0.9 | 42.0±1.2 | 88.7±0.7 | 52.4±1.1 | 41.7±1.1 | - | - | - | 7.1 | 8.0 | 7.3
CNAPS [66] | 52.3±1.0 | 88.4±0.7 | 80.5±0.6 | 72.2±0.9 | 58.3±0.7 | 72.5±0.8 | 47.4±1.0 | 86.0±0.5 | 60.2±0.9 | 42.6±1.1 | 92.7±0.4 | 61.5±0.7 | 50.1±1.0 | 6.6 | 6.0 | 6.4
BOHB-E [88] | 55.4±1.1 | 77.5±1.1 | 60.9±0.9 | 73.6±0.8 | 72.8±0.7 | 61.2±0.9 | 44.5±1.1 | 90.6±0.6 | 57.5±1.0 | 51.9±1.0 | - | - | - | 6.4 | 4.0 | 5.9
TaskNorm [89] | 50.6±1.1 | 90.7±0.6 | 83.8±0.6 | 74.6±0.8 | 62.1±0.7 | 74.8±0.7 | 48.7±1.0 | 89.6±0.5 | 67.0±0.7 | 43.4±1.0 | 92.3±0.4 | 69.3±0.8 | 54.6±1.1 | 4.7 | 4.8 | 4.8
SUR [90] | 56.3±1.1 | 93.1±0.5 | 85.4±0.7 | 71.4±1.0 | 71.5±0.8 | 81.3±0.6 | 63.1±1.0 | 82.8±0.7 | 70.4±0.8 | 52.4±1.1 | 94.3±0.4 | 66.8±0.9 | 56.6±1.0 | 3.1 | 2.6 | 2.9
URT [91] | 55.7±1.0 | 94.4±0.4 | 85.8±0.6 | 76.3±0.8 | 71.8±0.7 | 82.5±0.6 | 63.5±1.0 | 88.2±0.6 | 69.4±0.8 | 52.2±1.1 | 94.8±0.4 | 67.3±0.8 | 56.9±1.0 | 1.7 | 2.8 | 2.2
Simple CNAPS | 58.6±1.1 | 91.7±0.6 | 82.4±0.7 | 74.9±0.8 | 67.8±0.8 | 77.7±0.7 | 46.9±1.0 | 90.7±0.5 | 73.5±0.7 | 46.2±1.1 | 93.9±0.4 | 74.3±0.7 | 60.5±1.0 | 3.4 | 3.0 | 3.2
Transductive CNAPS | 58.8±1.1 | 93.9±0.4 | 84.1±0.6 | 76.8±0.8 | 69.0±0.8 | 78.6±0.7 | 48.8±1.1 | 91.6±0.4 | 76.1±0.7 | 48.7±1.0 | 95.7±0.3 | 75.7±0.7 | 62.9±1.0 | 2.1 | 1.6 | 1.9
TABLE I: Few-shot classification on Meta-Dataset, MNIST, and CIFAR10/100. Error intervals correspond to 95% confidence intervals, and bold values indicate statistically significant SoTA performance. Average rank is obtained by ranking methods on each dataset and averaging the ranks. |  | mini-ImageNet Acc (%) | tiered-ImageNet Acc (%)
---|---|---|---
|  |  5-way |  10-way |  5-way |  10-way
Model | T? |  1-shot |  5-shot |  1-shot |  5-shot |  1-shot |  5-shot |  1-shot |  5-shot
MAML [24] | BN | 48.7 | 63.1 | 31.3 | 46.9 | 51.7 | 70.3 | 34.4 | 53.3
MAML+ [31] | Yes | 50.8 | 66.2 | 31.8 | 48.2 | 53.2 | 70.8 | 34.8 | 54.7
Reptile [26] | No | 47.1 | 62.7 | 31.1 | 44.7 | 49.0 | 66.5 | 33.7 | 48.0
Reptile+BN [26] | BN | 49.9 | 66.0 | 32.0 | 47.6 | 52.4 | 71.0 | 35.3 | 52.0
ProtoNet [22] | No | 46.1 | 65.8 | 32.9 | 49.3 | 48.6 | 69.6 | 37.3 | 57.8
RelationNet [36] | BN | 51.4 | 67.0 | 34.9 | 47.9 | 54.5 | 71.3 | 36.3 | 58.0
TPN [31] | Yes | 55.5 | 69.8 | 38.4 | 52.8 | 59.9 | 73.3 | 44.8 | 59.4
AttWeightGen [92] | No | 56.2 | 73.0 | - | - | - | - | - | -
TADAM [29] | No | 58.5 | 76.7 | - | - | - | - | - | -
Simple CNAPS | No | 53.2 | 70.8 | 37.1 | 56.7 | 63.0 | 80.0 | 48.1 | 70.2
Transductive CNAPS | Yes | 55.6 | 73.1 | 42.8 | 59.6 | 65.9 | 81.8 | 54.6 | 72.5
LEO [93] | No | 61.8 | 77.6 | - | - | 66.3 | 81.4 | - | -
MetaOptNet [94] | No | 62.6 | 78.6 | - | - | 66.0 | 81.6 | - | -
MetaBaseline [95] | No | 63.2 | 79.3 | - | - | 68.6 | 83.7 | - | -
FEAT [96] | No | 66.8 | 82.0 | - | - | 70.8 | 84.8 | - | -
SimpleShot [97] | No | 62.8 | 80.0 | - | - | 71.3 | 86.6 | - | -
RFS [98] | No | 64.8 | 82.1 | - | - | 71.5 | 86.0 | - | -
FRN [99] | No | 66.5 | 82.8 | - | - | 71.2 | 86.0 | - | -
DeepEMD [100] | No | 68.8 | 84.1 | - | - | 74.3 | 87.0 | - | -
S2M2 [101] | No | 64.9 | 83.2 | - | - | 73.7 | 88.6 | - | -
LR+DC [102] | No | 68.6 | 82.9 | - | - | 78.2 | 89.9 | - | -
TABLE II: Few-shot visual classification results on 1/5-shot 5/10-way tasks on mini/tiered-ImageNet. See Table VII for error intervals. |  | mini-ImageNet Acc (%) | tiered-ImageNet Acc (%)
---|---|---|---
|  |  5-way |  10-way |  5-way |  10-way
Model | T? |  1-shot |  5-shot |  1-shot |  5-shot |  1-shot |  5-shot |  1-shot |  5-shot
RS-FSL [103] | No | 65.3 | - | - | - | - | - | - | -
AmdimNet [104] | No | 76.8 | 91.0 | - | - | - | - | - | -
Simple CNAPS \+ FETI | No | 77.4 | 90.3 | 63.5 | 83.1 | 71.4 | 86.0 | 57.1 | 78.5
Transductive CNAPS \+ FETI | Yes | 79.9 | 91.5 | 68.5 | 85.9 | 73.8 | 87.7 | 65.1 | 80.6
TABLE III: Few-shot visual classification results on 1/5-shot 5/10-way tasks
on mini/tiered-ImageNet with additional training data. For our models, “FETI”
indicates that the feature extractor used was trained on ImageNet [73]
excluding classes in the test splits of mini/tiered-ImageNet. See Table VIII
for error intervals.

Evaluation on Meta-Dataset:  In-domain, out-of-domain and overall rankings on
Meta-Dataset are shown in Table I. Following [21], we pretrain the feature
extractor on the Meta-Dataset specified training split of the ImageNet subset
of Meta-Dataset (See Requeima et al.[21]-C.1.1 for ResNet18 training details).
Note that this excludes any examples or classes present in the test/validation
sets of the Meta-Dataset’s ImageNet split. As shown in Table I, Transductive
CNAPS is able to establish state-of-the-art performance with an overall rank
of 1.9 while Simple CNAPS ranks fourth with an average rank of 3.2.

Evaluation on mini/tiered-ImageNet:  We consider two feature extractor
training settings on these benchmarks. First, we use the feature extractor
trained on the corresponding training split of the mini/tiered-ImageNet. As
shown in Table II, on tiered-ImageNet, Transductive CNAPS achieves best
accuracy on both 10-way settings with Simple CNAPS ranking second-best, as
compared to other previous work that report results on these settings. On
5-way tiered-ImageNet settings, however, despite outperforming a number of
major baselines, both methods trail behind a number of more recent works [94,
95, 96, 97, 98, 99, 100, 101, 102] in classification accuracy. It’s also
interesting to note that performance of Simple and Transductive CNAPS
generally ranks higher on the tiered-ImageNet benchmark as compared to mini-
ImageNet. We attribute this difference in performance between mini-ImageNet
and tiered-ImageNet to the fact that mini-ImageNet only provides 38,400
training examples, compared to 448,695 examples provided by tiered-ImageNet.
This results in a lower performing ResNet-18 feature extractor (which is
trained in a traditional supervised manner).

This hypothesis leads us to consider a second evaluation (denoted by “FETI”,
for “Feature Extractor Trained with ImageNet”, in Table III). In this model,
we train the feature extractor with a much larger subset of ImageNet, which
has been carefully selected to prevent any possible overlap (in examples or
classes) with the test sets of mini/tiered-ImageNet. Both Simple and
Transductive CNAPS are able to take advantage of the more example-rich feature
extractor, resulting in substantially better performance across the board,
even as compared to other baselines that employ additional data (such as RS-
FSL which pre-trains on the full ImageNet). Furthermore, Transductive CNAPS
outperforms Simple CNAPS by a large margin, even when using the same example-
rich feature extractor; this demonstrates that leveraging additional query set
information yields gains.

Performance vs. Class Shot:  In Figure 9, we examine the relationship between
class recall (i.e. accuracy among query examples belonging to the class
itself) and the number of support examples in the class (shot). As shown,
Simple CNAPS outperforms CNAPS with even as few as four support examples,
indicating its ability to produce useful estimates of covariance from few
examples. In addition, Transductive CNAPS is very effective when class shot is
below 10, showing large average recall improvements, especially at the 1-shot
level. However, as the class shot increases beyond 10, performance drops
compared to Simple CNAPS. This suggests that soft k-means learning of cluster
parameters can be effective when very few support examples are available.
Conversely, in high-shot classes, transductive updates can act as distractors.

| In-Domain Accuracy (%) | Out-of-Domain Accuracy (%) | Avg Acc.
---|---|---|---
Simple CNAPS + Metric |  ImageNet |  Omniglot |  Aircraft |  Birds |  DTD |  QuickDraw |  Fungi |  Flower |  Signs |  MSCOCO |  MNIST |  CIFAR10 |  CIFAR100 | In | Out | All
Negative Dot Product | 48.0±1.1 | 83.5±0.9 | 73.7±0.8 | 69.0±1.0 | 66.3±0.6 | 66.5±0.9 | 39.7±1.1 | 88.6±0.5 | 53.9±0.9 | 32.5±1.0 | 86.4±0.6 | 57.9±0.8 | 38.8±0.9 | 66.9 | 53.9 | 61.9
Cosine Similarity | 51.3±1.1 | 89.4±0.7 | 80.5±0.8 | 70.9±1.0 | 69.7±0.7 | 72.6±0.9 | 41.9±1.0 | 89.3±0.6 | 65.4±0.8 | 41.0±1.0 | 92.8±0.4 | 69.5±0.8 | 53.6±1.0 | 70.7 | 64.5 | 68.3
Absolute Distance ($L_{1}$) | 53.6±1.1 | 90.6±0.6 | 81.0±0.7 | 73.2±0.9 | 61.1±0.7 | 74.1±0.8 | 47.0±1.0 | 87.3±0.6 | 66.4±0.8 | 44.7±1.0 | 88.0±0.5 | 70.0±0.8 | 57.9±1.0 | 71.0 | 65.4 | 68.8
Squared Euclidean (${L_{2}}^{2}$) | 53.9±1.1 | 90.9±0.6 | 81.8±0.7 | 73.1±0.9 | 64.4±0.7 | 74.9±0.8 | 45.8±1.0 | 88.8±0.5 | 68.5±0.7 | 43.4±1.0 | 91.6±0.5 | 70.5±0.7 | 57.3±1.0 | 71.7 | 66.3 | 69.6
Squared Mahalanobis | 58.6±1.1 | 91.7±0.6 | 82.4±0.7 | 74.9±0.8 | 67.8±0.8 | 77.7±0.7 | 46.9±1.0 | 90.7±0.5 | 73.5±0.7 | 46.2±1.1 | 93.9±0.4 | 74.3±0.7 | 60.5±1.0 | 73.8 | 69.7 | 72.2
TABLE IV: Performance of various metric ablations on Meta-Dataset. Error intervals indicate 95% confidence intervals, and bold values indicate statistically significant state of the art performance. | In-Domain Accuracy (%) | Out-of-Domain Accuracy (%) | Avg Acc.
---|---|---|---
CNAPS Model |  ImageNet |  Omniglot |  Aircraft |  Birds |  DTD |  QuickDraw |  Fungi |  Flower |  Signs |  MSCOCO |  MNIST |  CIFAR10 |  CIFAR100 | In | Out | All
GMM | 45.3±1.0 | 88.0±0.9 | 80.8±0.8 | 71.4±0.8 | 61.1±0.7 | 70.7±0.8 | 42.9±1.0 | 88.1±0.6 | 68.9±0.7 | 37.2±0.9 | 91.4±0.5 | 64.5±0.7 | 46.6±0.9 | 68.5 | 61.7 | 65.9
GMM-EM | 52.3±1.0 | 92.0±0.5 | 84.3±0.6 | 75.2±0.8 | 64.3±0.7 | 72.6±0.8 | 44.6±1.0 | 90.8±0.5 | 71.4±0.7 | 44.7±0.9 | 93.0±0.4 | 71.1±0.7 | 56.4±0.9 | 72.0 | 67.3 | 70.2
Transductive+ | 53.3±1.1 | 92.3±0.5 | 81.2±0.7 | 75.0±0.8 | 72.0±0.7 | 74.8±0.8 | 45.1±1.0 | 92.1±0.4 | 71.0±0.8 | 44.0±1.1 | 95.9±0.3 | 71.1±0.7 | 57.3±1.1 | 73.2 | 67.9 | 71.2
FEOT | 57.3±1.1 | 90.5±0.7 | 82.9±0.7 | 74.8±0.8 | 67.3±0.8 | 76.3±0.8 | 47.7±1.0 | 90.5±0.5 | 75.8±0.7 | 47.1±1.1 | 94.9±0.4 | 74.3±0.8 | 61.2±1.0 | 73.4 | 70.7 | 72.4
COT | 58.8±1.1 | 95.2±0.3 | 84.0±0.6 | 76.4±0.7 | 68.5±0.8 | 77.8±0.7 | 49.7±1.0 | 92.7±0.4 | 70.8±0.7 | 47.3±1.0 | 94.2±0.4 | 75.2±0.7 | 61.2±1.0 | 75.4 | 69.7 | 73.2
Simple | 58.6±1.1 | 91.7±0.6 | 82.4±0.7 | 74.9±0.8 | 67.8±0.8 | 77.7±0.7 | 46.9±1.0 | 90.7±0.5 | 73.5±0.7 | 46.2±1.1 | 93.9±0.4 | 74.3±0.7 | 60.5±1.0 | 73.8 | 69.7 | 72.2
Transductive | 58.8±1.1 | 93.9±0.4 | 84.1±0.6 | 76.8±0.8 | 69.0±0.8 | 78.6±0.7 | 48.8±1.1 | 91.6±0.4 | 76.1±0.7 | 48.7±1.0 | 95.7±0.3 | 75.7±0.7 | 62.9±1.0 | 75.2 | 71.8 | 73.9
TABLE V: Performance of various ablations of Transductive and Simple CNAPS on Meta-Dataset. Error intervals indicate 95% confidence intervals, and bold values indicate statistically significant state of the art performance. | MNIST | CIFAR10 | CIFAR100
---|---|---|---
Method | Multi | Single | Multi | Single | Multi | Single
SI [61] | 99.3 | 57.6 | - | - | 73.2 | 22.8
EWC [62] | 99.3 | 55.8 | - | - | 72.8 | 23.1
VCL [63] |  98.5±0.4 | - | - | - | - | -
RWalk [64] | 99.3 | 82.5 | - | - | 74.2 | 34.0
CNAPS [66] \+ Moving Encoding |  98.9±0.2 |  80.9±0.9 | - | - |  76.0±0.5 |  37.2±0.6
Simple CNAPS \+ Moving Encoding |  83.8±3.2 |  19.2±0.2 |  83.0±3.0 |  18.1±1.2 |  67.7±0.8 |  12.1±2.0
Simple CNAPS \+ First Encoding |  96.7±0.3 |  69.5±1.0 |  87.4±0.8 |  41.9±1.0 |  69.0±0.7 |  34.2±0.8
Simple CNAPS \+ Averaging Encoding |  95.2±0.9 |  22.1±2.2 |  86.0±1.8 |  27.7±4.3 |  68.6±0.7 |  25.9±2.8
Transductive CNAPS \+ First Encoding |  90.7±1.4 |  9.3±0.3 |  86.0±0.5 |  10.0±0.3 |  66.7±0.6 |  1.0±0.1
Transductive CNAPS \+ Averaging Encoding |  88.7±0.5 |  9.2±0.3 |  85.7±0.7 |  9.9±0.3 |  66.8±0.9 |  1.0±0.1
. |  |  |  |  |  |
TABLE VI: “Out of the box” continual learning performance on MNIST, CIFAR10
and CIFAR100. Tasks here are generated with 100-shots per category.

Classification-Time Soft K-means Clustering: We use soft k-means iterative
updates of means and covariance at test-time only. It is natural to consider
training the feature adaptation network end-to-end through the soft k-means
transduction procedure. We provide this comparison in the bottom-half of Table
V, with “Transductive+” denoting this variation. Iterative updates during
training result in an average accuracy decrease of 2.5%, which we conjecture
to be due to training instabilities caused by applying this iterative
algorithm early in training on noisy features.

Transductive Feature Extraction vs. Classification: Our approach extends
Simple CNAPS in two ways: improved adaptation of the feature extractor using a
transductive task-encoding, and the soft k-means iterative estimation of class
means and covariances. We perform two ablations, “Feature Extraction Only
Transductive” (FEOT) and “Classification Only Transductive” (COT), to
independently assess the impact of these extensions. The results are presented
in Table V. As shown, both extensions outperform Simple CNAPS. The
transductive task-encoding is especially effective on out-of-domain tasks
whereas the soft k-mean learning of class parameters boosts accuracy on in-
domain tasks. Transductive CNAPS is able to leverage the best of both worlds,
allowing it to achieve statistically significant gains over Simple CNAPS
overall.

Comparison to Gaussian Mixture Models: We consider two GMM-based ablations of
our method where the log-determinant is introduced into the weight updates
(using a uniform class prior $\pi_{k}=1/K$). Note that with the exception of
how these ablations produce class probabilities (Equation 1 vs. Equations
18/19), all other aspects of GMM-based ablations remain identical to that of
Simple and Transductive CNAPS. Results are shown in Table V where GMM and GMM-
EM correspond to the GMM-based ablations of Simple and Transductive CNAPS. As
demonstrated, the GMM-based variations of our method and Simple CNAPS result
in a notable 4-8% loss in overall accuracy.

Metric Ablation: To test the significance of our choice of Mahalanobis
distance, we substitute it within our architecture with other distance metrics
- absolute difference ($L_{1}$), squared Euclidean ($L_{2}^{2}$), cosine
similarity and negative dot-product. Performance comparisons are shown in
Table IV. We observe that using the Mahalanobis distance results in the best
in-domain, out-of-domain, and overall average performance on all datasets.

Active Learning: We present active learning results on CIFAR10 and OMNIGLOT in
Figure 7. As shown, Transductive CNAPS and Simple CNAPS both benefit from
uncertainty-based label aquisition thanks to having well-calibrated measures
of uncertainty. It is interesting to note that the margin of performance gain
over random selection is larger in Simple CNAPS. This can suggest that
Transductive CNAPS is able to exploit much of the unlabelled information
through transductive learning. It may also imply that its measure of
uncertainty is less calibrated as soft-label class parameter estimates can be
more noisy. We also compare both methods against fixed feature extractor
(“FixedFE”) baselines. In these baselines, the adaptation networks are turned
off and the pre-trained ResNet18 is used to produce feature vectors without
any task-specific adaptation. As demonstrated, this results is lower
performance across the board, signifying the role of conditional adaptation
networks. Furthermore, the gap between Transductive and Simple FixedFE is
considerably smaller, indicating that the transductive feature extractor
adaptation module in Transductive CNAPS is responsible for a considerable
proportion of its performance gain over Simple CNAPS.

Continual Learning: We evaluate continual learning on MNIST, CIFAR10, and
CIFAR100 for both multi-head and single-head learning. In the multi-head
setting, each task is assigned a separate classification head that focuses on
the particular task’s categories. In our work, this corresponds to only
calculate distances to classes that are present in the task. In the single-
head case, by contrast, we evaluate with respect on all current and previously
seen classes.

As noted in Table VI, while neither variation is able to achieve state of the
art results on MNIST and CIFAR100, we observe that the task-encoding is
extremely important in achieving competitive performance. We explore this
further in Figure 8. As shown when using “Moving Encoding”, the performance on
previous tasks is completely lost in the single-head setting and significantly
reduced in the multi-head case. This suggests that Simple CNAPS and
Transductive CNAPS produce vastly different manifolds depending on the task at
hand, and previously estimated class parameters may no more be valid or useful
in these new feature spaces. This is further supported by the evidence we see
when using “First Encoding”. As shown, not only the best performance is
achieved here, but accuracy on previously learned classes is substantially
greater even in the single-head setting. Unsurprisingly, the best performance
is always scene at the query of the first task, as the feature space has been
adapted to maximize accuracy on that task. This shows the importance of
adapting the feature space to new tasks, which in part motivates the use of
the “Averaging Encoding” variation as a balance between the two. However, as
we see empirically, the differences in the feature space manifolds generated
with the averaged encoding still result in major loss of performance in the
single head, although in the multi-head setting, performance matching that of
“First Encoding” is achieved. It is interesting to note that in the single-
head “Averaging Encoding” variation, previous task performance tends to
improve as more tasks are seen. This is in part due to the fact that with more
tasks, the average encoding will be more stable and less likely to change
substantially with a new task, thus providing a more stable feature space
across the subsequent tasks. Overall, “out of the box” continual learning
remains an open question, even within the specific context of Simple and
Transductive CNAPS models.

##  7 Discussion

We propose two meta-learned few-shot image classification approaches, Simple
and Transductive CNAPS, that focus on effective estimation of class-wise
cluster parameters, including covariance. These models are able to accomplish
high accuracy, with the latter establishing state of the art performance on
Meta-Dataset, mini-ImageNet and tiered-ImageNet benchmarks. We further study
our methods in paradigms of active and continual learning, proposing
extensions for undertaking such tasks.

Future research on better continual learning of the task-wise feature
manifolds in both models can be beneficial in boosting the performance.
Additional research on generating support examples through learnable
augmentations, better cluster refinement algorithms, and further theoretical
understanding of the Mahalanobis-based classifiers in our methods are among
the many very interesting avenues for potentially fruitful future research.

##  8 Acknowledgments

We acknowledge the support of the Natural Sciences and Engineering Research
Council of Canada (NSERC), the Canada Research Chairs (CRC) Program, the
Canada CIFAR AI Chairs Program, Compute Canada, Intel, and DARPA under its D3M
and LWLL programs. Additionally, this material is based upon work supported by
the United States Air Force under Contract No. FA8750-19-C-0515.

## References

  * [1] L. Jiao, F. Zhang, F. Liu, S. Yang, L. Li, Z. Feng, and R. Qu, “A survey of deep learning-based object detection,” _CoRR_ , vol. abs/1907.09408, 2019\.
  * [2] M. Sornam, K. Muthusubash, and V. Vanitha, “A survey on image classification and activity recognition using deep convolutional neural network architecture,” in _International Conference on Advanced Computing (ICoAC)_ , 2017.
  * [3] M. Z. Hossain, F. Sohel, M. F. Shiratuddin, and H. Laga, “A comprehensive survey of deep learning for image captioning,” _ACM Comput. Surv._ , vol. 51, no. 6, Feb. 2019.
  * [4] G. Guz, P. Bateni, D. Muglich, and G. Carenini, “Neural RST-based evaluation of discourse coherence,” in _Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing_ , 2020, pp. 664–671.
  * [5] A. Krizhevsky, I. Sutskever, and G. Hinton, “Imagenet classification with deep convolutional neural networks,” _Neural Information Processing Systems_ , vol. 25, 2012.
  * [6] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * [7] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * [8] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards real-time object detection with region proposal networks,” _IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)_ , vol. 39, June 2015.
  * [9] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in _Advances in Neural Information Processing Systems_ , vol. 27, 2014, pp. 2672–2680.
  * [10] A. Ścibior, V. Lioutas, D. Reda, P. Bateni, and F. Wood, “Imagining the road ahead: Multi-agent trajectory prediction via differentiable simulation,” in _2021 IEEE International Intelligent Transportation Systems Conference (ITSC)_ , 2021, pp. 720–725.
  * [11] A. R. Feyjie, R. Azad, M. Pedersoli, C. Kauffman, I. B. Ayed, and J. Dolz, “Semi-supervised few-shot learning for medical image segmentation,” 2020.
  * [12] Y. Wang and Q. Yao, “Few-shot learning: A survey,” _CoRR_ , vol. abs/1904.05046, 2019.
  * [13] W. Wang, V. W. Zheng, H. Yu, and C. Miao, “A survey of zero-shot learning: Settings, methods, and applications,” _ACM Trans. Intell. Syst. Technol._ , vol. 10, no. 2, pp. 13:1–13:37, Jan. 2019.
  * [14] A. Bellet, A. Habrard, and M. Sebban, “A survey on metric learning for feature vectors and structured data,” _arXiv preprint arXiv:1306.6709_ , 2013.
  * [15] O. Vinyals, C. Blundell, T. Lillicrap, D. Wierstra _et al._ , “Matching networks for one shot learning,” in _Advances in neural information processing systems_ , 2016, pp. 3630–3638.
  * [16] J. Kim, T. Kim, S. Kim, and C. D. Yoo, “Edge-labeling graph neural network for few-shot learning,” _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * [17] G. Koch, R. Zemel, and R. Salakhutdinov, “Siamese neural networks for one-shot image recognition,” in _ICML deep learning workshop_ , vol. 2, 2015.
  * [18] V. G. Satorras and J. B. Estrach, “Few-shot learning with graph neural networks,” in _International Conference on Learning Representations (ICLR)_ , 2018.
  * [19] A. Frome, G. S. Corrado, J. Shlens, S. Bengio, J. Dean, M. A. Ranzato, and T. Mikolov, “Devise: A deep visual-semantic embedding model,” in _Advances in Neural Information Processing Systems_ , 2013, pp. 2121–2129.
  * [20] S. Gidaris and N. Komodakis, “Generating classification weights with gnn denoising autoencoders for few-shot learning,” _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * [21] J. Requeima, J. Gordon, J. Bronskill, S. Nowozin, and R. E. Turner, “Fast and flexible multi-task classification using conditional neural adaptive processes,” _Advances in Neural Information Processing Systems_ , 2019.
  * [22] J. Snell, K. Swersky, and R. Zemel, “Prototypical networks for few-shot learning,” in _Advances in Neural Information Processing Systems_ , 2017\.
  * [23] C. Xing, N. Rostamzadeh, B. N. Oreshkin, and P. O. Pinheiro, “Adaptive cross-modal few-shot learning,” _Advances in Neural Information Processing Systems_ , 2019.
  * [24] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in _International Conference on Machine Learning (ICML)_ , 2017.
  * [25] S. Ravi and H. Larochelle, “Optimization as a model for few-shot learning,” in _International Conference on Learning Representations (ICLR)_ , 2017.
  * [26] A. Nichol, J. Achiam, and J. Schulman, “On first-order meta-learning algorithms,” _CoRR_ , vol. abs/1803.02999, 2018.
  * [27] E. Perez, F. Strub, H. De Vries, V. Dumoulin, and A. Courville, “Film: Visual reasoning with a general conditioning layer,” in _Thirty-Second AAAI Conference on Artificial Intelligence_ , 2018.
  * [28] A. Banerjee, S. Merugu, I. S. Dhillon, and J. Ghosh, “Clustering with bregman divergences,” _Journal of Machine Learning Research_ , vol. 6, no. Oct, pp. 1705–1749, 2005.
  * [29] B. Oreshkin, P. Rodríguez López, and A. Lacoste, “Tadam: Task dependent adaptive metric for improved few-shot learning,” in _Advances in Neural Information Processing Systems_ , 2018, pp. 721–731.
  * [30] M. Ren, E. Triantafillou, S. Ravi, J. Snell, K. Swersky, J. B. Tenenbaum, H. Larochelle, and R. S. Zemel, “Meta-learning for semi-supervised few-shot classification,” _International Conference on Learning Representations (ICLR)_ , 2018.
  * [31] Y. Liu, J. Lee, M. Park, S. Kim, and Y. Yang, “Learning to propagate labels: Transductive propagation network for few-shot learning,” _International Conference on Learning Representations (ICLR)_ , 2019.
  * [32] E. Triantafillou, T. Zhu, V. Dumoulin, P. Lamblin, K. Xu, R. Goroshin, C. Gelada, K. Swersky, P.-A. Manzagol, and H. Larochelle, “Meta-dataset: A dataset of datasets for learning to learn from few examples,” _International Conference on Learning Representations (ICLR)_ , 2020.
  * [33] P. Bateni, R. Goyal, V. Masrani, F. Wood, and L. Sigal, “Improved few-shot visual classification,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * [34] P. Bateni, J. Barber, J.-W. van de Meent, and F. Wood, “Enhancing few-shot image classification with unlabelled examples,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_ , January 2022, pp. 2796–2805.
  * [35] P. Bateni, “On label-eﬀicient computer vision: building fast and effective few-shot image classifiers,” Ph.D. dissertation, University of British Columbia, 2021.
  * [36] F. Sung, Y. Yang, L. Zhang, T. Xiang, P. H. Torr, and T. M. Hospedales, “Learning to compare: Relation network for few-shot learning,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018\.
  * [37] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, “How transferable are features in deep neural networks?” _Advances in Neural Information Processing Systems_ , 2014.
  * [38] N. Mishra, M. Rohaninejad, X. Chen, and P. Abbeel, “Meta-learning with temporal convolutions,” _CoRR_ , vol. abs/1707.03141, 2017.
  * [39] M. Garnelo, D. Rosenbaum, C. J. Maddison, T. Ramalho, D. Saxton, M. Shanahan, Y. W. Teh, D. J. Rezende, and S. M. A. Eslami, “Conditional neural processes,” _International Conference on Machine Learning (ICML)_ , 2018\.
  * [40] S. Fort, “Gaussian prototypical networks for few-shot learning on omniglot,” _CoRR_ , vol. abs/1708.02735, 2017.
  * [41] S. Yang, L. Liu, and M. Xu, “Free lunch for few-shot learning: Distribution calibration,” in _International Conference on Learning Representations (ICLR)_ , 2021.
  * [42] L. Qiao, Y. Shi, J. Li, Y. Wang, T. Huang, and Y. Tian, “Transductive episodic-wise adaptive metric for few-shot learning,” in _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019.
  * [43] X. Song, Y. Dai, D. Zhou, L. Liu, W. Li, H. Li, and R. Yang, “Channel attention based iterative residual learning for depth map super-resolution,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * [44] W. H. Beluch, T. Genewein, A. Nurnberger, and J. M. Kohler, “The power of ensembles for active learning in image classification,” pp. 9368–9377, 2018\.
  * [45] J. A. Joshi, F. Porikli, and N. Papanikolopoulos, “Multi-class batch-mode active learning for image classification,” _Robotics and Automation_ , pp. 1873–1878, 2010.
  * [46] D. D. Lewis and W. A. Gale, “A sequential algorithm for training text classifiers,” pp. 3–12, 1994.
  * [47] H. Ranganathan, H. Venkateswara, S. Chakraborty, and S. Panchanathan, “Deep active learning for image classification,” pp. 3934–3938, 2017.
  * [48] H. S. Seung, M. Opper, and H. Sompolinsky, “Query by committee,” pp. 287–294, 1992.
  * [49] S. Tong and D. Koller, “Support vector machine active learning with applications to text classification,” _Journal of Machine Learning Research_ , vol. 2, no. 1, pp. 45–66, 2002.
  * [50] M. Bilgic and L. Getoor, “Link-based active learning,” 2009.
  * [51] Y. Gal, R. Islam, and Z. Ghahramani, “Deep bayesian active learning with image data,” _CoRR_ , 2017.
  * [52] Y. Guo, “Active instance sampling via matrix partition,” pp. 802–810, 2010.
  * [53] T. H. Nguyen and A. Smeulders, “Active learning using pre-clustering,” _ICML_ , pp. 79–79, 2004.
  * [54] A. Freytag, E. Rodner, and J. Denzler, “Selecting influential examples: Active learning with expected model output changes,” pp. 562–577, 2014.
  * [55] N. Roy and A. McCallum, “Toward optimal active learning through monte carlo estimation of error reduction,” _International Conference on Machine Learning_ , 2001.
  * [56] B. Settles, M. Craven, and S. Ray, “Multiple-instance active learning,” pp. 1289–1296, 2007.
  * [57] J. T. Ash, C. Zhang, A. Krishnamurthy, J. Langford, and A. Agarwal, “Deep batch active learning by diverse, uncertain gradient lower bounds,” _arXiv: Learning_ , 2019.
  * [58] C. Shui, F. Zhou, C. Gagne, and B. Wang, “Deep active learning: Unified and principled method for query and training.” _arXiv: Learning_ , 2019.
  * [59] C. Yin, B. Qian, S. Cao, X. Li, J. Wei, Q. Zheng, and I. Davidson, “Deep similarity-based batch mode active learning with exploration-exploitation,” in _2017 IEEE International Conference on Data Mining (ICDM)_ , 2017, pp. 575–584.
  * [60] S. S. Pouya Pezeshkpour, Zhengli Zhao, “On the utility of active instance selection for few-shot learning,” in _NeurIPS Workshop on Human And Model in the Loop Evaluation and Training Strategies_ , 2020.
  * [61] F. Zenke, B. Poole, and S. Ganguli, “Continual learning through synaptic intelligence,” in _International Conference on Machine Learning (ICML)_ , 2017, pp. 3987–3995.
  * [62] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell, “Overcoming catastrophic forgetting in neural networks,” _Proceedings of the National Academy of Sciences_ , vol. 114, 12 2016.
  * [63] C. V. Nguyen, Y. Li, T. D. Bui, and R. E. Turner, “Variational continual learning,” in _International Conference on Learning Representations (ICLR)_ , 2018.
  * [64] A. Chaudhry, P. K. Dokania, T. Ajanthan, and P. H. S. Torr, “Riemannian walk for incremental learning: Understanding forgetting and intransigence,” in _European Conference on Computer Vision (ECCV)_ , 2018, pp. 556–572.
  * [65] I. Melnykov and V. Melnykov, “On k-means algorithm with the use of mahalanobis distances,” _Statistics & Probability Letters_, vol. 84, pp. 88–95, 2014\.
  * [66] J. Requeima, J. Gordon, J. Bronskill, S. Nowozin, and R. E. Turner, “Fast and flexible multi-task classification using conditional neural adaptive processes,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 7957–7968.
  * [67] D. A. Cohn, Z. Ghahramani, and M. I. Jordan, “Active learning with statistical models,” _J. Artif. Int. Res._ , vol. 4, no. 1, p. 129–145, Mar. 1996.
  * [68] M. P. d. Carmo, _Riemannian geometry_. Birkhäuser, 1992.
  * [69] J. M. Lee, “Smooth manifolds,” in _Introduction to Smooth Manifolds_. Springer, 2013, pp. 1–31.
  * [70] D. Ramanan and S. Baker, “Local distance functions: A taxonomy, new algorithms, and an evaluation,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 33, no. 4, pp. 794–806, 2010.
  * [71] M. Belkin, P. Niyogi, and V. Sindhwani, “Manifold regularization: A geometric framework for learning from labeled and unlabeled examples.” _Journal of Machine Learning Research_ , vol. 7, no. 11, 2006.
  * [72] S. Hauberg, O. Freifeld, and M. J. Black, “A geometric take on metric learning,” in _Advances in Neural Information Processing Systems_ , 2012, pp. 2024–2032.
  * [73] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein _et al._ , “Imagenet large scale visual recognition challenge,” _International Journal of Computer Vision (IJCV)_ , vol. 115, no. 3, pp. 211–252, 2015.
  * [74] B. M. Lake, R. Salakhutdinov, and J. B. Tenenbaum, “Human-level concept learning through probabilistic program induction,” _Science_ , vol. 350, no. 6266, pp. 1332–1338, 2015.
  * [75] S. Maji, E. Rahtu, J. Kannala, M. Blaschko, and A. Vedaldi, “Fine-grained visual classification of aircraft,” _arXiv preprint arXiv:1306.5151_ , 2013\.
  * [76] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie, “The caltech-ucsd birds-200-2011 dataset,” 2011.
  * [77] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi, “Describing textures in the wild,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2014, pp. 3606–3613.
  * [78] J. Jongejan, H. Rowley, T. Kawashima, J. Kim, and N. Fox-Gieg, “The quick, draw!-ai experiment.(2016),” 2016.
  * [79] B. Schroeder and Y. Cui, “Fgvcx fungi classification challenge 2018,” 2018.
  * [80] M.-E. Nilsback and A. Zisserman, “Automated flower classification over a large number of classes,” in _IEEE Indian Conference on Computer Vision, Graphics & Image Processing_, 2008, pp. 722–729.
  * [81] S. Houben, J. Stallkamp, J. Salmen, M. Schlipsing, and C. Igel, “Detection of traffic signs in real-world images: The german traffic sign detection benchmark,” in _International Joint Conf. on Neural Networks (IJCNN)_ , 2013, pp. 1–8.
  * [82] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _European Conference on Computer Vision (ECCV)_ , 2014, pp. 740–755.
  * [83] Y. LeCun and C. Cortes, “MNIST handwritten digit database,” 2010.
  * [84] A. Krizhevsky, “Learning multiple layers of features from tiny images,” Tech. Rep., 2009.
  * [85] F. Sung, Y. Yang, L. Zhang, T. Xiang, P. H. Torr, and T. M. Hospedales, “Learning to compare: Relation network for few-shot learning,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 1199–1208.
  * [86] O. Vinyals, C. Blundell, T. Lillicrap, D. Wierstra _et al._ , “Matching networks for one shot learning,” in _Advances in neural information processing systems_ , 2016, pp. 3630–3638.
  * [87] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in _Proceedings of Machine Learning Research (PMLR)_ , 2017.
  * [88] T. Saikia, T. Brox, and C. Schmid, “Optimized generic feature learning for few-shot classification across domains,” 2020.
  * [89] J. Bronskill, J. Gordon, J. Requeima, S. Nowozin, and R. Turner, “Tasknorm: Rethinking batch normalization for meta-learning,” in _Proceedings of the 37th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research. PMLR, 2020, vol. 119.
  * [90] N. Dvornik, C. Schmid, and J. Mairal, “Selecting relevant features from a multi-domain representation for few-shot classification,” 2020.
  * [91] L. Liu, W. Hamilton, G. Long, J. Jiang, and H. Larochelle, “A universal representation transformer layer for few-shot image classification,” 2020.
  * [92] S. Gidaris and N. Komodakis, “Dynamic few-shot visual learning without forgetting,” _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , vol. abs/1804.09458, 2018.
  * [93] A. A. Rusu, D. Rao, J. Sygnowski, O. Vinyals, R. Pascanu, S. Osindero, and R. Hadsell, “Meta-learning with latent embedding optimization,” _International Conference on Learning Representations (ICLR)_ , 2018.
  * [94] K. Lee, S. Maji, A. Ravichandran, and S. Soatto, “Meta-learning with differentiable convex optimization,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2019.
  * [95] Y. Chen, Z. Liu, H. Xu, T. Darrell, and X. Wang, “Meta-baseline: exploring simple meta-learning for few-shot learning,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021.
  * [96] H.-J. Ye, H. Hu, D.-C. Zhan, and F. Sha, “Few-shot learning via embedding adaptation with set-to-set functions,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 8808–8817.
  * [97] Y. Wang, W.-L. Chao, K. Q. Weinberger, and L. van der Maaten, “Simpleshot: Revisiting nearest-neighbor classification for few-shot learning,” _arXiv preprint arXiv:1911.04623_ , 2019.
  * [98] Y. Tian, Y. Wang, D. Krishnan, J. B. Tenenbaum, and P. Isola, “Rethinking few-shot image classification: a good embedding is all you need?” _arXiv preprint arXiv:2003.11539_ , 2020.
  * [99] D. Wertheimer, L. Tang, and B. Hariharan, “Few-shot classification with feature map reconstruction networks,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2021, pp. 8012–8021.
  * [100] C. Zhang, Y. Cai, G. Lin, and C. Shen, “Deepemd: Few-shot image classification with differentiable earth mover’s distance and structured classifiers,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * [101] P. Mangla, N. Kumari, A. Sinha, M. Singh, B. Krishnamurthy, and V. N. Balasubramanian, “Charting the right manifold: Manifold mixup for few-shot learning,” in _The IEEE Winter Conference on Applications of Computer Vision_ , 2020, pp. 2218–2227.
  * [102] S. Yang, L. Liu, and M. Xu, “Free lunch for few-shot learning: Distribution calibration,” in _International Conference on Learning Representations (ICLR)_ , 2021.
  * [103] M. Afham, S. Khan, M. H. Khan, M. Naseer, and F. S. Khan, “Rich semantics improve few-shot learning,” 2021.
  * [104] D. Chen, Y. Chen, Y. Li, F. Mao, Y. He, and H. Xue, “Self-supervised learning for few-shot image classification,” _arXiv preprint arXiv:1911.06045_ , 2019\.
  * [105] C. Doersch, A. Gupta, and A. Zisserman, “Crosstransformers: spatially-aware few-shot transfer,” in _Advances in neural information processing systems_ , 2020.

|  Peyman Bateni is an AI entrepreneur, currently serving as CEO at Beam AI.
Previously, he was a graduate Machine Learning researcher at the University of
British Columbia and a research engineer at Inverted AI. Before that, he
completed his undergraduate studies in Computer Science at the University of
British Columbia. Peyman’s research primarily focuses on label-efficient
learning of computer vision models.
---|---
|  Jarred Barber received the M.S. degree in computer science in 2010 from
Johns Hopkins University. He is currently a research scientist at Amazon Alexa
AI, focusing on machine learning and signal processing techniques for speech
enhancement and recognition.
---|---
|  Raghav Goyal is a PhD student in Computer Vision and Machine Learning at
the University of British Columbia (UBC). Prior to this he spent three years
at Twenty Billion Neurons GmbH (20bn) in Berlin, Germany as an AI Engineer
working on video understanding under the supervision of Roland Memisevic, PhD.
He obtained an Integrated M.Tech. (5-year programme) from Indian Institute of
Technology (IIT) Delhi in Mathematics and Computing.
---|---
|  Vaden Masrani is a PhD student working in machine learning and approximate
inference at the UBC PLAI Group. Previously he received his M.S. degree in
computer science at the University of British Columbia where he developed
natural language processing techniques for early detection of Alzheimer’s
disease.
---|---
|  Jan-Willem van de Meent is assistant professor in the Khoury College of
Computer Sciences at Northeastern. His research combines probabilistic
programming with deep learning to develop probabilistic models for machine
learning, data science, and artificial intelligence. Prior to that, he was a
post-doctoral researcher at University of Oxford and Columbia University. He
received his PhD in Biophysics from University of Cambridge.
---|---
|  Leonid Sigal is an Associate Professor in the Department of Computer
Science at the University of British Columbia and a Senior Member of IEEE. He
holds NCERC Canada Research Chair (CRC II) in Computer Vision and Machine
Learning and a CIFAR Artificial Intelligence Chair at the Vector Institute for
AI, in Toronto. In addition, he serves as an Academic Advisor to Borealis AI.
Prior to this, from 2009 to 2017, he was a Senior Research Scientist at Disney
Research, Pittsburgh and an Adjunct Faculty of Carnegie Mellon University. He
completed his Ph.D. at Brown University; received his B.Sc. degrees in
Computer Science and Mathematics from Boston University (1999), his M.A. from
Boston University (1999), and his M.S. from Brown University (2003). From 1999
to 2001, he worked as a senior vision engineer at Cognex Corporation, where he
developed industrial vision applications for pattern analysis and
verification.
---|---
|  Frank Wood is an associate professor of computer science at the University
of British Columbia and a Canada CIFAR AI Chair at the Mila institute. He is
also the Co-Founder and Chief Scientist at Inverted AI. Before that Dr. Wood
was an associate professor in the Department of Engineering Science at the
University of Oxford and assistant professor of Statistics at Columbia
University. He formerly was a postdoctoral fellow of the Gatsby Computational
Neuroscience Unit of the University College London under Dr. Yee Whye Teh. He
received his PhD from Brown University in computer science under the
supervision of Dr. Michael Black and Dr. Tom Griffiths. Dr. Wood is a product
of the Illinois Mathematics and Science Academy from which he graduated in
1992. He began college at the University of Illinois at Chicago (UIC) but
transfered and received a B.S. in computer science from Cornell University in
1996. Prior to his academic career he was a successful entrepreneur having run
and sold the content-based image retrieval company ToFish! to Time Warner and
serving as CEO of Interfolio. He started his career working at both the
Cornell Theory Center and subsequently the Lawrence Berkeley National
Laboratory.
---|---

##  Appendix A Benchmarks and Training

###  A.1 Meta-Dataset

A brief description of the sampling procedure used in the Meta-Dataset setting
is already provided in Section 6.1. This sampling procedure, however, comes
with additional specifications that are uniform across all tasks (such as
count enforcing) and dataset specific details such as considering the class
hierarchy in ImageNet tasks. The full algorithm for sampling is outlined in
[32], and we refer the interested reader to Section 3.2 in [32] for complete
details. This procedure results in a task distribution where most tasks have
fewer than 10 classes and each class has fewer than 20 support examples. The
task frequency relative to the number of classes is presented in Figure 11a,
and the class frequency as compared to the class shot is presented in Figure
11b. The query set contains between 1 and 10 (inclusive) examples per class
for all tasks; fewer than 10 query examples occur only when there are not
enough total images to support 10 query examples.

###  A.2 mini/tiered-ImageNet

Task sampling across both mini-ImageNet and tiered-ImageNet first starts by
defining a constant number of ways and shots that will be used for each
generated task. For a $L$-shot $K$-way problem setting, first $K$ classes are
sampled from the dataset with uniform probability. Then, for each sampled
class, $L$ of the class images are sampled with uniform probability and used
as the support examples for the class. In addition, 10 query images (distinct
from the support images) are sampled per class.

Figure 10: Evaluating Transductive CNAPS on Meta-Dataset with different
minimum and maximum number of steps. Performances reported stem from five run
averages.

(a) Number of Tasks vs. Ways

(b) Number of Classes vs. Shots

Figure 11: Test-time task-way and class-shot frequency graphs. As shown, most tasks have fewer than 10 classes (way) and most classes have less than 20 support examples (shot). |  | mini-ImageNet Accuracy (%) | tiered-ImageNet Accuracy (%)
---|---|---|---
|  |  5-way |  10-way |  5-way |  10-way
Model | Transductive |  1-shot |  5-shot |  1-shot |  5-shot |  1-shot |  5-shot |  1-shot |  5-shot
MAML [24] | BN | 48.7±1.8 | 63.1±0.9 | 31.3±1.1 | 46.9±1.2 | 51.7±1.8 | 70.3±1.7 | 34.4±1.2 | 53.3±1.3
MAML+ [31] | Yes | 50.8±1.8 | 66.2±1.8 | 31.8±0.4 | 48.2±1.3 | 53.2±1.8 | 70.8±1.8 | 34.8±1.2 | 54.7±1.3
Reptile [26] | No | 47.1±0.3 | 62.7±0.4 | 31.1±0.3 | 44.7±0.3 | 49.0±0.2 | 66.5±0.2 | 33.7±0.3 | 48.0±0.3
Reptile+BN [26] | BN | 49.9±0.3 | 66.0±0.6 | 32.0±0.3 | 47.6±0.3 | 52.4±0.2 | 71.0±0.2 | 35.3±0.3 | 52.0±0.3
ProtoNet [22] | No | 46.1±0.8 | 65.8±0.7 | 32.9±0.5 | 49.3±0.4 | 48.6±0.9 | 69.6±0.7 | 37.3±0.6 | 57.8±0.5
RelationNet [36] | BN | 51.4±0.8 | 67.0±0.7 | 34.9±0.5 | 47.9±0.4 | 54.5±0.9 | 71.3±0.8 | 36.3±0.6 | 58.0±0.6
TPN [31] | Yes | 55.5±0.8 | 69.8±0.7 | 38.4±0.5 | 52.8±0.4 | 59.9±0.9 | 73.3±0.7 | 44.8±0.6 | 59.4±0.5
AttWeightGen [92] | No | 56.2±0.9 | 73.0±0.6 | - | - | - | - | - | -
TADAM [29] | No | 58.5±0.3 | 76.7±0.3 | - | - | - | - | - | -
Simple CNAPS | No | 53.2±0.9 | 70.8±0.7 | 37.1±0.5 | 56.7±0.5 | 63.0±1.0 | 80.0±0.8 | 48.1±0.7 | 70.2±0.6
Transductive CNAPS | Yes | 55.6±0.9 | 73.1±0.7 | 42.8±0.7 | 59.6±0.5 | 65.9±1.0 | 81.8±0.7 | 54.6±0.8 | 72.5±0.6
LEO [93] | No | 61.8±0.1 | 77.6±0.1 | - | - | 66.3±0.1 | 81.4±0.1 | - | -
MetaOptNet [94] | No | 62.6±0.6 | 78.6±0.5 | - | - | 66.0±0.7 | 81.6±0.5 | - | -
MetaBaseline [95] | No | 63.2±0.2 | 79.3±0.2 | - | - | 68.6±0.3 | 83.7±0.2 | - | -
FEAT [96] | No | 66.8±0.2 | 82.0±0.1 | - | - | 70.8±0.2 | 84.8±0.2 | - | -
SimpleShot [97] | No | 62.8±0.2 | 80.0±0.1 | - | - | 71.3±0.2 | 86.6±0.2 | - | -
RFS [98] | No | 64.8±0.6 | 82.1±0.4 | - | - | 71.5±0.7 | 86.0±0.5 | - | -
FRN [99] | No | 66.5±0.2 | 82.8±0.1 | - | - | 71.2±0.2 | 86.0±0.2 | - | -
DeepEMD [100] | No | 68.8±0.3 | 84.1±0.5 | - | - | 74.3±0.3 | 87.0±0.6 | - | -
S2M2 [101] | No | 64.9±0.2 | 83.2±0.1 | - | - | 73.7±0.2 | 88.6±0.1 | - | -
LR+DC [102] | No | 68.6±0.6 | 82.9±0.4 | - | - | 78.2±0.3 | 89.9±0.4 | - | -
TABLE VII: Few-shot visual classification results on 1/5-shot 5/10-way few-shot tasks on mini/tiered-ImageNet. Error intervals denote 95% confidence interval. |  | mini-ImageNet Accuracy (%) | tiered-ImageNet Accuracy (%)
---|---|---|---
|  |  5-way |  10-way |  5-way |  10-way
Model | Transductive |  1-shot |  5-shot |  1-shot |  5-shot |  1-shot |  5-shot |  1-shot |  5-shot
RS-FSL [103] | No | 65.3±0.8 | - | - | - | - | - | - | -
AmdimNet [104] | No | 76.8±0.2 | 91.0±0.1 | - | - | - | - | - | -
Simple CNAPS [33] \+ FETI | No | 77.4±0.8 | 90.3±0.4 | 63.5±0.6 | 83.1±0.4 | 71.4±1.0 | 86.0±0.6 | 57.1±0.7 | 78.5±0.5
Transductive CNAPS + FETI | Yes | 79.9±0.8 | 91.5±0.4 | 68.5±0.6 | 85.9±0.3 | 73.8±1.0 | 87.7±0.6 | 65.1±0.8 | 80.6±0.5
TABLE VIII: Few-shot visual classification results on 1/5-shot 5/10-way few-shot on mini/tiered-ImageNet with additional training data. For CNAP-based model, “FETI” indicates that the feature extractor used has been trained on ImageNet [73] exluding classes within the test splits of mini/tiered-ImageNet. Error intervals denote 95% confidence interval. |  |  | Out-of-Domain Accuracy (%) | Avg Rank
---|---|---|---|---
Model |  Backbone |  ImageNet |  Omniglot |  Aircraft |  Birds |  DTD |  QuickDraw |  Fungi |  Flower |  Signs |  MSCOCO |  MNIST |  CIFAR10 |  CIFAR100 | In | Out | All
RelationNet [85] | ResNet18 | 34.7±1.0 | 45.3±1.4 | 40.7±0.8 | 49.5±1.0 | 53.0±0.7 | 43.3±1.1 | 30.5±1.0 | 68.7±0.8 | 33.7±1.0 | 29.1±1.0 | - | - | - | 34.7 | 43.8 | 42.9
MatchingNet [86] | ResNet18 | 45.0±1.1 | 52.3±1.3 | 49.0±0.9 | 62.2±0.9 | 64.1±0.8 | 42.9±1.1 | 34.0±1.0 | 80.1±0.7 | 47.8±1.1 | 35.0±1.0 | - | - | - | 45.0 | 51.9 | 51.2
ProtoNet [22] | ResNet18 | 50.5±1.1 | 60.0±1.3 | 53.1±1.0 | 68.8±1.0 | 66.6±0.8 | 49.0±1.1 | 39.7±1.1 | 85.3±0.8 | 47.1±1.1 | 41.0±1.1 | - | - | - | 50.5 | 56.7 | 56.1
MAML [87] | ResNet18 | 45.5±1.1 | 55.5±1.5 | 56.2±1.1 | 63.6±1.1 | 68.0±0.8 | 44.0±1.3 | 32.1±1.1 | 81.7±0.8 | 50.9±1.5 | 35.3±1.2 | - | - | - | 45.5 | 54.2 | 53.3
ProtoMAML [32] | ResNet18 | 49.5±1.0 | 63.4±1.3 | 56.0±1.0 | 68.7±1.0 | 66.5±0.8 | 51.5±1.0 | 40.0±1.1 | 87.1±0.7 | 48.8±1.1 | 43.7±1.1 | - | - | - | 49.5 | 58.4 | 57.5
BOHB-E [88] | ResNet18 | 51.9±1.0 | 67.6±1.2 | 54.1±0.9 | 70.7±0.9 | 68.3±0.8 | 50.3±1.0 | 41.4±1.1 | 87.3±0.6 | 51.8±1.0 | 48.0±1.0 | - | - | - | 51.9 | 59.9 | 59.2
ProtoNet (Large) [105] | ResNet34 | 53.7±1.1 | 68.5±1.3 | 58.0±1.0 | 74.1±0.9 | 68.8±0.8 | 53.3±1.0 | 40.7±1.1 | 87.0±0.7 | 58.1±1.0 | 41.7±1.1 | - | - | - | 53.7 | 61.1 | 60.4
CTX [105] | ResNet34 | 62.8±1.0 | 82.2±1.0 | 79.5±0.9 | 80.6±0.9 | 75.6±0.6 | 72.7±0.8 | 51.6±1.1 | 95.3±0.4 | 82.6±0.8 | 59.9±1.0 | - | - | - | 62.8 | 75.6 | 74.3
Simple CNAPS | ResNet18 | 54.8±1.2 | 62.0±1.3 | 49.2±0.9 | 66.5±1.0 | 71.6±0.7 | 56.6±1.0 | 37.5±1.2 | 82.1±0.9 | 63.1±1.1 | 45.8±1.0 | 81.2±0.6 | 69.9±0.8 | 59.4±1.0 | 54.8* | 59.4* | 58.9*
Transductive CNAPS | ResNet18 | 54.1±1.1 | 62.9±1.3 | 48.4±0.9 | 67.3±0.9 | 72.5±0.7 | 58.0±1.0 | 37.7±1.1 | 82.8±0.8 | 61.8±1.1 | 45.8±1.0 | 83.9±0.7 | 68.9±0.8 | 60.0±1.1 | 54.1* | 59.7* | 59.1*
TABLE IX: Few-shot classification on Meta-Dataset, MNIST, and CIFAR10/100 with ImageNet-only training (all remaining datasets were kept for out-of-domain evaluation. Error intervals correspond to 95% confidence intervals, and bold values indicate statistically significant SoTA performance. *Accuracy averages for Simple and Transductive CNAPS exclude MNIST, CIFAR10 and CIFAR100 as other baselines do not report performance on these datasets. | In-Domain Accuracy (%) | Out-of-Domain Accuracy (%) | Avg Acc.
---|---|---|---
Simple CNAPS + Metric |  ImageNet |  Omniglot |  Aircraft |  Birds |  DTD |  QuickDraw |  Fungi |  Flower |  Signs |  MSCOCO |  MNIST |  CIFAR10 |  CIFAR100 | In | Out | All
Root Riemannian* | 48.2±1.0 | 88.4±0.7 | 78.0±0.7 | 70.2±0.8 | 61.7±0.7 | 74.5±0.7 | 44.3±1.0 | 79.9±0.7 | 60.4±1.1 | 31.1±1.1 | 94.1±0.4 | 62.4±0.8 | 54.7±1.0 | 68.2 | 60.5 | 65.2
Squared Mahalanobis | 58.6±1.1 | 91.7±0.6 | 82.4±0.7 | 74.9±0.8 | 67.8±0.8 | 77.7±0.7 | 46.9±1.0 | 90.7±0.5 | 73.5±0.7 | 46.2±1.1 | 93.9±0.4 | 74.3±0.7 | 60.5±1.0 | 73.8 | 69.7 | 72.2
TABLE X: Performance of root Riemannian ablation vs. squared Mahalanobis. Error intervals indicate 95% confidence intervals, and bold values indicate statistically significant state of the art performance. *Note that the “Root Riemannian” ablation was trained with gradient clipping (L2-norm capped at 5.0) to become numerically stable during training. | In-Domain Accuracy (%) | Out-of-Domain Accuracy (%) | Avg Accuracy
---|---|---|---
Model |  ImageNet |  Omniglot |  Aircraft |  Birds |  DTD |  QuickDraw |  Fungi |  Flower |  Signs |  MSCOCO |  MNIST |  CIFAR10 |  CIFAR100 | In | Out | All
Simple CNAPS | 58.6±1.1 | 91.7±0.6 | 82.4±0.7 | 74.9±0.8 | 67.8±0.8 | 77.7±0.7 | 46.9±1.0 | 90.7±0.5 | 73.5±0.7 | 46.2±1.1 | 93.9±0.4 | 74.3±0.7 | 60.5±1.0 | 73.8 | 69.7 | 72.2
No Refinements | 57.3±1.1 | 90.5±0.7 | 82.9±0.7 | 74.8±0.8 | 67.3±0.8 | 76.3±0.8 | 47.7±1.0 | 90.5±0.5 | 75.8±0.7 | 47.1±1.1 | 94.9±0.4 | 74.3±0.8 | 61.2±1.0 | 73.4 | 70.7 | 72.4
No Min/Max | 58.7±1.1 | 94.0±0.4 | 84.0±0.6 | 76.4±0.8 | 68.9±0.8 | 77.9±0.7 | 48.0±1.0 | 91.6±0.5 | 74.0±0.8 | 48.3±1.0 | 95.7±0.3 | 75.5±0.7 | 61.3±1.0 | 74.9 | 71.0 | 73.4
Min 2 Max 4 | 58.8±1.1 | 93.9±0.4 | 84.1±0.6 | 76.8±0.8 | 69.0±0.8 | 78.6±0.7 | 48.8±1.1 | 91.6±0.4 | 76.1±0.7 | 48.7±1.0 | 95.7±0.3 | 75.7±0.7 | 62.9±1.0 | 75.2 | 71.8 | 73.9
TABLE XI: Evaluating min/max refinement restrictions in Transductive CNAPS on
Meta-Dataset, MNIST, and CIFAR10/100.

###  A.3 Meta-Dataset Training/Testing

Following [33] and [21], we train our ResNet18 feature extractor as a
supervised multi-class image classifier on the training split of the ImageNet
subset of the Meta-Dataset. We directly follow the procedure described in
C.1.1 of [21] and use their released checkpoint. Images from the 712 ImageNet
classes designated for training by Meta-Dataset [32] are first resized to
84x84. The ResNet18 in Transductive CNAPS is then trained as a 712-class image
recognition task using this data. Training is done using the cross-entropy
loss for 125 epochs using SGD with momentum of 0.9, weight decay of 0.0001,
batch size of 256, and a learning rate of 0.1 that is reduced by a factor of
10 every 25 epochs. During training, the dataset was augmented with random
crops, horizontal flips, and color jitter.

Once the ResNet18 is trained, we freeze the parameters and proceed to train
the adaptation network using Episodic training [22, 24] where tasks themselves
are used as training examples. For each iteration of Episodic training, a task
(with additional ground truth query labels) is generated, and the adaptation
network is trained to minimize classification error (cross entropy) of the
query set given the task.

We train for a total of 110K tasks, with 16 tasks per batch, resulting in 6875
gradient updates. We train using Adam optimizer with learning rate of $5\times
10^{-4}$. We evaluate on the validation splits of all 8 in-domain and 1 out-
of-domain (MSCOCO) datasets, saving the best performing checkpoint for test-
time evaluation.

###  A.4 mini/tiered-ImageNet Training/Testing

Similar to Meta-Dataset, we first train the ResNet18 feature extractor. This
is done with respect to two settings: first, we directly use the training data
from mini-ImageNet and tiered-ImageNet, with 38,400 and 448,695 images
respectively. Second, we consider a larger training set of 825 classes from
ImageNet [73] that don’t overlap with the test sets of the benchmarks. In both
cases, we follow the same procedure as the one described for Meta-Dataset in
A.3 with the exception of training for 90 epochs and reducing the learning
rate every 30 epochs.

After training the ResNet18, the weights are frozen while we train the
adaptation network using Episodic training: at each iteration, a task is
generated, and we backpropagate the query set classification loss through the
adaptation network. For mini/tiered-ImageNet, we train for a total of 20K
tasks, validating performance every 2K tasks and saving the best checkpoint
for test-time evaluation. We, similarly, use the Adam optimiser with learning
rate of $5\times 10^{-4}$, and use a batch size of 16, for a total of 1250
gradient steps.

(a) 5-way 1-shot

(b) 5-way 5-shot

Figure 12: Mini-ImageNet performance as the number of query examples per class
increases on 1/5-shot 5-way tasks.

##  Appendix B Additional Experimental Results

(a) Atlantean

(b) Atlantean (FixedFE)

(c) Aurek-Besh

(d) Aurek-Besh (FixedFE)

(e) Avesta

(f) Avesta (FixedFE)

(g) Ge Ez

(h) Ge Ez (FixedFE)

(i) Glagolitic

(j) Glagolitic (FixedFE)

(k) Gurmukhi

(l) Gurmukhi (FixedFE)

(m) Kannada

(n) Kannada (FixedFE)

(o) Keble

(p) Keble (FixedFE)

(q) Malayalam

(r) Malayalam (FixedFE)

(s) Manipuri

(t) Manipuri (FixedFE)

(u) Mongolian

(v) Mongolian (FixedFE)

(w) Old Church

(x) Old Church (FixedFE)

Figure 13: “Out of the box” active learning results on 12 of the remaining
OMNIGLOT languages.

###  B.1 Evaluation on mini/tiered-ImageNet

Few-shot image classification results with error intervals on mini-ImageNet
and tiered-ImageNet in Table VII. Classification results with error intervals
when additional data is present on both mini- and tiered-ImageNet benchmarks
is shown in Table VIII.

###  B.2 Active Learning

“Out of the box” active learning results on additional OMNIGLOT language have
been provided in Figures 13 and 14.

###  B.3 Squared Mahalanobis vs. Root Riemannian

Given the theoretical grounding of our methods in Riemannian metric learning,
the root Riemannian distance is a natural baseline to consider. That is,
instead of producing class probabilities using Equation 1, we employ the
Mahalanobis distance itself as oppose to the squared Mahalanobis distance,
such that

$p(y^{*}=k\mid\mathbf{z}^{*})\propto\exp\big{(}-\sqrt{(\mathbf{z}^{*}-\bm{\mu}_{k})^{T}\mathbf{Q}_{k}^{-1}(\mathbf{z}^{*}-\bm{\mu}_{k})}\big{)}.$
(31)

At test time, given the same model checkpoints, both variations perform the
same as the decision boundaries remain intact because they are not impacted by
the square root function. The main difference observed is during training
where the square root function results in a differing optimization manifold,
and by consequence, gradient decent.

Extensive experiments were conducted exploring this and the root Riemannian
variation was observed to be incredibly unstable numerically. Standard
settings for training Simple and Transductive CNAPS resulted in exploding
gradients during backpropagation, and reducing step size proved ineffective.
Stable runs were only achieved with extreme gradient clipping (L2-norm capped
at 5.0). As demonstrated in Table X, this results in significantly worse
performance as compared to the squared Mahalanobis distance.

###  B.4 ImageNet-Only Training on Meta-Dataset

The Meta-Dataset benchmark also proposes an alternative experimental setting
where only the ImageNet training split is used for episodic training. All
remaining sub-datasets are kept for out-of-domain evaluation. This variation
has received comparatively little attention as it lacks the examples diversity
provided by the original Meta-Dataset setup during training.

We report ImageNet-only training results on the Meta-Dataset in Table IX. As
shown, Simple and Transductive CNAPS demonstrate competitive performances,
ranking second to that of CTX [105] whilst matching BOHB-E [88] and ResNet34
Prototypical Networks [105]. When compared to other methods with ResNet18
backbones, our models match the SoTA performance of BOHB-E.

(a) Oriya

(b) Oriya (FixedFE)

(c) Sylheti

(d) Sylheti (FixedFE)

(e) Syriac (Serto)

(f) Syriac (Serto) (FixedFE)

(g) Tengwar

(h) Tengwar (FixedFE)

(i) Tibetan

(j) Tibetan (FixedFE)

(k) ULOG

(l) ULOG (FixedFE)

Figure 14: “Out of the box” active learning results on the last 6 remaining
OMNIGLOT languages.

###  B.5 Performance vs. Number of Query Examples

Figure 12b shows the performance of Transductive CNAPS on 1/5-shot 5-way Mini-
ImageNet tasks as the number of query examples per category increases from 10
to 50. As shown, with a greater number of query examples, performance improves
as our method is able to exploit more unlabelled data.

###  B.6 Maximum and Minimum Number of Refinements in Transductive CNAPS

In our experiments, we use a minimum number of 2 refinement steps of class
parameters, with the maximum set to 4 on the Meta-Dataset and 10 on the
mini/tiered-ImageNet benchmarks. As shown in Table XI, the refinement criteria
itself, without any step constraints, results in a significant performance
gain as compared to performing no refinements. In fact, it accounts for the
majority of the accuracy gain for Transductive CNAPS. We further explore the
impact of these step-hyperparameters on the performance on Transductive CNAPS
on the Meta-Dataset in Figure 10.

As shown, requiring the same number of refinement steps for every task results
in sub-optimal performance. This is demonstrated by the fact that the peak
performance for each minimum number of steps is achieved with larger number of
maximum steps, showcasing the importance of allowing different numbers of
refinement steps depending on the task. In addition, we observe that as the
number of minimum refinement steps increases, the performance improves up to
two steps while declining after. This suggests that, unlike [30] where only a
single refinement step leads to the best performance, our Mahalanobis-based
approach can leverage extra steps to further refining the class parameters. We
do see a decline in performance with a higher number of steps; this suggests
that while our refinement criteria can be effective at performing different
number of steps depending on the task, it can potentially lead to over-
fitting, justifying the need for a well chosen maximum number of steps.