# One-Class Meta-Learning: Towards Generalizable Few-Shot Open-Set
Classification

Jedrzej Kozerawski
Department of Electrical & Computer Engineering
University of California, Santa Barbara
Santa Barbara, CA
<EMAIL_ADDRESS>
&Matthew Turk
Toyota Technological Institute at Chicago
6045 South Kenwood Ave,
Chicago, IL
<EMAIL_ADDRESS>

###### Abstract

Real-world classification tasks are frequently required to work in an open-set
setting. This is especially challenging for few-shot learning problems due to
the small sample size for each known category, which prevents existing open-
set methods from working effectively; however, most multiclass few-shot
methods are limited to closed-set scenarios. In this work, we address the
problem of few-shot open-set classification by first proposing methods for
few-shot one-class classification and then extending them to few-shot
multiclass open-set classification. We introduce two independent few-shot one-
class classification methods: Meta Binary Cross-Entropy (Meta-BCE), which
learns a separate feature representation for one-class classification, and
One-Class Meta-Learning (OCML), which learns to generate one-class classifiers
given standard multiclass feature representation. Both methods can augment any
existing few-shot learning method without requiring retraining to work in a
few-shot multiclass open-set setting without degrading its closed-set
performance. We demonstrate the benefits and drawbacks of both methods in
different problem settings and evaluate them on three standard benchmark
datasets, miniImageNet, tieredImageNet, and Caltech-UCSD-Birds-200-2011, where
they surpass the state-of-the-art methods in the few-shot multiclass open-set
and few-shot one-class tasks.

_Keywords_ meta-learning $\cdot$ few-shot learning $\cdot$ open-set $\cdot$
one-class

##  1 Introduction

|
---|---
(a) Few-Shot One-Class | (b) Few-Shot Multiclass Open-Set
Figure 1:  (a) Few-Shot One-Class classification and (b) Few-Shot Multiclass
Open-Set problems. The training set (top) contains few (one in this case)
labeled training examples from a single category (a) or five categories (b).
The test set (bottom) contains unseen examples from known categories present
in the training set treated as a known categories (Known - bottom left) and
examples from unknown categories not present in the training set (Unknown -
bottom right). The goal is to train a model given examples from the training
set such that it would be able to differentiate between examples coming from
the known categories (Known examples) and from other, unknown categories
(Unknown examples), while simultaneously correctly assigning examples
predicted as Known into one of $n$ known categories.

Deep learning methods are able to achieve high performance on large-scale
visual recognition tasks [1, 2, 3, 4], but the quality of the learned
representations greatly depends on the amount of available training data. In
many classification tasks this quantity is not sufficient to properly train a
neural network that would generalize well to unseen examples. Obtaining more
training data is often not feasible for numerous reasons, such as the natural
rarity of specific categories (_e.g._ , rare diseases or events), necessity of
fast adaptation to novel tasks (_e.g._ , early detection of new disease such
as COVID-19 or recognition of a new type of car by the autonomous driving
system), financial constraints, and researchers/scientists needing to train a
specialized model on their own small-scale dataset.

Few-shot learning [5, 6] is a problem in image classification dealing with
small sets of training data per class, where the task is to train a classifier
on data from a small, labeled support set $\mathcal{S}$ (consisting of data
from $n$ categories, each represented with $k$ examples – $k$-shot $n$-way
classification) and use a query set $\mathcal{Q}$ to assess the quality of
this classifier. Recent meta-learning approaches [6, 7, 8], designed for
better generalization capabilities of trained models to unseen data, help
increase the performance while simultaneously reducing the training time
necessary to adjust to the novel data. However, existing approaches
unfortunately assume that the query set $\mathcal{Q}$ has to contain examples
only from the same $n$ categories as the $\mathcal{S}$ (a closed-set setting),
which is very restrictive. Recently Liu  _et al._ [9] introduced the few-shot
open-set setting, where during the inference time the query set $\mathcal{Q}$
might contain additional unknown open-set categories $\mathcal{U}$ (that are
not present in the support set $\mathcal{S}$) that need to be differentiated
from the known classes $\mathcal{K}$
($\mathcal{Q}=\mathcal{K}\cup\mathcal{U}$) before performing the multiclass
classification step. Standard open-set approaches [10, 11, 12] assume access
to high number of per-category examples in order to model their distribution
and detect out-of-distribution samples (unknown $\mathcal{U}$), however a
small samples size in few-shot learning prevents those methods from working
well (or frequently at all when number of per-category examples drops to one).
This indicates the need for new meta-learning approaches capable of learning
to detect the unknown examples. Initial approach introduced by Liu  _et al._
[9] unfortunately provides only a relative ranking score for all examples
without any method to clearly differentiate between known examples
$\mathcal{K}$ and unknown examples $\mathcal{U}$, provides low closed-set
accuracy and cannot address a problem when $n=1$ since it operates on softmax
scores. In this work we propose to address these issues by studying meta-
learning approaches capable of out-of-distribution detection given only few
training examples (few-shot one-class classification), and extending them to
few-shot multiclass open-set problem setting (as seen in Fig. 1). Proposed
approaches will have the following benefits:

  * •

They can be used to augment any existing few-shot multiclass classification
approach (such as FEAT [7] or PEELER [9]) to operate in an open-set setting
without requiring retraining and without a performance drop in the closed-set
setting.

  * •

After training, our approach can work in both the few-shot open-set and
closed-set problem settings with any number of categories ($n$-way) and per-
category examples ($k$-shot), even when both $n=1$ and $k=1$ (one-shot one-
class).

  * •

They do not require separate background (unknown) categories present during
training (contrary to existing open-set methods).

Our contributions in this work can be summarized as:

  * •

We present two novel meta-learning methods for few-shot one-class image
classification that are capable of augmenting any existing few-shot multiclass
classification approach to work in few-shot multiclass open-set classification
setting.

  * •

We verify the value of the proposed approaches using few-shot one-class and
few-shot multiclass open-set experiments by reporting performance on two
benchmark few-shot datasets.

##  2 Related work

###  2.1 Few-shot classification

Few-shot classification refers to a problem where a model is trained to
generalize to novel, unseen samples in the query set $\mathcal{Q}$ represented
by a small number of examples in the support set $\mathcal{S}$. There are two
main types of approaches when addressing this problem. One type of methods is
concentrated around metric learning for better similarity and relation
embeddings. Siamese networks [13] compute the similarity score between two
images, while Matching Networks [14] learn classifiers for novel categories
based on a mapping from a small support set of examples (input-label pairs) to
a classifier for the given example. Snell  _et al._ [6] used Prototypical
Networks for few-shot learning by representing each class by the mean of its
examples in an embedding space learned by the neural network. Sung  _et al._.
[15] presented Relation Networks that consist of two modules, an embedding
module and a relation module, learning the appropriate relation between a
query image and each of $k$ images in $k$-way classification. Liu  _et al._
[9] used a modified prototypical network approach to tackle the few-shot open-
set problem. Qi  _et al._ [16] proposed to imprint the weights of a new
classifier with an embedding vector extracted from a base classifier pre-
trained on known categories. This idea is very similar to the Gidaris and
Komodakis approach [8]. Ye  _et al._ [7] proposed to use a transformer on top
of the prototypical network embeddings to learn a better mapping for class
representations. Another set of ideas focused on optimization approaches to
solve this problem. Ravi and Larochelle [5] used an LSTM to optimize updates
while training a network on a meta-set consisting of multiple datasets. Li
_et al._ [17] presented a Meta-SGD approach based on Stochastic Gradient
Descent. Finn  _et al._ [18] introduced Model-Agnostic Meta-Learning (MAML).

###  2.2 One-class classification

In one-class classification only data from positive category is available
during training. Schölkopf  _et al._ [19] addressed this with a One-Class
Support Vector Machine (OC-SVM) which maximizes the margin between the origin
and one-class samples. Chen  _et al._ [20] also used the One-Class SVM in the
image retrieval problem. Tax and Duin [21, 22] introduced Support Vector Data
Description (SVDD) and later augmented the method by generating artificial
outliers [23]. Ruff  _et al._ [24] proposed Deep Support Vector Data
Description (Deep-SVDD) to train a deep feature extractor jointly with the
one-class classification objective. Perera and Patel [25] introduced two loss
functions (compactness and descriptiveness loss) together with a template
matching matching framework for deep one-class classification. Sabokrou  _et
al._ [26] utilized a two-network architecture trained in a GAN-style
adversatial learning framework for adversarially learned one-class
classification. Kemmler  _et al._ [27] utilized Gaussian Process (GP) priors
for one-class classification. Kozerawski and Turk [28] proposed CLEAR to
predict the hyperparameters of a one-class SVM classifier given a single
positive example (one-shot one-class classification).

###  2.3 Open-set classification

Open-set classification is a machine learning problem when during the
inference stage the set of observable examples (the query set $\mathcal{Q}$)
can include unknown examples $\mathcal{U}$ coming from unknown categories
apart from known examples $\mathcal{K}$ coming from known categories (present
in the support set $\mathcal{S})$. Scheirer  _et al._ [11] introduced a new
variant of SVM (1-vs-set machine) based on the risk minimization. Bendale  _et
al._ [10] proposed OpenMax, a method using extreme value theory to re-evaluate
logit values. Ge  _et al._ [29] augmented OpenMax to a generative variant
called G-OpenMax. Neal  _et al._ [30] also proposed a generative approach
sythesizing open-set images while training, which helps detecting unknown
examples during the inference time. Liu  _et al._ [31] introduced a method for
long-tail open-set recognition with distance-based methods. Dhamija  _et al._
[32] proposed two losses (Entropic Open-Set and Objectosphere) to maximize the
difference in the softmax output for known and unknown samples. Yoshihashi
_et al._ [33] proposed CROSR (Classification-Reconstruction learning for Open-
Set Recognition) where they jointly perform classification and reconstruction
of the input data. Rudd  _et al._ [12] introduced extreme value machines which
utilize extreme value theory to model the probability of example coming from
unknown category. Liu  _et al._ [9] proposed a new problem setting of few-shot
open-set recognition and utilized method based on Prototypical Networks [6]
combined with entropy-based loss function.

##  3 Meta Learning for Few-Shot Open-Set Classification

Many traditional open-set classification approaches [12, 34, 35, 10]
differentiate known from unknown examples by modeling the distribution of
known classes and frequently focus on modeling the tail of this distribution
using the Extreme Value Theory. In few-shot learning modeling the distribution
of a known category and analyzing the tail of this distribution when the
number of examples is close to zero is not feasible (or even not possible in
case of number of examples $k=1$). Additionally, in few-shot learning the set
of known categories is different during meta-training and meta-testing phase,
thus preventing standard open-set methods from working efficiently. For these
reasons there is a need for an open-set meta-learning approach that can work
well in few-shot setting. Lack of sufficient training examples is also
problematic for existing one-class classification approaches [19, 36, 24] that
need abundance of positive training examples to model well the distribution of
the positive class in order to detect any anomalous examples (out-of-
distribution detection). We propose to solve mentioned problems with an
introduction of two separate few-shot one-class meta-learning approaches
capable of detection of unknown examples in both one-class and multiclass
open-set settings. Proposed methods work as separate modules that can augment
any existing closed-set few-shot multiclass classification method to work in
few-shot multiclass open-set setting without a need to retrain it.

Figure 2: An overview of OCML and Meta-BCE classification methods. Both
approaches are independent and standalone modules (indicated with dashed
lines) trained and utilized separately from each other. $k$ images belonging
to the known support class $c$ have their features extracted using a CNN
$f_{\phi}$ and a prototype representation $\bar{x}_{c}$ is calculated, which
is used for closed-set classification and as an input to OCML. In the example
above, $k=2$ with two upper images ( blue and  green) as the known examples
from the support class $c$, and the  red example is a query example. One-class
classification is performed either using either Eq. 2 (when using Meta-BCE) or
Eq. 4 (when using OCML).

###  3.1 Meta Binary Cross-Entropy (Meta-BCE)

Let us divide the dataset $\mathcal{D}$ by following a standard few-shot meta-
learning setting [14, 5] into three separate meta-sets: meta-training meta-set
$\mathcal{D}_{meta-train}$, meta-validation meta-set $\mathcal{D}_{meta-val}$,
and meta-testing meta-set $\mathcal{D}_{meta-test}$. Each of the meta-sets has
a separate, non-overlapping set of categories. We utilize episodic training as
is a standard practice in few-shot learning [14, 6, 7].

Following the work of Liu  _et al._ [9], let us use a separate branch of the
feature extractor $f_{\phi}$ indicated as $f_{\phi}^{\prime}$, but instead of
using it for multiclass features (as done by Liu) let us use it to learn a
separate feature representation for one-class classification (instead of only
multiclass classification). We hypothesize that high quality feature
representation for one-class classification differs from that for multiclass
classification (which is backed by our results). Standard multiclass
classifiers answer the question “Which of the $n$ classes does this new,
unknown example resemble the most?”. Answering this question depends highly on
the composition of the few-shot task (i.e., how similar are the categories and
how many of them there are). In order to eliminate that dependency, we need to
learn a different feature space representation where the probability of a new
example belonging or not belonging to the known category depends only on the
known positive examples from that category irrespective of other categories.
We propose to use a binary cross-entropy loss in a meta-learning setting to
simultaneously learn a one-class feature representation on the branch
$f_{\phi}^{\prime}$ and a one-class classification decision boundary:

$\mathcal{L}_{Meta\mbox{-}BCE}=-\frac{1}{N}\sum_{(x_{i},y_{i})\in\mathcal{D}_{meta-
train}}\sum_{c=1}^{N}(y_{i,c}\log
p(y_{c}|x_{i})+(1-y_{i,c})(1-p(y_{c}|x_{i})))$ (1)

where $y_{i,c}$ is a binary indicator (0 or 1) if class label $c$ is the
correct classification for observation $i$, and $p(y_{c}|x_{i})$ is the
probability of the example $x_{i}$ belonging to the category $y_{c}$:

$\begin{split}p(y=c|x_{i})=\frac{1}{1+exp(-(-d(f_{\phi}^{\prime}(x_{i}),\bar{x}_{c})-t))}=\frac{1}{1+exp(d(f_{\phi}^{\prime}(x_{i}),\bar{x}_{c})+t)}\end{split}$
(2)

where $t$ is a learnable parameter. You can see the feature extractor,
$f_{\phi}$, one-class branch $f_{\phi}^{\prime}$, and Meta-BCE approach in
Figure 2

###  3.2 One-Class Meta-Learning (OCML)

One-class classifiers should be ideally adapted to the class of interest,
which in few-shot scenario is difficult to achieve given small sample size. We
propose to use meta-learning to learn how to generate dynamically parameters
of a one-class classifier for a few-shot category. Kozerawski and Turk
addressed this issued by allowing to dynamically predict the parameters of a
one-class SVM classifier [28], however the method was limited to only a one-
shot setting as the transfer learning module was using a feature vector of a
single image as an input, and the method had multiple, separate stages of
training with a final classification limited to using SVMs or logistic
regression. To overcome these limitation we propose One-Class Meta-Learning
(OCML), a method that is able to dynamically create a one-class neural network
classifier for a novel category with more than a single image in the support
set $\mathcal{S}$ and is trained jointly with the feature extractor in a
single stage. OCML has a transfer learning module $g_{\theta}$ that learns how
to transform the feature representation of a category $c$ ($\bar{x}_{c}$) to a
weight vector $w_{c}$ of the one-class classifier for the category $c$:

$w_{c}=g_{\theta}(f_{\phi}(x_{c}))$ (3)

After obtaining the weight of the one-class classifier $w_{c}$, we can
calculate the probability of a novel example $x_{i}$ belonging to the category
$c$:

$p(y=c|x_{i})=\frac{1}{1+exp(-w_{c}\cdot f_{\phi}(x_{i}))}$ (4)

We use the binary cross-entropy loss using the probabilities calculated with
the Eq. 4 to train jointly both networks $f_{\phi}$ and $g_{\theta}$. The
overview of OCML can be seen in Figure 2.

###  3.3 Few-Shot Open-Set

In the standard open-set setting we have access to an abundance of examples
from known classes; however, in the few-shot open-set setup the known classes
will be revealed in the meta-testing phase (as the model is trained on a
separate, non-overlapping set of classes in meta-training). This means that
the algorithm should be able to work on any given set of known categories. To
address this issue, we utilize a divide-and-conquer approach and divide the
problem of detecting samples not belonging to any of $n$ known categories into
$n$ smaller problems of detecting samples not belonging to a single category
(one-class problem). This allows the use of meta-learning more efficiently, as
one-class learning depends only on positive examples from a single class,
whereas multiclass open-set depends also on the composition of the task (what
classes are known and how many of them). Additionally, few-shot one-class
classification can be thought of as a special case of few-shot multiclass
open-set classification, where in the $n$-way $k$-shot scenario, $n=1$. In
order to adapt both OCML and Meta-BCE to a multiclass scenario ($n\geq 2$) we
treat the few-shot open-set problem as an ensemble of $n$ few-shot one-class
problems and modify the prediction equation for Meta-BCE:

$p_{(}y\in\mathcal{U}|x_{i})=\max_{c\in
n}(1-\frac{1}{1+exp(-(-d(f_{\phi}^{\prime}(x_{i}),\bar{x}_{c})-t))})=\min_{c\in
n}(\frac{1}{1+exp(d(f_{\phi}^{\prime}(x_{i}),\bar{x}_{c})+t)})\ $ (5)

and for OCML:

$p(y\in\mathcal{U}|x_{i})=\max_{c\in
n}(1-\frac{1}{1+exp(-g_{\theta}(f_{\phi}(\bar{x}_{c}))\cdot
f_{\phi}(x_{i}))})=\min_{c\in
n}(\frac{1}{1+exp(-g_{\theta}(f_{\phi}(\bar{x}_{c}))\cdot f_{\phi}(x_{i}))})$
(6)

##  4 Experimental Results

###  4.1 Datasets and performance metrics

We conducted experiments on three benchmark datasets for few-shot learning:
miniImageNet [14], CUB-200-2011 [37], and tieredImageNet [38]. The
miniImageNet dataset is a set of 100 categories (a subset of ImageNet
categories [2]) with 600 images per category. We follow the meta-
training/meta-validation/meta-testing split of the 100 categories to 64/16/20
[5]. The CUB-200-2011 dataset has 200 visual categories and we follow meta-
training/meta-validation/meta-testing split as in Ye  _et al._ [7] to
100/50/50 categories respectively. The tieredImageNet dataset has 608 visual
categories and we follow meta-training/meta-validation/meta-testing split as
in Ye  _et al._ [38] to 351/97/60 categories respectively.

To evaluate all methods we have used three metrics in the few-shot one-class
settings: accuracy, F1-score, and AUROC (Area Under ROC curve) score. To
evaluate the methods in the few-shot open-set setting we have used four
metrics: closed-set accuracy (dubbed here accuracy), normalized accuracy (NA),
F1-open score, and AUROC score. For details on the performance metrics used
here please see Appendix A.

###  4.2 Few-Shot One-Class

In Table 1 we present the results for our two proposed approaches (Meta-BCE
and OCML) on the newly-introduced few-shot one-class task on the miniImageNet
dataset [14]. We compare our methods with benchmark one-class approaches such
as DeepSVDD [24], One-Class SVM [19], SVDD [36], DeepAnomaly [39], and with a
one-shot one-class approach introduced by Kozerawski and Turk [28] (CLEAR). As
hypothesised, standard many-shot one-class approaches do not translate well to
the few-shot setting. DeepSVDD [24] and DeepAnomaly provide only a relative
ranking score for all examples, thus allowing to calculate only AUROC score
without accuracy or F1-score. Their AUROC scores are lower than SVDD’s [36]
and OCSVM’s [19], which would indicate that a deep network approach might
require more training examples to work correctly. In the $1$-shot setting,
both OCSVM and SVDD completely overfit to the single examples resulting in a
random classification performance, and CLEAR [28] has slightly better
classification scores, but lower AUROC score. OCML has the best performance in
the $1$-shot setting with an accuracy of $68.05\%$, F1-score of $0.620$ and
second highest AUROC score of $0.759$. Meta-BCE has the second highest
performance in the $1$-shot setting (accuracy of $57.57\%$ and F1-score of
$0.283$) and the highest AUROC score ($0.764$). In the $5$-shot setting, Meta-
BCE becomes the best method with accuracy of $77.38\%$ and F1-score of
$0.793$, better than OCML (accuracy $77.74\%$ and F1-score $0.693$). Both
introduced methods (Meta-BCE and OCML) perform much better than any other
approaches, with Meta-BCE performing the best in the $5$-shot setting and OCML
in the $1$-shot setting. We also provide an upper-bound accuracy obtained
using FEAT [7] for a $k$-shot $2$-way classification setting when the unknown
class is treated as a known one with provided training examples. Calculated
upper-bound accuracy shows there is still room for progress as $1$-shot
accuracy is $79.76\%$ and $5$-shot accuracy is $88.61\%$. The difference in
performance between threshold and Meta-BCE confirms that multiclass and one-
class classification requires different feature representations. The results
on the CUB-200-2011 dataset [37] and on tieredImageNet [38] follow the above
conclusions, with OCML achieving best performance in the $1$-shot setting
($0.701$ F1-score on CUB and $0.800$ on tieredImageNet), and Meta-BCE
performing best in the $5$-shot setting on CUB ($0.783$ F1-score), while OCML
performs best on tieredImageNet ($0.878$ F1-score). For details on the
CUB-200-2011 and tieredImageNet results please see Appendix C.1.

|  | Accuracy (%) $\uparrow$ | F1-score $\uparrow$ | AUROC $\uparrow$
---|---|---|---|---
Method | Arch. | 1-shot
Proto Net [6] \+ DeepSVDD [24] | Conv64 | - | - | $0.675\pm 0.029$
Proto Net [6] \+ DeepAnomaly [39] | Conv64 | - | - | $0.633\pm 0.011$
Proto Net [6] \+ SVDD [36] | Conv64 | $50.00\pm 0.00$ | $0.000\pm 0.000$ | $0.703\pm 0.010$
Proto Net [6] \+ OCSVM [19] | Conv64 | $50.00\pm 0.00$ | $0.000\pm 0.000$ | $0.702\pm 0.011$
Proto Net [6] \+ Threshold | Conv64 | $53.18\pm 6.65$ | $0.131\pm 0.279$ | $0.706\pm 0.012$
CLEAR [28] | Conv64 | $50.42\pm 1.03$ | $0.077\pm 0.102$ | $0.507\pm 0.009$
Proto Net [6] \+ Meta-BCE [ours] | Conv64 | $57.57\pm 0.78$ | $0.283\pm 0.019$ | $\bf{0.764\pm 0.016}$
Proto Net [6] \+ OCML [ours] | Conv64 | $\bf{68.05\pm 0.99}$ | $\bf{0.620\pm 0.014}$ | $\bf{0.759\pm 0.012}$
Upper-bound (supervised FEAT [7])1 | Conv64 | $79.76\pm 0.91$ | - | -
|  | 5-shot
Proto Net [6] \+ DeepSVDD [24] | Conv64 | - | - | $0.714\pm 0.009$
Proto Net [6] \+ DeepAnomaly [39] | Conv64 | - | - | $0.729\pm 0.011$
Proto Net [6] \+ SVDD [36] | Conv64 | $50.12\pm 0.02$ | $0.005\pm 0.001$ | $0.747\pm 0.016$
Proto Net [6] \+ OCSVM [19] | Conv64 | $51.22\pm 0.01$ | $0.053\pm 0.006$ | $0.746\pm 0.016$
Proto Net [6] \+ Threshold | Conv64 | $60.93\pm 6.09$ | $0.584\pm 0.168$ | $0.742\pm 0.019$
Proto Net [6] \+ Meta-BCE [ours] | Conv64 | $\bf{77.38\pm 0.67}$ | $\bf{0.793\pm 0.005}$ | $\bf{0.850\pm 0.007}$
Proto Net [6] \+ OCML [ours] | Conv64 | $74.74\pm 0.88$ | $0.693\pm 0.013$ | $\bf{0.854\pm 0.009}$
Upper-bound (supervised FEAT [7])1 | Conv64 | $88.61\pm 0.63$ | - | -
Table 1: Experimental results on miniImageNet dataset for few-shot one-class
classification. The best results are shown in bold. 1 Supervised two-class
classification.

###  4.3 Few-Shot Open-Set

In Table 2 we provide the experimental results for the few-shot open-set task
on the miniImageNet dataset [14]. We compare our approaches with Gaussian
Embedding (GaussE) [9] and PEELER [9] by Liu  _et al._ , where they introduced
the few-shot open-set problem. We also compare our method with existing (non-
few-shot) open-set state-of-the-art methods such as Open-Max [10],
Counterfactual [30], Entropic Open-Set Loss [32], and Objectosphere Loss [32].
Following Liu  _et al._ [9] we perform experiments with ResNet-18 network and
in accordance to standard few-shot learning practices we add experiments with
Conv64 backbone as well.

|  | Accuracy (%) $\uparrow$ | NA (%) $\uparrow$ | F1-open $\uparrow$ | AUROC $\uparrow$
---|---|---|---|---|---
Method | Arch. | 1-shot
GaussE [9] \+ OpenMax [10]1 | Res18 | $57.89\pm 0.59$ | - | - | $0.589\pm 0.006$
GaussE [9] \+ Counterfactual [30]1 | Res18 | $57.89\pm 0.59$ | - | - | $0.522\pm 0.006$
GaussE [9]1 | Res18 | $57.89\pm 0.59$ | - | - | $0.587\pm 0.006$
PEELER [9]1 | Res18 | $58.31\pm 0.58$ | - | - | $0.617\pm 0.006$
PEELER [9] \+ threshold | Res18 | $56.98\pm 0.88$ | $44.77\pm 0.70$ | $0.357\pm 0.009$ | $0.626\pm 0.008$
PEELER [9] \+ Entropic Loss [32] | Res18 | $56.10\pm 0.86$ | $48.36\pm 0.69$ | $0.367\pm 0.008$ | $0.631\pm 0.008$
PEELER [9] \+ Objectosphere [32] | Res18 | $52.63\pm 0.86$ | $27.31\pm 0.37$ | $0.060\pm 0.007$ | $0.548\pm 0.008$
PEELER [9] \+ Meta-BCE [ours] | Res18 | $56.98\pm 0.88$ | $36.94\pm 0.44$ | $0.252\pm 0.007$ | $0.606\pm 0.008$
PEELER [9] \+ OCML [ours] | Res18 | $56.98\pm 0.88$ | $57.60\pm 0.76$ | $0.380\pm 0.006$ | $0.607\pm 0.007$
FEAT [7] \+ threshold | Res18 | $\bf{66.42\pm 0.65}$ | $56.53\pm 0.89$ | $\bf{0.442\pm 0.005}$ | $\bf{0.686\pm 0.008}$
FEAT [7] \+ Meta-BCE [ours] | Res18 | $\bf{66.42\pm 0.65}$ | $34.70\pm 0.31$ | $0.232\pm 0.006$ | $0.624\pm 0.006$
FEAT [7] \+ OCML [ours] | Res18 | $\bf{66.42\pm 0.65}$ | $\bf{59.79\pm 0.54}$ | $\bf{0.440\pm 0.005}$ | $0.623\pm 0.006$
PEELER [9] \+ threshold | Conv64 | $51.38\pm 0.81$ | $42.48\pm 0.75$ | $0.310\pm 0.009$ | $0.568\pm 0.007$
PEELER [9] \+ Entropic Loss [32] | Conv64 | $51.11\pm 0.82$ | $41.81\pm 0.68$ | $0.299\pm 0.008$ | $0.594\pm 0.008$
PEELER [9] \+ Objectosphere [32] | Conv64 | $49.01\pm 0.85$ | $33.79\pm 0.68$ | $0.204\pm 0.009$ | $0.519\pm 0.009$
PEELER [9] \+ Meta-BCE [ours] | Conv64 | $51.38\pm 0.81$ | $36.11\pm 0.41$ | $0.233\pm 0.007$ | $0.600\pm 0.008$
PEELER [9] \+ OCML [ours] | Conv64 | $51.38\pm 0.81$ | $\bf{55.80\pm 0.78}$ | $0.349\pm 0.006$ | $0.613\pm 0.008$
FEAT [7] \+ threshold | Conv64 | $\bf{55.01\pm 0.62}$ | $38.80\pm 0.40$ | $0.279\pm 0.005$ | $0.557\pm 0.006$
FEAT [7] \+ Meta-BCE [ours] | Conv64 | $\bf{55.01\pm 0.62}$ | $33.64\pm 0.30$ | $0.205\pm 0.006$ | $0.616\pm 0.006$
FEAT [7] \+ OCML [ours] | Conv64 | $\bf{55.01\pm 0.62}$ | $\bf{55.81\pm 0.53}$ | $\bf{0.373\pm 0.005}$ | $\bf{0.626\pm 0.006}$
|  | 5-shot
GaussE [9] \+ OpenMax [10]1 | Res18 | $75.31\pm 0.76$ | - | - | $0.675\pm 0.007$
GaussE [9] \+ Counterfactual [30]1 | Res18 | $75.31\pm 0.76$ | - | - | $0.533\pm 0.006$
GaussE [9]1 | Res18 | $75.31\pm 0.76$ | - | - | $0.665\pm 0.007$
PEELER [9]1 | Res18 | $75.08\pm 0.72$ | - | - | $0.699\pm 0.007$
PEELER [9] \+ threshold | Res18 | $73.04\pm 0.68$ | $52.78\pm 0.83$ | $0.472\pm 0.009$ | $0.677\pm 0.008$
PEELER [9] \+ Entropic Loss [32] | Res18 | $70.98\pm 0.64$ | $59.26\pm 0.69$ | $0.491\pm 0.006$ | $0.688\pm 0.007$
PEELER [9] \+ Objectosphere [32] | Res18 | $66.27\pm 0.77$ | $26.29\pm 0.19$ | $0.034\pm 0.005$ | $0.564\pm 0.008$
PEELER [9] \+ Meta-BCE [ours] | Res18 | $73.04\pm 0.68$ | $70.98\pm 0.58$ | $0.495\pm 0.004$ | $0.646\pm 0.005$
PEELER [9] \+ OCML [ours] | Res18 | $73.04\pm 0.68$ | $65.64\pm 0.49$ | $0.498\pm 0.004$ | $0.661\pm 0.006$
FEAT [7] \+ threshold | Res18 | $\bf{80.26\pm 0.45}$ | $63.89\pm 1.02$ | $0.537\pm 0.003$ | $\bf{0.727\pm 0.007}$
FEAT [7] \+ Meta-BCE [ours] | Res18 | $\bf{80.26\pm 0.45}$ | $\bf{73.32\pm 0.49}$ | $\bf{0.553\pm 0.003}$ | $0.667\pm 0.005$
FEAT [7] \+ OCML [ours] | Res18 | $\bf{80.26\pm 0.45}$ | $64.36\pm 0.48$ | $0.544\pm 0.004$ | $0.682\pm 0.006$
PEELER [9] \+ threshold | Conv64 | $66.92\pm 0.66$ | $47.24\pm 0.83$ | $0.399\pm 0.009$ | $0.608\pm 0.007$
PEELER [9] \+ Entropic Loss [32] | Conv64 | $65.06\pm 0.71$ | $47.77\pm 0.80$ | $0.386\pm 0.009$ | $0.625\pm 0.008$
PEELER [9] \+ Objectosphere [32] | Conv64 | $62.04\pm 0.71$ | $35.10\pm 0.73$ | $0.219\pm 0.010$ | $0.526\pm 0.009$
PEELER [9] \+ Meta-BCE [ours] | Conv64 | $66.92\pm 0.66$ | $\bf{68.74\pm 0.56}$ | $0.460\pm 0.004$ | $0.645\pm 0.005$
PEELER [9] \+ OCML [ours] | Conv64 | $66.92\pm 0.66$ | $63.57\pm 0.48$ | $0.466\pm 0.004$ | $0.666\pm 0.006$
FEAT [7] \+ threshold | Conv64 | $\bf{70.70\pm 0.50}$ | $45.85\pm 0.46$ | $0.389\pm 0.005$ | $0.595\pm 0.006$
FEAT [7] \+ Meta-BCE [ours] | Conv64 | $\bf{70.70\pm 0.50}$ | $\bf{69.10\pm 0.45}$ | $\bf{0.487\pm 0.004}$ | $0.669\pm 0.005$
FEAT [7] \+ OCML [ours] | Conv64 | $\bf{70.70\pm 0.50}$ | $61.29\pm 0.48$ | $\bf{0.490\pm 0.004}$ | $\bf{0.680\pm 0.006}$
Table 2: Experimental results on miniImageNet dataset for few-shot $5$-way
open-set classification with $5$ open-set categories. The best results are
shown in bold. 1 Results from Liu  _et al._ [9]

First of all, it is worth noticing that PEELER and Gaussian Embedding (GaussE)
introduced by Liu  _et al._ [9] do not provide a method for clearly
differentiating between known and unknown examples at inference time and only
provide a relative AUROC score and accuracy on closed-set (known) examples.
Additionally, their method operates on softmax scores thus making it not
suitable for scenario when $n=1$ (few-shot one-class). We have reproduced
PEELER approach using the original authors’ implementation and augmented it
with different existing methods of detecting unknown examples in the query
set: threshold method, Entropic Open-Set Loss [32], and Objectosphere Loss
[32]. Out of all methods, Objectosphere Loss [32] performs the worst,
obtaining the lowest normalized accuracy, F1-open score and AUROC score both
in $1$-shot and $5$-shot settings with ResNet-18 and Conv64 networks, and
results in a lower closed-set accuracy compared to other methods. This
confirms that methods performing well in standard open-set tasks, do not have
to transfer well to few-shot open-set tasks. In this specific scenario,
Objectosphere [32] learns to produce features of lower magnitude for unseen
classes, however, in the few-shot setting all classes in the meta-testing
meta-set $\mathcal{D}_{meta-test}$ will be considered as unseen since there is
no overlap in between categories present in the meta-training meta-set and
meta-testing meta-set. Entropic Open-Set Loss [32] aims to increase entropy
for unknown examples, which seems to translate better to few-shot setting
compared to simple threshold or Objectosphere [32] for both settings ($1$-shot
and $5$-shot) and both networks (ResNet-18 and Conv64), however it results in
a lower closed-set accuracy.

When we augment PEELER [9] with an ensemble of our proposed few-shot one-class
methods (Meta-BCE or OCML) we can significantly increase the open-set
performance without influencing the closed-set accuracy. The results confirm
the conclusion from the experiments on few-shot one-class task: Meta-BCE
results in a better performance for larger values of $k$ (per-category
examples) with normalized accuracy of $70.98\%$ (ResNet-18 and $k=5$) and OCML
results in a better performance for smaller values of $k$ with a normalized
accuracy of $57.60\%$ (ResNet-18 and $k=1$). The great benefit of our
approaches is that they do not degrade the performance of closed-set accuracy
(compared to Entropic Open-Set Loss [32] and Objectosphere [32]) as they work
as separate modules trained on top of existing few-shot learning methods.
Additionally, since OCML and Meta-BCE are trained as few-shot one-class
methods, they do not require separate background (or unknown) categories
present in the training set (contrary to PEELER [9], Entropic Open-Set Loss
[32], Objectosphere [32] or OpenMax [10]).

Standard multiclass open-set methods such as PEELER, OpenMax, or Entropic
Open-Set Loss cannot be adapted to the special case when number of known
categories ($n$) drops to one (few-shot one-class setting) since all of them
are softmax-based, which prevents them from working in the single-category
scenario. Our proposed approaches tackle both problems with superior
performance. Additionally, in the few-shot open-set setting, existing meta-
learning models due to the training process are tailored to the closed-set
setting, thus achieving worse performance on the open-set setting – as seen in
Table 2 through the comparison of Prototypical Networks [6] or FEAT [7] when
using the multiclass feature space (“+ theshold”) versus when using a one-
class feature space (Meta-BCE).

Since both Meta-BCE and OCML can be added to any existing few-shot closed-set
approaches to allow them to work in open-set settings, we combine them with
state-of-the-art few-shot closed-set method (FEAT) proposed by Ye  _et al._
[7]. FEAT results with Meta-BCE and OCML significantly outperform all the
other existing methods in all performance metrics with normalized accuracy of
$59.79\%$ with OCML for $k=1$ and $73.32\%$ with Meta-BCE for $k=5$ (with
ResNet-18). The results on the CUB-200-2011 dataset [37] and on tieredImageNet
[38] uphold these conclusions, as FEAT + OCML achieves the best performance in
the $1$-shot setting ($60.21\%$ normalized accuracy on CUB and $67.76\%$ on
tieredImageNet) and FEAT + Meta-BCE performs best in the $5$-shot setting on
the CUB dataset ($72.04\%$ normalized accuracy), while FEAT + OCML achieves
the best performance on the tieredImageNet dataset ($77.96\%$ normalized
accuracy). For details on the CUB-200-2011 and tieredImageNet results, please
see Appendix C.2.

##  5 Conclusions

We proposed two novel methods for few-shot one-class classification (Meta-BCE
and OCML) that can augment any existing few-shot learning method (such as
PEELER [9] or FEAT [7]) to work in the few-shot open-set setting. These
methods do not require retraining of the existing few-shot method, do not
degrade its performance in the closed-set setting, and (contrary to existing
open-set methods) do not require separate background categories present during
the training phase. Our approaches surpass the state-of-the-art methods in
few-shot one-class and few-shot multiclass open-set classification, with Meta-
BCE performing better when the number of per-category examples is higher, and
OCML performing optimally for smaller values of per-category examples.
Training high-quality models quickly and efficiently with smaller amounts of
data and with the ability to work in an open-set setting is an important
future direction for machine learning.

## References

  * [1] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , 2017, pp. 4700–4708.
  * [2] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein _et al._ , “Imagenet large scale visual recognition challenge,” _International journal of computer vision_ , vol. 115, no. 3, pp. 211–252, 2015.
  * [3] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 2818–2826.
  * [4] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 770–778.
  * [5] S. Ravi and H. Larochelle, “Optimization as a model for few-shot learning,” in _International Conference on Learning Representations_ , 2017.
  * [6] J. Snell, K. Swersky, and R. Zemel, “Prototypical networks for few-shot learning,” in _Advances in neural information processing systems_ , 2017, pp. 4077–4087.
  * [7] H.-J. Ye, H. Hu, D.-C. Zhan, and F. Sha, “Few-shot learning via embedding adaptation with set-to-set functions,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 8808–8817.
  * [8] S. Gidaris and N. Komodakis, “Dynamic few-shot visual learning without forgetting,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 4367–4375.
  * [9] B. Liu, H. Kang, H. Li, G. Hua, and N. Vasconcelos, “Few-shot open-set recognition using meta-learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 8798–8807.
  * [10] A. Bendale and T. E. Boult, “Towards open set deep networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 1563–1572.
  * [11] W. J. Scheirer, A. de Rezende Rocha, A. Sapkota, and T. E. Boult, “Toward open set recognition,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 35, no. 7, pp. 1757–1772, 2012.
  * [12] E. M. Rudd, L. P. Jain, W. J. Scheirer, and T. E. Boult, “The extreme value machine,” _IEEE transactions on Pattern Analysis and Machine Intelligence_ , vol. 40, no. 3, pp. 762–768, 2017.
  * [13] G. Koch, R. Zemel, and R. Salakhutdinov, “Siamese neural networks for one-shot image recognition,” in _ICML Deep Learning Workshop_ , 2015.
  * [14] O. Vinyals, C. Blundell, T. Lillicrap, K. Kavukcuoglu, and D. Wierstra, “Matching networks for one shot learning,” in _Advances in Neural Information Processing Systems_ , 2016, pp. 3630–3638.
  * [15] F. Sung, Y. Yang, L. Zhang, T. Xiang, P. H. Torr, and T. M. Hospedales, “Learning to compare: Relation network for few-shot learning,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 1199–1208.
  * [16] H. Qi, M. Brown, and D. G. Lowe, “Low-shot learning with imprinted weights,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 5822–5830.
  * [17] Z. Li, F. Zhou, F. Chen, and H. Li, “Meta-sgd: Learning to learn quickly for few shot learning,” _arXiv preprint arXiv:1707.09835_ , 2017.
  * [18] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in _International Conference on Machine Learning_ , 2017, pp. 1126–1135.
  * [19] B. Schölkopf, R. C. Williamson, A. J. Smola, J. Shawe-Taylor, and J. C. Platt, “Support vector method for novelty detection,” in _Advances in neural information processing systems_ , 2000, pp. 582–588.
  * [20] Y. Chen, X. S. Zhou, and T. S. Huang, “One-class svm for learning in image retrieval,” in _Image Processing, 2001. Proceedings. 2001 International Conference on_ , vol. 1. IEEE, 2001, pp. 34–37.
  * [21] D. M. Tax and R. P. Duin, “Data domain description using support vectors.” in _ESANN_ , vol. 99, 1999, pp. 251–256.
  * [22] ——, “Support vector domain description,” _Pattern recognition letters_ , vol. 20, no. 11, pp. 1191–1199, 1999.
  * [23] ——, “Uniform object generation for optimizing one-class classifiers,” _Journal of machine learning research_ , vol. 2, no. Dec, pp. 155–173, 2001\.
  * [24] L. Ruff, R. A. Vandermeulen, N. Görnitz, L. Deecke, S. A. Siddiqui, A. Binder, E. Müller, and M. Kloft, “Deep one-class classification,” in _Proceedings of the 35th International Conference on Machine Learning_ , vol. 80, 2018, pp. 4393–4402.
  * [25] P. Perera and V. M. Patel, “Learning deep features for one-class classification,” _IEEE Transactions on Image Processing_ , vol. 28, no. 11, pp. 5450–5463, 2019.
  * [26] M. Sabokrou, M. Khalooei, M. Fathy, and E. Adeli, “Adversarially learned one-class classifier for novelty detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 3379–3388.
  * [27] M. Kemmler, E. Rodner, E.-S. Wacker, and J. Denzler, “One-class classification with gaussian processes,” _Pattern Recognition_ , vol. 46, no. 12, pp. 3507–3518, 2013.
  * [28] J. Kozerawski and M. Turk, “Clear: Cumulative learning for one-shot one-class image recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 3446–3455.
  * [29] S. D. Zongyuan Ge and R. Garnavi, “Generative openmax for multi-class open set classification,” pp. 42.1–42.12, September 2017. [Online]. Available: https://dx.doi.org/10.5244/C.31.42
  * [30] L. Neal, M. Olson, X. Fern, W.-K. Wong, and F. Li, “Open set learning with counterfactual images,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 613–628.
  * [31] Z. Liu, Z. Miao, X. Zhan, J. Wang, B. Gong, and S. X. Yu, “Large-scale long-tailed recognition in an open world,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 2537–2546.
  * [32] A. R. Dhamija, M. Günther, and T. E. Boult, “Reducing network agnostophobia,” in _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , 2018, pp. 9175–9186.
  * [33] R. Yoshihashi, W. Shao, R. Kawakami, S. You, M. Iida, and T. Naemura, “Classification-reconstruction learning for open-set recognition,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 4016–4025.
  * [34] W. Liu, X. Wang, J. D. Owens, and Y. Li, “Energy-based out-of-distribution detection,” in _Advances in neural information processing systems_ , 2020\.
  * [35] X. Sun, Z. Yang, C. Zhang, K.-V. Ling, and G. Peng, “Conditional gaussian distribution learning for open set recognition,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 13 480–13 489.
  * [36] D. M. Tax and R. P. Duin, “Support vector data description,” _Machine learning_ , vol. 54, no. 1, pp. 45–66, 2004.
  * [37] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie, “The Caltech-UCSD Birds-200-2011 Dataset,” California Institute of Technology, Tech. Rep. CNS-TR-2011-001, 2011.
  * [38] M. Ren, E. Triantafillou, S. Ravi, J. Snell, K. Swersky, J. B. Tenenbaum, H. Larochelle, and R. S. Zemel, “Meta-learning for semi-supervised few-shot classification,” _arXiv preprint arXiv:1803.00676_ , 2018.
  * [39] I. Golan and R. El-Yaniv, “Deep anomaly detection using geometric transformations,” in _Advances in neural information processing systems_ , 2018.
  * [40] P. R. M. Júnior, R. M. De Souza, R. d. O. Werneck, B. V. Stein, D. V. Pazinato, W. R. de Almeida, O. A. Penatti, R. d. S. Torres, and A. Rocha, “Nearest neighbors distance ratio open-set classifier,” _Machine Learning_ , vol. 106, no. 3, pp. 359–386, 2017.
  * [41] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, high-performance deep learning library,” in _Advances in Neural Information Processing Systems 32_ , H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, Eds. Curran Associates, Inc., 2019, pp. 8024–8035. [Online]. Available: http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf

##  Appendix A Performance metrics

Few-Shot One-Class metrics. For the few-shot one-class experiments we have
used multiple metrics to capture the performance of tested models on the meta-
testing meta-set: classification accuracy, AUROC (Area Under ROC curve) score,
and F1-score.

Few-Shot Open-Set metrics. To obtain fair comparison with Liu  _et al._ [9] we
use the same metrics in the few-shot experiments: closed-set accuracy (dubbed
here accuracy) and AUROC (Area Under ROC curve) score. However, these measures
alone are not sufficient to capture the quality of open-set methods (impact of
unknown examples on the classification performance on known or closed-set
samples), and for this reason we will utilize two commonly used metrics in
open-set classification introduced by Júnior  _et al._ [40]: F1-open score and
Normalized Accuracy. Where F1-open score can be calculated using following
formula:

$\mbox{F1-open}=2\times\frac{P_{mi}\times R_{mi}}{P_{mi}+R_{mi}}$ (7)

$P_{mi}=\frac{\sum_{i=1}^{N}TP_{i}}{\sum_{i=1}^{N}(TP_{i}+FP_{i})},~{}~{}~{}R_{mi}=\frac{\sum_{i=1}^{N}TP_{i}}{\sum_{i=1}^{N}(TP_{i}+FN_{i})}$
(8)

And Normalized Accuracy is:

$NA=\lambda\times AKS+(1-\lambda)\times AUS$ (9)

where $\lambda$ is a weight hyperparameter balancing the AKS and AUS (set here
to $\lambda=0.5$). The AKS is the Accuracy on Known Samples:

$AKS=\frac{\sum_{i=1}^{N}(TP_{i}+TN_{i})}{\sum_{i=1}^{N}(TP_{i}+TN_{i}+FP_{i}+FN_{i})}$
(10)

and AUS is the Accuracy on Unknown Samples:

$AUS=\frac{TU}{TU+FU}$ (11)

where TP is the number of True Positives, TN is the number of True Negatives,
FP is the number of False Positives, FN is the number of False Negatives, TU
is the number of True Unknowns, and FU is the number of False Unknowns.

The proposed way of measuring AKS (Eq. 10) in a multiclass scenario results in
weighing heavily the number of True Negatives (TN) in the formula thus skewing
the quality of the metric. We propose to utilize the following way of
measuring the AKS, which is more frequently used in a multiclass
classification:

$AKS=\frac{1}{M}\sum_{i=1}^{M}I(\hat{y}_{i}=y_{i})$ (12)

where $M$ is number of known examples, $y_{i}$ is the true label of an i-th
examples, $\hat{y}_{i}$ is the predicted label for the i-th example, and $I()$
is the indicator function, which returns 1 if the prediction matches the
ground truth and 0 otherwise.

Open-Set Classification Rate Curve (OSCRC) metric introduced by Dhamija _et
al._ [32] assumes that the open-set method does not provide a clear
classification decision (whether a novel examples in known or unknown) and
scans throug different softmax threshold values in order to produce the curve.
Our approaches provide a clear classification decision (a single operating
point on the OSCRC curve), thus calculation of OSCRC metric is not possible.

##  Appendix B Testing procedure

###  B.1 Few-Shot One-Class

Every method is tested using the same testing procedure. The testing procedure
consists of $M_{episodes}$ testing episodes, each consisting of a support set
$\mathcal{S}$ in form of $1$-way $k$-shot setting, a query set $\mathcal{Q}$
with $q$ unseen examples from the category present in the support set and with
$q$ examples from another, unknown category. All episodes contain data from
the meta-testing meta-set $\mathcal{D}_{meta-test}$. We report the average
performance with a $95\%$ confidence interval. In all the experiments we have
set $q=15$, $M_{episodes}=10000$.

###  B.2 Few-Shot Open-Set

Every method is tested using the same testing procedure. The testing procedure
consists of $M_{episodes}$ testing episodes, each consisting of a support set
$\mathcal{S}$ in form of $n$-way $k$-shot setting, a query set $\mathcal{Q}$
with $q$ per-category unseen examples from $N$ categories present in the
support set and with $q$ per-category examples from $n_{U}$ unknown
categories. All episodes contain data from the meta-testing meta-set
$\mathcal{D}_{meta-test}$. We report the average performance with a $95\%$
confidence interval. In all the experiments we have set $q=15$,
$M_{episodes}=10000$, and $n_{U}=N$.

##  Appendix C Additional experiments: CUB-200-2011 and tieredImageNet

###  C.1 Few-Shot One-Class

In Tables 3 and 4 we present the results for few-shot one-class experiments on
the CUB-200-2011 dataset [37] and tieredImageNet dataset [38] respectively.
The performance of all methods confirms the conclusions from the miniImageNet
experiments. Standard one-class approaches (DeepSVDD [24], DeepAnomaly [39],
OCSVM [19], SVDD [36]) do not work well in the few-shot setting (especially
when considering accuracy and F1-score metrics in $1$-shot setting).
Prototypical Networks [6] with OCML perform the best in the $1$-shot setting
with $72.14\%$ accuracy, $0.701$ F1-score, and $0.801$ AUROC score on CUB
dataset and with $72.13\%$ accuracy, $0.724$ F1-score, and $0.800$ AUROC score
on tieredImageNet dataset. We can see that the supervised upper-bound
calculated using FEAT [7] still gives a lot of room for improvement with
accuracy of $84.25\%$ on CUB and $88.54\%$ on tieredImageNet dataset. In the
$5$-shot setting, on CUB dataset OCML has the best accuracy ($78.74\%$) and
AUROC score ($0.874$), while Meta-BCE has the best F1-score of $0.783$. On
tieredImageNet OCML has the best accuracy ($78.89\%$), highest F1-score
($0.803$) and AUROC score ($0.878$). The supervised two-class upper-bound of
FEAT [7] rises higher as well to $91.26\%$ on CUB and to $94.56\%$ on
tieredImageNet.

|  | Accuracy (%) $\uparrow$ | F1-score $\uparrow$ | AUROC $\uparrow$
---|---|---|---|---
Method | Arch. | 1-shot
Proto Net [6] \+ DeepSVDD [24] | Conv64 | - | - | $0.603\pm 0.007$
Proto Net [6] \+ DeepAnomaly [39] | Conv64 | - | - | $0.655\pm 0.011$
Proto Net [6] \+ SVDD [36] | Conv64 | $50.00\pm 0.00$ | $0.000\pm 0.000$ | $0.599\pm 0.030$
Proto Net [6] \+ OCSVM [19] | Conv64 | $50.00\pm 0.00$ | $0.000\pm 0.000$ | $0.633\pm 0.015$
Proto Net [6] \+ Threshold | Conv64 | $50.05\pm 0.04$ | $0.002\pm 0.001$ | $0.697\pm 0.005$
CLEAR [28] | Conv64 | $50.24\pm 0.36$ | $0.008\pm 0.012$ | $0.620\pm 0.120$
Proto Net [6] \+ Meta-BCE [ours] | Conv64 | $54.72\pm 1.29$ | $0.169\pm 0.044$ | $0.774\pm 0.009$
Proto Net [6] \+ OCML [ours] | Conv64 | $\bf{72.14\pm 0.56}$ | $\bf{0.701\pm 0.016}$ | $\bf{0.801\pm 0.006}$
Upper-bound (supervised FEAT [7])1 | Conv64 | $84.25\pm 0.89$ | - | -
|  | 5-shot
Proto Net [6] \+ DeepSVDD [24] | Conv64 | - | - | $0.656\pm 0.007$
Proto Net [6] \+ DeepAnomaly [39] | Conv64 | - | - | $0.762\pm 0.010$
Proto Net [6] \+ SVDD [36] | Conv64 | $50.61\pm 0.41$ | $0.036\pm 0.021$ | $0.609\pm 0.036$
Proto Net [6] \+ OCSVM [19] | Conv64 | $52.40\pm 0.45$ | $0.130\pm 0.019$ | $0.656\pm 0.019$
Proto Net [6] \+ Threshold | Conv64 | $64.92\pm 1.29$ | $0.590\pm 0.063$ | $0.725\pm 0.003$
Proto Net [6] \+ Meta-BCE [ours] | Conv64 | $76.16\pm 0.58$ | $\bf{0.783\pm 0.007}$ | $0.839\pm 0.006$
Proto Net [6] \+ OCML [ours] | Conv64 | $\bf{78.74\pm 1.07}$ | $0.774\pm 0.019$ | $\bf{0.874\pm 0.010}$
Upper-bound (supervised FEAT [7])1 | Conv64 | $91.26\pm 0.56$ | - | -
Table 3: Experimental results on CUB-200-2011 dataset for few-shot one-class classification. The best results are shown in bold. 1 Supervised two-class classification. |  | Accuracy (%) $\uparrow$ | F1-score $\uparrow$ | AUROC $\uparrow$
---|---|---|---|---
Method | Arch. | 1-shot
Proto Net [6] \+ DeepSVDD [24] | Conv64 | - | - | $0.656\pm 0.004$
Proto Net [6] \+ DeepAnomaly [39] | Conv64 | - | - | $0.603\pm 0.012$
Proto Net [6] \+ SVDD [36] | Conv64 | $50.00\pm 0.00$ | $0.000\pm 0.000$ | $0.669\pm 0.003$
Proto Net [6] \+ OCSVM [19] | Conv64 | $50.00\pm 0.00$ | $0.000\pm 0.000$ | $0.676\pm 0.003$
Proto Net [6] \+ Threshold | Conv64 | $50.48\pm 0.11$ | $0.017\pm 0.003$ | $0.704\pm 0.008$
CLEAR [28] | Conv64 | $50.53\pm 0.53$ | $0.552\pm 0.006$ | $0.509\pm 0.008$
Proto Net [6] \+ Meta-BCE [ours] | Conv64 | $55.87\pm 0.38$ | $0.208\pm 0.010$ | $0.755\pm 0.009$
Proto Net [6] \+ OCML [ours] | Conv64 | $\bf{72.13\pm 0.33}$ | $\bf{0.724\pm 0.003}$ | $\bf{0.800\pm 0.004}$
Upper-bound (supervised FEAT [7])1 | ResNet12 | $88.54\pm 0.81$ | - | -
|  | 5-shot
Proto Net [6] \+ DeepSVDD [24] | Conv64 | - | - | $0.678\pm 0.003$
Proto Net [6] \+ DeepAnomaly [39] | Conv64 | - | - | $0.719\pm 0.011$
Proto Net [6] \+ SVDD [36] | Conv64 | $50.52\pm 0.04$ | $0.021\pm 0.001$ | $0.724\pm 0.003$
Proto Net [6] \+ OCSVM [19] | Conv64 | $52.37\pm 0.09$ | $0.106\pm 0.003$ | $0.712\pm 0.003$
Proto Net [6] \+ Threshold | Conv64 | $61.90\pm 0.54$ | $0.426\pm 0.011$ | $0.741\pm 0.008$
Proto Net [6] \+ Meta-BCE [ours] | Conv64 | $75.62\pm 0.65$ | $0.774\pm 0.005$ | $0.831\pm 0.007$
Proto Net [6] \+ OCML [ours] | Conv64 | $\bf{78.89\pm 0.67}$ | $\bf{0.803\pm 0.006}$ | $\bf{0.878\pm 0.007}$
Upper-bound (supervised FEAT [7])1 | ResNet12 | $94.56\pm 0.51$ | - | -
Table 4: Experimental results on tieredImageNet dataset for few-shot one-class
classification. The best results are shown in bold. 1 Supervised two-class
classification.

###  C.2 Few-Shot Open-Set

In Tables 5 and 6 we can see the performance comparison of few-shot open-set
methods on CUB-200-2011 dataset [37] and tieredImageNet [38]. We can see that
both Meta-BCE and OCML do not degrade the closed-set accuracy (dubbed accuracy
here), hence methods initially performing well in this setting (_e.g._ FEAT
[7]) keep their superior performance. On CUB dataset OCML performs the best in
the $1$-shot setting with normalized accuracy of $60.21\%$ and F1-open score
of $0.414$ (with FEAT [7]) and AUROC score of $0.654$ with Prototypical
Networks [6]. In the $5$-shot setting the best performing method is Meta-BCE
with normalized accuracy of $72.04\%$ and F1-open score of $0.541$. Similarly
to $1$-shot setting, Prototypical Networks [6] with OCML has the best AUROC
score of $0.736$. On tieredImageNet dataset OCML performs the best in the
$1$-shot setting with normalized accuracy of $67.76\%$ and F1-open score of
$0.478$ (with FEAT [7]). In the $5$-shot setting the best performing method is
OCML with normalized accuracy of $77.96\%$ and F1-open score of $0.577$.

|  | Accuracy (%) $\uparrow$ | NA (%) $\uparrow$ | F1-open $\uparrow$ | AUROC $\uparrow$
---|---|---|---|---|---
Method | Arch. | 1-shot
PEELER [9] | Conv64 | $52.56\pm 0.73$ | - | - | $0.549\pm 0.003$
PEELER [9] \+ threshold | Conv64 | $52.56\pm 0.73$ | $26.51\pm 1.48$ | $0.042\pm 0.036$ | $0.605\pm 0.007$
Proto Nets [6] \+ threshold | Conv64 | $41.72\pm 1.03$ | $25.03\pm 0.02$ | $0.001\pm 0.001$ | $0.566\pm 0.009$
Proto Nets [6] \+ Meta-BCE [ours] | Conv64 | $41.72\pm 1.03$ | $35.66\pm 4.85$ | $0.229\pm 0.060$ | $0.613\pm 0.019$
Proto Nets [6] \+ OCML [ours] | Conv64 | $41.72\pm 1.03$ | $48.61\pm 5.72$ | $0.371\pm 0.035$ | $\bf{0.654\pm 0.010}$
FEAT [7] \+ threshold | Conv64 | $\bf{63.16\pm 0.74}$ | $32.69\pm 0.29$ | $0.199\pm 0.005$ | $0.537\pm 0.007$
FEAT [7] \+ Meta-BCE [ours] | Conv64 | $\bf{63.16\pm 0.74}$ | $32.70\pm 0.26$ | $0.190\pm 0.005$ | $0.648\pm 0.006$
FEAT [7] \+ OCML [ours] | Conv64 | $\bf{63.16\pm 0.74}$ | $\bf{60.21\pm 0.80}$ | $\bf{0.414\pm 0.005}$ | $0.556\pm 0.005$
|  | 5-shot
Proto Nets [6] \+ OpenMax [10] | Conv64 | $70.97\pm 0.99$ | $25.60\pm 0.15$ | $0.02\pm 0.005$ | $0.574\pm 0.011$
PEELER [9] | Conv64 | $71.65\pm 0.79$ | - | - | $0.637\pm 0.004$
PEELER [9] \+ threshold | Conv64 | $71.65\pm 0.79$ | $42.79\pm 6.17$ | $0.348\pm 0.062$ | $0.665\pm 0.010$
Proto Nets [6] \+ threshold | Conv64 | $70.97\pm 0.99$ | $37.62\pm 1.56$ | $0.298\pm 0.022$ | $0.606\pm 0.009$
Proto Nets [6] \+ Meta-BCE [ours] | Conv64 | $70.97\pm 0.99$ | $68.59\pm 2.64$ | $0.509\pm 0.009$ | $0.661\pm 0.031$
Proto Nets [6] \+ OCML [ours] | Conv64 | $70.97\pm 0.99$ | $47.70\pm 9.15$ | $0.426\pm 0.103$ | $\bf{0.736\pm 0.007}$
FEAT [7] \+ threshold | Conv64 | $\bf{77.47\pm 0.55}$ | $35.31\pm 0.34$ | $0.261\pm 0.005$ | $0.568\pm 0.007$
FEAT [7] \+ Meta-BCE [ours] | Conv64 | $\bf{77.47\pm 0.55}$ | $\bf{72.04\pm 0.43}$ | $\bf{0.541\pm 0.004}$ | $0.713\pm 0.005$
FEAT [7] \+ OCML [ours] | Conv64 | $\bf{77.47\pm 0.55}$ | $66.95\pm 0.60$ | $0.513\pm 0.004$ | $0.573\pm 0.005$
Table 5: Experimental results on CUB-200-2011 dataset for few-shot $5$-way open-set classification with $5$ open-set categories. The best results are shown in bold.  |  | Accuracy (%) $\uparrow$ | NA (%) $\uparrow$ | F1-open $\uparrow$ | AUROC $\uparrow$
---|---|---|---|---|---
Method | Arch. | 1-shot
PEELER [9] | Conv64 | $38.55\pm 0.43$ | - | - | $0.546\pm 0.004$
Proto Nets [6] \+ threshold | Conv64 | $43.77\pm 0.47$ | $25.38\pm 0.05$ | $0.011\pm 0.001$ | $0.576\pm 0.004$
Proto Nets [6] \+ Meta-BCE [ours] | Conv64 | $43.77\pm 0.47$ | $33.74\pm 0.21$ | $0.339\pm 0.004$ | $0.623\pm 0.004$
Proto Nets [6] \+ OCML [ours] | Conv64 | $43.77\pm 0.47$ | $56.06\pm 0.47$ | $0.299\pm 0.003$ | $0.659\pm 0.004$
FEAT [7] \+ threshold | ResNet12 | $\bf{70.88\pm 0.72}$ | $50.97\pm 0.06$ | $0.463\pm 0.007$ | $\bf{0.697\pm 0.006}$
FEAT [7] \+ Meta-BCE [ours] | ResNet12 | $\bf{70.88\pm 0.72}$ | $39.26\pm 0.36$ | $0.308\pm 0.006$ | $0.592\pm 0.006$
FEAT [7] \+ OCML [ours] | ResNet12 | $\bf{70.88\pm 0.72}$ | $\bf{67.76\pm 0.86}$ | $\bf{0.478\pm 0.005}$ | $0.610\pm 0.006$
|  | 5-shot
Proto Nets [6] \+ OpenMax [10] | Conv64 | $71.04\pm 0.40$ | $25.73\pm 0.04$ | $0.024\pm 0.001$ | $0.585\pm 0.003$
PEELER [9] | Conv64 | $67.71\pm 0.42$ | - | - | $0.635\pm 0.004$
Proto Nets [6] \+ threshold | Conv64 | $71.04\pm 0.40$ | $38.08\pm 0.25$ | $0.302\pm 0.004$ | $0.615\pm 0.004$
Proto Nets [6] \+ Meta-BCE [ours] | Conv64 | $71.04\pm 0.40$ | $70.46\pm 0.37$ | $0.495\pm 0.003$ | $0.672\pm 0.004$
Proto Nets [6] \+ OCML [ours] | Conv64 | $71.04\pm 0.40$ | $72.87\pm 0.36$ | $0.509\pm 0.003$ | $0.727\pm 0.004$
FEAT [7] \+ threshold | ResNet12 | $\bf{84.22\pm 0.53}$ | $67.91\pm 0.05$ | $0.545\pm 0.005$ | $\bf{0.768\pm 0.005}$
FEAT [7] \+ Meta-BCE [ours] | ResNet12 | $\bf{84.22\pm 0.53}$ | $76.71\pm 0.73$ | $0.570\pm 0.004$ | $0.636\pm 0.005$
FEAT [7] \+ OCML [ours] | ResNet12 | $\bf{84.22\pm 0.53}$ | $\bf{77.96\pm 0.63}$ | $\bf{0.577\pm 0.003}$ | $0.657\pm 0.006$
Table 6: Experimental results on tieredImageNet dataset for few-shot $5$-way
open-set classification with $5$ open-set categories. The best results are
shown in bold.

##  Appendix D Implementation details

We have utilized PyTorch [41] to train all models. In order to implement
results with PEELER [9] and FEAT [7] we have used the original implementations
supplied by the authors of PEELER111https://github.com/BoLiu-SVCL/meta-open
and FEAT222https://github.com/Sha-Lab/FEAT. We have used Prototypical Networks
implementation introduced from the authors of FEAT as well. Meta-BCE uses a
separate branch of the main feature extractor ($f_{\phi}^{\prime}$) which in
case of Conv64 network is the last convnet block (2 last convolutional
layers). OCML uses a transfer learning module $g_{\theta}$ which in case of a
Conv64 architecture is a single fully connected linear layer with input and
output dimension of $1600$. As both our methods augment existing few-shot
closed-set approaches, all training hyperparameters are consistent with
original hyperparameters introduced by the authors of the closed-set
approaches (PEELER, FEAT or Prototypical Networks). We will release the code
necessary to reproduce the results from this paper along all pre-trained
models upon the publication.

##  Appendix E Ablation studies

###  E.1 OCML $g_{\theta}$ architecture

$g_{\theta}$ architecture name | Layer 1 | Layer 2
---|---|---
1 layer | FC layer (1600, 1600) | -
2 layers, middle dim 100 | FC layer (1600, 100) | FC layer (100, 1600)
2 layers, middle dim 500 | FC layer (1600, 500) | FC layer (500, 1600)
2 layers, middle dim 1000 | FC layer (1600, 1000) | FC layer (1000, 1600)
Table 7: Tested architectures for the transfer learning module $g_{\theta}$ for OCML approach. |
---|---
(a) One-Class Accuracy | (b) AUROC
Figure 3: Ablation study results for different architectures of the transfer
learning module $g_{\theta}$ for OCML approach. Results for four different
architectures are presented for different $k$-shot scenarios (different sizes
of the support set $\mathcal{S}$ of the meta-testing meta-set
$\mathcal{D}_{meta-test}$).

OCML method utilizes a transfer learning module $g_{\theta}$ transforming a
class-level representation $\bar{x}_{c}$ into the weights $w_{c}$ of a
classifier for a category $c$. We performed an ablation study comparing the
impact of the architecture of the transfer learning module $g_{\theta}$ on the
performance of the network. In Table 7 we can see four architectures of the
transfer learning module $g_{\theta}$ being tested. All four architectures
consist of only fully connected layers (FC) with input, output dimensions in
the parenthesis. The ablation study is performed on miniImageNet dataset with
the Conv64 network, which has a feature space dimensionality of $1600$, hence
the basic architecture for $g_{\theta}$ has only a single FC layer keeping the
dimensionality the same. With three additional architectures we have tested
whether adding more layers to the transfer learning module is beneficial and
what is the impact of the latent space dimensionality (middle dim in Table 7).
All experiments were repeated five times and the average performance along the
$95\%$ confidence interval was reported.

The results of the experiments for all four settings present in Table 7 are
visible in Figure 3. On the left we can see the impact of the architecture on
the accuracy of the method for various sizes of the support set category in
the meta-testing meta-set (various $k$ in the $k$-shot scenario). On the right
of Figure 3 we can see the impact of the architecture on the AUROC score. The
single-layer architecture achieves the best performance (both accuracy and
AUROC score) across all $k$ in the $k$-shot setting. Among the two-layer
architectures, the one with the latent dimensionality of $100$ achieves the
lowest performance across all $k$, which might indicate that the low-
dimensional space is not enough to reliably compress the necessary
information. Two-layer architectures with a higher latent dimensionality (500
and 1000) have similar performance, although lower than the single-layer
architecture. The performance of all the methods increases with with number of
examples ($k$) in the support set $\mathcal{S}$ of the meta-testing meta-set
$\mathcal{D}_{meta-test}$.

###  E.2 Meta-BCE separate branch

|  | Accuracy (%) $\uparrow$ | NA (%) $\uparrow$ | F1-open $\uparrow$ | AUROC $\uparrow$
---|---|---|---|---|---
Method | Arch. | 1-shot
Proto Nets [6] \+ Meta-BCE [ours] | Conv64 | $41.72\pm 1.03$ | $35.66\pm 4.85$ | $0.229\pm 0.060$ | $0.613\pm 0.019$
Proto Nets [6] \+ Meta-BCEC [ours] | Conv64 | $41.72\pm 1.03$ | $28.66\pm 3.78$ | $0.080\pm 0.077$ | $0.629\pm 0.009$
|  | 5-shot
Proto Nets [6] \+ Meta-BCE [ours] | Conv64 | $70.97\pm 0.99$ | $68.59\pm 2.64$ | $0.509\pm 0.009$ | $0.661\pm 0.031$
Proto Nets [6] \+ Meta-BCEC [ours] | Conv64 | $70.97\pm 0.99$ | $37.13\pm 1.53$ | $0.233\pm 0.162$ | $0.684\pm 0.011$
Table 8: Experimental results on CUB-200-2011 dataset for few-shot $5$-way
open-set classification with $5$ open-set categories.

The main method of Meta-BCE is using an auxiliary branch $f_{\phi}^{\prime}$
to produce features for the one-class classifier. We can also modify Meta-BCE
to use the main branch of the feature extractor $f_{\phi}$ to calculate the
one-class feature vectors. In order to do this, we will use a separate module
$h_{\chi}$ to transform the multiclass classification feature vector
$f_{\phi}(x)$ to a Meta-BCE one-class classification feature vector
$h_{\chi}(f_{\phi}(x))$ and use it for one-class predictions:

$\begin{split}p_{(}y=c|x)&=\frac{1}{1+exp(-(-d((h_{\chi}(f_{\phi}(x)),\bar{x}_{c})-t))}\\\
&=\frac{1}{1+exp(d(h_{\chi}(f_{\phi}(x)),\bar{x}_{c})+t)}\end{split}$ (13)

This version of the Meta-BCE will be called Meta-BCEC. We can see the accuracy
comparison of the above two approaches on the CUB-200-2011 dataset [37] in
Table 8. Both methods do not degrade the closed-set accuracy, however the
results for Meta-BCE using a separate branch to calculate the embeddings
($f_{\phi}^{\prime}$) are significantly better than when using the main branch
(Meta-BCEC). This pertains to normalized accuracy and F1-open scores, where
Meta-BCE achieves $35.66\%$ normalized accuracy in $1$-shot setting (compared
to $28.66\%$ by Meta-BCEC) and $68.59\%$ in $5$-shot setting (compared to
$37.13\%$ by Meta-BCEC). However, the AUROC scores are slightly better when
using the main branch (Meta-BCEC). This might indicate that multiclass
embeddings obtained in the main branch of the feature extractor help slightly
with the overall ranking of known vs unknown examples (as indicated by AUROC
score), but a separate feature extraction branch leading to a separate,
dedicated one-class embedding allows to obtain better separability of the
feature space leading to better classification metrics (normalized accuracy
and F1-open score).

###  E.3 Impact of $k$-shot test setting

Number of per-category examples ($k$) has a big effect on the performance of
every few-shot approach. In Figure 4 we provide a comparison between studied
approaches where we showcase impact of number of per-category examples on four
performance metrics (closed-set accuracy, AUROC score, normalized accuracy,
and F1-open score) for a $5$-way $k$-shot classification with $n_{U}=5$ open-
set (unknown) categories. The results are from experiments on the miniImageNet
dataset [14].

|
---|---
(a) Closed-Set Accuracy | (b) AUROC
|
(d) Normalized Accuracy | (e) F1-open score
Figure 4: Impact of $k$ (number of per-category examples) on the performance
for $5$-way $k$-shot classification with $5$ open-set categories.

In Figure 4(a) we can see the comparison between methods and their impact on
closed-set accuracy. Methods based on PEELER [9] have overall lower accuracy
for all values of $k$ than those based on FEAT [7]. Additionally, it is clear
that both our methods (Meta-BCE and OCML) do not degrade closed-set accuracy,
compared to Entropic Open-Set Loss [32]. In Figure 4(b) we can see the impact
on AUROC score. FEAT [7] \+ OCML has the best performance for all values of
$k$, closely followed by PEELER [9] \+ OCML and FEAT [7] \+ Meta-BCE.
Threshold-based methods and Entropic Open-Set Loss [32] have the lowest AUROC
score across all values of $k$. In Figure 4(c) we can see the normalized
accuracy values for the tested methods. Important to see are the high values
for OCML methods (with PEELER [9] and FEAT [7]) when $k=1$ and a very high
gain of both Meta-BCE methods (with PEELER [9] and FEAT [7]) when increasing
values of $k$ (surpassing OCML performance when $k\geq 3$). Very similar
behavior can be observed in Figure 4(d) for F1-open score.

###  E.4 Impact of $N$-way test setting

We have performed also a more thorough analysis of studied methods when
varying both number of categories and number of per-category examples in the
open-set setting. We provide the results for this analysis below in Figure 5
(for closed-set accuracy), Figure 6 (for AUROC score), Figure 7 (for
normalized accuracy), and Figure 8 (for F1-open score). In all experiments
number of unknown categories ($n_{U}$) was equal to the number of known
categories ($n_{u}=n$).

When analyzing the closed-set accuracy (Figure 5) we can see that for all
methods (as expected) higher number of per-category examples ($k$) increases
the performance and higher number of categories ($n$) decreases the
performance. Threshold, Meta-BCE, and OCML do not impact the closed-set
performance (thus their plots are comparable) and Entropic Open-Set Loss [32]
reduces the closed-set accuracy of PEELER [9]. AUROC score comparison (Figure
6 indicates that Meta-BCE and OCML has much higher scores for all values of
$k$ and $n$ than original PEELER [9] or PEELER with Entropic Open-Set Loss
[32]. And we can further increase the performance by substituting PEELER [9]
with FEAT [7] as the closed-set training method.

Figures showcasing the classification metrics (normalized accuracy in Figure 7
and F1-open score in Figure 8 indicate few interesting properties. Both PEELER
[9] and PEELER with Entropic Open-Set Loss [32] have low scores, and for some
values of $n$ they classify all query examples as unknown examples. The reason
for this behavior is the discrepancy between the training setting of $5$-way
$1$-shot and the test setting showcased in Figures and the fact that both
methods are basing their classification decision (whether a sample is known or
unknown) based on the threshold learned during the training phase. We can see
that such problem does not occur in any other method. Another important
property is the difference in behavior between OCML and Meta-BCE. OCML starts
with high performance (when $k=1$) and slowly increases it when $k$ increases.
Meta-BCE on the other hand has low initial performance (when $k=1$), but it
has a very rapid gain in both normalized accuracy and F1-open score when $k$
increases achieving values for normalized accuracy above $85\%$ when $n=2$ and
$k=20$, which is $\sim 12\%$ higher than OCML, $\sim 23\%$ higher than
possible with a simple thresholding, and $\sim 40\%$ higher than with Entropic
Open-Set Loss [32].

|  |
---|---|---
(a) PEELER + threshold | (b) PEELER + Entropic Loss | (c) PEELER + Meta-BCE
|  |
(d) PEELER + OCML | (e) FEAT + threshold | (f) FEAT + Meta-BCE
|  |
(d) FEAT + OCML |  |
Figure 5: Closed-set accuracy |  |
---|---|---
(a) PEELER + threshold | (b) PEELER + Entropic Loss | (c) PEELER + Meta-BCE
|  |
(d) PEELER + OCML | (e) FEAT + threshold | (f) FEAT + Meta-BCE
|  |
(d) FEAT + OCML |  |
Figure 6: AUROC score |  |
---|---|---
(a) PEELER + threshold | (b) PEELER + Entropic Loss | (c) PEELER + Meta-BCE
|  |
(d) PEELER + OCML | (e) FEAT + threshold | (f) FEAT + Meta-BCE
|  |
(d) FEAT + OCML |  |
Figure 7: Normalized accuracy |  |
---|---|---
(a) PEELER + threshold | (b) PEELER + Entropic Loss | (c) PEELER + Meta-BCE
|  |
(d) PEELER + OCML | (e) FEAT + threshold | (f) FEAT + Meta-BCE
|  |
(d) FEAT + OCML |  |
Figure 8: F1-open score