# Plug-and-Play Feature Generation for Few-Shot Medical Image Classification
††thanks: *: Corresponding Authors.

Qianyu Guo  School of Computer Science
Fudan University
Shanghai, China
<EMAIL_ADDRESS>Huifang Du  College of Design and Innovation
Tongji University
Shanghai, China
<EMAIL_ADDRESS>Xing Jia  School of Computer Science
Fudan University
Shanghai, China
<EMAIL_ADDRESS>Shuyong Gao  School of Computer Science
Fudan University
Shanghai, China
<EMAIL_ADDRESS>Yan Teng  Governance Research Center
Shanghai Artificial Intelligence Laboratory
Shanghai, China
<EMAIL_ADDRESS>Haofen Wang* College of Design and Innovation
Tongji University
Shanghai, China
<EMAIL_ADDRESS>Wenqiang Zhang* School of Computer Science
Fudan University
Shanghai, China
<EMAIL_ADDRESS>
###### Abstract

Few-shot learning (FSL) presents immense potential in enhancing model
generalization and practicality for medical image classification with limited
training data; however, it still faces the challenge of severe overfitting in
classifier training due to distribution bias caused by the scarce training
samples. To address the issue, we propose MedMFG, a flexible and lightweight
plug-and-play method designed to generate sufficient class-distinctive
features from limited samples. Specifically, MedMFG first re-represents the
limited prototypes to assign higher weights for more important information
features. Then, the prototypes are variationally generated into abundant
effective features. Finally, the generated features and prototypes are
together to train a more generalized classifier. Experiments demonstrate that
MedMFG outperforms the previous state-of-the-art methods on cross-domain
benchmarks involving the transition from natural images to medical images, as
well as medical images with different lesions. Notably, our method achieves
over 10% performance improvement compared to several baselines. Fusion
experiments further validate the adaptability of MedMFG, as it seamlessly
integrates into various backbones and baselines, consistently yielding
improvements of over 2.9% across all results.

###### Index Terms:

Few-shot Learning, Medical Image Classification, Feature Generation, Plug-and-
Play

##  I Introduction

Few-shot learning (FSL) [1, 2, 3] has been widely adopted to address the
challenges of limited annotated data and the long-tail distribution in medical
image classification, as it enables rapid acquisition of new category
knowledge from limited samples. In few-shot medical image classification, the
feature extraction model is trained on a large number of natural or common
medical images in the source domain. Subsequently, the target domain images
extract prototype features and train the classifier. Nevertheless, the limited
data in the target domain fails to represent the class distribution
accurately. Additionally, due to the semantic gap between the source and
target domains, the pre-trained model struggles to accurately capture the
prototype features of the target domain data. Noisy prototypes further
contribute to the errors in the category distribution. Consequently, the
biased representation and scarcity of training samples result in a skewed
category distribution, leading to severe overfitting during classifier
training and impacting the performance of few-shot medical image
classification.

Existing methods in medical few-shot learning can be broadly categorized into
transfer learning-based and meta-learning-based approaches. Transfer learning-
based methods [4, 5, 6, 7] aim to adapt to new medical data by training a
highly generalized representation network on large-scale generic datasets.
Meta-learning approaches [8, 9, 10, 11]. aim to acquire general knowledge,
such as feature extraction capabilities, across a plethora of diverse tasks.
Subsequently, the models rapidly iterate to achieve better results on small-
sample data in the target domain. These approaches focus on enhancing the
generalization of the feature extraction network for obtaining more accurate
representations of target domain features, while they fail to address the
problem of learning the accurate category distribution with limited data,
which directly leads to the suboptimal performance of the classifier.

Figure 1: Comparison of training and testing process between transfer learning
(a) and meta-learning (b). (c) is the process of our MedMFG. The bold parts in
the stage boxes refer to the modules that are involved in training, while the
others refer to the parts that are frozen. An example of an N-way-K-shot task
is shown on the left.

To address this issue, we propose MedMFG, a plug-and-play feature generation
method that focuses on augmenting sufficient data with a few prototype
features. We propose augmenting the feature data by generating additional
samples from the limited prototype, inspired by the variational autoencoder
(VAE) [12, 13]. This augmentation and process further enhance classifier
training. Moreover, we incorporate multiple reconstruction methods before
augmentation, motivated by attention mechanisms for feature representation
[14, 15]. In contrast to existing approaches that completely rely on backbones
like ResNet-50, our reconstruction method employs a few simple transformer
layers. These layers can be seamlessly incorporated after diverse pre-trained
models to amplify the discriminative information of prototype features.

Concretely, our method comprises three modules: a self-construction feature
module, an inter-construction feature module, and a variational sample
generation module. Firstly, the self-construction feature module utilizes
self-attention mechanisms to assign higher weights to the discriminative
information of prototype features. Subsequently, the inter-construction
feature module re-expresses the information related to the test samples in the
prototypes through cross-attention mechanisms. Finally, the variational sample
generation module generates a large number of effective feature samples with
category-specific characteristics. These augmented features alleviate the
distribution bias arising from limited prototype data. Finally, the generated
features, combined with the prototype features, are utilized to train the
classifier, significantly improving its generalization performance.

In summary, this paper has the following contributions:

  * •

We introduce MedMFG, a flexible and lightweight plug-and-play method for
generating features in few-shot medical image classification. It effectively
addresses distribution bias stemming from overfitting by seamlessly
integrating with diverse backbones and baselines.

  * •

We propose the self-construction and inter-construction modules to increase
the weights of important information in the prototype features. Additionally,
we introduce a variational sample generation module to augment these
prototypes into suffcient class-discriminative features.

  * •

The efficacy and generalization of our approach are demonstrated through
extensive experiments on CDFSL (cross-domain from natural to medical images)
and FHIST (cross-domain across different lesions in medical images)
benchmarks. Furthermore, fusion experiments illustrate multiple backbones and
baselines can benefit from MedMFG.

Figure 2: The framework of MedMFG is composed of the self-construction feature
module (SFM), the inter-construction feature module (IFM), and the variational
sample generation module (VSGM) which includes the encoder and decoder
components.

##  II Preliminary and Problem Setting

Few-shot learning (FSL) [16] aims to recognize samples from unseen classes
with limited training data, which is expected to reduce labeling costs and
achieve a low-cost and quick model deployment. Annotated samples are often
hard to obtain in many specialized tasks, making FSL highly applicable in
domains like fine-grained species recognition [17], industrial defect
classification [18], and medical diagnosis [19, 9]. To simulate real-world
scenarios, FSL datasets are typically split into a pre-training dataset and a
testing dataset. Notably, the model has no prior exposure to the classes
present in the testing dataset during training. Therefore, FSL requires online
training of classifiers for new tasks with a few images.

Specifically, there are two main differences between FSL and general
classification. Firstly, the train set and test set in FSL have no overlapping
classes. Specifically, FSL models are pretrained on the base dataset
$\mathcal{T}_{base}$ with base classes $\mathcal{C}_{base}$, and they need to
predict novel classes $\mathcal{C}_{novel}$ on the novel dataset
$\mathcal{T}_{novel}$, where ${C}_{base}\cap{C}_{novel}=\oslash$. Secondly,
FSL utilizes a meta-task with the $N$-way-$K$-shot approach during both the
training and testing phases, as depicted in the left part of Fig. 1. In each
meta-test ($i^{th}$ meta-test), the support sets $\mathcal{S}_{i}$ and the
query set $\mathcal{Q}_{i}$ are randomly drawn from the $\mathcal{T}_{novel}$.
Specifically, $N_{i}$ categories are randomly selected from
$\mathcal{C}_{novel}$, and $K$ images per category are used as
$\mathcal{S}_{i}$. Obviously, the limited availability of $\mathcal{S}_{i}$
often leads to severe overfitting of the classifiers, causing difficulties in
accurately predicting the categories of $\mathcal{Q}_{i}$.

In few-shot medical image classification, we adhere to the setting rules of
classical few-shot learning (FSL). Apart from this, there is a notable
difference in our experiments, as the domain gap between $\mathcal{T}_{base}$
and $\mathcal{T}_{novel}$ is more pronounced. For instance, in the CDFSL
benchmark [2], $\mathcal{T}_{base}$ comprises natural images, whereas
$\mathcal{T}_{novel}$ consists of two different modalities of medical images.
This larger modality and the semantic gap in few-shot medical image
classification make it difficult for pre-trained models to accurately
represent the target classes in medical images. The inaccurately represented
features exacerbate distribution bias when handling new class distributions.
Therefore, obtaining accurate feature representation through re-representation
and acquiring precise class distribution from limited data are key to
improving the performance of few-shot medical image classification.

##  III Related Work

###  III-A Transfer learning for FSL

Transfer learning-based methods [20, 3, 21] primarily aim to enhance the
generalization ability of the representation model on cross-domain data. As
illustrated in Fig. 1(a), these methods learn a universal image representation
capability on the pre-training data $\mathcal{T}_{base}$. During the meta-
testing phase, the feature extraction part of the model is typically frozen,
and the classifier is trained online using $\mathcal{S}_{i}$, allowing
predictions to be made on $\mathcal{Q}_{i}$. Although transfer learning-based
representation learning methods have been extensively researched and applied
in traditional few-shot learning tasks, medical images present distinct
challenges including diverse modalities, closer inter-class similarity, and
the difficulty of directly transferring representation abilities from natural
images. Consequently, our approach focuses on improving the performance of
medical few-shot learning by directly increasing the number of training
samples for medical target categories.

###  III-B Meta learning for FSL

Meta-learning-based methods [22, 23] strive to equip machines with the ability
to learn how to learn. The objective is to achieve a universal classification
capability through meta-training, enabling rapid adaptation to new tasks, as
depicted in Fig. 1(b). During the training process, meta-learning methods
iterate tasks using the same $N$-way-$K$-shot setup as in testing. This
iterative process results in a feature extractor that possesses robust
generalization ability. Although meta-learning methods have shown promising
performance in classical few-shot learning tasks, where they learn
representation [24, 25, 26, 27], classification [28], or optimization [29],
they often prove inadequate when faced with more complex domains like medicine
and industry. Drawing inspiration from meta-learning approaches, we
incorporate meta-training strategies, as in Fig. 1(c). Specifically, in each
training episode, we deliberately define visible and unseen samples to augment
the model’s capability to fully consider information from unseen samples and
facilitate feature interactions.

##  IV Method

###  IV-A Method overview

We propose MedMFG, as in Fig. 2, consisting of three modules: the self-
construction feature module (SFM), the inter-construction feature module
(IFM), and the variational sample generation module (VSGM). After feature
extraction, SFM and IFM enhance the class characteristics of the prototype
features. Then, VSGM can generate sufficient class-discriminative samples for
each prototype. These augmented samples are trained together with the
prototype features to improve the generalization ability and classification
accuracy of the classifier. Clearly, the training and utilization of MedMFG
are independent of the feature extraction backbone, meaning that MedMFG can
flexibly and widely integrate with various backbones or existing
representation learning methods.

During the training phase, MedMFG adopts a meta-training approach. A training
set $T_{Rj}$ is sampled, with $S_{Rj}$ serving as the support set and $Q_{Rj}$
as the query set. We follow exactly the same pre-training method as in other
work to get the feature extraction backbone [30, 31]. Then, freezing the
backbone, we train MedMFG using classification loss, KL divergence, and
reconstruction loss.

In the meta-test phase, the backbone and MedMFG are both frozen. We only train
classifiers online for each task. After feature extraction, SFM, and IFM, we
obtain the support features $\hat{s}^{i}_{k}$ with updated weights from the
original support images, as well as the query features $\hat{q}_{j}$ form the
query image. Next, we generate $n$ new samples for each support feature by
VSGM. In the $N$-way-$K$-shot setup, we obtain $N$-way-$(k\times n)$ shot
representative samples. Subsequently, we train the classifier using the
generated new features along with the prototype features $N\times
k\times(n+1)$ to classify $\hat{q}_{j}$.

###  IV-B Model structure and loss function

Figure 3: The self-construction feature module (SFM).

SFM: SFM increases the weight of category-specific information in prototype
features through self-attention mechanisms. As depicted in Fig. 3, the support
samples $x_{i}$ are fed into the embedding module $f_{\theta}$, resulting in
the prototype features $\hat{x}_{i}=f_{\theta}(x_{i})\in\mathbb{R}^{d\times
h\times w}$, where $d$ denotes the number of channels, and $h$ and $w$
represent the height and width of the features, respectively. In SFM, the
inputs are $z_{i}=x_{i}+E_{pos}$, where $E_{pos}$ employs sinusoidal position
encoding. The output of SFM is computed using the standard self-attention
operation in the Transformer Encoder [32], performing multi-attention
calculations using Eq. (1)

${Attention}(Q,K,V)={softmax}(\frac{{QK^{T}}}{\sqrt{d_{k}}}){V}.$ (1)

Thus, we obtain the re-characterized features $\hat{z}_{i}$ as:

$\hat{z}_{i}={Attention}(z_{i}W^{Q}_{\phi},z_{i}W^{K}_{\phi},z_{i}W^{V}_{\phi}),\hat{z}_{i}\in\mathbb{R}^{r\times
d},$ (2)

where $W^{Q}_{\phi},W^{k}_{\phi},W^{V}_{\phi}$ are the learnable weights with
$d\times d$ size. Finally, $z^{i}$ is calculated continually by a layer
normalization and an MLP.

$\hat{z}_{i}=MLP(LN(z_{i}+\hat{z}_{i})).$ (3)

IFM: IFM enhances the weight of information related to the test samples in
prototype features through interactive attention mechanisms. After we obtain
the self-reconstructed feature $\hat{z}_{i}$ of the support set, as well as
the pooled query prototype feature ${q}_{i}$, we construct the IFM module
shown as Fig. 4 to embed the attention weights between ${q}_{i}$ and
$\hat{z}_{i}$ into the features of $\hat{z}_{i}$, resulting in the new support
feature $\dot{z}_{i}$:

$\dot{z}_{i}={Attention}(\hat{z}_{i}W^{Q}_{r},{q}_{i}W^{K}_{r},{q}_{i}W^{V}_{r}),\dot{z}_{i}\in\mathbb{R}^{r\times
d},$ (4)

where $Attention$ follows Eq.(1). Therefore, in order to better learn the
feature interaction capability of the IFM module, we adopt a meta-training
approach to better simulate the embedding of unknown features with known
features.

Figure 4: The inter-construction feature module (IFM).

VSGM: VSGM utilizes an MLP-based VAE to perform variational augmentation on
the input features and train a more generalized classifier. As in Fig. 2, the
VAE consists of two parts: an encoder $E(x,\theta_{e})$ which map the input
feature $\dot{z}_{i}$ to a latent code $\tilde{z}$, and a decoder
$G(\tilde{z},\theta_{g})$, which reconstructs $z_{i}$ from $\dot{z}_{i}$,
where $\theta_{e}$ and $\theta_{g}$ are weight parameters of $E$ and $G$. Same
as the traditional VAE [33], we approximate the true posterior distribution
$p(z|X)$ with another distribution $q(z|X)$. Thus, the KL scatter between the
two distributions is:

${KL(q(z|X)||p(z|X))}=\int_{Z}q(Z|X)\log\frac{q(z|X)}{(p(z|X)}.$ (5)

TABLE I: Comparison between MedMFG and other state-of-the-arts of 5-way few-
shot accuracies on FHIST benchmark with ResNet-18.

Method | CRC-TP$\rightarrow$ NCT | CRC-TP$\rightarrow$ LC25000 | CRC-TP$\rightarrow$ BreakHis
---|---|---|---
1-shot | 5-shot | 10-shot | 1-shot | 5-shot | 10-shot | 1-shot | 5-shot | 10-shot
ProtoNet [30] | $58.1$ | $72.3$ | $74.4$ | $58.6$ | $68.5$ | $70.8$ | $32.0$ | $40.8$ | $44.0$
MetaOpt [34] | $48.6$ | $64.4$ | $70.0$ | $49.2$ | $72.9$ | $78.5$ | $28.3$ | $42.5$ | $49.1$
SimpleShot [35] | $63.2$ | $76.1$ | $78.4$ | $62.1$ | $73.8$ | $76.6$ | $36.2$ | $47.5$ | $51.8$
Distill [36] | $61.9$ | $77.3$ | $80.9$ | $63.2$ | $75.9$ | $79.5$ | $36.2$ | $50.2$ | $55.7$
Finetune [37] | $64.7$ | $81.4$ | $85.0$ | $62.1$ | $79.2$ | $83.7$ | $36.4$ | $56.9$ | $66.7$
MedMFG | 69.2 | 85.2 | 87.7 | 65.3 | 82.0 | 84.4 | 39.2 | 59.9 | 67.6

Loss function: We design a three-part loss function to optimize MedMFG: the KL
scatter, the reconstruction loss, and the classification loss. To diversify
the samples generated by VSGM, we set the prior distribution of $\tilde{z}$ to
a centered isotropic multivariate Gaussian: $p(\tilde{z})=N(0,I)$. For the
posterior distribution, we assume it to be a multivariate Gaussian with
diagonal covariance: $q({z}_{i}|X^{i})=N(\mu^{i},\sigma^{i})$. The KL
divergence$KL(q({z}_{i}|X^{i})||p(\tilde{z}))$ can be calculated by Eq. (5):

$L_{KL}=\int_{z}q({z}_{i}|X^{i})\log\frac{q({z}_{i}|X^{i})}{p(\tilde{z}|X^{i})}.$
(6)

To drive VSGM to generate high-quality features, we design the reconstruction
feature loss:

$L_{cons}=||z_{i}-\dot{z}_{i}||^{2}.$ (7)

After obtaining augmented samples $z_{i}$, for each meta-training task
$R_{j}$, we train an online classifier $W_{R_{j}}$ to predict the class of the
query $q_{i}$, resulting in a classification loss:

$L_{cls}=L_{(cross-entropy)}(W_{Rj}(q_{i}),y_{i}),$ (8)

where $y_{i}$ refers to the true label of $q_{i}$. The classification loss is
crucial as it can help the SFM and IFM maintain the class properties when
reconstructing features, and the features generated by VSGM do not deviate
from the original category boundaries. Thus, the overall loss function is a
weighted combination of the aforementioned terms:

$L=L_{cons}+\alpha L_{KL}+\beta L_{cls}.$ (9)

The performance is experimentally verified to be optimal for $\alpha=0.0001$
and $\beta=1$.

TABLE II: Comparison between MedMFG and other state-of-the-arts of 5-way few-
shot accuracies on miniImageNet$\rightarrow$ChestX/ISIC.

Method | miniImageNet$\rightarrow$ChestX | miniImageNet$\rightarrow$ISIC
---|---|---
5-shot | 20-shot | 50-shot | 5-shot | 20-shot | 50-shot
MatchingNet [31] | $22.40$ | $23.61$ | $22.12$ | $36.74$ | $45.72$ | $54.58$
MAML [22] | $23.48$ | $27.53$ | $--$ | $40.13$ | $52.36$ | $--$
ProtoNet [30] | $24.05$ | $28.21$ | $29.32$ | $39.57$ | $49.50$ | $51.99$
FWT [38] | $21.26$ | $23.23$ | $23.01$ | $30.40$ | $32.01$ | $33.17$
FT [38] | $24.28$ | $--$ | $--$ | $40.87$ | $--$ | $--$
FT-All [2] | $25.97$ | $31.32$ | $35.49$ | $48.11$ | $59.31$ | 66.48
AFA [39] | $24.32$ | $--$ | $--$ | $44.91$ | $--$ | $--$
ConFeSS [40] | $27.09$ | $33.57$ | $39.02$ | $48.85$ | $60.10$ | $65.34$
AF [41] | $25.02$ | $--$ | $--$ | $46.01$ | $--$ | $--$
RDC [42] | $25.91$ | $--$ | $--$ | $41.28$ | $--$ | $--$
MedMFG | 29.83 | 34.00 | 39.66 | 50.03 | 60.52 | $65.74$

TABLE III: Comparison experiments of MedMFG with multiple bakcbones and
baselines of 5-way few-shot accuracies on
miniImageNet$\rightarrow$ChestX/ISIC.

Method | Backbone | miniImageNet $\rightarrow$ ChestX | miniImageNet $\rightarrow$ ISIC
---|---|---|---
5-shot | 20-shot | 50-shot | 5-shot | 20-shot | 50-shot
Baseline-1 [30] | Conv-4 | $24.04$ | $26.23$ | $28.49$ | $35.32$ | $40.17$ | $44.69$
+MedMFG | Conv-4 | $27.50_{{\color[rgb]{1,0,0}+3.46}}$ | $29.97_{{\color[rgb]{1,0,0}+3.74}}$ | $31.50_{{\color[rgb]{1,0,0}+3.01}}$ | $38.25_{{\color[rgb]{1,0,0}+2.93}}$ | $44.52_{{\color[rgb]{1,0,0}+4.35}}$ | $48.24_{{\color[rgb]{1,0,0}+3.55}}$
Baseline-1 [30] | ResNet-10 | $24.05$ | $28.21$ | $29.32$ | $39.57$ | $49.50$ | $51.99$
+MedMFG | ResNet-10 | $28.30_{{\color[rgb]{1,0,0}+4.25}}$ | $31.17_{{\color[rgb]{1,0,0}+2.96}}$ | $33.00_{{\color[rgb]{1,0,0}+3.68}}$ | $44.52_{{\color[rgb]{1,0,0}+4.95}}$ | $53.13_{{\color[rgb]{1,0,0}+3.63}}$ | $55.50_{{\color[rgb]{1,0,0}+3.51}}$
Baseline-1 [30] | ResNet-18 | $23.13$ | $29.02$ | $33.20$ | $42.83$ | $53.48$ | $58.51$
+MedMFG | ResNet-18 | $26.94_{{\color[rgb]{1,0,0}+3.81}}$ | $32.22_{{\color[rgb]{1,0,0}+3.20}}$ | $39.55_{{\color[rgb]{1,0,0}+6.35}}$ | $49.00_{{\color[rgb]{1,0,0}+6.17}}$ | $56.71_{{\color[rgb]{1,0,0}+3.23}}$ | $62.11_{{\color[rgb]{1,0,0}+3.60}}$
Baseline-2 [31] | ResNet-10 | $22.40$ | $23.61$ | $24.12$ | $36.74$ | $45.72$ | $54.58$
+MedMFG | ResNet-10 | $25.98_{{\color[rgb]{1,0,0}+3.58}}$ | $26.86_{{\color[rgb]{1,0,0}+3.25}}$ | $27.11_{{\color[rgb]{1,0,0}+2.99}}$ | $39.80_{{\color[rgb]{1,0,0}+3.06}}$ | $48.99_{{\color[rgb]{1,0,0}+3.27}}$ | $57.60_{{\color[rgb]{1,0,0}+3.02}}$
Baseline-3 [43] | ResNet-10 | $22.96$ | $26.63$ | $28.45$ | $39.41$ | $41.77$ | $49.32$
+MedMFG | ResNet-10 | $26.00_{{\color[rgb]{1,0,0}+3.04}}$ | $30.17_{{\color[rgb]{1,0,0}+3.54}}$ | $32.05_{{\color[rgb]{1,0,0}+3.60}}$ | $42.67_{{\color[rgb]{1,0,0}+3.26}}$ | $45.21_{{\color[rgb]{1,0,0}+3.44}}$ | $52.44_{{\color[rgb]{1,0,0}+3.12}}$

##  V Experiments

###  V-A Experimental setup

Datasets: We validate our method on two benchmarks: CDFSL [2] and FHIST [44],
following the standard few-shot setting as in previous works [45, 46, 3].
Specifically, on CDFSL, we regard miniImageNet [31], a dataset excerpted from
ImageNet, as the base set while ISIC [47], a dermoscopic image dataset and
chestX [48], a frontal-view X-ray images dataset as the novel set
respectively. On FHIST, we train MedMFG on CRC-TP [49] (colorectal tissues
images) and test it on LC25000 (colorectal and lung tissues images) [50],
BreakHis (breast tissues images) [51], and NCT-CRC-HE-100k (colorectal tissues
images) [52]. It is important to note that we follow the inductive setting,
ensuring no information leakage among different queries.

Learnable parameters: MedMFG is designed to be lightweight and portable. Among
the three modules, SFM consists of one transformer block with 1.66M
parameters, IFM consists of another transformer block with 1.24M parameters,
and the VAE module has 0.79M parameters. In total, MedMFG has 3.69M
parameters. To ensure a fair comparison, we use the same feature extractor as
the comparison methods.

Implementation details: During the meta-training phase, we follow the
5-way-1-shot approach for training. In the meta-testing phase, after
extracting the features of support and novel data, we apply MedMFG to amplify
each support feature into $5$ samples. We report the accuracy on standard
N-way K-shot settings with $15$ query samples per class. All testing
experiments are randomly sampled over 600 episodes. The learning rate of
MedMFG is set to 0.00001, and it is optimized with Adam.

###  V-B Result analysis

Comparison to state-of-the-Art (SOTA) methods: We compare our method with the
SOTA methods on CDFSL and FHIST benchmarks. On FHIST, as shown in Tab. I,
MedMFG achieves the new state-of-the-art in all three migration settings. For
instance, on CRC-TP$\rightarrow$ NCT, MedMFG outperforms Finetune [37] by
$4.5\%$, $3.8\%$, and $2.7\%$ on 5-way 1-shot, 5-shot, and 10-shot settings,
respectively. In other scenarios like CRC-TP$\rightarrow$ LC25000 and CRC-
TP$\rightarrow$ BreakHis, our method surpasses the current optimal results by
$3.2\%$, $2.8\%$, $0.7\%$, $2.8\%$, $3.0\%$, and $0.9\%$. It is worth noting
that Finetune [37] requires retraining during testing, whereas our approach is
completely frozen during testing without any additional consumption of time
and space. These results demonstrate the superiority and practicality of
MedMFG in a wide range of medical domain migrations.

On CDFSL, as shown in Tab. II, MedMFG achieves a new SOTA on most of the
results. For example, on miniImageNet$\rightarrow$ChestX and ISIC, MedMFG
outperforms ConFess [40] by $2.74\%$, $0.43\%$, $1.18\%$, and $0.42\%$ on
5-way 5-shot and 20-shot settings. In the 50-shot setting, our method reaches
the current optimum. These results highlight the strong generalization ability
of MedMFG.

In summary, whether it involves cross-domain transfer from natural images to
medical images or cross-category transfer on medical images, MedMFG
demonstrates stronger generalization capabilities. This proves the
effectiveness of the feature design and augmentation methods in medical
images, providing a novel approach to address the challenges posed by the
limited availability of medical data.

Fusion experiments: To demonstrate the generalizability and flexibility of our
method, we extract features using multiple backbones and baselines, and then
fuse MedMFG amplification samples. As shown in Tab. III, we validate the
fusion results of MedMFG with three feature extractors, Conv-4 [53], ResNet-10
[54], and ResNet-18 [54], as well as three methods [30, 31, 43]. MedMFG
consistently improves the performance by more than 2.9% across all settings.
Particularly, it surpasses the baseline by 6% in the two results with
ResNet-18. Notably, it exhibits superior performance even with fewer support
instances, indicating that the generated feature samples accurately capture
class characteristics. These results highlight the versatility of MedMFG,
which is not restricted to specific datasets or feature extractors. As a
result, various works based on different backbones can benefit from our
approach.

TABLE IV: Ablation experiments of modules and the number of MLP layers in VAE.

Method | SFM | IFM | Layers | CRC-TP$\rightarrow$ NCT
---|---|---|---|---
1-shot | 5-shot | 10-shot
Baseline [30] | $\times$ | $\times$ | $\times$ | 58.1 | 72.3 | 74.4
MedMFG-1 | $\checkmark$ | $\times$ | $\times$ | 58.7 | 73.6 | 75.7
MedMFG-2 | $\checkmark$ | $\checkmark$ | $\times$ | 59.0 | 75.3 | 79.0
MedMFG-3 | $\checkmark$ | $\checkmark$ | one | 67.3 | 84.0 | 84.3
MedMFG-4 | $\checkmark$ | $\times$ | two | 63.1 | 77.0 | 76.3
MedMFG-5 | $\times$ | $\checkmark$ | two | 66.5 | 83.9 | 83.7
MedMFG-6 | $\checkmark$ | $\checkmark$ | three | 69.0 | 85.7 | 86.9
MedMFG-7 | $\times$ | $\times$ | two | 60.0 | 75.6 | 76.1
MedMFG | $\checkmark$ | $\checkmark$ | two | 69.2 | 85.2 | 87.7

Ablation studies: In Tab. IV, we analyze the effectiveness of the three
modules in MedMFG: SFM, IFM, and VSGM. Additionally, we compare the
performance with varying numbers of layers in the VAE. The results show that
using SFM or IFM alone does not lead to significant improvements, especially
in the 1-shot setting. This is because, without the feature augmentation
module, a single sample fails to adequately represent the entire distribution
of a class, resulting in a substantial bias. However, the last three rows
clearly demonstrate the crucial role of SFM and IFM in conjunction with
feature augmentation. The absence of SFM or IFM results in a performance
decrease of more than 2.5%. This indicates that feature reconstruction
effectively highlights class characteristics and reduces intra-class
distances, contributing to improved performance. Moreover, when comparing
MedMFG-3 with MedMFG, it can be observed that the VAE with a two-layer MLP
structure better simulates the characteristics of real samples. However,
blindly increasing the number of MLP layers provides minimal benefits while
adding more parameters and increasing training complexity. Additionally, in
the 20-shot setting, MedMFG-6 performs worse than MedMFG, suggesting that
excessive feature expression can limit feature diversity. Therefore, we
ultimately choose the VAE composed of a two-layer MLP as the final structure.

Figure 5: For a 5-way-2-shot task, we visualize the feature descriptors
generated by the feature backbone (Left) and after amplification by MedMFG
(Right) with t-SNE.

###  V-C Visualization

We assert that MedMFG is capable of generating representative samples to
enhance classification results. To visualize this, we present the original
features generated by the feature extractor and the augmented features of
MedMFG on the NCT dataset in Fig. 5. In this example, the true label of the
query is TMU. However, due to the fine-grained nature of the support classes,
the difference in distance between the query and each support feature is
small, resulting in overfitting of the linear regression classifier and
misclassification errors. Nevertheless, after variational feature generation
by MedMFG to adjust the distribution, the query is accurately included within
the distribution boundary of TMU.

##  VI Conclusion

Summary: In this paper, to enhance the generalization ability of the
classifier in few-shot medical image classification, we propose MedMFG. It is
a plug-and-play method that can augment sufficient class-discriminative
features from a small number of samples. Through comparison experiments, we
demonstrate the superiority of our method by achieving new SOTA in multiple
benchmarks. Furthermore, fusion experiments showcase the flexibility and
applicability of our method.

Limitations: By comparing the baseline with MedMFG-1, as well as MedMFG-4 and
MedMFG-7 in Tab. IV, it is apparent that employing the self-attention
mechanism alone yields limited improvement over the baseline. In contrast,
when combined with VSGM, the self-attention mechanism demonstrates promising
performance. The cross-attention mechanism exhibits similar characteristics.
We will investigate the generalizability of this phenomenon through additional
experiments. Furthermore, we aim to explore additional medical modalities like
CT and MRI, enhancing the practicality of MedMFG.

## References

  * [1] Y. Wang, Q. Yao, J. T. Kwok, and L. M. Ni, “Generalizing from a few examples: A survey on few-shot learning,” _ACM computing surveys (csur)_ , vol. 53, no. 3, pp. 1–34, 2020.
  * [2] Y. Guo, N. C. Codella, L. Karlinsky, J. V. Codella, J. R. Smith, K. Saenko, T. Rosing, and R. Feris, “A broader study of cross-domain few-shot learning,” in _European conference on computer vision_. Springer, 2020, pp. 124–141.
  * [3] Q. Guo, G. Haotong, X. Wei, Y. Fu, Y. Yu, W. Zhang, and W. Ge, “Rankdnn: Learning to rank for few-shot learning,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 37, no. 1, 2023, pp. 728–736.
  * [4] Z. Dai, J. Yi, L. Yan, Q. Xu, L. Hu, Q. Zhang, J. Li, and G. Wang, “Pfemed: Few-shot medical image classification using prior guided feature enhancement,” _Pattern Recognition_ , vol. 134, p. 109108, 2023.
  * [5] A. Paul, Y.-X. Tang, T. C. Shen, and R. M. Summers, “Discriminative ensemble learning for few-shot chest x-ray diagnosis,” _Medical image analysis_ , vol. 68, p. 101911, 2021.
  * [6] M. Yazdanpanah and P. Moradi, “Visual domain bridge: A source-free domain adaptation for cross-domain few-shot learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 2868–2877.
  * [7] Y. He, W. Liang, D. Zhao, H. Zhou, W. Ge, Y. Yu, and W. Zhang, “Attribute surrogates learning and spectral tokens pooling in transformers for few-shot learning,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022_. IEEE, 2022, pp. 9109–9119.
  * [8] H. Jiang, M. Gao, H. Li, R. Jin, H. Miao, and J. Liu, “Multi-learner based deep meta-learning for few-shot medical image classification,” _IEEE Journal of Biomedical and Health Informatics_ , vol. 27, no. 1, pp. 17–28, 2022.
  * [9] R. Singh, V. Bharti, V. Purohit, A. Kumar, A. K. Singh, and S. K. Singh, “Metamed: Few-shot medical image classification using gradient-based meta-learning,” _Pattern Recognition_ , vol. 120, p. 108111, 2021.
  * [10] V.-K. Vo-Ho, K. Yamazaki, H. Hoang, M.-T. Tran, and N. Le, “Meta-learning of nas for few-shot learning in medical image applications,” _arXiv preprint arXiv:2203.08951_ , 2022.
  * [11] P. Khandelwal and P. Yushkevich, “Domain generalizer: A few-shot meta learning framework for domain generalization in medical imaging,” in _Domain Adaptation and Representation Transfer, and Distributed and Collaborative Learning_. Springer, 2020, pp. 73–84.
  * [12] S. Foti, B. Koo, D. Stoyanov, and M. J. Clarkson, “3d shape variational autoencoder latent disentanglement via mini-batch feature swapping for bodies and faces,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022_. IEEE, 2022, pp. 18 709–18 718.
  * [13] D. Danks and C. Yau, “Basisdevae: Interpretable simultaneous dimensionality reduction and feature-level clustering with derivative-based variational autoencoders,” in _Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event_ , ser. Proceedings of Machine Learning Research, M. Meila and T. Zhang, Eds., vol. 139. PMLR, 2021, pp. 2410–2420.
  * [14] H. Zhao, J. Jia, and V. Koltun, “Exploring self-attention for image recognition,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 10 076–10 085.
  * [15] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [16] J. Lu, P. Gong, J. Ye, J. Zhang, and C. Zhang, “A survey on machine learning from few samples,” _Pattern Recognit._ , vol. 139, p. 109480, 2023.
  * [17] L. Tang, D. Wertheimer, and B. Hariharan, “Revisiting pose-normalization for fine-grained few-shot recognition,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 14 352–14 361.
  * [18] Y. Cao, W. Zhu, J. Yang, G. Fu, D. Lin, and Y. Cao, “An effective industrial defect classification method under the few-shot setting via two-stream training,” _Optics and Lasers in Engineering_ , vol. 161, p. 107294, 2023.
  * [19] H. Tang, X. Liu, S. Sun, X. Yan, and X. Xie, “Recurrent mask refinement for few-shot medical image segmentation,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2021, pp. 3918–3928.
  * [20] M. N. Rizve, S. Khan, F. S. Khan, and M. Shah, “Exploring complementary strengths of invariant and equivariant representations for few-shot learning,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 10 836–10 846.
  * [21] W.-Y. Chen, Y.-C. Liu, Z. Kira, Y.-C. F. Wang, and J.-B. Huang, “A closer look at few-shot classification,” _arXiv preprint arXiv:1904.04232_ , 2019.
  * [22] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in _International conference on machine learning_. PMLR, 2017, pp. 1126–1135.
  * [23] A. Nichol and J. Schulman, “Reptile: a scalable metalearning algorithm,” _arXiv preprint arXiv:1803.02999_ , vol. 2, no. 3, p. 4, 2018.
  * [24] A. Nichol, J. Achiam, and J. Schulman, “On first-order meta-learning algorithms,” _CoRR_ , vol. abs/1803.02999, 2018.
  * [25] A. Antoniou, H. Edwards, and A. J. Storkey, “How to train your MAML,” in _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net, 2019.
  * [26] X. Song, W. Gao, Y. Yang, K. Choromanski, A. Pacchiano, and Y. Tang, “ES-MAML: simple hessian-free meta learning,” in _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net, 2020.
  * [27] L. Collins, A. Mokhtari, S. Oh, and S. Shakkottai, “MAML and ANIL provably learn representations,” in _International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA_ , ser. Proceedings of Machine Learning Research, K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, Eds., vol. 162. PMLR, 2022, pp. 4238–4310.
  * [28] C. Kao, W. Chiu, and P. Chen, “MAML is a noisy contrastive learner in classification,” in _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net, 2022.
  * [29] M. Andrychowicz, M. Denil, S. G. Colmenarejo, M. W. Hoffman, D. Pfau, T. Schaul, and N. de Freitas, “Learning to learn by gradient descent by gradient descent,” in _Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain_ , D. D. Lee, M. Sugiyama, U. von Luxburg, I. Guyon, and R. Garnett, Eds., 2016, pp. 3981–3989.
  * [30] J. Snell, K. Swersky, and R. S. Zemel, “Prototypical networks for few-shot learning,” in _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett, Eds., 2017, pp. 4077–4087.
  * [31] O. Vinyals, C. Blundell, T. Lillicrap, K. Kavukcuoglu, and D. Wierstra, “Matching networks for one shot learning,” in _Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain_ , D. D. Lee, M. Sugiyama, U. von Luxburg, I. Guyon, and R. Garnett, Eds., 2016, pp. 3630–3638.
  * [32] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.
  * [33] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” in _2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings_ , Y. Bengio and Y. LeCun, Eds., 2014.
  * [34] K. Lee, S. Maji, A. Ravichandran, and S. Soatto, “Meta-learning with differentiable convex optimization,” _arXiv e-prints_ , 2019.
  * [35] Y. Wang, W. Chao, K. Q. Weinberger, and L. van der Maaten, “Simpleshot: Revisiting nearest-neighbor classification for few-shot learning,” _CoRR_ , vol. abs/1911.04623, 2019.
  * [36] Y. Tian, Y. Wang, D. Krishnan, J. B. Tenenbaum, and P. Isola, “Rethinking few-shot image classification: A good embedding is all you need?” in _Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV_ , ser. Lecture Notes in Computer Science, A. Vedaldi, H. Bischof, T. Brox, and J. Frahm, Eds., vol. 12359. Springer, 2020, pp. 266–282.
  * [37] W. Y. Chen, Y. C. Liu, Z. Kira, Y. Wang, and J. B. Huang, “A closer look at few-shot classification,” in _arXiv_ , 2019.
  * [38] H.-Y. Tseng, H.-Y. Lee, J.-B. Huang, and M.-H. Yang, “Cross-domain few-shot classification via learned feature-wise transformation,” _arXiv preprint arXiv:2001.08735_ , 2020.
  * [39] H. Wang and Z.-H. Deng, “Cross-domain few-shot classification via adversarial task augmentation,” _arXiv preprint arXiv:2104.14385_ , 2021.
  * [40] D. Das, S. Yun, and F. Porikli, “Confess: A framework for single source cross-domain few-shot learning,” in _International Conference on Learning Representations_ , 2022.
  * [41] Y. Hu and A. J. Ma, “Adversarial feature augmentation for cross-domain few-shot classification,” in _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XX_. Springer, 2022, pp. 20–37.
  * [42] P. Li, S. Gong, C. Wang, and Y. Fu, “Ranking distance calibration for cross-domain few-shot learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 9099–9108.
  * [43] F. Sung, Y. Yang, L. Zhang, T. Xiang, P. H. S. Torr, and T. M. Hospedales, “Learning to compare: Relation network for few-shot learning,” in _2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018_. Computer Vision Foundation / IEEE Computer Society, 2018, pp. 1199–1208.
  * [44] F. Shakeri, M. Boudiaf, S. Mohammadi, I. Sheth, M. Havaei, I. B. Ayed, and S. E. Kahou, “Fhist: A benchmark for few-shot classification of histological images,” _arXiv preprint arXiv:2206.00092_ , 2022.
  * [45] A. Parnami and M. Lee, “Learning from few examples: A summary of approaches to few-shot learning,” _arXiv preprint arXiv:2203.04291_ , 2022.
  * [46] Y. He, W. Liang, D. Zhao, H.-Y. Zhou, W. Ge, Y. Yu, and W. Zhang, “Attribute surrogates learning and spectral tokens pooling in transformers for few-shot learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 9119–9129.
  * [47] P. Tschandl, C. Rosendahl, and H. Kittler, “The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions,” _Scientific data_ , vol. 5, no. 1, pp. 1–9, 2018.
  * [48] X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri, and R. M. Summers, “Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 2097–2106.
  * [49] S. Javed, A. Mahmood, N. Werghi, K. Benes, and N. Rajpoot, “Multiplex cellular communities in multi-gigapixel colorectal cancer histology images for tissue phenotyping,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 9204–9219, 2020.
  * [50] A. A. Borkowski, M. M. Bui, L. B. Thomas, C. P. Wilson, L. A. DeLand, and S. M. Mastorides, “Lung and colon cancer histopathological image dataset (lc25000),” _arXiv preprint arXiv:1912.12142_ , 2019.
  * [51] F. A. Spanhol, L. S. Oliveira, C. Petitjean, and L. Heutte, “A dataset for breast cancer histopathological image classification,” _Ieee transactions on biomedical engineering_ , vol. 63, no. 7, pp. 1455–1462, 2015.
  * [52] B. Á. Pataki, A. Olar, D. Ribli, A. Pesti, E. Kontsek, B. Gyöngyösi, Á. Bilecz, T. Kovács, K. A. Kovács, Z. Kramer _et al._ , “Huncrc: annotated pathological slides to enhance deep learning applications in colorectal cancer screening,” _Scientific Data_ , vol. 9, no. 1, p. 370, 2022.
  * [53] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in _Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States_ , P. L. Bartlett, F. C. N. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, Eds., 2012, pp. 1106–1114.
  * [54] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016_. IEEE Computer Society, 2016, pp. 770–778.


11institutetext: The University of Tennessee, Knoxville, TN, USA
11email<EMAIL_ADDRESS>22institutetext: Applied Research
Associates, Raleigh, NC, USA
22email<EMAIL_ADDRESS>
# Representative-Discriminative Learning for Open-set Land Cover
Classification of Satellite Imagery

Razieh Kaviani Baghbaderani 11 Ying Qu Corresponding author11 Hairong Qi 11
Craig Stutts 22

###### Abstract

Land cover classification of satellite imagery is an important step toward
analyzing the Earth’s surface. Existing models assume a closed-set setting
where both the training and testing classes belong to the same label set.
However, due to the unique characteristics of satellite imagery with extremely
vast area of versatile cover materials, the training data are bound to be non-
representative. In this paper, we study the problem of open-set land cover
classification that identifies the samples belonging to unknown classes during
testing, while maintaining performance on known classes. Although inherently a
classification problem, both representative and discriminative aspects of data
need to be exploited in order to better distinguish unknown classes from
known. We propose a representative-discriminative open-set recognition (RDOSR)
framework, which 1) projects data from the raw image space to the embedding
feature space that facilitates differentiating similar classes, and further 2)
enhances both the representative and discriminative capacity through
transformation to a so-called abundance space. Experiments on multiple
satellite benchmarks demonstrate effectiveness of the proposed method. We also
show the generality of the proposed approach by achieving promising results on
open-set classification tasks using RGB images.

###### Keywords:

Hyperspectral image classification, open-set recognition

##  1 Introduction

Recent advancements in computer vision, especially the advent of Convolutional
Neural Networks (CNN), have significantly improved the performance of image
classification [1, 2, 3], detection [4, 5], and segmentation [6, 7] tasks,
enabling their deployment in many different fields. One of such field of
applications is satellite image analysis that includes resource management,
urban development planning, and climate control. Land cover classification or
material classification is one of the building blocks of satellite image
analysis, providing essential inputs to a series of subsequent tasks including
object segmentation, 3D reconstruction and modeling, as well as texture
mapping. Supervised land cover classification involves categorization of
multispectral or hyperspectral image pixels into predefined material classes,
e.g., asphalt, tree, concrete, water, metal, soil, etc. Note that both
multispectral and hyperspectral images (MSI and HSI) try to provide additional
spectral information, beyond the visible spectra, to reveal extra details and
compensate for the coarse spatial resolution of these images.

Figure 1: Open-set land-cover classification: Data samples corresponding to
ground truth categories are from the known class set (K). It is likely that
some categories are not known during training and will be encountered at
testing, i.e., samples from unknown class set (U). The goal is to identify
pixels coming from (U), while correctly classify any pixel belonging to (K).
From left to right, a satellite image from the Pavia University dataset [8]
showing unknown materials surfaces with yellow bounding boxes, the ground
truth labels, and visualization of the feature space for both known and
unknown classes using tSNE [9].

Although inherently a classification problem, material classification in
satellite imagery faces a unique challenge: the vast area covered by the
satellite imagery makes the task of generating representative training samples
almost impossible, as there are a large variety of materials existed on the
Earth’s surface, especially those not well-exploited regions. Therefore, one
of the most essential capabilities of land cover classification is to be able
to automatically identify which test image and which area or pixel location of
the image, has a higher probability of hosting new classes of materials. This
would provide essential guideline to human operators in collecting training
samples for the new classes.

The vast majority of existing works for land cover classification have been
done under the “static closed world” assumption, meaning that the training and
testing sets are drawn from the same label set. As a result, a system
observing any unknown class is forced to misclassify it as one of the known
classes, thus, weakening the recognition performance. A more realistic
scenario is to work in a non-stationary and open environment that not all
categories are known a priori and testing samples from unseen classes can
emerge unexpectedly. Recognition of known and unknown pixels in a given image
and correctly classifying known pixels is defined as “open-set land cover
classification”. Fig. 1 explains this process using a real-world satellite
image.

In this paper, we present a multi-tasking representative-discriminative open-
set recognition (RDOSR) framework to address the challenging land cover
classification problem, where both the representative and discriminative
aspects of data are exploited in order to best characterize the differences
between known and unknown classes. We propose the representative and
discriminative learning among three spaces, as shown in Fig. 2, including 1)
the transformation from the raw image space to the embedding feature space,
and 2) the transformation from the embedding feature space to a so-called
abundance space. See Supplement A for an illustration of the effect in
different spaces.

Figure 2: Representative-discriminative learning through the transformation
among 3 spaces: the raw image space, the embedding space, and the abundance
space.

The contributions of this paper are thus summarized as follows: First, unlike
other open-set recognition methods applied on the raw image space directly, we
propose to first learn a classification network that would transform from the
raw image space to an embedding feature space such that a more discernible
input is fed into the subsequent open-set learning network. Second, we propose
to use the so-called Dirichlet-net to transform data from the embedding
feature space to the abundance space. Due to the resolution issue, each pixel
in a satellite image covers a large area with more than one constituent
material, resulting in “mixed pixel”. The mixtures are generally assumed to be
a linear combination of a few spectral bases, with the corresponding mixing
coefficients (or abundances). This way, instead of looking at the mixed pixel,
we study the mixing coefficients of each spectral basis in making up the
mixture. Thus the abundance space provides a finer-scale representation.
Third, to the best of our knowledge, this work is the first attempt to address
the critical open-set land cover classification problem essential for
analyzing the Earth’s surface. Fourth, while the proposed method was motivated
by satellite imagery analysis, it is generalizable to RGB images and achieves
promising results.

##  2 Related Work

Conventional Land Cover Classification. These methods mainly employ a
traditional classifier on spectral information, with its discriminative power
further enhanced through feature engineering algorithms such as minimum noise
fraction (MNF) [10], independent component analysis (ICA) [11], morphological
profiles [12], and spectral unmixing [13, 14, 15]. The advent of deep learning
has enabled the extraction of hierarchical features automatically and achieved
unprecedented performance. [16, 17] applied a 1D-CNN framework in the spectral
domain to take into account the correlation between adjacent spectral bands.
Several works use a patch surrounding the desired pixels by adopting a 2D-CNN
structure [18, 19] to incorporate the spatial correlation as well. More
recently, integration of both spectral and spatial domains using 3D-CNN
structures have been employed to further improve the classification accuracy
[20, 21].

Although each approach has its own merit, all the existing land cover
classification approaches work under the closed-set assumption where the
training and testing sets share the same label set.

Open-set Recognition. Open-set recognition has gained considerable attention
due to its handling of unknown class samples based on incomplete knowledge of
the data during model training. Early studies are based on traditional
classification models including Nearest Neighbor, Support Vector Machine
(SVMs), Sparse Representation, etc. The open-set version of Nearest Neighbor
was developed based on the distance of the testing samples to the known
samples [22]. The SVM-based approaches employed different regularization terms
or kernels to detect unknown samples [23, 24]. In [25], the residuals from the
Sparse Representation-based Classification (SRC) algorithm were used as the
score for unknown class detection.

In the context of deep networks, [26] employed a statistical model to
calibrate the SoftMax scores and produced an alternative layer, called
OpenMax. [27] improved upon the OpenMax layer approach by maximizing the
inter-class distance and minimizing the intraclass distance on the penultimate
layer. The work of [28] proposed a k-sigmoid activation-based loss function
for training a neural network to be able to find an operating threshold on the
final activation layer. [29] incorporated the latent representation for
reconstruction along with the discriminative features obtained from a
classification model to enhance the feature vector used for open-set
detection. Unlike previous methods, [30] utilized reconstruction error
obtained from a multi-task learning framework as a detection score. Recently,
[31] proposed using self-supervision and augmented the input image to learn
richer features to improve separation between classes.

More recent works try to simulate open-set classes in order to provide
explicit probability estimation over unknown classes. Ge et al. [32] extended
OpenMax [26] by synthesizing the unknown samples using a Generative
Adversarial Network (GAN) based framework. Along the same line, [33] proposed
the counterfactual image generation (OSRCI) framework which employs a GAN to
generate samples placing between decision boundaries that can be treated as
unknown examples. [34] proposed class-conditioned auto-encoder (C2AE)
algorithm where conditional reconstruction helps learning of both known and
unknown score distributions.

It should be noted that there are related problems in the literature including
outlier detection [35, 36] and anomaly detection [37, 38] that have some
overlap with open-set recognition and can be treated as a relaxed version of
open-set recognition. These problems assume the availability of one abnormal
class during training. However, general open-set recognition problems usually
do not provide information about the type or the number of the unknown classes
in advance.

##  3 Proposed Approach

We propose a representative-discriminative open-set recognition (RDOSR)
structure, as shown in Fig. 3. The network mainly consists of two components,
1) a closed-set embedding component to project the data from the original
image domain to the embedding domain, such that different classes with similar
spectral characteristics are more distinguishable, and 2) a multi-task
representative-discriminative learning component to learn a better
representation scheme at a finer scale in the abundance space, such that
unknown classes can be better differentiated from known classes.

###  3.1 Network Architecture

Figure 3: An overview of the proposed framework: i) Closed-set embedding
learning: the classifier $F$ is trained on the spectral domain $X$ to produce
latent discriminative embedding $\mathbf{z_{F}}$. ii) Representative-
discriminative feature learning: the encoder $E$ takes the embedding feature
$\mathbf{z_{F}}$ and derives the representative features $S$ using a
Dirichlet-Net. The classifier $C$ applied on $S$ enhances the discriminative
aspect of $S$, and the reconstruction error between the decoder output
($\mathbf{\hat{z}_{F}}$) and input to encoder ($\mathbf{z_{F}}$) enhances the
representative aspect of $S$.

One challenging issue of the open-set satellite land cover classification
problem is that different classes may possess similar spectral
characteristics. Thus, it is likely that an unknown class, whose spectral
profile is close to that of a known class, may be misclassified as the known
class. To address this issue, instead of detecting unknown classes on the
image domain, we detect them on the embedding domain projected by a closed-set
embedding layer, as shown in Fig. 3.i. The closed-set embedding layer
increases the discriminative power of network to a large extent, such that
unknown classes can be better recognized even if their spectra are similar to
those of the known classes. The weights of the closed-set embedding layer are
trained with a classifier $F$, which is further elaborated in Sec. 3.2.

To recognize unknown classes in the embedding domain, we propose a multi-task
representative-discriminative feature learning framework to boost both the
representative and discriminative power of the extracted feature vector, such
that it is more informative and effective to recognize unknown samples. This
is shown in Fig. 3.ii. The network consists of an encoder-decoder architecture
with the representative features $S$ extracted using a sparse Dirichlet
encoder $E$, and a decoder formed by the bases shared among known classes. A
classifier $C$ applied on $S$ is also included to further increase its
discriminative capability. In this way, the data from unknown classes fed into
the network would produce higher reconstruction error, thus can be detected
accordingly. The details of network design are further elaborated in Sec. 3.3.

###  3.2 Closed-set Embedding Learning

Given the set of sample pixels
$X_{k}=\\{\mathbf{x}_{1},\mathbf{x}_{2},\dots,\mathbf{x}_{N_{k}}\\}$ from the
known classes with each pixel, $\mathbf{x}_{i}$, being a high-dimensional
vector recording the reflectance readings of different spectral bands in the
hyperspectral image, the corresponding labels are denoted with
$Y_{k}=\\{y_{1},y_{2},\dots,y_{N_{k}}\\}$, where $N_{k}$ is the number of
known pixels and $\forall y_{i}\in\\{1,2,\dots,L\\}$, where $L$ is the number
of known classes. To distinguish classes with similar spectral distributions,
we project the input data $X_{k}$ from the image domain to the embedding
domain $Z_{F}$. The projection is learned through a classifier, $F$, with
parameters $\Theta_{F}$ and the embedded features, $\mathbf{z_{F}}$ is forced
to be discriminative through the cross-entropy loss,

$\mathcal{L}_{f}(\Theta_{F})=-\frac{1}{N_{k}}\displaystyle\sum_{i=1}^{N_{k}}y_{i}\log[F(\mathbf{x}_{i})],$
(1)

where $y_{i}$ is a one-hot encoded label and $F(\mathbf{x}_{i})$ denotes the
vector carrying the predicted probability score of the $i^{th}$ known sample.
Such vector is generated by applying a softmax function on the features
$\mathbf{z_{F}}$ in the embedding domain.

This general structure is sufficient for a common classification problem where
the encountered classes are known. However, our goal is to increase the
discriminative power of the features from classes with similar spectral
characteristics. Therefore, we further increase the discriminative capacity of
the embedded features with the $l_{1}$-norm sparse constraint defined by

$\mathcal{L}_{z}(\Theta_{F})=\frac{1}{N_{k}}\displaystyle\sum_{i=1}^{N_{k}}{\left\lVert\mathbf{z_{F}}_{i}\right\rVert},$
(2)

where $\mathbf{z_{F}}_{i}$ is the embedding feature vector learned by the
classifier $F$. With such constraint, the embedded features from samples of
different classes are more discriminative, even if their spectra are similar
in the image domain.

###  3.3 Multi-Task Representative-Discriminative Feature Learning

With the proposed closed-set embedding layer, the samples are projected from
the image domain to the embedding domain possessing more distinguishable
features. In order to better identify unknown samples, both discriminative and
representative nature of the samples need to be exploited. Previous approaches
[30, 34] usually train a general auto-encoder to reconstruct samples from
known classes. When the samples from unknown classes are fed into the network,
the reconstruction error is expected to be larger than that of the known
classes in the ideal case, since the network weights are optimized during the
training procedure using known samples. However, the challenge lies in the
scenario when the unknown classes especially the ones close to the known
classes may potentially contribute to small reconstruction error too which
would lead to failure in detection.

In this work, instead of adopting a general-purpose auto-encoder, we propose a
multi-task representative-discriminative feature learning framework to improve
the detection accuracy. The purpose of this network is to decrease the
reconstruction error of the known classes while increasing the reconstruction
error of unknown classes intensively.

Due to the resolution issue, each pixel in a satellite image usually covers a
large geographical area or footprint (e.g., 30$\times$30m for Landsat-8),
resulting in the so-called “mixed pixel” (i.e., each pixel tends to cover more
than one constituent materials). These mixtures are generally assumed to be a
linear combination of a few spectral bases with the corresponding mixing
coefficients (or abundance). The proposed method is designed based on this
assumption as shown in Eq. 3. Assume that the feature vector of a sample from
known classes $\mathbf{z_{F}}$ is a linear combination of a few bases $B$, and
such bases are shared among features of the known classes. Thus, each sample
of known classes can be decomposed with

$\mathbf{z_{F}}=\mathbf{s}B,$ (3)

where $\mathbf{s}$ denotes the proportional coefficients of the shared bases,
which serves as a form of “representation” of the embedding feature, that we
refer to as the abundance. The abundance vector, or representation, should
satisfy two physical constraints, i.e., non-negative and sum-to-one. The
samples from unknown classes are also able to be decomposed by Eq. 3 using
shared bases of the known classes, $B$. However, since $B$ does not include
the bases of the unknown classes, the distribution of its representations
$\mathbf{s}$ should deviate from that of the known classes. Therefore, we
design a network following the model of Eq. 3, which enforces $\mathbf{s}$
from known classes to follow a certain distribution. And if the network can
extract $\mathbf{s}$ from unknown classes with similar distributions, then we
expect they have high reconstruction errors.

Figure 4: The flowchart of the multi-task representative-discriminative
feature learning framework.

The flowchart of the proposed multi-task representative-discriminative feature
learning is detailed in Fig. 4. The network performs both the reconstruction
task and the classification task. The reconstruction branch consists of a
sparse Dirichlet-based encoder $E$ with weights $\Theta_{E}$ and a decoder $D$
with weights $\Theta_{D}$. The encoder and decoder can be defined by the
functions $E:Z_{F}\to S$ and $D:S\to Z_{F}$, respectively, where $Z_{F}$ is
the embedding space obtained by the closed-set classifier $F$, and $S$ is the
abundance space of latent representations projected by the encoder $E$. The
representations $\mathbf{s}$ in the latent space $S$ is enforced to follow a
Dirichlet distribution. And a sparse constraint is introduced to enhance the
representativeness of $\mathbf{s}$. More details and justifications are
provided below. In addition, $S$ is also enforced to be discriminative by the
classifier $C$, which can be defined by the function $C:S\to Y$ with weights
$\Theta_{C}$, where $Y$ is the space of known labels.

####  3.3.1 Representative Feature Learning with Reconstruction

The reconstruction branch is constructed according to Eq. 3, where the shared
bases are embedded in the decoder $D$ of the network and the corresponding
representation $\mathbf{s}$ is extracted with the encoder $E$. Since
$\mathbf{s}$ denotes the proportional coefficients of the bases, we enforce it
to follow a Dirichlet distribution meeting the non-negative and sum-to-one
physical constraints. Following the work of [39, 40, 41], we adopt the stick-
breaking structure in the encoder to enforce the representations $\mathbf{s}$
to follow the Dirichlet distribution.

In the stick-breaking structure, a single element $s_{j}$ in $\mathbf{s}$ can
be expressed by

$s_{j}=\left\\{\begin{array}[]{ll}v_{1}&\text{for}\quad j=1\\\
v_{j}\prod_{o<j}(1-v_{o})&\text{for}\quad j>1,\end{array}\right.$ (4)

where $v_{j}$ is drawn from a Kumaraswamy distribution, i.e.,
$v_{j}\sim\text{Kuma}(u,1,\beta)$ as shown in Eq. (5),

$v_{o}\sim(1-(1-u^{\frac{1}{\beta}})).$ (5)

Then, there are two parameters used to extract representations $\mathbf{s}$,
i.e., $u$ and $\beta$, both of which are hidden layers in the encoder of the
network. A softplus activation function is adopted on the layer $\beta$ due to
its non-negative property, and a sigmoid is used to map $u$ into the $(0,1)$
range at the layer $u$. More details of the stick-breaking structure can be
found in [40] and [41].

In addition, the entropy function [42] is adopted to reinforce the sparsity of
the representation layer. Let $\hat{s}_{j}=\frac{|s_{j}|}{\|\mathbf{s}\|}$,
for each pixel, the entropy function is defined as,

$\mathcal{H}_{s}(\Theta_{E})=-\sum_{j=1}^{c}\hat{s}_{j}\log\hat{s}_{j}.$ (6)

where $c$ is the dimension of the representation $\mathbf{s}$. The
reconstruction loss $\mathcal{L}_{r}$ is adopted to reduce the reconstruction
error of the known classes. It is defined by,

$\displaystyle\mathcal{L}_{r}(\\{\Theta_{E},\Theta_{D}\\})=\frac{1}{N_{k}}\displaystyle\sum_{i=1}^{N_{k}}{\left\lVert\mathbf{z_{F}}_{i}-\mathbf{\hat{z}_{F}}\right\rVert}_{2},$
(7)

where $\mathbf{z_{F}}_{i}$ is the embedding feature vector fed into the
encoder $E$, and $\mathbf{\hat{z}_{F}}$ is the reconstructed
$\mathbf{z_{F}}_{i}$ obtained from the decoder $D$.

####  3.3.2 Discriminative Feature Learning with Classification Branch.

To further increase the discriminative capacity of the representations, a
classifier is adopted on the representations $\mathbf{s}$ with the
classification loss $\mathcal{L}_{c}$ defined as,

$\displaystyle\mathcal{L}_{c}(\\{\Theta_{E},\Theta_{C}\\})=\frac{1}{N_{k}}\displaystyle\sum_{i=1}^{N_{k}}y_{i}\log[E(\mathbf{z_{F}}_{i})],$
(8)

where $y_{i}$ is the ground truth label and $E(\mathbf{z_{F}}_{i})$ denotes
the representative feature vector of the $i^{th}$ known sample. Note that the
weights of both the reconstruction branch and classifier $C$ are updated
together, such that the learned representations can be both representative and
discriminative.

###  3.4 Training Procedure and Network Settings

We first learn the embedding projection by optimizing the weights $\Theta_{F}$
of the classifier $F$ with the loss function,

$\displaystyle\displaystyle{\min_{\Theta_{F}}\lambda_{f}\mathcal{L}_{f}+\lambda_{z}\mathcal{L}_{z}},$
(9)

where $\lambda_{f}$ and $\lambda_{z}$ are two parameters to balance the trade-
off between the cross-entropy loss and the sparsity loss.

Then, having the learned embedding layer, the multi-task representative-
discriminative feature learning network is trained to minimize both the
reconstruction loss and the classification error of the known classes with
loss function,

$\displaystyle\displaystyle{\min_{\Theta_{E},\Theta_{D},\Theta_{C}}\lambda_{r}\mathcal{L}_{r}+\lambda_{s}\mathcal{H}_{s}+\lambda_{c}\mathcal{L}_{c}},$
(10)

where $\lambda_{r}$, $\lambda_{s}$, and $\lambda_{c}$ are parameters balancing
the trade-off between the reconstruction loss, the sparsity loss, and the
classification loss.

The structures of the four networks, $F$, $E$, $D$, and $C$ are listed in
Table 1.

Table 1: The nodes in the proposed network Networks | $F$ | $E$ | $D$ | $C$
---|---|---|---|---
nodes | [512,1024,512,32,$L$] | [3,3,3,3, 10] | [10,10,$L$] | [$L$]

##  4 Experiments and Results

In this section, the effectiveness of the proposed RDOSR method is evaluated
on several widely used benchmark hyperspectral image datasets. In addition, we
demonstrate the generalization capacity of the proposed approach on RGB image
datasets. Furthermore, the contribution from each component of the proposed
framework is analyzed through ablation study.

###  4.1 Implementation Details

We train the network, described in Sec. 3.1, using an Adam optimizer [43],
with a learning rate of $10^{-3}$. The classifier $F$ and the joint structure
of encoder-decoder-classifier ($E$-$D$-$C$) are trained separately for a total
number of $15$K epochs. However, the other methods which do not have two
separate components were trained for $6$K epochs.

For training the classifier $F$, $\lambda_{f}$ and $\lambda_{z}$ are set equal
to $1$ and $0.1$, respectively. The weights for the reconstruction
$\lambda_{r}$, sparsity $\lambda_{s}$, and classification $\lambda_{c}$ losses
in training the $E$-$D$-$C$ structure are set equal to $0.5$, $10^{-3}$, and
$0.5$, respectively. The sparsity weight $\lambda_{s}$ is decayed with a
weight decay of 0.9977. The classifier $F$ is trained until its accuracy
reaches $0.9988$. It should be noted that all input data of the datasets are
normalized to their mean value and unit variance. In addition, the feature
vector obtained from the classifier $F$ is divided by 10 to avoid divergence.

One of the factors that affects the performance of the open-set recognition
algorithm is Openness [44] of the problem, defined as,

$\displaystyle Openness=1-\sqrt{\frac{2\times
N_{train}}{N_{test}+N_{target}}},$ (11)

where $N_{train}$, $N_{test}$ and $N_{target}$ are the number of classes known
during training, the number of classes given during testing, and the number of
classes that need to be recognized correctly during testing phase,
respectively. In the experiments, classes of each dataset is partitioned into
known and unknown sets according to the Openness.

The code is written in TensorFlow, and all the experiments are performed on a
desktop computer having GeForce GPU of 10 GB Memory. The code is available at
https://github.com/raziehkaviani/rdosr.

###  4.2 Metrics

To compare performance of different methods, there are several metrics
including overall accuracy or F-score on a combination of known and unknown
classes, and Receiver Operating Characteristic (ROC). The first two metrics do
not characterize the performance of the model well due to their sensitivity
not only to the performance of the model in classifying the known classes, but
also an arbitrary operating threshold for detecting unknown samples.

On the other hand, the ROC curve would illustrate the ability of a binary
classification system (here, known vs. unknown detection) as a discrimination
threshold is varied from the minimum to the maximum value of the given
detection measure (here, reconstruction error). Thus, it provides a measure
free from calibration. To have a quantitative comparison, the area under the
ROC (AUC) is computed in the experiments.

###  4.3 Open-set Recognition for Hyperspectral Data

The experiments are conducted on three hyperspectral image datasets:

Pavia University (PU) and Pavia Center (PC). Both PU and PC datasets were
gathered over Northern Italy in 2011 by the Reflective Optics Systems Imaging
Spectrometer which has a resolution of 1.3 m. The dimension of the PU dataset
is $1096\times 715$ pixels with 103 spectral bands, ranging from 430 to 860
nm. The PC dataset has $610\times 340$ pixels with 102 spectral bands. The PU
and PC datasets both include nine land cover categories.

Indian Pines (IN). The IN dataset was collected over Northwest Indiana in 1912
by Airborne Visible/Infrared Imaging Spectrometer (AVIRIS). It has a dimension
of $145\times 145$ with a resolution of 20 m by pixel and 200 spectral bands.
Its ground truth consists of 16 land cover classes.

We compare the performance of the proposed approach with three methods
described in the following:

SoftMax: In a neural network classifier, a common confidence-based approach to
detect open-set examples is thresholding the SoftMax scores. We use network
structure of the classifier $F$ without considering sparsity constraint.

OpenMax [26]: This approach calibrates the SoftMax scores in a classifier and
augments them with a $N_{k}+1$ class for an unknown category. The replaced
SoftMax layer with an OpenMax layer is used for open-set recognition. We adopt
the classifier mentioned earlier in the SoftMax method, and use the Weibull
fitting approach with parameter $Weibull\,tail\,size=10$ to generate OpenMax
layer values.

AE+CLS: Fully connected version of MLOSR [30] which utilizes a multi-task
learning framework, composed of a classifier and a decoder with a shared
feature extraction part, to detect open-set examples. To have a fair
comparison with our approach, the encoder, decoder, and classifier are
designed as our $E$ (without the Dirichlet-Net), $D$, $C$, and trained with
$\mathcal{L}_{r}$ and $\mathcal{L}_{c}$ loss, with weights of 0.5.

First, each of the $L$ classes is assumed to be unknown which equates to an
openness of $2.99\%$, $2.99\%$, and $1.63\%$ for PU, PC, IN, respectively. The
AUC values corresponding to choosing each of the $L$ labels as unknown are
averaged and reported for each method in Table 2. It can be observed that the
proposed method outperforms other methods on all three datasets. See
Supplement B for detailed comparison on the PU dataset. The minor improvement
on the PC dataset can be justified by its differentiated spectrum of different
classes which diminishes the effect of the classifier $F$.

Second, for the Openness equal to $6.46\%$, the ROC curves of different
methods for the PU and IN datasets are illustrated in Fig. 5. As seen from the
results on both datasets, the AE+CLS+Dirichlet method which adopts the
Dirichlet net to the AE+CLS framework and the proposed method lie above all
other methods. It should be noticed that our proposed method is able to detect
unknown classes with $60\%$ and above $90\%$ accuracy and almost zero false
detection for the PU and PC datasets, respectively.

Table 2: Area under the ROC curve for open-set detection. Results are averaged over $L$ partitioning of the selected dataset to $L-1$ known and $1$ unknown classes. Method |  PU | PC |  IN
---|---|---|---
SoftMax |  0.385 | 0.816 |  0.555
OpenMax [26] |  0.441 | 0.884 |  0.415
AE+CLS [30] |  0.586 | 0.757 |  0.669
AE+CLS+Dirichlet |  0.714 | 0.927 |  0.681
RDOSR (Ours) |  0.773 | 0.963 |  0.802

Third, the histograms of reconstruction error for both known and unknown sets
with Openness$=2.99\%$ are shown in Fig. 6. It can be observed that the
reconstruction errors corresponding to the known set have small values.
However, the unknown set produces larger error due to mismatches in terms of
representative and discriminative features learned from the known classes
examples.

(a) Open-set detection on PU dataset

(b) Open-set detection on PC dataset

Figure 5: Receiver Operating Curve curves for open-set recognition for PU and
PC datasets, for $L=7$ (openness=$6.46\%$).

(a) PU dataset

(b) PC dataset

Figure 6: Reconstruction error distribution of known and unknown classes using
the proposed method for PU and PC datasets, for $L=8$.

###  4.4 Open-set Recognition for RGB Images

To show the generalization capacity of the proposed method, we evaluate the
performance of the proposed approach on two RGB datasets and compare with
several state-of-the-art methods. For this purpose, the classifier $F$
performing pixel-wise classification is substituted with a DenseNet structure
which takes a 2D image as input.

Following the protocol in [33], we sample 4 known classes from CIFAR10 [45] to
have Openness=$13.39\%$ and 20 known classes out of 200 categories of
TinyImageNet [46] resulting in an Openness of $57.35\%$. Table 3 summarizes
the results where the values other than the proposed RDOSR are taken from
[31]. It can be observed that the proposed method has better performance over
the compared methods, except for GDOSR [31], on CIFAR10. However, it achieves
significant improvement on TinyImageNet. It may be due to the similarity
between the classes in TinyImageNet which hinders detecting unknown samples in
the image space while RDOSR addresses this issue by operating in the embedding
space.

Table 3: Area under the ROC curve for Open-set recognition. Method |  CIFAR10 | TinyImageNet
---|---|---
SoftMax |  0.677 | 0.577
OpenMax [26] |  0.695 | 0.576
OSRCI [33] |  0.699 | 0.586
C2AE [34] |  0.711 | 0.581
GDOSR [31] |  0.807 | 0.608
RDOSR (Ours) |  0.744 | 0.752

###  4.5 Ablation Study

Figure 7: Ablation study of the proposed method on PU dataset

Starting with a baseline, AE+CLS, each component is gradually added to the
framework to show its effectiveness. The results corresponding to the ablation
study are shown in Fig. 7. It can be seen that employing the baseline
structure applied on the spectra domain has the worst performance. However,
adding the Dirichlet-based network makes a major improvement due to applying
physical constraints on the latent space learned by the encoder $E$. Directly
performing open-set recognition on an embedding space causes instability
problem which is confirmed by a performance drop compared to the
AE+CLS+Dirichlet method. Our proposed method addresses the instability issue
by adopting a sparsity constraint on the embedding feature vector
$\mathbf{z_{F}}$. As seen from Fig. 7, our proposed method achieves the
highest AUC value compared to three other baseline methods.

##  5 Conclusions

We studied the challenging problem of open-set land cover recognition in
satellite images. Although inherently a classification problem, both
representative and discriminative features need to be learned in order to best
characterize the difference between known and unknown classes. We presented
the transformation among three spaces, that is, the original image space, the
embedding feature space, and the abundance space, where features with both
representative and discriminative capacity can be learned to maximize success
rate. The proposed multi-tasking representative-discriminative learning
structure was evaluated on three hyperspectral and two RGB image datasets and
exhibited significant improvement over state-of-the-art open-set recognition
algorithms.

## References

  * [1] Krizhevsky, A., Sutskever, I., Hinton, G.E.:  Imagenet classification with deep convolutional neural networks.  In: Advances in neural information processing systems. (2012) 1097–1105
  * [2] He, K., Zhang, X., Ren, S., Sun, J.:  Deep residual learning for image recognition.  In: Proceedings of the IEEE conference on computer vision and pattern recognition. (2016) 770–778
  * [3] Simonyan, K., Zisserman, A.:  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556 (2014)
  * [4] Redmon, J., Divvala, S., Girshick, R., Farhadi, A.:  You only look once: Unified, real-time object detection.  In: Proceedings of the IEEE conference on computer vision and pattern recognition. (2016) 779–788
  * [5] Ren, S., He, K., Girshick, R., Sun, J.:  Faster r-cnn: Towards real-time object detection with region proposal networks.  In: Advances in neural information processing systems. (2015) 91–99
  * [6] Long, J., Shelhamer, E., Darrell, T.:  Fully convolutional networks for semantic segmentation.  In: Proceedings of the IEEE conference on computer vision and pattern recognition. (2015) 3431–3440
  * [7] Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.:  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  IEEE transactions on pattern analysis and machine intelligence 40(4) (2017) 834–848
  * [8] http://lesun.weebly.com/hyperspectral-data-set.html.
  * [9] Maaten, L.v.d., Hinton, G.:  Visualizing data using t-sne.  Journal of machine learning research 9(Nov) (2008) 2579–2605
  * [10] Green, A.A., Berman, M., Switzer, P., Craig, M.D.:  A transformation for ordering multispectral data in terms of image quality with implications for noise removal.  IEEE Transactions on geoscience and remote sensing 26(1) (1988) 65–74
  * [11] Comon, P.:  Independent component analysis, a new concept?  Signal processing 36(3) (1994) 287–314
  * [12] Plaza, A., Martinez, P., Plaza, J., Perez, R.:  Dimensionality reduction and classification of hyperspectral image data using sequences of extended morphological transformations.  IEEE Transactions on Geoscience and remote sensing 43(3) (2005) 466–479
  * [13] Luo, B., Chanussot, J.:  Unsupervised classification of hyperspectral images by using linear unmixing algorithm.  In: 2009 16th IEEE International Conference on Image Processing (ICIP), IEEE (2009) 2877–2880
  * [14] Dópido, I., Zortea, M., Villa, A., Plaza, A., Gamba, P.:  Unmixing prior to supervised classification of remotely sensed hyperspectral images.  IEEE Geoscience and Remote Sensing Letters 8(4) (2011) 760–764
  * [15] Baghbaderani, R.K., Wang, F., Stutts, C., Qu, Y., Qi, H.:  Hybrid spectral unmixing in land-cover classification.  In: International Geoscience and Remote Sensing Symposium (IGARSS). (2019)
  * [16] Hu, W., Huang, Y., Wei, L., Zhang, F., Li, H.:  Deep convolutional neural networks for hyperspectral image classification.  Journal of Sensors 2015 (2015)
  * [17] Song, Y., Zhang, Z., Baghbaderani, R.K., Wang, F., Stutts, C., Qi, H.:  Land cover classification for satellite images through 1d cnn.  In: Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing (WHISPERS). (2019)
  * [18] Cao, X., Zhou, F., Xu, L., Meng, D., Xu, Z., Paisley, J.:  Hyperspectral image classification with markov random fields and a convolutional neural network.  IEEE Transactions on Image Processing 27(5) (2018) 2354–2367
  * [19] Sharma, V., Diba, A., Tuytelaars, T., Van Gool, L.:  Hyperspectral cnn for image classification & band selection, with application to face recognition.  Technical report KUL/ESAT/PSI/1604, KU Leuven, ESAT, Leuven, Belgium (2016)
  * [20] Hamida, A.B., Benoit, A., Lambert, P., Amar, C.B.:  3-d deep learning approach for remote sensing image classification.  IEEE Trans. on geoscience and remote sensing 56(8) (2018) 4420–4434
  * [21] Zhong, Z., Li, J., Luo, Z., Chapman, M.:  Spectral–spatial residual network for hyperspectral image classification: A 3-d deep learning framework.  IEEE Transactions on Geoscience and Remote Sensing 56(2) (2017) 847–858
  * [22] Júnior, P.R.M., De Souza, R.M., Werneck, R.d.O., Stein, B.V., Pazinato, D.V., de Almeida, W.R., Penatti, O.A., Torres, R.d.S., Rocha, A.:  Nearest neighbors distance ratio open-set classifier.  Machine Learning 106(3) (2017) 359–386
  * [23] Scheirer, W.J., Jain, L.P., Boult, T.E.:  Probability models for open set recognition.  IEEE transactions on pattern analysis and machine intelligence 36(11) (2014) 2317–2324
  * [24] Júnior, P.R.M., Boult, T.E., Wainer, J., Rocha, A.:  Specialized support vector machines for open-set recognition.  arXiv preprint arXiv:1606.03802 (2016)
  * [25] Zhang, H., Patel, V.M.:  Sparse representation-based open set recognition.  IEEE transactions on pattern analysis and machine intelligence 39(8) (2016) 1690–1696
  * [26] Bendale, A., Boult, T.E.:  Towards open set deep networks.  In: Proceedings of the IEEE conference on computer vision and pattern recognition. (2016) 1563–1572
  * [27] Hassen, M., Chan, P.K.:  Learning a neural-network-based representation for open set recognition.  arXiv preprint arXiv:1802.04365 (2018)
  * [28] Shu, L., Xu, H., Liu, B.:  Doc: Deep open classification of text documents.  arXiv preprint arXiv:1709.08716 (2017)
  * [29] Yoshihashi, R., Shao, W., Kawakami, R., You, S., Iida, M., Naemura, T.:  Classification-reconstruction learning for open-set recognition.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (2019) 4016–4025
  * [30] Oza, P., Patel, V.M.:  Deep cnn-based multi-task learning for open-set recognition.  arXiv preprint arXiv:1903.03161 (2019)
  * [31] Perera, P., Morariu, V.I., Jain, R., Manjunatha, V., Wigington, C., Ordonez, V., Patel, V.M.:  Generative-discriminative feature representations for open-set recognition.  In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. (2020) 11814–11823
  * [32] Ge, Z., Demyanov, S., Chen, Z., Garnavi, R.:  Generative openmax for multi-class open set classification.  arXiv preprint arXiv:1707.07418 (2017)
  * [33] Neal, L., Olson, M., Fern, X., Wong, W.K., Li, F.:  Open set learning with counterfactual images.  In: Proceedings of the European Conference on Computer Vision (ECCV). (2018) 613–628
  * [34] Oza, P., Patel, V.M.:  C2ae: Class conditioned auto-encoder for open-set recognition.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (2019) 2307–2316
  * [35] Xia, Y., Cao, X., Wen, F., Hua, G., Sun, J.:  Learning discriminative reconstructions for unsupervised outlier removal.  In: Proceedings of the IEEE International Conference on Computer Vision. (2015) 1511–1519
  * [36] You, C., Robinson, D.P., Vidal, R.:  Provable self-representation based outlier detection in a union of subspaces.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (2017) 3395–3404
  * [37] Chalapathy, R., Menon, A.K., Chawla, S.:  Robust, deep and inductive anomaly detection.  In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Springer (2017) 36–51
  * [38] Golan, I., El-Yaniv, R.:  Deep anomaly detection using geometric transformations.  In: Advances in Neural Information Processing Systems. (2018) 9758–9769
  * [39] Sethuraman, J.:  A constructive definition of dirichlet priors.  Statistica Sinica (1994) 639–650
  * [40] Nalisnick, E., Smyth, P.:  Stick-breaking variational autoencoders.  International Conference on Learning Representations (ICLR) (2017)
  * [41] Qu, Y., Qi, H., Kwan, C.:  Unsupervised sparse dirichlet-net for hyperspectral image super-resolution.  Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018) 2511–2520
  * [42] Huang, S., Tran, T.D.:  Sparse signal recovery via generalized entropy functions minimization.  IEEE Transactions on Signal Processing 67(5) (2018) 1322–1337
  * [43] Kingma, D.P., Ba, J.:  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980 (2014)
  * [44] Scheirer, W.J., de Rezende Rocha, A., Sapkota, A., Boult, T.E.:  Toward open set recognition.  IEEE transactions on pattern analysis and machine intelligence 35(7) (2012) 1757–1772
  * [45] Krizhevsky, A., Hinton, G., et al.:  Learning multiple layers of features from tiny images.  (2009)
  * [46] Le, Y., Yang, X.:  Tiny imagenet visual recognition challenge.  CS 231N 7 (2015)

Supplementary Material: Representative-Discriminative Learning for Open-set
Land Cover Classification of Satellite Imagery

## Supplementary Contents

Section A presents a visual comparison of data used for open-set recognition
among the three spaces, i.e., the image space ($X$), the embedding space
($Z_{f}$), and the abundance space ($S$). Section B shows the performance of
open-set recognition if conducted on the image space ($X$) or the embedded
space ($Z_{f}$).

## A A visual comparison among spaces $X$, $Z_{f}$, and $S$

Figures S1, S2, and S3 illustrate the mean of samples belonging to different
classes in spaces $X$, $Z_{f}$, and $S$, respectively, using the PU dataset.
The feature vectors learned through $F$ and $E$, shown in Figs. S2 and S3,
respectively, are sparse due to the sparsity constraint.

It can be seen from Fig. S1 that the spectrum of samples belonging to class 3
and 8 are close. However, the feature vectors, learned through $F$,
corresponding to class 3 and 8 are more discriminative. Further, the
discriminative and representative characteristics of the features are enhanced
through encoder $E$, as illustrated in Fig. S3.

Figure S1: Mean of samples in space $X$, belonging to classes 1 to 9, using
the PU dataset

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

Figure S2: Mean of samples in space $Z_{f}$, belonging to classes 1 to 9,
using the PU dataset

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

Figure S3: Mean of samples in space $S$, belonging to classes 1 to 9, using
the PU dataset

## B Comparison of open-set recognition performing on space $X$ and $Z_{f}$

To better compare the effectiveness of performing open-set recognition in
spaces $X$ and $Z_{f}$, we show the results of performing in each space
separately using the PU dataset in Table S1.

Comparing the results of $AE+CLS$ and $AE+CLS+Dirichlet$ approaches performed
on spaces $X$ and $Z_{f}$, it can be seen that discriminative features learned
through the classifier $F$ contribute to a substantial improvement. In
addition, the Dirichlet network plays more critical role when performing open-
set recognition in space $X$ as compared to space $Z_{f}$.

Table S1: Area under the ROC curve for open-set recognition. Results are from partitioning PU dataset to $L-1$ known and the mentioned unknown classes, (openness=$2.99\%$). Note that $L$ denotes the number of classes in the original PU dataset Space | Method | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | Avg.
---|---|---|---|---|---|---|---|---|---|---|---
$X$ | SoftMax | 0.54 | 0.52 | 0.51 | 0.42 | 0.14 | 0.38 | 0.23 | 0.64 | 0.09 | 0.39
OpenMax [26] | 0.67 | 0.37 | 0.45 | 0.40 | 0.99 | 0.35 | 0.12 | 0.57 | 0.04 | 0.44
AE+CLS [30] | 0.51 | 0.53 | 0.54 | 0.83 | 1.0 | 0.46 | 0.48 | 0.46 | 0.46 | 0.59
AE+CLS+Dirichlet | 0.79 | 0.69 | 0.47 | 0.90 | 1.0 | 0.64 | 0.47 | 0.48 | 0.97 | 0.71
$Z_{f}$ | AE+CLS | 0.91 | 0.70 | 0.68 | 0.72 | 1.0 | 0.62 | 0.46 | 0.66 | 0.94 | 0.74
AE+CLS+Dirichlet | 0.91 | 0.70 | 0.71 | 0.72 | 1.0 | 0.68 | 0.51 | 0.80 | 0.93 | 0.77