# Multi-Teacher Multi-Objective Meta-Learning
for Zero-Shot Hyperspectral Band Selection

Jie Feng, Xiaojian Zhong, Di Li, Weisheng Dong, Ronghua Shang, and Licheng
Jiao J. Feng, X. Zhong, D. Li, W. Dong, R. Shang and L. Jiao are with the Key
Laboratory of Intelligent Perception and Image Understanding of Ministry of
Education of China, Xidian University, Xi’an 710071, P.R. China (e-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>lchjiao@mail.xidian.edu.cn).

###### Abstract

Band selection plays a crucial role in hyperspectral image classification by
removing redundant and noisy bands and retaining discriminative ones. However,
most existing deep learning-based methods are aimed at dealing with a specific
band selection dataset, and need to retrain parameters for new datasets, which
significantly limits their generalizability. To address this issue, a novel
multi-teacher multi-objective meta-learning network (M3BS) is proposed for
zero-shot hyperspectral band selection. In M3BS, a generalizable graph
convolution network (GCN) is constructed to generate dataset-agnostic base,
and extract compatible meta-knowledge from multiple band selection tasks. To
enhance the ability of meta-knowledge extraction, multiple band selection
teachers are introduced to provide diverse high-quality experiences.strategy
Finally, subsequent classification tasks are attached and jointly optimized
with multi-teacher band selection tasks through multi-objective meta-learning
in an end-to-end trainable way. Multi-objective meta-learning guarantees to
coordinate diverse optimization objectives automatically and adapt to various
datasets simultaneously. Once the optimization is accomplished, the acquired
meta-knowledge can be directly transferred to unseen datasets without any
retraining or fine-tuning. Experimental results demonstrate the effectiveness
and efficiency of our proposed method on par with state-of-the-art baselines
for zero-shot hyperspectral band selection.

###### Index Terms:

Band selection, hyperspectral image, multi-objective learning, meta-learning,
graph convolutional network.

##  I Introduction

With the rapid development of remote sensing technologies, hyperspectral
images (HSIs) have become an indispensable embranchment in land cover
discrimination. In comparison with multispectral imaging, hyperspectral
imaging measures hundreds of spectral bands for the same spatial area and
provides a continuous spectrum with finer wavelength resolution and richer
semantic information, which is eligible for image classification [1], change
detection [2], and anomaly detection [3]. Over the past decades, it has been
applied in a wide variety of fields, ranging from environmental monitoring,
agriculture, mineral and geological exploration, and military applications
[4].

The abundance of spectral bands enables hyperspectral images to perceive and
identify land covers accurately. However, it brings noisy and irrelevant
information. In addition, as the dimensionality increases, the amount of
samples needed for training a robust classification model grows exponentially
[5]. Furthermore, these hundreds of spectral bands implicitly require
significant computing and storage resources. Therefore, dimensionality
reduction is an imperative step for eliminating spectral noise and redundancy,
and improving the performance of downstream tasks.

Dimensionality reduction can be fulfilled mainly through two approaches:
feature extraction and feature selection [6]. By applying linear or nonlinear
transformations, feature extraction transforms the original feature space into
a lower-dimensional one and generates the representations completely different
from the original ones.

When feature selection comes to HSIs, it is referred to as an alias for band
selection. As the name suggested, band selection aims at identifying a subset
from hundreds of spectral bands to represent the overall spectral information
by removing redundant and noisy bands and retaining discriminative ones. As
opposed to feature extraction, band selection excels at retaining original
physical information [7], making it valuable for real-world applications.

Band selection methods can be roughly divided into filter-based, wrapper-based
and embedding-based [6]. Filter-based methods measure the performance of
candidate band subsets by utilizing certain predefined criteria, which are
usually independent of the chosen classifiers and thus can be calculated
efficiently. In [8], a representative filter-based method, called minimal-
redundancy maximal-relevance (mRMR), was introduced, where the redundancy
between any two spectral bands is minimized, and the dependency of bands with
class labels is maximized.

Wrapper-based methods treat the classification performance of the chosen
classifiers as the evaluation metric of candidate band subsets. Different from
filter-based methods, both evaluation and classification of the selected bands
are implemented by the same classifier, which leads to better classification
performance and more time-consuming training. Dynamic classifier selection,
abbreviated as DCS, was proposed in [9] as a representative wrapper-based
method. DCS utilizes new samples to be classified to select the base
classifiers and generates pseudo-labels through edge-preserving filtering. In
[10], fractional-order Darwinian particle swarm optimization (FODPSO) is
employed to search for candidate band subsets, and its fitness evaluation is
achieved by a support vector machine. Following this, a particle ranking
strategy [11] was proposed to efficiently select features in the multi-
objective swarm optimization space.

Embedding-based methods construct an end-to-end framework by combining band
selection and classification into a single task, where selection results could
be obtained after the training of the chosen classifier. Hence, these methods
not only produce better classification performance than wrapper-based methods
but also maintain competitive speed on par with filter-based methods. In [12],
a SVM based on recursive feature elimination (RFE-SVM) was proposed to
maximize the margin by using the weights of the sequential backward selection.
Zhou et al. [13] integrated a genetic algorithm with a SVM to encode and
optimize the preliminarily selected feature subsets. Instead of sorting weight
values, [14] proposed a kernel-based SVM to differentiate bands by sorting the
magnitude of SVM coefficients, which is more suitable for training SVM
classifiers.

Over the past decades, deep learning has made significant advances in solving
problems that resisted the best practice of the artificial intelligence
community for years [15]. Among numerous deep learning techniques suitable for
band selection, convolutional neural network (CNN) grows by leaps and bounds
due to its excellent spatial representation capacity and powerful nonlinear
fitting ability.

During the early stage, CNN-based methods, such as self-improving CNN (SICNN)
[16], utilized traditional algorithms like FODPSO to search for candidate
subsets. To evaluate each candidate subset, a two-dimensional CNN is involved
in SICNN and needs to be retrained during each iteration. To mitigate the time
consumption caused by retraining, a band selection algorithm based on distance
density (DDCNN) [17] was proposed. In DDCNN, a distance density among all the
bands is calculated to select band combinations. Then, a customized one-
dimensional CNN is pretrained with original full-band HSIs as an evaluation
metric. Nevertheless, the absence of retraining also brings a decline for the
ability of evaluation. Unlike SICNN and DDCNN, Feng et al. [18] constructed a
novel ternary weight CNN (TWCNN) to indicate whether the corresponding band is
selected. TWCNN combines band selection, feature extraction and classification
into a unified end-to-end optimization procedure, comprising a depth-wise
convolutional layer, subsequent convolutional layers and fully-connected
layers. Later, a bandwise-independent binary convolution and a novel coarse-
to-fine loss are introduced in BHCNN [19] to improve the optimization
interpretability brought by discrete weights. Equipped with a novel attention
mechanism, an attention-based one-dimensional CNN (ABCNN) [20] is coupled with
an anomaly detection technique to assign scores to spectral bands and select
the most discriminative ones.

In addition to traditional fully supervised settings, unsupervised and semi-
supervised methods have also attracted great attentions due to their cheaper
annotations. Cai et al. [21] proposed a novel BS-Nets to formalize band
selection as an unsupervised spectral reconstruction task, where the weights
are sparsed by l1-regular constraints for band selection of HSIs. Taking both
labeled and unlabeled samples into consideration, Sellami et al. [22] proposed
a semi-supervised 3D-CNN based on adaptive dimensionality reduction to extract
spectral and spatial features for classification.

Apart from convolutional neural networks, a wide variety of advanced
techniques in deep learning are energizing band selection by constantly
breaking through the theoretical limit of this research field. Mou et al. [23]
framed unsupervised band selection as a Markov decision process and further
exploited reinforcement learning to solve it by training an agent to learn a
band-selection policy with two reward schemes. As in the previous practice,
Feng et al. [24] formalized band selection as a sequential decision-making
process in deep reinforcement learning. For a better measurement on how well a
band subset performs, a semi-supervised CNN is constructed as an efficient
evaluation criterion. To make full use of the structural information, Cai et
al. [25] incorporated graph convolutional layers into a self-representation
model for a more robust coefficient matrix to determine an informative band
subset. Similarly, in [26], insufficient mining towards inter-band correlation
can be handled by a dual-graph convolutional network based on a band attention
map with a sparse constraint.

It should be noted that, although deep learning-based band selection methods
emerge in endlessly and continue to achieve new state-of-the-arts, a common
issue still remains unsolved. Specifically, without considering inherent
correlation among different datasets, most existing deep learning-based
methods are aimed at handling a specific dataset. When encountering a new
dataset, the model needs to be trained from scratch. The inherent correlation
among different datasets can be referred to as meta-knowledge, that is, the
invarient knowledge across different datasets. Therefore, how to design a
dataset-invariant meta-knowledge extractor in zero-shot band selection of HSIs
to maintain adaptability over multiple datasets has become a practical but
challenging topic.

In this paper, we propose a novel multi-teacher multi-objective meta-learning
network (M3BS) for zero-shot hyperspectral band selection. Specifically, a
spatial-spectral graph is first built to encode spectral bands into a discrete
non-Euclidean space, where the vertices are regarded as bands and the edges
are determined according to the spatial and spectral relationship among these
bands. On this basis, a generalizable GCN is constructed as a dataset-
invariant meta-knowledge extractor which decomposites the parameters to
learnable dataset-agnostic bases and dataset-specific coefficients. After
that, subsequent CNN-based classifier is attached. To guarantee a better
generalization ability, several band selection taechers with diversity
ensemble strategy is designed to provide extra supervision. Finally, an
uncertainty-based multi-objective meta-learning procedure is established to
jointly optimize band selection and classification in an end-to-end trainable
way. Once the optimization is finished, the dataset-specific coefficients can
characterize unseen samples, and the acquired dataset-agnostic bases can be
immediately transferred to new datasets without any retraining or fine-tuning.

The main contributions of this paper are listed as follows:

  1. 1.

The generalizable GCN can achieve meta-knowledge extraction that is compatible
with various datasets, and make it possible for M3BS to achieve zero-shot band
selection of HSIs. As the training progresses, inconsistent representations
among different datasets will be synchronized by the unified meta-knowledge.

  2. 2.

Owing to miscellaneous optimization directions, the addition of multifarious
experiences from multiple representative and reliable teacher models can
maintain generalization and adaptability of M3BS on unseen samples in zero-
shot learning, and the generalizable GCN is more prone to accelerate the
training procedure and converge in a reasonable time.

  3. 3.

Using the multi-objective meta-learning end-to-end optimization procedure,
diverse optimization objectives can be automatically coordinated without any
manual intervention, while multiple band selection tasks from multiple
datasets can be simultaneously co-improved. As a result, the acquired meta-
knowledge can be directly transferred from seen samples to unseen samples
without any retraining or fine-tuning.

The remainder of this paper is constituted by the following sections. With a
brief introduction to preliminaries relevant to GCN, multi-objective learning
and meta-learning in Section II, our unified architecture M3BS for zero-shot
hyperspectral band selection is described in Section III. Section IV depicts
quantitative statistical experiments and qualitative theoretical analysis with
other competitive band selection algorithms. Section V ends this paper with a
concise conclusion.

Figure 1: The overall architecture of the proposed M3BS for zero-shot
hyperspectral band selection.

##  II Background

###  A Graph Convolutional Network

GNN [27] stands for a series of models capable of extracting and analyzing
complicated relationships from a discrete graph. In contrast to CNN, GNN has
shown superior performance and great potential in processing data in a non-
Euclidean space. Since being proposed, GNNs have proven their powerful
abstraction ability in various applications and domains, e.g., recommendation
system [28], traffic forecasting [29], and molecular modeling [30].

By generalizing the idea of convolution operators from a two-dimensional grid
space to a discrete graph space, feature representations can be obtained by
aggregating the adjacent vertices with a graph convolutional layer. Let
$\mathcal{G}=\\{\mathcal{V},\mathcal{E},\mathbf{A}\\}$ be an undirected
acyclic graph, where $\mathcal{V}$ and $\mathcal{E}$ are the sets of $n$
vertices and $m$ edges respectively. Before explaining how a graph convolution
operates, the adjacency matrix $\mathbf{A}\in\mathbb{R}^{n\times n}$ is
necessary for describing the similarity between two vertices of the graph
$\mathcal{G}$. Let $\mathbf{X}\in\mathbb{R}^{n\times d}$ denote an optional
matrix associated with feature vectors of each vertex. With a standard radial
basis function [31], each element in the adjacency matrix can be defined as
follows:

$\mathbf{A}_{i,j}=\exp{\left(-\frac{\|\mathbf{X}_{i}-\mathbf{X}_{j}\|_{2}^{2}}{\sigma^{2}}\right)}$
(1)

where $\sigma$ is the width constant, $\mathbf{X}_{i}$ and $\mathbf{X}_{j}$
are two rows of the feature matrix, corresponding to the feature vector of the
$i$-th and $j$-th vertex, respectively. For a better generalization ability
[32], the symmetric normalized Laplacian matrix $\mathbf{L}$ is expressed as:

$\mathbf{L}=\mathbf{I}-\mathbf{D}^{-\frac{1}{2}}\mathbf{A}\mathbf{D}^{-\frac{1}{2}}$
(2)

where $\mathbf{D}$ is the degree matrix of the adjacency matrix, that is,
$\mathbf{D}_{i,j}=\sum_{j}\mathbf{A}_{i,j}$. The Laplacian matrix can be
decomposited by a spectral decomposition
$\mathbf{L}=\mathbf{U}\mathbf{\Lambda}\mathbf{U}^{\top}$. Given two functions
$f(\cdot)$ and $g(\cdot)$, the graph convolution operator $*_{\mathcal{G}}$
can be defined as Eq. (3) by regarding $g_{\theta}=\mathbf{U}^{\top}g$ as the
convolutional kernel:

$f*_{\mathcal{G}}g=\mathbf{U}\\{(\mathbf{U}^{\top}f)\cdot(\mathbf{U}^{\top}g)\\}=\mathbf{U}g_{\theta}\mathbf{U}^{\top}f$
(3)

This formula imitates the way a traditional convolution $*$ is defined, i.e.,
transforming the graph into the spectral domain by a Fourier transform
$\mathbf{U}^{\top}\cdot$, performing a matrix multiplication, and converting
back to the spatial domain by an inverse Fourier transform $\mathbf{U}\cdot$.

As one of the most famous and practical forms of GCNs, the propagation rule
was proposed by Kipf et al. [33], which is also used in this paper. It
diminishes the computational cost by introducing a constrained Chebyshev
polynomial significantly:

$\mathbf{Y}=f(\tilde{\mathbf{D}}^{-\frac{1}{2}}\tilde{\mathbf{A}}\tilde{\mathbf{D}}^{-\frac{1}{2}}\mathbf{X}\mathbf{W}+\mathbf{b})$
(4)

where $\tilde{\mathbf{A}}=\mathbf{A}+\mathbf{I}$ and
$\tilde{\mathbf{D}}_{i,j}=\sum_{j}\tilde{\mathbf{A}}_{i,j}$ are renormalized
from $\mathbf{A}$ and $\mathbf{D}$ for a better training stability, $f(\cdot)$
is a custom activation function, $\mathbf{W}\in\mathbb{R}^{d\times
d^{\prime}}$ and $\mathbf{b}\in\mathbb{R}^{d^{\prime}}$ are weights and bias
that serve as learnable parameters. This graph convolutional layer takes a
feature matrix $\mathbf{W}$ with $n$ $d$-dimensional vectors as the input, and
outputs another feature matrix $\mathbf{Y}$ with the same number of
$d^{\prime}$-dimensional vectors.

###  B Multi-Objective Learning

Multi-objective learning (MTL) [34] aims to handle multiple different tasks
with a single neural network, in which parameters are divided into shared ones
and independent ones [35]. Compared to conventional single-task learning, MTL
does better in reducing the amount of parameters by efficiently leveraging
common knowledge and similarities among associated tasks with shared layers.
For example, MTL-enhanced feature extraction layers inside a CNN are
compatible with various computer vision tasks such as classification,
detection and segmentation [36]. Furthermore, inductive biases from multiple
tasks during training enhance the robustness of the model in disguise, making
it less susceptible to noise samples. The generality of MTL makes it suitable
for introducing auxiliary tasks to improve the accuracy of the main task,
e.g., classifying gender and pose for facial landmark detection [37],
discriminating speakers for voice trigger detection [38]. In addition to
designing model structures to support multiple inputs and outputs, it is also
crucial for MTL to consider how to balance loss functions from different
tasks. In this paper, a single-input-multi-output MTL framework with hard
parameter sharing is adopted to determine the loss weights of both band
selection and classification tasks automatically.

###  C Meta-Learning

Since Donald Maudsley coined the meta-learning in 1979 [39] to describe
internalized perception, inquiry, learning and growth, it has increasingly
attracted attentions and is considered as a golden key to achieve general
artificial intelligence. The idea behind meta-learning is the so-called
“learn-to-learn”, that is, taking advantages of knowledge and experiences from
a variety of existing tasks to acquire the learning ability and quickly
generalize to new tasks. Different from the well-known machine learning
pipeline, the minimum granularity of meta-learning is no longer a single
sample, but a complete task composed of multiple samples. Besides, instead of
a function that maps from images to labels, the goal of meta-learning is a
meta-function that generates functions for specific tasks. Regarded as a sub-
field of transfer learning, meta-learning transfers knowledge from the source
domain of seen tasks to the target domain of unseen tasks. Existing approaches
towards meta-learning can be roughly divided into three categories: learning
weight initializations, training meta-models that generates model parameters,
as well as designing transferable optimizers [40]. Promising applications of
meta-learning have been presented in various areas spanning few-shot learning
[41], meta reinforcement learning [42] and neural architecture search [43]. In
this paper, zero-shot optimization for band selection is formalized as a two-
stage meta-learning procedure.

Figure 2: The first graph convolutional layer inside the generalizable GCN,
comprising consecutive matrix multiplications. The weight matrix is
parameterized as dataset-agnostic bases and dataset-specific coefficients.
These bases are shared by different datasets, while these coefficients, which
are specific to each dataset, are computed by a simple MLP network.

##  III Methodology

The overall architecture of the proposed M3BS for zero-shot hyperspectral band
selection is illustrated in Fig. 1. Specifically, the generalizable GCN with
spatial-spectral graphs is constructed for dataset-agnostic band selection in
Sec. A, followed by subsequent classification in Sec. B. In Sec. C, auxiliary
supervision from multiple band selection teachers is introduced with a
diversity ensemble strategy. Sec. E focuses on zero-shot optimization, where
all the modules are integrated into an unified uncertainty-based multi-
objective meta-learning framework to jointly optimize band selection and
classification tasks.

###  A Spatial-Spectral Generalizable GCN for Band Selection

Aiming at zero-shot band selection, we construct a generalizable GCN with
spatial-spectral graphs to calculate an importance score for each band. It is
served as a dataset-invariant meta-knowledge extractor to decomposite the
parameters to learnable dataset-agnostic bases and dataset-specific
coefficients. Thus, it can synchronize inconsistent representations among
different datasets, and finally learn “how to perform band selection” and
acheive zero-shot inference on any new dataset.

####  A1 Spatial-Spectral Graph

In contrast to the traditional grid structure, the graph structure can capture
complex and irregular associations among individuals, which makes it
particularly eligible for modeling correlation among spectral bands in HSIs.
In this way, the bands and their spatial and spectral relationships are
regarded as vertices and edges, respectively.

Suppose there are ${n^{\text{ds}}}$ different HSI datasets
$\\{\mathcal{D}_{1},\dots,\mathcal{D}_{n^{\text{ds}}}\\}$, each of which
consists of a different number of bands ${n^{\text{band}}}$ but shares a
common patch size $0pt\times 0pt$. As for the $k$-th dataset
$\mathcal{D}_{k}$, the proposed spatial-spectral band graph is represented as
$\mathcal{G}=\\{\mathcal{V},\mathcal{E},\mathbf{A}\\}$. $\mathcal{V}$ is the
set of vertices corresponding to spectral bands of HSIs, and
$v_{i}\in\mathcal{V}$ denotes the $i$-th spectral band. $\mathcal{E}$ is the
set of edges indicating the relationships among all the spectral bands, and
$e_{i,j}=(v_{i},v_{j})\in\mathcal{E}$ denotes the correlation between the
$i$-th and $j$-th bands. By flattening all the pixels in a single patch, the
resulting feature vector $\mathbf{x}_{i}\in\mathbb{R}^{0pt0pt}$ corresponds to
the $i$-th band $v_{i}$, and the feature matrix
$\mathbf{X}\in\mathbb{R}^{{n^{\text{band}}}\times 0pt0pt}$ is formed by
stacking together all these feature vectors. As illustrated in Fig. 2, the
“blue” row of the feature matrix corresponds to the “blue” spectral band, the
“green” row corresponds to the “green” spectral band, and so on.

The adjacency matrix, denoted as
$\mathbf{A}\in\mathbb{R}^{{n^{\text{band}}}\times{n^{\text{band}}}}$ is
utilized for describing the similarity between two vertices. We exploit both
spatial and spectral information with a threshold for truncation
$\theta^{\text{adj}}$ to calculate the adjacency matrix as follows:

$\mathbf{A}_{i,j}=\begin{cases}\mathbf{A}_{i,j}^{\text{spa}}+\mathbf{A}_{i,j}^{\text{spec}},&\text{if
}\mathbf{A}_{i,j}^{\text{spa}}+\mathbf{A}_{i,j}^{\text{spec}}\geq\theta^{\text{adj}}\\\
0,&\text{otherwise}\end{cases}$ (5)
$\mathbf{A}_{i,j}^{\text{spa}}=\begin{cases}\exp\left(-\cfrac{1}{{n^{\text{band}}}}|i-j|\right),&\text{if
}i\neq j\\\ 0,&\text{otherwise}\end{cases}$ (6)
$\mathbf{A}_{i,j}^{\text{spec}}=\begin{cases}\exp\left(-\cfrac{1}{0pt0pt}\|\mathbf{x}_{i}-\mathbf{x}_{j}\|_{2}\right),&\text{if
}i\neq j\\\ 0,&\text{otherwise}\end{cases}$ (7)

where $\mathbf{A}^{\text{spa}}$ and $\mathbf{A}^{\text{spec}}$ are critical to
characterizing the similarity among spectral bands from different aspects.
Concretely, $\mathbf{A}_{i,j}^{\text{spa}}$ indicates the $1$-dimensional
spatial distance between the $i$-th and $j$-th bands. The closer the two bands
are, the closer the value is to $1$. The spatial distance reflects the fact
that the intensity of adjacent bands is tend to be close due to their similar
light reflectance. $\mathbf{A}_{i,j}^{\text{spec}}$ indicates the $l^{2}$-norm
of the difference of the feature vector $\mathbf{x}_{i}$ and $\mathbf{x}_{j}$,
in the sense of the $0pt0pt$-dimensional Euclidean distance. As shown in Eq.
(6) and (7), the Gaussian kernel function is utilized to constrain the numeric
range of the two similarity matrices $\mathbf{A}^{\text{spa}}$ and
$\mathbf{A}^{\text{spec}}$ between $0$ and $1$. Besides, as introduced in
Section II, we still apply renormalization to the adjacency matrix
$\widetilde{\mathbf{A}}=\mathbf{A}+\mathbf{I}$, thereby enhancing stability in
the training process.

####  A2 Dataset-Agnostic Generalizable GCN

Meta-knowledge can be understood as generic knowledge acquired from continuous
adaptation to different tasks. For meta-knowledge extraction, we propose a
more effective representation of learnable parameters, which decomposites
these parameters into dataset-agnostic bases and dataset-specific
coefficients. These dataset-agnostic bases act as unified meta-knowledge to
synchronize inconsistent representations among different datasets, while
dataset-specific coefficients can effectively deal with unseen samples due to
their dynamically generated characteristics.

Based on the abstraction of the spatial-spectral band graph, a spatial-
spectral generalizable GCN is constructed for band selection in M3BS,
comprising two graph convolutional layers with batch normalization (BN) as
shown in Eq. (8) and (9):

$\hat{\mathbf{H}}=\operatorname{ReLU}(\operatorname{BN}(\widetilde{\mathbf{D}}^{-\frac{1}{2}}\widetilde{\mathbf{A}}\widetilde{\mathbf{D}}^{-\frac{1}{2}}\mathbf{X}\mathbf{W}^{\text{GCN(1)}}))$
(8)
$\hat{\mathbf{s}}=\operatorname{\sigma}(\operatorname{BN}(\widetilde{\mathbf{D}}^{-\frac{1}{2}}\widetilde{\mathbf{A}}\widetilde{\mathbf{D}}^{-\frac{1}{2}}\hat{\mathbf{H}}\mathbf{W}^{\text{GCN(2)}}))$
(9)

where $\widetilde{\mathbf{A}}$ and $\widetilde{\mathbf{D}}$ are the
renormalized forms of the adjacency matrix and the degree matrix respectively,
$\hat{\mathbf{H}}\in\mathbb{R}^{{n^{\text{band}}}\times 256}$ denotes the
output feature matrix of the first graph convolutional layer,
$\hat{\mathbf{s}}\in[0,1]^{{n^{\text{band}}}}$ indicates the importance score
of each band for determining which subset should be selected,
$\mathbf{W}^{\text{GCN(1)}}\in\mathbb{R}^{0pt0pt\times
256},\mathbf{W}^{\text{GCN(2)}}\in\mathbb{R}^{256}$ are learnable parameters,
which will be updated by gradient descent during training. The output
dimensionality of each vertex in the first layer is $256$. The second layer is
normalized by a Sigmoid function $\operatorname{\sigma}(\cdot)$ to ensure that
the score of each band falls within $[0,1]$.

Instread of static forward propagation, we parameterize the above-mentioned
graph convolutional kernels $\mathbf{W}^{\text{GCN(1)}}$ and
$\mathbf{W}^{\text{GCN(2)}}$ as linear combinations of ${n^{\text{base}}}$
bases:

$\mathbf{W}^{\text{GCN(:)}}=\alpha_{1}\mathbf{W}_{1}^{\text{GCN(:)}}+\dots+\alpha_{n^{\text{base}}}\mathbf{W}_{n^{\text{base}}}^{\text{GCN(:)}}$
(10)

where
$\mathbf{W}_{1}^{\text{GCN(:)}}\dots\mathbf{W}_{n^{\text{base}}}^{\text{GCN(:)}}$
are relatively orthogonal bases decomposited from the original parameter,
$\alpha_{1}\cdots\alpha_{n^{\text{base}}}$ are linear combination coefficients
from an average pooling layer, a fully-connected layer and a sigmoid
activation function:

$\alpha_{1}\dots\alpha_{n^{\text{base}}}=\operatorname{\sigma}(\frac{1}{{n^{\text{band}}}}\mathbf{1}^{\top}\mathbf{X}\mathbf{W}^{\text{FC}})$
(11)

where
$\frac{1}{{n^{\text{band}}}}\mathbf{1}^{\top}\in\mathbb{R}^{1\times{n^{\text{band}}}}$
calculates the mean along each column of the feature matrix $\mathbf{X}$,
$\mathbf{W}^{\text{FC}}\in\mathbb{R}^{0pt0pt\times{n^{\text{base}}}}$ maps the
pooled feature matrix into ${n^{\text{base}}}$ scalars corresponding to
${n^{\text{base}}}$ parameter bases. Meta-knowledge is explicitly represented
as dataset-agnostic bases, while dataset-specific coefficients can be used to
characterize unique attributes of different datasets.

According to the extracted score vector $\hat{\mathbf{s}}$ for all the bands,
it is available to determine which subset should be selected. Generally, the
number of selected bands is fixed as a hyperparameter $n^{\text{sband}}$ by
users in advance ($n^{\text{sband}}<{n^{\text{band}}}$) [19]. To make it
available to attach the following classification network, a binarization
operator is devised to convert scores into a band mask:

$\theta^{\text{band}}=\left\\{\hat{\mathbf{s}}_{o_{1}},\hat{\mathbf{s}}_{o_{2}},\cdots,\hat{\mathbf{s}}_{o_{n^{\text{band}}}}\right\\}_{n^{\text{sband}}}$
(12)
$\hat{\mathbf{M}}=\phi(\hat{\mathbf{s}},\theta^{\text{band}})=\begin{cases}0,&\text{if
}\hat{\mathbf{s}}_{i}\geq\theta^{\text{band}}\\\
1,&\text{otherwise}\end{cases}$ (13)

Specifically, the scores of all the bands are firstly sorted in the descending
order $o_{1},o_{2},\cdots,o_{{n^{\text{band}}}}$, and the threshold
$\theta^{\text{band}}$ is taken to be the $n^{\text{sband}}$-th score. Then, a
binarization operator $\phi(\cdot)$ is applied to extract a band mask
$\hat{\mathbf{M}}\in\\{0,1\\}^{{n^{\text{band}}}}$ by providing a conditional
comparative constraint, where the corresponding value is set to $0$ when the
score is smaller than or equals to the threshold. Finally, this band mask is
multiplied to the HSIs as the input of the classifier.

###  B CNN-based Image Classifier

TABLE I: The detailed structure of the CNN-based image classifier. Stage | Operator | Resolution | #Channels
---|---|---|---
$i$ | $\hat{\mathcal{F}_{i}}$ | $\hat{0pt}_{i}\times\hat{0pt}_{i}$ | $\hat{C}_{i}$
0 | - | $33\times 33$ | $n^{\text{sband}}$
1 | ConvBNReLU, 5$\times$5 w/ pad=2 | $33\times 33$ | $64$
1 | MaxPool, 2$\times$2 | $16\times 16$ | $64$
2 | ConvBNReLU, 5$\times$5 w/ pad=2 | $16\times 16$ | $128$
2 | MaxPool, 2$\times$2 | $8\times 8$ | $128$
3 | ConvBNReLU, 5$\times$5 w/ pad=2 | $8\times 8$ | $256$
3 | MaxPool, 2$\times$2 | $4\times 4$ | $256$
4 | ConvBNReLU, 5$\times$5 w/ pad=2 | $4\times 4$ | $512$
4 | MaxPool, 2$\times$2 | $2\times 2$ | $512$
5 | ConvBNReLU, 5$\times$5 w/ pad=2 | $2\times 2$ | $1024$
5 | MaxPool, 2$\times$2 | $1\times 1$ | $1024$
6 | Flatten | - | $1024$
6 | Dropout & FC & Softmax | - | $n^{\text{class}}$

To embed an auxiliary classification network to evaluate band combinations, we
construct a 2-dimensional CNN to extract spatial features and further output a
probability distribution for classification.

Suppose that there are $n^{\text{class}}$ categories in the $k$-th HSI
dataset. The detailed network structure of the CNN for image classification is
illustrated in Table I, consisting of five stages of spatial convolutions and
one stage of dense connection. With the hyperspectral image patch
$x\in\mathbb{R}^{{n^{\text{band}}}\times 0pt\times 0pt}$ as the input, the
probability distribution $\hat{\mathbf{y}}\in[0,1]^{n^{\text{class}}}$ can be
predicted by the $k$-th CNN with parameters $\mathbf{W}^{\text{CNN}}_{k}$.

Following the settings of embedding-based band selection methods, the
classification loss for a single batch is defined as a multi-class cross-
entropy loss as shown in Eq. (14). $y_{i}\in\\{1,2,\cdots,n^{\text{class}}\\}$
denotes which category the center pixel of the $i$-th patch actually belongs
to, and $\mathbf{1}\\{\cdot\\}$ is a vectorized indicator function to perform
one-hot encoding over scalars.

$\mathcal{L}^{\text{cls}}=-\frac{1}{n^{\text{batch}}}\sum\limits_{i=1}^{n^{\text{batch}}}\mathbf{1}\\{y_{i}\\}\log\hat{\mathbf{y}}_{i}$
(14)

###  C Multi-Teacher Diversity Ensemble

As a typical preprocessing procedure for downstream tasks, band selection does
not involve any explicit ground truths like other fully supervised tasks, such
as image classification. Thus, it is usually formalized as an unsupervised
task [21] or evaluated by the performance of an auxiliary classifier [9].
These band selection methods either produce poor performance or consume
numerous training time. Moreover, they fail to generalize to unseen samples,
which makes it challenging to acheive zero-shot hyperspectral band selection.

In addition to the classification loss in Sec. B, we exploit diverse high-
quality experiences from multiple band selection teachers, generating an
auxiliary loss function. This extra supervision not only facilitates
generalization and adaptability for the previous meta-knowledge, but also
accelerates the convergence procedure and alleviates the training time
consumption.

Specifically, multiple representative and reliable teachers from filter-based,
wrapper-based and embedding-based methods respectively, are pretrained to
prepare for high-quality experiences in advance. Considering how to make a
balance among multiple teachers, we formulate a diversity ensemble strategy to
choose the spectral bands according to their popularity. After the band
subsets are selected by these teachers respectively, a counting function
$\text{cnt}(i)$ can be defined to denote the number of received votes of the
$i$-th band. As stated in Equation 15 and 16, we can obtain a band sequence
and take the top $n^{\text{sband}}$ bands as the ground truth by sorting these
bands by the number of received votes in the descending order:

$\mathbb{S}=\left\\{\underbrace{i_{1},\dots}_{\text{cnt}(i)=3}\underbrace{i_{7},i_{8},\dots\dots}_{\text{cnt}(i)=2}\underbrace{i_{22},i_{23},\dots\dots\dots}_{\text{cnt}(i)=1}\right\\}_{1,\dots,n^{\text{sband}}}$
(15) $\mathbf{s}_{i}=\begin{cases}1,&\text{if }i\in\mathbb{S}\\\
0,&\text{otherwise}\end{cases}$ (16)

where $\mathbf{s}$ denotes the label for directly supervising the band
selection task. For the sake of fairness, for those bands with the same number
of votes, we randomly pick a subset with an appropriate number of bands. With
this ensemble strategy, multiple teachers can be integrated into a more
powerful teacher. Thanks to the various optimizing directions from different
teachers, the lack of generalization can be effectively alleviated, and this
kind of diversity is beneficial to generalizing better to unseen samples in
zero-shot learning.

The band selection loss for the entire batch is defined as a multi-label
binary cross-entropy loss of the continuous importance scores
$\hat{\mathbf{s}}\in[0,1]^{{n^{\text{band}}}}$ from the GCN and the discrete
selected bands $\mathbf{s}\in\\{0,1\\}^{{n^{\text{band}}}}$ from the
integrated teacher:

$\mathcal{L}^{\text{bs}}=\mathbf{s}\log\hat{\mathbf{s}}+(\mathbf{1}-\mathbf{s})\log(\mathbf{1}-\hat{\mathbf{s}})$
(17)

###  D Uncertainty-based Multi-Objective Learning

Both the classification loss in Sec. B and the selection loss in Sec. C are
beneficial for the expected band selection task. It inspires us to
simultaneously optimize them with an integrated loss function. Intuitively,
combining multiple losses could be simply accomplished by applying a weighted
sum and constituting the overall loss function $\mathcal{L}$ as follows:

$\mathcal{L}=\lambda^{\text{bs}}\mathcal{L}^{\text{bs}}+\lambda^{\text{cls}}\mathcal{L}^{\text{cls}}$
(18)

where $\lambda^{\text{bs}}\in[0,1]$ and $\lambda^{\text{cls}}\in[0,1]$ are
relative weights for the band selection loss and the classification loss
respectively. In ${n^{\text{ds}}}$ different HSI datasets, there are
$2{n^{\text{ds}}}$ extra hyperparameters needed to be tuned and denoted as
$\\{\lambda_{1}^{\text{cls}},\dots,\lambda_{n^{\text{ds}}}^{\text{cls}},\lambda_{1}^{\text{bs}},\dots,\lambda_{n^{\text{ds}}}^{\text{bs}}\\}$.
Acutually, the performance of the band selection task is heavily sensitive to
the propotion of these weights [44]. However, it is difficult to tune these
hyperparameters manually.

It is more ideal to allow the loss weights to be updated together with network
parameters during the training process. Inspried by [45], a multi-objective
learning approach is proposed, which weighs multiple loss functions by
considering the uncertainty of each task. [45] stated that the accidential
error between model outputs and sample labels can be modeled as a
homoscedastic uncertainty, which allows us to decompose the multi-objective
loss into the product of multiple likelihoods. By estimating these two tasks
with a Sigmoid likelihood $\operatorname{\sigma}(\cdot)$ and a Softmax
likelihood $\operatorname{Softmax}(\cdot)$ respectively, an uncertainty-based
minimization objective can be defined as follows:

$\displaystyle\mathcal{L}$ $\displaystyle=-\log\sigma(\hat{\mathbf{s}}_{k};\
\mathbf{W}^{\text{GCN}},\lambda^{\text{bs}})$
$\displaystyle\qquad\qquad\cdot\operatorname{Softmax}(\hat{\mathbf{y}}_{k};\
\mathbf{W}^{\text{GCN}},\mathbf{W}_{k}^{\text{CNN}},\lambda^{\text{cls}})$
$\displaystyle\propto\lambda^{\text{bs}}\mathcal{L}^{\text{bs}}+\lambda^{\text{cls}}\mathcal{L}^{\text{cls}}+\log\sqrt{\frac{1}{\lambda^{\text{bs}}}}+\log\sqrt{\frac{1}{\lambda^{\text{cls}}}}$
(19)

where $\lambda^{\text{bs}}$ and $\lambda^{\text{cls}}$ are two weights
corresponding to the above-mentioned two objectives. As $\lambda^{\text{bs}}$
or $\lambda^{\text{cls}}$ decreases, $1/(\lambda^{\text{bs}})^{2}$ or
$1/(\lambda^{\text{cls}})^{2}$ increases, which brings a greater influence to
the corresponding loss function. These relative weights are also discouraged
from decreasing excessively by the last regularization term
$\log\sqrt{1/{\lambda^{\text{bs}}}}$ and
$\log\sqrt{1/{\lambda^{\text{cls}}}}$.

By applying Eq. (D) rather than Eq. (18) to optimize multiple objectives,
massive hyperparameter tuning costs can be avoided since the weights are
determined automatically by gradient descent without any manual intervention,
and different training stages can be adapted since the weights are updated
dynamically as training progresses.

###  E Multi-Objective Meta-Learning for Zero-Shot Optimization

Figure 3: The multi-objective multi-objective meta-learning procedure for
zero-shot optimization, where different colors correspond to different HSI
datasets.

When it comes to zero-shot inference, we try to figure out how to perform
hyperspectral band selection on a new dataset without any of its samples
during training. To achieve zero-shot optimization, we construct a multi-
objective meta-learning procedure to formalize an intuitive trial-and-error
process. It makes our framework not only suitable for both the band selection
and classification objectives, but also applicable to various datasets with
different characteristics, thereby ensuring maximum compatibility.

Following how previous works preprocess data for meta-learning [41, 42, 43],
we divide HSI datasets into meta-training and meta-testing tasks. Our goal is
to acheive meta-knowledge extraction from meta-training tasks and apply them
to meta-testing tasks. Specifically, let HSI datasets
$\mathcal{D}_{1}^{\text{train}},\dots,\mathcal{D}_{n^{\text{ds}}}^{\text{train}}$
be different meta-training tasks for training, $\mathcal{D}^{\text{test}}$ be
the meta-testing task for zero-shot inference. What we want to achieve is to
directly inference on $\mathcal{D}^{\text{test}}$ after training on
$\mathcal{D}_{1}^{\text{train}},\dots,\mathcal{D}_{n^{\text{ds}}}^{\text{train}}$,
and none of samples in $\mathcal{D}^{\text{test}}$ are provided during the
training phrase. Besides, each meta-training task (i.e. dataset)
$\mathcal{D}_{k}^{\text{train}}$ is further divided into a support set
$\mathcal{D}_{k}^{\text{spt}}$ and a query set $\mathcal{D}_{k}^{\text{qry}}$
according to a predefined percentage.

The overall procedure of M3BS is described elaborately in Alg. 1. Inspired by
model-agnostic meta-learning [40], the whole meta-training process can be
viewed as a trial-and-error process in Fig. 3, where the support stage makes
an attempt and the query stage verifies the effectiveness of this attempt.
Significantly different from [40], instead of applying the same loss function
for both stages, we optimize the multi-objective loss in Sec. D in the query
stage, thereby utilizing the stronger supervision from classification to more
effectively verify whether the attempt made in the support stage is
reasonable. In the support stage, only the band selection loss
$\mathcal{L}^{\text{bs}}$ is involved for a tentative update step for the
temporary parameters $\mathbf{W}^{\text{GCN}}_{k}$. In the query stage, the
multi-objective loss $\mathcal{L}$ is used to verify whether the previous step
is beneficial for classification or not. For verification, only the
derivatives of the loss with respect to the parameters
$\partial{\mathcal{L}}/\partial{\mathbf{W}^{\text{GCN}}_{k}}$ are calculated,
and the parameters $\mathbf{W}^{\text{GCN}}_{k}$ are not actually updated.
After iterating over all ${n^{\text{ds}}}$ meta-training tasks, all these
derivatives are translated to the original parameters
$\mathbf{W}^{\text{GCN}}|_{t}$ for gradient descent, leading to a more
generalizable version $\mathbf{W}^{\text{GCN}}|_{t+1}$.

During meta-testing, discriminative and informative band subsets can be
obtained by applying the compatible meta-knowledge extracted by the
generalizable GCN to perform zero-shot inference.

Algorithm 1 M3BS

0:
$\mathcal{D}_{1}^{\text{train}},\dots,\mathcal{D}_{n^{\text{ds}}}^{\text{train}},\mathcal{D}^{\text{test}}$:
meta-training and meta-testing tasks; $\alpha$: learning rate; $\beta$: meta-
learning rate; $T$: number of epochs.

0:
$\mathbf{W}^{\text{GCN(:)}},\mathbf{W}^{\text{CNN}}_{1},\dots,\mathbf{W}^{\text{CNN}}_{n^{\text{ds}}}$:
well-train parameters.

1: Initialize relative weights: $\lambda^{\text{bs}}|_{t=1}\leftarrow
1,\lambda^{\text{cls}}|_{t=1}\leftarrow 1$

2: for epoch $t=1,\dots,T$ do

3: for each meta-training task $\mathcal{D}^{\text{train}}_{k}$ in
$\\{\mathcal{D}^{\text{train}}_{1},\dots,\mathcal{D}^{\text{train}}_{n^{\text{ds}}}\\}$
do

4: Split samples for support and query:
$\mathcal{D}_{k}\rightarrow\\{\mathcal{D}^{\text{spt}}_{k},\mathcal{D}^{\text{qry}}_{k}\\}$

5: Temporarily duplicate parameters:
$\mathbf{W}^{\text{GCN}}_{k}\leftarrow\mathbf{W}^{\text{GCN}}$

6: for each batch in $\mathcal{D}^{\text{spt}}_{k}$ do

7: Flatten all pixels into a feature matrix $\mathbf{X}$

8: Characterize an adjacency matrix $\mathbf{A}$ by (5)-(7)

9: Build a spatial-spectral graph
$\mathcal{G}=\\{\mathcal{V},\mathcal{E},\mathbf{A}\\}$

10: Assign importance scores $\hat{\mathbf{s}}$ with $\mathbf{W}^{\text{GCN}}$
by (8)-(11)

11: Vote a band subset $\mathbf{s}$ from teachers by (15)-(16)

12: Measure the band selection loss $\mathcal{L}^{\text{bs}}$ by (17)

13: Backpropagate gradients to obtain
$\partial{\mathcal{L}^{\text{bs}}}/\partial{\mathbf{W}_{k}^{\text{GCN}}}$

14: Update $\mathbf{W}^{\text{GCN}}_{k}$ with gradient
descent:$\mathbf{W}^{\text{GCN}}_{k}|_{t+1}=\mathbf{W}^{\text{GCN}}_{k}|_{t}+\alpha\cdot\partial{\mathcal{L}^{\text{bs}}}/\partial{\mathbf{W}^{\text{GCN}}_{k}|_{t}}$

15: end for

16: for each batch in $\mathcal{D}^{\text{qry}}_{k}$ do

17: Obtain $\hat{\mathbf{s}}$ and $\mathcal{L}^{\text{bs}}$ by following the
same steps as above

18: Binarize $\hat{\mathbf{s}}$ into a band mask $\hat{\mathbf{M}}$ by
(12)-(13)

19: for each patch $x_{i}$ and category $y_{i}$ in this batch do

20: Mask $x_{i}$ with $\hat{\mathbf{M}}$ by multiplication: $x_{i}\leftarrow
x_{i}\cdot\hat{\mathbf{M}}$

21: Classify $x_{i}$ into a probability vector $\hat{\mathbf{y}}_{i}$ with
$\mathbf{W}^{\text{CNN}}_{k}$

22: end for

23: Measure the classification loss $\mathcal{L}^{\text{cls}}$ by (14)

24: Measure the multi-objective loss $\mathcal{L}$ by (D)

25: Backpropagate gradients to
obtain$\partial{\mathcal{L}}/\partial\lambda^{\text{bs}}$,
$\partial{\mathcal{L}}/\partial\lambda^{\text{cls}}$,
$\partial{\mathcal{L}}/\partial{\mathbf{W}^{\text{GCN}}_{k}}$ and
$\partial{\mathcal{L}}/\partial{\mathbf{W}^{\text{CNN}}_{k}}$

26: Accumulate $\partial{\mathcal{L}}/\partial{\mathbf{W}^{\text{GCN}}_{k}}$
for $\mathbf{W}_{k}^{\text{GCN}}$

27: Update $\lambda^{\text{bs}}$, $\lambda^{\text{cls}}$ and
$\mathbf{W}_{k}^{\text{CNN}}$ with gradient
descent:$\lambda^{\text{bs}}|_{t+1}=\lambda^{\text{bs}}|_{t}+\alpha\cdot\partial{\mathcal{L}}/\partial{\lambda^{\text{bs}}|_{t}}$$\lambda^{\text{cls}}|_{t+1}=\lambda^{\text{cls}}|_{t}+\alpha\cdot\partial{\mathcal{L}}/\partial{\lambda^{\text{cls}}|_{t}}$$\mathbf{W}^{\text{CNN}}_{k}|_{t+1}=\mathbf{W}^{\text{CNN}}_{k}|_{t}+\alpha\cdot\partial{\mathcal{L}}/\partial{\mathbf{W}^{\text{CNN}}_{k}|_{t}}$

28: end for

29: Average $\partial{\mathcal{L}}/\partial{\mathbf{W}^{\text{GCN}}_{k}}$ over
all batches for $\mathbf{W}_{k}^{\text{GCN}}$

30: Summarize $\partial{\mathcal{L}}/\partial{\mathbf{W}^{\text{GCN}}_{k}}$ to
$\partial{\mathcal{L}}/\partial{\mathbf{W}^{\text{GCN}}}$ for
$\mathbf{W}^{\text{GCN}}$:$\partial{\mathcal{L}}/\partial{\mathbf{W}^{\text{GCN}}}\leftarrow\partial{\mathcal{L}}/\partial{\mathbf{W}^{\text{GCN}}}+\partial{\mathcal{L}}/\partial{\mathbf{W}^{\text{GCN}}_{k}}$

31: end for

32: Update $\mathbf{W}^{\text{GCN}}$ with gradient descent for meta-
learning:$\mathbf{W}^{\text{GCN}}|_{t+1}=\mathbf{W}^{\text{GCN}}|_{t}+\beta\cdot\partial{\mathcal{L}}/\partial{\mathbf{W}^{\text{GCN}}}$

33: end for

##  IV Experimental Results and Analysis

In this section, quantitative statistical experiments and qualitative
theoretical analysis are conducted over three commonly used hyperspectral
datasets to validate the effectiveness and efficiency of our method. Detailed
descriptions of these datasets and experimental settings are provided in Sec.
A and Sec. B, respectively.

###  A Dataset Description

TABLE II: Categories and Respective Numbers of Samples Dataset | No. | Category | #Samples
---|---|---|---
Indian Pines | 1 | Alfalfa | 46
2 | Corn-notill | 1428
3 | Corn-mintill | 830
4 | Corn | 237
5 | Grass-pasture | 483
6 | Grass-trees | 730
7 | Grass-pasture-mowed | 28
8 | Hay-windrowed | 478
9 | Oats | 20
10 | Soybean-notill | 972
11 | Soybean-mintill | 2455
12 | Soybean-clean | 593
13 | Wheat | 205
14 | Woods | 1265
15 | Buildings-Grass-Trees-Drives | 386
16 | Stone-Steel-Towers | 93
total | 10249
Pavia University | 1 | Asphalt | 6631
2 | Meadows | 18649
3 | Gravel | 2099
4 | Trees | 3064
5 | Painted metal sheets | 1345
6 | Bare Soil | 5029
7 | Bitumen | 1330
8 | Self-Blocking Bricks | 3682
9 | Shadows | 947
total | 42776
University of Houston | 1 | Healthy grass | 1251
2 | Stressed grass | 1254
3 | Synthetic grass | 697
4 | Trees | 1244
5 | Soil | 1242
6 | Water | 325
7 | Residential | 1268
8 | Commercial | 1244
9 | Road | 1252
10 | Highway | 1227
11 | Railway | 1235
12 | Parking Lot 1 | 1233
13 | Parking Lot 2 | 469
14 | Tennis Court | 428
15 | Running Track | 660
total | 15029

(a)

(b)

(c)

Figure 4: The false-color images composited by three different spectral bands
from (a) Indian Pines, (b) Pavia University, and (c) University of Houston.

Our crafted experiments involve a total of 6 hyperspectral datasets, namely
Indian Pines, Pavia University, University of Houston, Salinas, Washington DC
Mall and Kennedy Space Center. Following the same practice of previous works,
all samples are randomly divided into a training set and a testing set at a
ratio of 5:95 for Indian Pines and University of Houston, and a ratio of 3:97
for Pavia University. All training sets are further subdivided into support
sets and query sets for meta-learning optimization at a unified ratio of 3:7,
which guarantees that all categories are present during training.

####  A1 Indian Pines

Being a publicly recognized hyperspectral band selection dataset, Indian Pines
was collected in 1992 by the Airborne Visual InfraRed Imaging Spectrometer
(AVIRIS) over a landscape in Indiana, USA. Technically, this dataset consists
of a hyperspectral image with 145$\times$145 pixels.

With a wavelength range of 0.4-2.5 $\mathrm{\SIUnitSymbolMicro m}$, the
airborne visible infrared imager captures a total of 224 spectral reflectance
bands, and discards 24 bands that cover the region of water absorption.

16 categories are involved as in Table II, and a false-color image composited
by the bands 50, 27 and 17 is shown in Fig. 4 (a).

####  A2 Pavia University

As a partial scene of Pavia, northern Italy, this dataset was acquired by the
German Airborne Reflective Optical Spectral Imager Hyperspectral Data Spectral
Imager (ROSIS) in 2003. It contains 115 consecutive bands, 12 bands of which
are previously removed due to noise interference. The hyperspectral image has
a spatial resolution of 610$\times$340 and contains a total of 2,207,400
pixels, including a large number of background pixels. 9 land cover categories
are differenciated for classification as in Table II, and Fig. 4 (b) shows the
extracted bands 53, 31 and 8.

####  A3 University of Houston

This dataset was obtained by the ITRES CASI-1500 sensor over a campus and its
neighboring urban area and served as a contest for the 2013 IEEE GRSS Data
Fusion Competition. It is a hyperspectral image of size 349$\times$1905 and
contains 144 bands in the spectral range from 364 $\mathrm{nm}$ to 1046
$\mathrm{nm}$. Table II makes a list of all 15 ground object categories and
their corresponding numbers of samples. Fig. 4 (c) displays a false-color
image with pixel-level ground truths composed of the bands 28, 45, and 65.

####  A4 Salinas

This scene was gathered by the AVIRIS sensor over Salinas Valley, California
with a high spatial resolution of 512$\times$217\. Same as Indian Pines
described before, only 204 spectral bands are preserved by discarding 20 bands
covering water absorption. This dataset incorporates 16 categories such as
grapes, stubble and celery.

####  A5 Washington DC Mall

This 191-band dataset was taken by the Hyperspectral Digital Imagery
Collection Experiment (HYDICE) sensor with a wavelength range of 0.4-2.4
$\mathrm{\SIUnitSymbolMicro m}$. The hyperspectral image contains
750$\times$307 pixels, each of which is classified into one of 9 categories
such as roof and grass.

####  A6 Kennedy Space Center

Generally abbreviated as KSC, this dataset was collected by the AVIRIS sensor
in 1996. The spatial resolution of this dataset is 512$\times$616\. The number
of spectral bands is also cut down from 224 to 176 by removing both water
absorption and low SNR bands. To achieve pixel-level classification, land
cover objects are grouped into 13 categories, e.g. scrub, slash pine and salt
marsh.

TABLE III: Classification Results of M3BS and 9 Other Algorithms on a Single Dataset Indian Pines Category |  mRMR [8] |  BS-Nets [21] |  GCSR-BS [25] |  DRLBS [23] |  DGLAnet [46] |  TSC [47] |  SICNN [16] |  ABCNN [20] |  BHCNN [19] | M3BS
---|---|---|---|---|---|---|---|---|---|---
OA (%) | 71.7$\pm$0.6 | 68.7$\pm$0.9 | 68.3$\pm$2.5 | 75.2$\pm$1.2 | 81.7$\pm$0.4 | 71.4$\pm$0.7 | 87.6$\pm$1.0 | 81.5$\pm$1.7 | 96.1$\pm$0.7 | 96.5$\pm$0.4
AA (%) | 63.1$\pm$3.4 | 59.9$\pm$1.8 | 74.6$\pm$1.1 | 67.4$\pm$1.6 | 79.9$\pm$1.0 | 67.3$\pm$3.2 | 82.7$\pm$2.6 | 74.2$\pm$2.0 | 92.2$\pm$1.3 | 95.3$\pm$1.3
Kappa$\times$100 | 67.6$\pm$0.7 | 64.3$\pm$1.0 | 71.0$\pm$1.2 | 71.7$\pm$1.4 | 79.1$\pm$0.1 | 67.3$\pm$0.7 | 85.9$\pm$1.1 | 78.8$\pm$1.4 | 95.6$\pm$1.0 | 95.9$\pm$0.3
TABLE IV: Classification Results of M3BS and 9 Other Algorithms on a Single Dataset Pavia University Category |  mRMR [8] |  BS-Nets [21] |  GCSR-BS [25] |  DRLBS [23] |  DGLAnet [46] |  TSC [47] |  SICNN [16] |  ABCNN [20] |  BHCNN [19] | M3BS
---|---|---|---|---|---|---|---|---|---|---
OA (%) | 80.5$\pm$0.5 | 79.8$\pm$0.9 | 82.6$\pm$0.5 | 88.4$\pm$0.4 | 91.9$\pm$0.2 | 83.9$\pm$0.3 | 95.3$\pm$0.9 | 91.3$\pm$0.1 | 99.0$\pm$0.2 | 99.5$\pm$0.2
AA (%) | 73.8$\pm$0.3 | 71.6$\pm$2.0 | 86.6$\pm$0.4 | 85.6$\pm$0.6 | 89.3$\pm$0.5 | 80.8$\pm$0.7 | 93.1$\pm$0.7 | 88.1$\pm$0.4 | 98.6$\pm$0.3 | 99.1$\pm$0.1
Kappa$\times$100 | 73.5$\pm$0.6 | 72.2$\pm$1.4 | 82.1$\pm$0.6 | 84.4$\pm$0.6 | 89.3$\pm$0.1 | 78.5$\pm$0.4 | 93.7$\pm$0.7 | 88.5$\pm$1.3 | 98.7$\pm$0.2 | 99.3$\pm$0.2
TABLE V: Classification Results of M3BS and 9 Other Algorithms on a Single Dataset University of Houston Category |  mRMR [8] |  BS-Nets [21] |  GCSR-BS [25] |  DRLBS [23] |  DGLAnet [46] |  TSC [47] |  SICNN [16] |  ABCNN [20] |  BHCNN [19] | M3BS
---|---|---|---|---|---|---|---|---|---|---
OA(%) | 82.2$\pm$0.2 | 83.3$\pm$0.5 | 88.1$\pm$0.5 | 89.7$\pm$0.6 | 91.4$\pm$0.3 | 83.8$\pm$0.3 | 93.8$\pm$0.7 | 91.6$\pm$0.4 | 96.5$\pm$0.5 | 96.7$\pm$0.4
AA(%) | 82.0$\pm$0.2 | 82.3$\pm$0.7 | 87.1$\pm$0.5 | 88.6$\pm$0.7 | 90.3$\pm$0.6 | 83.6$\pm$0.2 | 92.7$\pm$1.7 | 91.2$\pm$0.9 | 96.4$\pm$0.6 | 96.8$\pm$0.3
Kappa$\times$100 | 80.8$\pm$0.3 | 82.0$\pm$0.6 | 87.1$\pm$0.5 | 88.9$\pm$0.7 | 90.8$\pm$0.4 | 82.9$\pm$0.3 | 93.4$\pm$0.9 | 90.9$\pm$0.8 | 96.2$\pm$0.5 | 96.5$\pm$0.4

###  B Experimental Setups

To more convincingly demonstrate the effectiveness of our algorithm in various
aspects, up to 9 existing band selection algorithms are chosen for performance
comparison, including mRMR [8], BS-Nets [21], GCSR-BS [25], DRLBS [23],
DGLAnet [46], TSC [47], SICNN [16], ABCNN [20] and BHCNN [19]. Among all these
methods, mRMR is a traditional filter-based method. TSC is a recently
published band selection algorithm based on subspace clustering. The rest of 9
comparison algorithms are either based on deep learning or partially assisted
with deep learning technologies. Among them, GCSR-BS and DRLBS introduce GCN
and deep reinforcement learning for a more robust representation of spectral
bands in HSIs. BS-Nets, DGLAnet, SICNN, ABCNN and BHCNN are 5 modern CNN-based
methods with strong spatial representation capacity.

The methods, mRMR, GCSR-BS and TSC, require additional classifiers. Here, a
SVM classifier is attached by grid searching $c$ and $\gamma$ in the range of
{1, 10, 100, 1000, 10000} and {0.01, 0.1, 1, 10, 100}, respectively. For BS-
Nets with a trainable CNN, the learning rate is fixed to 0.001, the number of
training epochs is set to 500, and the regularization coefficient is set to
0.02. For deep learning-based methods, BS-Nets, GCSR-BS, DRLBS, DGLAnet,
SICNN, ABCNN and BHCNN, the hyperparameters such as batch size, learning rate
and training epochs are determined either with a trial-and-error procedure or
by following the settings of the original paper.

For our M3BS, both the learning rate $\alpha$ for optimizing classification
and the meta-learning rate $\beta$ for optimizing band selection are set to
0.001, and further adopt an exponential scheduler to decay themselves every
epoch. Both the generalizable GCN and the CNN-based image classifiers are
optimized with a gradient-based stochastic algorithm Adam [48]. The batch size
$n^{\text{batch}}$ is set to 128, and the number of epochs $T$ is set to 400.
For any batch of any dataset, the spatial size of the hyperspectral image
patch is fixed to 33$\times$33\. By controlling the truncation threshold
$\theta^{\text{adj}}$ for the adjacency matrix $\mathbf{A}$, the number of
edges is limited to less than 1000 to prevent over-densification. The number
of dataset-agnostic bases for parameterizing the graph convolutional kernels
is set to 3. After extracting importance scores for each band, the number of
selected bands $n^{\text{sband}}$ is fixed to 20 for a fair comparison among
different datasets. From a diversity perspective, BS-Nets, SICNN and TWCNN,
which represents filter-based, wrapper-based and embedding-based methods
respectively, are chosen to be our teachers. For the sake of reproducibility,
detailed experimental procedures and well-commented implementations in PyTorch
[49]. All the experiments are conducted on a computer with a AMD Ryzen 5950X
CPU and a Nvidia GeForce RTX 3090 GPU.

###  C Classification Results on a Single Dataset

In this subsection, to preliminarily confirm the effectiveness of our network
structure, we verify the classification performance of all the comparison
algorithms. This scenario has nothing to do with zero-shot learning. Three
well-known classification metrics, named the overall accuracy (OA), the
average accuracy (AA), and the Kappa coefficient (Kappa), are used for
quantitative comparison of the classification performance among different
algorithms.

####  C1 Indian Pines

The dataset is randomly divided into 5% training set and 95% testing set, and
a subset consisting of 20 bands is selected from the original 200 spectral
bands. Table III records the classification results of each algorithm,
including the OA, AA and Kappa. The classification accuracy of each class is
available Table S1 in the Supplementary Material. The corresponding standard
deviations of each accuracy is given by running each algorithm 30 times
independently. Among all these results from 10 competitive algorithms, the
best ones are bolded and highlighted in gray. Among all these algorithms, with
an OA below 70%, BS-Nets and GCSR-BS have the worst classification
performance. The former formalizes band selection as an spectral
reconstruction task, and the latter does not utilize any label information for
classification. As band selection methods with auxiliary classifiers, mRMR,
DRLBS and TSC have slightly better results due to their reasonable
optimization strategies. However, for these algorithms, the classification is
acheived by an independent SVM and optimized separately from the band
selection task, which hinders mutual promotion between different tasks. By
integrating an attention module for richer spatial-spectral and global-local
features, DGLAnet receives nearly 10% performance improvement. Thanks to the
spatial representation and nonlinear fitting ability of CNNs, the CNN-based
algorithms, including SICNN, ABCNN, BHCNN and M3BS, are significantly better
than the previous ones. Among them, the accuracy of BHCNN is about 10% greater
than that of SICNN due to its bandwise-independent convolution layers, which
are more suitable for band selection. Finally, our M3BS acheives the best
accuracy on most categories and on all the three classification metrics.

####  C2 Pavia University

The dataset is splitted into 3% training samples and 97% testing samples. For
all competitive algorithms, 20 bands are selected from the original 103 bands.
Table IV demonstrates the quantitative comparison of classification ability
among M3BS and 9 other algorithms. The classification accuracy of each class
is available Table S2 in the Supplementary Material. Similar to the situation
in Indian Pines, end-to-end deep learning algorithms including SICNN, ABCNN,
BHCNN and M3BS, have an accuracy improvement of 5%-15% compared to traditional
algorithms such as mRMR and GCSR-BS. Meanwhile, among all deep learning
algorithms, M3BS achieves the best results on most of the 9 categories.
Compared with the representative CNN-based algorithm SICNN, M3BS gains an
improvement over the overall accuracy by 4.1%, the average accuracy by 5.7%,
and the Kappa coefficient by 5.2%. In addition, it is difficult for most
algorithms to correctly classify the three categories Gravel, Trees and Self-
Blocking Bricks, while M3BS improves these categories by at most 69.7%, 21.5%
and 22.4%, and at least 1.1%, 0.5% and 0.7% respectively.

TABLE VI: Zero-Shot Classification Results of M3BS and 9 Other Algorithms on the Unseen Dataset Indian Pines |  mRMR [8] |  BS-Nets [21] |  GCSR-BS [25] |  DRLBS [23] |  DGLAnet [46] |  TSC [47] |  SICNN [16] |  ABCNN [20] |  BHCNN [19] | M3BS
---|---|---|---|---|---|---|---|---|---|---
OA (%) | 70.8$\pm$0.8 | 74.0$\pm$0.3 | 70.5$\pm$1.4 | 72.6$\pm$1.2 | 78.6$\pm$0.8 | 69.3$\pm$2.6 | 85.0$\pm$3.2 | 77.6$\pm$2.1 | 90.5$\pm$1.3 | 96.2$\pm$0.5
AA (%) | 60.9$\pm$1.3 | 64.3$\pm$1.6 | 65.5$\pm$2.6 | 60.0$\pm$1.5 | 76.9$\pm$1.7 | 63.5$\pm$2.4 | 78.6$\pm$3.4 | 70.6$\pm$3.4 | 78.6$\pm$2.5 | 94.3$\pm$0.9
Kappa$\times$100 | 66.6$\pm$0.9 | 70.0$\pm$0.3 | 66.2$\pm$1.6 | 68.5$\pm$1.4 | 76.4$\pm$1.3 | 63.6$\pm$1.4 | 82.8$\pm$3.7 | 75.4$\pm$2.2 | 89.1$\pm$1.5 | 95.7$\pm$0.5
TABLE VII: Zero-Shot Classification Results of M3BS and 9 Other Algorithms on the Unseen Dataset Pavia University |  mRMR [8] |  BS-Nets [21] |  GCSR-BS [25] |  DRLBS [23] |  DGLAnet [46] |  TSC [47] |  SICNN [16] |  ABCNN [20] |  BHCNN [19] | M3BS
---|---|---|---|---|---|---|---|---|---|---
OA (%) | 73.4$\pm$0.4 | 79.5$\pm$0.4 | 82.6$\pm$0.3 | 86.2$\pm$0.7 | 87.3$\pm$0.5 | 79.7$\pm$1.2 | 94.5$\pm$0.6 | 81.5$\pm$1.1 | 94.2$\pm$1.6 | 99.5$\pm$0.2
AA (%) | 63.2$\pm$1.0 | 66.7$\pm$1.1 | 76.6$\pm$0.7 | 79.5$\pm$2.9 | 86.1$\pm$0.7 | 77.6$\pm$1.1 | 90.4$\pm$1.6 | 80.9$\pm$2.3 | 89.8$\pm$3.6 | 98.5$\pm$0.3
Kappa$\times$100 | 63.3$\pm$0.5 | 71.2$\pm$0.6 | 76.2$\pm$0.4 | 81.2$\pm$1.0 | 86.2$\pm$0.7 | 77.3$\pm$0.9 | 92.7$\pm$0.9 | 81.1$\pm$1.7 | 92.3$\pm$2.1 | 99.2$\pm$0.2
TABLE VIII: Zero-Shot Classification Results of M3BS and 9 Other Algorithms on the Unseen Dataset University of Houston |  mRMR [8] |  BS-Nets [21] |  GCSR-BS [25] |  DRLBS [23] |  DGLAnet [46] |  TSC [47] |  SICNN [16] |  ABCNN [20] |  BHCNN [19] | M3BS
---|---|---|---|---|---|---|---|---|---|---
OA (%) | 80.6$\pm$1.0 | 79.7$\pm$0.5 | 88.5$\pm$1.2 | 79.9$\pm$0.8 | 88.6$\pm$0.2 | 81.4$\pm$0.6 | 88.9$\pm$1.7 | 83.7$\pm$1.6 | 91.4$\pm$2.5 | 96.5$\pm$0.5
AA (%) | 79.5$\pm$1.1 | 79.2$\pm$0.4 | 87.4$\pm$1.2 | 78.9$\pm$0.9 | 87.0$\pm$0.5 | 81.1$\pm$0.2 | 87.3$\pm$1.9 | 83.0$\pm$1.3 | 90.8$\pm$2.3 | 95.9$\pm$0.3
Kappa$\times$100 | 79.0$\pm$1.1 | 78.1$\pm$0.5 | 87.6$\pm$1.2 | 78.2$\pm$0.8 | 87.9$\pm$0.6 | 80.7$\pm$0.3 | 88.0$\pm$1.8 | 82.4$\pm$1.8 | 90.6$\pm$2.7 | 96.3$\pm$0.4

####  C3 University of Houston

Same as Indian Pines, the dataset is divided into a training set and a testing
set at a ratio of 5:95. For all 10 algorithms for comparison, 20 bands among
176 bands are selected for hyperspectral band selection. Table V lists the
classification accuracy of each algorithm on the entire dataset. The
classification accuracy of each class is available Table S3 in the
Supplementary Material. Our M3BS performs best on most of the 15 categories,
and maintains a classification accuracy above 94% on almost all the
categories. Compared with ABCNN, M3BS exhibits at least 5.1%, 5.6% and 5.6% in
terms of OA, AA and Kappa, respectively. For some hard-to-classify categories,
such as Commercial and Road, M3BS maintains an accuracy of more than 93%,
while the accuracy of almost all other competitive algorithms is below 90%.

TABLE IX: Classification Results of M3BS with different numbers of teachers on different datasets Dataset | Condition | OA (%) | AA (%) | Kappa$\times$100
---|---|---|---|---
Indian Pines | M3BS w/o teachers | 93.9 | 90.3 | 93.1
M3BS w/ 1 teacher (BS-Nets) | 95.6 | 91.4 | 95.0
M3BS w/ 2 teachers (BS-Nets and SICNN) | 95.8 | 92.3 | 95.2
M3BS w/ 3 teachers (BS-Nets, SICNN and TWCNN) | 96.2 | 94.3 | 95.7
Pavia University | M3BS w/o teachers | 97.2 | 95.7 | 96.4
M3BS w/ 1 teacher (BS-Nets) | 98.2 | 96.7 | 97.7
M3BS w/ 2 teachers (BS-Nets and SICNN) | 98.4 | 97.1 | 97.9
M3BS w/ 3 teachers (BS-Nets, SICNN and TWCNN) | 98.5 | 97.5 | 98.1
University of Houston | M3BS w/o teachers | 93.4 | 92.9 | 93.0
M3BS w/ 1 teacher (BS-Nets) | 95.0 | 94.5 | 94.7
M3BS w/ 2 teachers (BS-Nets and SICNN) | 95.9 | 95.3 | 95.6
M3BS w/ 3 teachers (BS-Nets, SICNN and TWCNN) | 96.5 | 95.9 | 96.3
TABLE X: Classification Results of M3BS with different multi-teacher fusion strategies on different datasets Dataset | Condition | OA (%) | AA (%) | Kappa$\times$100
---|---|---|---|---
Indian Pines | union | 95.4 | 92.4 | 94.8
normalized sum | 96.1 | 91.7 | 95.5
diversity ensemble | 96.2 | 94.3 | 95.7
Pavia University | union | 98.0 | 96.7 | 97.4
normalized sum | 98.2 | 96.8 | 97.7
diversity ensemble | 98.5 | 97.5 | 98.1
University of Houston | union | 96.4 | 96.1 | 96.2
normalized sum | 92.7 | 92.6 | 92.2
diversity ensemble | 96.5 | 95.9 | 96.3
TABLE XI: Classification Results of M3BS with Different Multi-Loss Weighting Schemes on Different Datasets Scenario | Dataset | Weights | OA (%) | AA (%) | Kappa$\times$100
---|---|---|---|---|---
$\lambda^{\text{bs}}$ | $\lambda^{\text{cls}}$
only $\mathcal{L}^{\text{cls}}$ | Indian Pines | 0 | 1 | 93.9 | 90.3 | 93.1
Pavia University | 0 | 1 | 97.2 | 95.7 | 96.4
University of Houston | 0 | 1 | 93.4 | 92.9 | 93.0
hand-crafted weighting | Indian Pines | 0.01 | 0.99 | 95.5 | 92.2 | 94.9
Pavia University | 0.01 | 0.99 | 98.1 | 96.8 | 97.6
University of Houston | 0.01 | 0.99 | 95.0 | 94.3 | 94.7
Indian Pines | 0.05 | 0.95 | 95.7 | 92.1 | 95.2
Pavia University | 0.05 | 0.95 | 98.0 | 96.7 | 97.5
University of Houston | 0.05 | 0.95 | 94.9 | 94.6 | 94.6
Indian Pines | 0.1 | 0.9 | 95.7 | 93.0 | 95.2
Pavia University | 0.1 | 0.9 | 98.0 | 96.6 | 97.5
University of Houston | 0.1 | 0.9 | 95.0 | 94.3 | 94.6
Indian Pines | 0.3 | 0.7 | 96.0 | 93.1 | 95.5
Pavia University | 0.3 | 0.7 | 98.2 | 97.1 | 97.8
University of Houston | 0.3 | 0.7 | 95.1 | 94.5 | 94.8
Indian Pines | 0.7 | 0.3 | 96.2 | 93.3 | 95.7
Pavia University | 0.7 | 0.3 | 98.2 | 96.9 | 97.7
University of Houston | 0.7 | 0.3 | 95.0 | 94.5 | 94.7
Indian Pines | 0.9 | 0.1 | 95.9 | 92.5 | 95.3
Pavia University | 0.9 | 0.1 | 98.2 | 96.8 | 97.7
University of Houston | 0.9 | 0.1 | 95.0 | 94.5 | 94.7
averaged weighting | Indian Pines | 0.5 | 0.5 | 95.6 | 92.3 | 95.0
Pavia University | 0.5 | 0.5 | 98.3 | 97.0 | 97.9
University of Houston | 0.5 | 0.5 | 95.0 | 94.4 | 94.7
uncertainty weighting | Indian Pines | auto | 96.2 | 94.3 | 95.7
Pavia University | 98.5 | 97.5 | 98.1
University of Houston | 96.5 | 95.9 | 96.3

###  D Zero-Shot Classification Results on the Unseen Dataset

What we initially want to implement is zero-shot band selection, that is,
selecting spectral bands for a new dataset without any of its samples during
training. To verify the performance of our M3BS in this scenario, we introduce
three additional hyperspectral datasets, namely Salinas, Washington DC Mall
and KSC, to be meta-training tasks, and regard the above-mentioned Indian
Pines, Pavia University and University of Houston datasets as meta-testing
tasks. Similar to the practice of previous datasets, these meta-training
datasets are splited into training sets and testing sets at a ratio of 10:90.
In other words, what we want to achieve is to train our algorithm only with
three datasets and directly perform band selection on the other three
datasets. It should be noted that, this scenario differs from unsupervised
learning, in which only labels rather than samples are absent.

None of the 9 comparison algorithms are optimized for zero-shot learning, and
some of them are not compatible with multiple training datasets. To make
training feasible, we apply some intuitive modifications to these algorithms.
For multiple datasets to be trained on, we firstly find the dataset containing
the most spectral bands, and then fill in the missing bands with 0 for other
datasets with less bands. As a result, all training datasets are aligned in
terms of the number of spectral bands. For mRMR, after calculating the score
of the selected band combination for each training dataset, the average of
these scores are taken to be the final score for further updates. For BS-Nets,
we set up as many reconstruction networks as there are datasets, and take the
average of the output of these networks as the final result. For CNN-based
algorithms SICNN and ABCNN, their CNN-based classifiers are retrained for
different band subsets during the training phase. For BHCNN, multiple
independent classifiers instead of one classifier are attached for parallel
optimization.

It should be pointed out that, since our M3BS is originally designed for zero-
shot learning, it is naturally compatible with multiple training datasets and
does not require an additional dataset alignment operation. Taking the first
graph convolutional layer as an example, the size $0pt0pt\times 256$ of the
weight matrix $\mathbf{W}^{\text{GCN(1)}}$ is independent of the number of
bands ${n^{\text{band}}}$ for any HSI dataset. As long as the patch size
$0pt\times 0pt$ is fixed, the same learnable parameters are appliable to
datasets with any number of bands. The relationship between the network
structure and the amount of bands is decoupled in M3BS.

Table VI-VIII shows the average of the classification metrics OA, AA and Kappa
and their corresponding standard deviations on three datasets by 30
independent runs of each algorithm. Among these accuracies, the best ones are
bolded and highlighted as in Table III-V. It can be seen at a glance from
these tables that, our M3BS significantly outperforms all the other 9
algorithms, and maintains a consistently high level over all three datasets
that are absent throughout the whole training phase. Quantitatively, for any
of these meta-testing datasets, M3BS can improve at least 4.3% OA, 5.1% AA,
and 5.7% Kappa compared to the best performing one among other algorithms.
This is predominantly because both the network structure and the optimization
procedure in M3BS are designed for adapting to zero-shot band selection.
Specifically, the generalizable GCN with the additional supervision from
multiple teachers can achieve meta-knowledge extraction that is compatible
with various datasets. The multi-objective meta-learning optimization
procedure makes M3BS appliable to different objectives and different datasets.
As a result, the extracted meta-knowledge can be directly transferred to
unseen datasets without any retraining or fine-tuning. In contrast, the
knowledge learned by other comparison algorithms is not transferable, which
means that these algorithms perform well on seen datasets but poorly on unseen
datasets.

For a more intuitive comparison, the ground truth (G.T.) and the visual
classification results on three unseen hyperspectral datasets of these
algorithms are shown in Fig.S1-Fig.S3 in the Supplementary Material. It should
be noted that, the full-image classification predictions of the dataset
University of Houston are given due to the scattered distribution of labeled
pixel samples. As shown in (b-j), a large number of pixel samples are
misclassified into other categories. For mRMR, BS-Nets, GCSR-BS, DRLBS, TSC
and ABCNN, the network input is constructed at the pixel level, which does not
consider the relationship among adjacent pixels, and lacks local spatial
information. As a result, the output of these methods contains lots of noise
points and discontinuous areas, espacially for categories like Grass-trees,
Soybean-notill, Soybean-mintill in Indian Pines and Bare Soil in Pavia
University. For CNN-based algorithms SICNN and BHCNN, the spatial network
takes the spatial window around the pixel as input, which significantly
improves local spatiality and category consistency in the same area. However,
misclassified pixels still appear at some boundaries among different
categories. In M3BS, pixel-level samples are used in the process of
constructing spatial-spectral graphs, and spatial windows are regarded as
input for the classifier. Compared with the above methods, M3BS not only
utilizes a more efficient graph construction strategy, but also takes local
spatial information into consideration. As a result, M3BS obtains the best
classification visualization among all comparison algorithms, which preserves
detailed boundaries, has the least noise points and is closest to the ground
truth.

###  E Ablation Experiments on the Unseen Dataset

####  E1 Multiple Teachers

This ablation aims to verify whether multiple teachers are beneficial for band
selection. To quantitatively analyze the contribution of each teacher, Table
IX records the classification results of M3BS with different numbers of
teachers.

As shown in Table IX, as the number of band selection teachers increases, the
classification performance of M3BS keeps improving. The number of teachers
being zero corresponds to the absence of both the band selection loss
$\mathcal{L}^{\text{bs}}$ and the multi-objective loss $\mathcal{L}$. At this
time, the classification accuracy is about 3% lower than the original M3BS,
which proves that the classification task alone can’t provide sufficient meta-
knowledge for zero-shot band selection. As different teachers join, the
classification ability becomes stronger and stronger since different types of
teachers can provide guidance from different perspectives. Additionally, the
order in which teachers join shows that, the supervision from deep learning
algorithms like SICNN and TWCNN, is of higher quality than that from
unsupervised algorithms like BS-Nets.

####  E2 Multi-Teacher Fusion

After confirming that multiple teachers are indeed beneficial for zero-shot
band selection, this ablation try to figure out the best strategy to fuse
these teachers into one integrated teacher.

Our original M3BS utilizes a strategy called diversity ensemble, which chooses
the spectral bands according to their popularity. Bands with more received
votes will be selected first, and bands with the same number of received votes
will be randomly selected. For comparison, we compare M3BS with two intuitive
fusion strategies, named union and normalized sum. The former treats the band
combinations selected by teachers as sets, and takes the union of these sets
as the result. The latter multiplies each teacher’s output score by one-third,
and add these scores together as the ground truth.

Table X records the classification results of M3BS with different multi-
teacher fusion strategies.

As shown in Table X, our novel fusion strategy outperforms the other two
strategies with an accuracy increase of about 1%. For the strategy union, we
suspect that this is due to the so-called over-selection problem. In other
words, when the bands selected by different teachers do not have much overlap,
too many bands will be used as fusion results, and some bands with low
discriminability may be added, which in turn downgrades the classification
performance. For the strategy normalized sum, we think that it may lack the
introduction of random noise, thus leaving the supervision for each epoch
unchanged and reducing the generalization ability. On the contrary, the
diversity ensemble strategy we proposed not only limits the number of selected
bands, but also introduces a certain degree of randomness, thereby ensuring
the effective fusion of multiple teachers.

####  E3 Multi-Loss Weighting

This ablation is used to prove the necessity of balancing multiple losses, and
to make comparison among hand-crafted and automatically learned loss weights.
When encountering multiple loss functions in a deep learning algorithm, the
intuitive idea is to manually set up different weights for different losses,
and keep these weights unchanged throughout the training process. Here, we
design 6 hand-crafted weight combinations in sequence. Also as usual, the sum
of all the loss weights in a single combination is fixed to 1. Instead of
manual intervention on these weights, our M3BS utilizes a homoscedastic
uncertainty-based multi-objective loss to automatically learn them. For a more
refined and convincing comparison, Table XI elaborately lists the
classification results of M3BS with different multi-loss weighting schemes.

As shown in Table XI, the uncertainty-based weighting scheme introduced in
M3BS dominates all other weighting schemes. First, we do not show the
classification performance when only optimizing the band selection loss
$\mathcal{L}^{\text{bs}}$. When the classification loss
$\mathcal{L}^{\text{cls}}$ is absent, the CNN-based classifier of each dataset
is not trained any more, leading to random network parameters for inferring,
which makes no sense. Next, the case of only optimizing the classification
loss $\mathcal{L}^{\text{cls}}$ is actually the same as the case without
multiple teachers in Table IX, and the accuracy is about 3% lower than M3BS.
Here, the same assertion can be applied: a single classification task is not
enough for sufficient meta-knowledge for zero-shot band selection. Among these
hand-crafted weight combinations, we can observe that for the dataset Indian
Pines, Pavia University and University of Houston, the optimal weight ratios
are approximately 0.7:0.3, 0.5:0.5 and 0.3:0.7, respectively. Finally, the
weight combination learned by our uncertainty weighting scheme is better than
any hand-crafted weight. It’s reasonable to speculate that, since these
weights are updated together with the network parameters, they can be adapted
to different training stages. For example, in the early stage, band selection
is entrusted to dominate the network training, leading to a preliminary
understanding of spectral bands; in the later stage, the classification takes
the lead to further strengthening the feature representation ability. This is
only available for multi-objective learning, not for static weights.

##  V Conclusion

In this paper, a novel multi-teacher multi-objective meta-learning framework
M3BS is proposed for zero-shot hyperspectral band selection. For compatible
meta-knowledge extraction for various datasets, M3BS constructs a
generalizable GCN with spatial-spectral graphs. The decomposition of dataset-
agnostic bases and dataset-specific coefficients makes it possible to
explicitly characterize unseen samples with the unified meta-knowledge. For a
better generalization ability and a faster convergence speed, M3BS ensemble
multiple teachers to provide high-quality experiences. For zero-shot
optimization over multiple datasets and multiple objectives simultaneously,
all these modules are integrated into a multi-objective meta-learning
procedure. Experimental results over three commonly used hyperspectral
datasets prove that M3BS can produce better band combinations than many other
state-of-the-art baselines for zero-shot learning. Moreover, it provides a new
design paradigm for subsequent researchers, and reminds them to pay more
attention to the performance when transferring to unseen datasets.

## References

  * [1] Y. Xu, B. Du, and L. Zhang, “Robust self-ensembling network for hyperspectral image classification,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2022.
  * [2] J. Qu, W. Dong, Y. Yang, T. Zhang, Y. Li, and Q. Du, “Cycle-refined multidecision joint alignment network for unsupervised domain adaptive hyperspectral change detection,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2023.
  * [3] J. Lian, L. Wang, H. Sun, and H. Huang, “Gt-had: Gated transformer for hyperspectral anomaly detection,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2024.
  * [4] G. Tejasree and L. Agilandeeswari, “An extensive review of hyperspectral image classification and prediction: techniques and challenges,” _Multimedia Tools and Applications_ , pp. 1–98, 2024.
  * [5] R. Bellman, “Dynamic programming,” _Science_ , vol. 153, no. 3731, pp. 34–37, 1966.
  * [6] R. Vaddi, B. P. Kumar, P. Manoharan, L. Agilandeeswari, and V. Sangeetha, “Strategies for dimensionality reduction in hyperspectral remote sensing: A comprehensive overview,” _The Egyptian Journal of Remote Sensing and Space Sciences_ , vol. 27, no. 1, pp. 82–92, 2024.
  * [7] Q. Wang, Q. Li, and X. Li, “Hyperspectral band selection via adaptive subspace partition strategy,” _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , vol. 12, pp. 4940–4950, 2019.
  * [8] H. Peng, F. Long, and C. Ding, “Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 27, pp. 1226–1238, 2003.
  * [9] X. Cao, C. Wei, Y. Ge, J. Feng, J. Zhao, and L. Jiao, “Semi-supervised hyperspectral band selection based on dynamic classifier selection,” _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , vol. 12, pp. 1289–1298, 2019.
  * [10] P. Ghamisi, M. S. Couceiro, and J. A. Benediktsson, “A novel feature selection approach based on fodpso and svm,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 53, pp. 2935–2947, 2015.
  * [11] A. Rashno, M. Shafipour, and S. Fadaei, “Particle ranking: An efficient method for multi-objective particle swarm optimization feature selection,” _Knowl. Based Syst._ , vol. 245, p. 108640, 2022.
  * [12] I. R. Subramanian, J. Weston, S. D. Barnhill, and V. N. Vapnik, “Gene selection for cancer classification using support vector machines,” _Machine Learning_ , vol. 46, pp. 389–422, 2002.
  * [13] Y. Zhou, R. Zhang, S. Wang, and F. Wang, “Feature selection method based on high-resolution remote sensing images and the effect of sensitive features on classification accuracy,” _Sensors (Basel, Switzerland)_ , vol. 18, 2018.
  * [14] B.-C. Kuo, H.-H. Ho, C.-H. Li, C.-C. Hung, and J.-S. Taur, “A kernel-based feature selection method for svm with rbf kernel for hyperspectral image classification,” _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , vol. 7, pp. 317–326, 2014.
  * [15] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” _Nature_ , vol. 521, pp. 436–444, 2015.
  * [16] P. Ghamisi, Y. Chen, and X. Zhu, “A self-improving convolution neural network for the classification of hyperspectral data,” _IEEE Geoscience and Remote Sensing Letters_ , vol. 13, pp. 1537–1541, 2016.
  * [17] Y. Zhan, D. Hu, H. Xing, and X. Yu, “Hyperspectral band selection based on deep convolutional neural network and distance density,” _IEEE Geoscience and Remote Sensing Letters_ , vol. 14, pp. 2365–2369, 2017.
  * [18] J. Feng, D. Li, J. Chen, X. Zhang, X. Tang, and X. Wu, “Hyperspectral band selection based on ternary weight convolutional neural network,” _IGARSS 2019 - 2019 IEEE International Geoscience and Remote Sensing Symposium_ , pp. 3804–3807, 2019.
  * [19] J. Feng, J. Chen, Q. Sun, R. Shang, X. Cao, X. Zhang, and L. Jiao, “Convolutional neural network based on bandwise-independent convolution and hard thresholding for hyperspectral band selection,” _IEEE Transactions on Cybernetics_ , vol. 51, pp. 4414–4428, 2020.
  * [20] P. R. Lorenzo, L. Tulczyjew, M. Marcinkiewicz, and J. Nalepa, “Hyperspectral band selection using attention-based convolutional neural networks,” _IEEE Access_ , vol. 8, pp. 42 384–42 403, 2020.
  * [21] Y. Cai, X. Liu, and Z. Cai, “Bs-nets: An end-to-end framework for band selection of hyperspectral image,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 58, pp. 1969–1984, 2019.
  * [22] A. Sellami, M. Farah, I. R. Farah, and B. Solaiman, “Hyperspectral imagery classification based on semi-supervised 3-d deep neural network and adaptive band selection,” _Expert Syst. Appl._ , vol. 129, pp. 246–259, 2019.
  * [23] L. Mou, S. Saha, Y. Hua, F. Bovolo, L. Bruzzone, and X. Zhu, “Deep reinforcement learning for band selection in hyperspectral image classification,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 60, pp. 1–14, 2021.
  * [24] J. Feng, D. Li, J. Gu, X. Cao, R. Shang, X. Zhang, and L. Jiao, “Deep reinforcement learning for semisupervised hyperspectral band selection,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 60, pp. 1–19, 2021.
  * [25] Y. Cai, Z. Zhang, X. Liu, and Z. Cai, “Efficient graph convolutional self-representation for band selection of hyperspectral image,” _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , vol. 13, pp. 4869–4880, 2020.
  * [26] J. Feng, Z. Ye, S. Liu, X. Zhang, J. Chen, R. Shang, and L. Jiao, “Dual-graph convolutional network based on band attention and sparse constraint for hyperspectral band selection,” _Knowl. Based Syst._ , vol. 231, p. 107428, 2021.
  * [27] F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini, “The graph neural network model,” _IEEE Transactions on Neural Networks_ , vol. 20, pp. 61–80, 2009.
  * [28] R. Ying, R. He, K. Chen, P. Eksombatchai, W. L. Hamilton, and J. Leskovec, “Graph convolutional neural networks for web-scale recommender systems,” _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 2018.
  * [29] Y. Li, R. Yu, C. Shahabi, and Y. Liu, “Diffusion convolutional recurrent neural network: Data-driven traffic forecasting,” _arXiv: Learning_ , 2017.
  * [30] A. Fout, J. Byrd, B. Shariat, and A. Ben-Hur, “Protein interface prediction using graph convolutional networks,” in _NIPS_ , 2017.
  * [31] M. D. Buhmann, “Radial basis functions - theory and implementations,” in _Cambridge monographs on applied and computational mathematics_ , 2003.
  * [32] U. Feige, “Spectral graph theory,” _Zeta and $L$-functions in Number Theory and Combinatorics_, 2019.
  * [33] T. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” _ArXiv_ , vol. abs/1609.02907, 2016.
  * [34] R. Caruana, “Multitask learning: A knowledge-based source of inductive bias,” in _International Conference on Machine Learning_ , 1993.
  * [35] W. Hou, Y. Dong, B. Zhuang, L. Yang, J. Shi, and T. Shinozaki, “Large-scale end-to-end multilingual speech recognition and language identification with multi-task learning,” in _Interspeech_ , 2020.
  * [36] K. He, G. Gkioxari, P. Dollár, and R. B. Girshick, “Mask r-cnn,” _2017 IEEE International Conference on Computer Vision (ICCV)_ , pp. 2980–2988, 2017.
  * [37] Z. Zhang, P. Luo, C. C. Loy, and X. Tang, “Facial landmark detection by deep multi-task learning,” in _European Conference on Computer Vision_ , 2014.
  * [38] S. Sigtia, P. Clark, R. Haynes, H. B. Richards, and J. S. Bridle, “Multi-task learning for voice trigger detection,” _ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 7449–7453, 2020.
  * [39] D. Maudsley, _A Theory of Meta-learning and Principles of Facilitation : an Organismic Perspective_. Thesis (Ed.D.)–University of Toronto, 1979. [Online]. Available: https://books.google.com.hk/books?id=nIiYmQEACAAJ
  * [40] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in _International Conference on Machine Learning_ , 2017.
  * [41] A. Shaban, S. Bansal, Z. Liu, I. Essa, and B. Boots, “One-shot learning for semantic segmentation,” _ArXiv_ , vol. abs/1709.03410, 2017.
  * [42] K. Young, B. Wang, and M. E. Taylor, “Metatrace actor-critic: Online step-size tuning by meta-gradient descent for reinforcement learning control,” in _International Joint Conference on Artificial Intelligence_ , 2018.
  * [43] A. E. Shaw, W. Wei, W. Liu, L. Song, and B. Dai, “Meta architecture search,” in _Neural Information Processing Systems_ , 2018.
  * [44] S. Liu, E. Johns, and A. J. Davison, “End-to-end multi-task learning with attention,” _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 1871–1880, 2018.
  * [45] A. Kendall, Y. Gal, and R. Cipolla, “Multi-task learning using uncertainty to weigh losses for scene geometry and semantics,” _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 7482–7491, 2017.
  * [46] K. He, W. SUN, G. Yang, X. Meng, K. Ren, J. Peng, and Q. Du, “A dual global-local attention network for hyperspectral band selection,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. PP, pp. 1–1, 2022. [Online]. Available: https://api.semanticscholar.org/CorpusID:248310116
  * [47] S. Huang, H. Zhang, J. Xue, and A. Pižurica, “Heterogeneous regularization-based tensor subspace clustering for hyperspectral band selection,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 34, pp. 9259–9273, 2022. [Online]. Available: https://api.semanticscholar.org/CorpusID:247498481
  * [48] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _CoRR_ , vol. abs/1412.6980, 2014.
  * [49] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, high-performance deep learning library,” _ArXiv_ , vol. abs/1912.01703, 2019.