11institutetext: Computing Sciences, Tampere University, Finland

# Spectral Graph Reasoning Network for Hyperspectral Image Classification

Huiling Wang ID<EMAIL_ADDRESS>
###### Abstract

Convolutional neural networks (CNNs) have achieved remarkable performance in
hyperspectral image (HSI) classification over the last few years. Despite the
progress that has been made, rich and informative spectral information of HSI
has been largely underutilized by existing methods which employ convolutional
kernels with limited size of receptive field in the spectral domain. To
address this issue, we propose a spectral graph reasoning network (SGR)
learning framework comprising two crucial modules: 1) a spectral decoupling
module which unpacks and casts multiple spectral embeddings into a unified
graph whose node corresponds to an individual spectral feature channel in the
embedding space; the graph performs interpretable reasoning to aggregate and
align spectral information to guide learning spectral-specific graph
embeddings at multiple contextual levels 2) a spectral ensembling module
explores the interactions and interdependencies across graph embedding
hierarchy via a novel recurrent graph propagation mechanism. Experiments on
two HSI datasets demonstrate that the proposed architecture can significantly
improve the classification accuracy compared with the existing methods with a
sizable margin.

Figure 1: Illustration of our proposed network architecture.

##  1 Introduction

Hyperspectral image comprises hundreds of continuous spectral bands throughout
the electromagnetic spectrum with high spectral resolution, which facilities
the precise differentiation of chemical properties of scene materials
remotely. Consequently hyperspectral images have been considered as a crucial
data source in various fields, such as environmental monitoring, mining,
agriculture, and land-cover mapping.

HSI classification involves assigning a categorical class label to each
individual pixel location given the corresponding spectral feature. Various
classification approaches have been proposed to address the hyperspectral
image classification problem. Early approaches largely adopted traditional
machine learning methods which were trained on hand-crafted features from HSI
data to empirically encode the spectral information, _e.g_. SVM [19], KNN
[25], dictionary learning [3], graphical model [29], extreme learning machine
[22] and among others. Other methods [62, 60] also exploited both spectral and
spatial information since utilizing only the spectral information regardless
of the spatial correlation is difficult in accurately classifying different
land-cover categories.

With the advent of new remote sensing instruments and increased temporal
resolutions, the deluge of high dimensional HSI data is posing new challenges
on the limited discriminative power of the empirically designed features and
the traditional classification methods. Deep learning based HSI classification
methods have been proposed inspired by its success in visual recognition of
natural images. Those methods benefit from the strong representation
capability and end-to-end learning. Chen _et al._ [5] developed the first deep
neural networks (DNN) method which utilized stacked autoencoders to learn
high-level features. Mou _et al._ [26] introduced a Recurrent Neural Network
(RNN) based architecture. More powerful end-to-end Convolutional Neural
Network (CNN) based architectures [4, 10, 15] have been proposed recently. Lee
_et al._ [21] explored local contextual interactions by jointly exploiting
local spatio-spectral relationships of neighboring pixels. Residual learning
[12] was introduced by Song _et al._ [30] to build very deep network for
extracting more discriminative feature representations for HSI classification.
Li _et al_. [23] introduced a 3D-CNN architecture which was improved by He _et
al_. [13] to jointly learn Multi-scale 2D spatial feature and 1D spectral
feature from HSI data. Graph convolutional network (GCN) based approaches [28,
50] have also been proposed which operated on graphs constructed in the
_spatial_ domain, aggregating and transforming feature information from the
neighbors of every graph node given their relevance.

Despite of the significant improvements by the DNN based methods, they suffer
from a number of inherent pitfalls. Specifically, all existing methods utilize
convolutional kernels with limited size of receptive field to encode the very
high dimensional and informative spectral domain. As a consequence, output
constituting a certain new spectral channel is derived from only a fraction of
all spectral channels which blocks information flow between distant spectra
and fails to capture longer term contextual information. Furthermore, spectral
domain normally contains abundant yet noisy information which can lead to
spurious classifications due to this weakened representation based on local
contexts.

Graphical models and graph neural networks (GNNs) have emerged as powerful
tools in the field of computer vision [49, 51, 38, 48, 27, 40, 57, 64, 65, 54,
41, 59], offering a versatile framework for representing and processing
complex visual data. These approaches leverage the inherent structure and
relationships within images [55, 53, 37, 50, 58, 11] and videos [45, 43, 2],
allowing for more nuanced and context-aware analysis. By representing visual
elements as nodes and their interactions as edges, graphical models and GNNs
can capture spatial, temporal, and semantic dependencies that are crucial for
tasks such as visual information retrieval [14], stylization [47, 44, 46],
object detection [36, 39, 61], scene understanding [9], and image or video
segmentation [56, 42, 66, 24, 35]. In recent years, these techniques have also
found significant applications in remote sensing and geospatial data analysis.
For instance, in the domain of Global Navigation Satellite Systems (GNSS),
graphical models have been employed to enhance positioning accuracy by
modeling the intricate relationships between satellite signals, atmospheric
conditions, and ground-based receivers [16, 17].

In this paper, we propose a novel spectral graph reasoning network (SGR) to
explicitly address the above issues. Specifically, we firstly unpack and cast
multiple spectral embeddings into a unified graph with each node corresponding
to an individual spectral embedding channel via a novel spectral decoupling
module. Graph reasoning facilitates the interactions between spectra which in
turn boosts the representational capacity of the network by discriminating the
significance of different spectral bands. A spectral-specific graph embedding
hierarchy is generated through the spectral decoupling module which encodes a
hierarchical representation of the spectral information. Each level of the
hierarchy carries different degrees of discriminative power for HSI
classification. To the best of our knowledge, this is the first method to
model and interpret HSI data using graph neural network in the spectral
domain.

We further introduce a spectral ensembling module that correlates the multi-
contextual spectral-specific graph embeddings across the spectral hierarchy.
This is achieved through a novel recurrent graph propagation mechanism which
learns to map from a sequence of input graphs with varying size, each
representing a state of transformed spectra with its intrinsic structure, to
an ensembled spectral representation containing the key discriminative
features from each contextual level to improve the classification accuracy.

##  2 Method

We propose a novel end-to-end spectral-specific graph representation learning
framework for HSI classification, which comprises two core modules _i.e_.
spectral decoupling module and spectral ensembling module, as illustrated in
Fig. 1.

###  2.1 Graph Construction

We define a graph structure for mining correlations and constraints among
spectral embedding channels. Specifically, we define an undirected graph
$\mathcal{G}=(\mathcal{V},\mathcal{E},\mathcal{A})$. Given the feature map
$X\in\mathbb{R}^{N\times H\times W}$ produced by the encoder network from an
input HSI data volume centered at each pixel location, we seek to construct
graph node set $\mathcal{V}=\\{v_{i}\\}_{i=1}^{N}$ and each node
$v_{i}\in\mathcal{V}$ corresponds to a spectral feature from the embedding and
the associated feature vector with $C=H\times W$ dimensions indicates
$x_{i}\in\mathbb{R}^{C}$. The set of edges $\mathcal{E}=\\{e_{ij}\\}$ indicate
the connection between nodes, and $\mathbf{A}=(e_{ij})_{N\times N}$ is the
adjacency matrix, with $a_{i,j}=0$ if $(i,j)\notin\mathcal{E}$, and
$a_{i,j}=1$ if $(i,j)\in\mathcal{E}$. We adopt K-Nearest Neighbor Graph (KNN-
Graph) construction [52, 31, 33, 34] in which two vertices $v_{i}$ and $v_{j}$
are connected by an edge, if the distance between $v_{i}$ and $v_{j}$ is among
the k-th smallest distances measured by cosine distance. The normalized graph
Laplacian [7, 32] is computed as $L=I_{N}-D^{-\frac{1}{2}}AD^{-\frac{1}{2}}$,
where $D^{ii}=\sum_{j}A^{ij}$.

The spectral convolutions on graphs [8] can be formulated as the
multiplication of input $x\in\mathbb{R}^{n}$ with operator $g_{\theta}$ in
Fourier domain,

$\displaystyle y=g_{\theta}(L)*x=Ug_{\theta}(\Lambda)U^{T}x$ (1)

where graph Fourier basis $U$ is the matrix of the eigenvectors of the
normalized graph Laplacian $L$ such that $L=U\theta(\Lambda)U^{T}$ with
$\Lambda$ being its corresponding eigenvalues, and $U^{T}x$ represents the
graph Fourier transform of $x$. To reduce the computational complexity, Kipf
and Welling [18] stacked multiple localized graph convolutional layers with
the first-order approximation of graph Laplacian,

$\displaystyle Y=\tilde{D}^{-\frac{1}{2}}\tilde{A}\tilde{D}^{-\frac{1}{2}}XW$
(2)

where $W\in\mathbb{R}^{M\times F}$ is the matrix of filter parameters,
$\tilde{A}$ and $\tilde{D}$ are the normalized versions with
$\tilde{A}=A+I_{N}$ and $\tilde{D}^{ii}=\sum_{j}\tilde{A}^{ij}$.

###  2.2 Spectral Decoupling

The spectral decoupling module aims to learn spectral-specific feature
representation by reasoning within spectral embedding at multiple scales.
Necessitated by the excessive information of the broad and noisy spectral
domain, this module explores the long-range contextual dependencies in the
original feature space to form multiple lower dimensional intermediate
representations in order to capture the essence of the spectral information.

We adopt a graph pyramid of spectral features - at each scale of the graph
pyramid, graph convolution (GCN) is applied to capture the interdependence of
graph via message passing between the nodes of graphs, _i.e_. spectral
features, as illustrated in Fig. 1. Assuming that filters in later layers of
an encoder are responsive to various materials that make up the scanned scene,
the proposed band GCN captures correlations and occurrences between more
abstract characteristics in the scene like constituent materials. Graph
pooling is applied to reduce the resolution of spectral features, whose result
forms the input of the following graph convolution layer.

Graph pooling aims to enable down-sampling on graph data while preserving as
much information as possible from the original graph. To this end, we employ a
single neuron linear layer parameterized by $W_{\theta}$, followed by a
sigmoid function and top-$k$ pooling. This is implemented as multiplying the
input at $l$th layer $X^{l}\in\mathbb{R}^{N\times C}$ with
$W_{\theta}^{l}\in\mathbb{R}^{C}$ followed by sigmoid operation, whose result
is an 1D vector $\mathbf{s}\in\mathbb{R}^{N}$ measuring how much information
of nodes can be retained when projected onto the direction $W_{\theta}^{l}$.
Graph coarsening can thus be done by selecting nodes with the largest scalar
projection values on $W_{\theta}^{l}$ to form a new graph. The location of
selected nodes in the original graph is recorded for graph unpooling
operation. The feature of the selected nodes is adjusted by its projection
values as $X^{l+1}=X^{l}\odot\mathbf{s}$ where $\odot$ is the Hadamard
product. Intuitively the selected nodes represent the “cluster centers” of
spectral features and the edges measure the similarity between these clusters.

Figure 2: Illustration of the encoder network.

###  2.3 Spectral Ensembling

Once obtaining a pyramid of graph embeddings after graph pooling, _i.e_.
$\mathcal{X}=\\{X_{1},X_{2},\cdots,X_{L}\\}$, with each level of the hierarchy
encoding the key spectral-specific feature at a certain scale, we explore
their interactions across the hierarchy via a novel recurrent spectral
ensembling module. There are mainly two challenges in ensembling a sequence of
multi-level graph embeddings. Firstly, interaction between graphs of different
number of nodes is not as straightforward as grid-like data _e.g_. images and
texts due to the lack of spatial locality and order information. Note, the
number of nodes in the graph corresponds to the cardinality of spectral
embeddings. Secondly, recall that each of the previous graph pooling layer
selected some important nodes to form a new graph for high-level feature
encoding, the nodes in the new graph might become isolated due to the related
edges have been removed while removing nodes.

In order to address the above issues, we propose a recurrent message passing
ensembling mechanism. Specifically, for every two neighboring graph
embeddings, the smaller but higher level graph is firstly unpooled to match
the size of the bigger yet lower level graph. Then two graphs are interacting
in a way that higher level graph is used as the hidden state encoding
historical information, transferring its graph embedding and node
interdependency to guide the message passing within the lower level graph.
Fig. 1 illustrates the unrolled recurrent spectral ensembling where all the
intermediate graph representations are processed as a sequence of input
instances in the recurrent unit with shared weights.

Formally, indicating $\mathbf{h}_{0}=X_{L}$, the graph unpooling layer $l$ is
defined as

$\displaystyle\mathbf{\breve{h}}_{l}=\text{MessagePassing}(f(0_{N\times
C},\mathbf{h}_{l},\mathcal{I}),\mathcal{G}_{L-l})$ (3)

where $\mathcal{I}$ represents the set of indices of selected nodes in the
corresponding graph pooling layer $L-l$ which reduced the number of graph
nodes from N to k (due to top-$k$ pooling),
$\mathbf{h}_{l}\in\mathbb{R}^{k\times C}$ is the feature map of current graph,
$0_{N\times C}$ is the empty feature matrix for the new graph, $f(\cdot)$
indicates the mapping operation distributing row vectors of $\mathbf{h}_{l}$
into $0_{N\times C}$ according to the corresponding indices in $\mathcal{I}$,
$\mathcal{G}_{L-l}$ indicates the graph definition before graph pooling layer
$L-l$ which consists of affinity matrix and edge weights, and MessagePassing
defines the message passing process [18] as

$\displaystyle\mathbf{\breve{h}}^{i}_{l}=\sum_{j\in\mathcal{N}(i)\cup\\{i\\}}\mathbf{\Theta}\cdot\mathbf{h}^{j}_{l},$
(4)

where neighboring node features are first transformed by the weight matrix
$\mathbf{\Theta}$ passed via $\mathcal{G}_{L-l}$. The potential isolation of
upsampled nodes in the new graph due to the removed related edges is properly
compensated by the message passing based on the original graph interdependency
information.

Then the framework updates the hidden state based on spectral feature
$X_{L-l}$ and hidden state $\mathbf{h}_{l-1}$ from previous step via a gated
mechanism similar to Gated Recurrent Unit (GRU) [6] as

$\begin{split}\mathbf{z}_{l}&=\sigma\left(\mathbf{W}_{z}X_{L-l}+\mathbf{U}_{z}\mathbf{\breve{h}}_{l-1}\right),\\\
\mathbf{r}_{l}&=\sigma\left(\mathbf{W}_{r}X_{L-l}+\mathbf{U}_{r}\mathbf{\breve{h}}_{l-1}\right),\\\
\mathbf{\tilde{h}}_{l}&=\text{tanh}\left(\mathbf{W}_{o}X_{L-l}+\mathbf{U}_{o}(\mathbf{r}_{l}\odot\mathbf{\breve{h}}_{l-1})\right),\\\
\mathbf{h}_{l}&=(1-\mathbf{z}_{l})\odot\mathbf{\breve{h}}_{l-1}+\mathbf{z}_{l}\odot\mathbf{\tilde{h}}_{l}\end{split}$
(5)

where $\sigma(\cdot)$ is the logistic sigmoid function, $\text{tanh}(\cdot)$
is the hyperbolic tangent function, and $\odot$ is the Hadamard product.
Interactions between all levels in the graph embedding hierarchy are enabled
by encoding higher level contextual spectral information in the hidden states.
The process is repeated L times according to the levels of hierarchy, and the
final hidden state $\mathbf{h}_{L}$ goes through another GCN layer before
being concatenated with the feature map from the encoder network for
predicting the category label of the input HSI volume with a fully connected
layer.

###  2.4 Encoder Network

We adopt a multi-path dilated 3D convolutional residual network to encode the
orginal HSI data as illustrated in Fig. 2. Specifically, the encoder starts
with a 3D convolutional layer with kernel size $15\times 3\times 3$
corresponding to the spectral dimension and two spatial dimensions
respectively, which is followed by two multi-path ($3\times 3\times 3$,
$5\times 3\times 3$, $7\times 3\times 3$ and $9\times 3\times 3$ kernels
respectively) dilated (rate=2) convolutional residual modules with identity
shortcuts. Note that our proposed framework is agnostic to the utilized
encoder network.

Class No. | Class Name | Training | Testing
---|---|---|---
1 | Asphalt | 50 | 6581
2 | Meadows | 50 | 18599
3 | Gravel | 50 | 2049
4 | Trees | 50 | 3014
5 | Painted metal sheets | 50 | 1295
6 | Bare Soil | 50 | 4979
7 | Bitumen | 50 | 1280
8 | Self-Blocking Bricks | 50 | 3632
9 | Shadows | 50 | 897
| Total | 450 | 42776
Table 1: Sample size for the University of Pavia dataset. Class No. | Class Name | Training | Testing
---|---|---|---
1 | Alfalfa | 40 | 6
2 | Corn-notill | 100 | 1328
3 | Corn-mintill | 100 | 730
4 | Corn | 100 | 137
5 | Grass-pasture | 100 | 383
6 | Grass-trees | 100 | 630
7 | Grass-pasture-mowed | 20 | 8
8 | Hay-windrowed | 100 | 378
9 | Oats | 15 | 5
10 | Soybean-notill | 100 | 872
11 | Soybean-mintill | 100 | 2355
12 | Soybean-clean | 100 | 493
13 | Wheat | 100 | 105
14 | Woods | 100 | 1165
15 | Buildings-Grass-Trees-Drives | 100 | 286
16 | Stone-Steel-Towers | 80 | 13
| Total | 1355 | 8894
Table 2: Sample size for the Indian Pines dataset

##  3 Experimental Results

We evaluate our method on two publicly available hyperspectral image datasets
with four metrics including per-class accuracy, overall accuracy (OA), average
accuracy (AA), and Kappa coefficient. The network architecture of the proposed
method SGR is identical for both datasets. Specifically, we use an
architecture of L=2 and the input HSI is sampled on 7$\times$7 image patches
in the spatial domain. We adopt stochastic gradient descent with momentum set
to 0.9, weight decay of 0.0005, and with a batch size of 30. We initialize an
equal learning rate for all trainable layers to 0.05, which is manually
decreased by a factor of 10 when the validation error plateaus. The number of
training epochs is set to 500. All the reported accuracies are calculated
based on the average of five training sessions to obtain stable results.

###  3.1 Datasets

The University of Pavia dataset was captured with the ROSIS sensor in 2001
which consists of 610$\times$340 pixels with a spatial resolution of 1.3
m$\times$1.3 m and has 103 spectral channels in the wavelength range from 0.43
$\mu$m to 0.86 $\mu$m after removing noisy bands. This dataset includes 9
land-cover classes and the false color image and ground-truth map are shown in
Fig. 3.

The Indian Pines dataset was captured with Airborne Visible/Infrared Imaging
Spectrometer sensor comprising 145$\times$145 pixels with a spatial resolution
of 20 m$\times$20 m and 220 spectral channels covering the range from 0.4
$\mu$m to 2.5 $\mu$m. This dataset includes 16 land-cover classes and the
false color image and ground-truth map are shown in Fig. 4.

Table 1 and Table 2 provide information about all the classes of both datasets
with their corresponding training and test samples respectively. During
training, 90% of the training samples are used to learn the network parameters
and 10% are used as validation set for tuning the hyperparameters.

###  3.2 Classification Results

Class No. | FCN [20] | MS 3D-CNN [13] | SS 3D-CNN [23] | 3D-CNN [10] | SGR
---|---|---|---|---|---
1 | 61.98 | 92.47 | 91.98 | 91.86 | 95.91
2 | 72.16 | 94.22 | 93.88 | 90.55 | 98.27
3 | 12.90 | 86.38 | 81.46 | 85.89 | 88.47
4 | 39.52 | 92.63 | 94.98 | 93.65 | 99.83
5 | 99.46 | 98.57 | 100.00 | 98.59 | 99.91
6 | 41.78 | 80.45 | 74.34 | 70.49 | 84.75
7 | 50.49 | 83.60 | 84.94 | 84.66 | 86.42
8 | 67.99 | 90.30 | 86.92 | 88.58 | 91.43
9 | 97.55 | 97.43 | 99.94 | 98.82 | 98.24
OA | 61.21 | 91.35 | 90.30 | 88.14 | 94.37
AA | 60.42 | 90.67 | 89.82 | 89.23 | 93.69
Kappa | 48.44 | 88.56 | 87.15 | 84.46 | 91.26
Table 3: Per-class accuracy, OA, AA (%), and Kappa coefficient achieved by
different methods on the University of Pavia dataset.

####  3.2.1 The University of Pavia Dataset

The quantitative results obtained by different approaches on the University of
Pavia dataset are reported in Table 3. Our method outperforms the compared
methods on 7 out of 9 categories and achieves the best overall results, _i.e_.
OA, AA and Kappa. Generally, all other networks significantly outperform FCN
[20]. One possible reason could be that FCN heavily squeezed the spectral
dimension and utilized small 2D convolutional kernels in the spatial domain.
3D-CNN [10] adopted more 3D convolutional layers to perform simultaneous
spatial-spectral convolution. Compared with [20], [10] used a large number of
3D filters per layer to improve the feature representation over the very high
dimensional spectral domain which increases the accuracy by $\sim 2\%$. [13]
addresses the limitations of single scale 3D CNN architectures by introducing
a multi-scale 3D CNN architecture, which improves the accuracy by $\sim 1\%$
compared to [10]. Nevertheless, none of the existing methods has explicitly
performed reasoning among the spectral embedding channels in a global manner,
but rather modeling the local context by convolutional kernels with limited
size of receptive field. Our method bridges this gap by casting spectral
embeddings into a unified graph whose node corresponds to an individual
spectral feature in the embedding space and thus the interaction between graph
nodes guides learning spectral-specific graph representations. Quantitative
results shows that our method outperforms the best competing method [13] with
a significant margin of $3.02\%$.

Fig. 3 shows a visual comparison of our proposed SGR and the best competing
method [13] on the University of Pavia dataset. The observation reveals that
[13] suffers more mis-classifications than SGR in large semantically coherent
regions, due to its weaker representation capability of encoding the
discriminative spectral features with the presence of noise. The qualitative
result that our SGR produces more accurate and coherent predictions.

Class No. | FCN [20] | SS 3D-CNN [23] | MS 3D-CNN [13] | 3D-CNN [10] | SGR
---|---|---|---|---|---
1 | 6.7 | 80.00 | 100.00 | 30.77 | 99.56
2 | 64.81 | 76.41 | 77.52 | 70.90 | 79.32
3 | 74.67 | 78.06 | 79.36 | 76.12 | 80.47
4 | 59.21 | 76.65 | 80.85 | 77.46 | 83.72
5 | 92.19 | 92.16 | 96.43 | 90.96 | 97.62
6 | 97.66 | 99.13 | 99.84 | 96.86 | 99.92
7 | 47.56 | 40.00 | 100.00 | 48.48 | 100.00
8 | 97.04 | 99.87 | 100.00 | 97.70 | 100.0
9 | 16.95 | 58.82 | 100.00 | 71.43 | 100.00
10 | 78.71 | 76.75 | 79.84 | 70.07 | 81.55
11 | 76.59 | 80.09 | 82.50 | 77.83 | 85.26
12 | 65.23 | 78.03 | 76.56 | 68.31 | 80.97
13 | 98.51 | 97.65 | 99.53 | 95.57 | 99.92
14 | 94.88 | 94.64 | 97.34 | 96.12 | 98.75
15 | 70.75 | 73.96 | 85.53 | 71.70 | 85.97
16 | 49.33 | 89.66 | 100.00 | 78.79 | 100.00
OA | 79.20 | 83.37 | 85.79 | 80.35 | 87.83
AA | 68.17 | 80.74 | 90.96 | 76.19 | 92.06
Kappa | 75.89 | 80.89 | 83.58 | 77.19 | 85.42
Table 4: Per-class accuracy, OA, AA (%), and Kappa coefficient achieved by
different methods on the Indian Pines dataset

####  3.2.2 The Indian Pines Dataset

Table 4 gives the quantitative results on the Indian Pines dataset, which
shows that the proposed method obtains the best overall accuracy of $87.83\%$.
Similar observations with the University of Pavia dataset can be found that
all 3D CNN architectures outperform FCN [20] which adopts small 2D
convolutional kernels as majority components. 3D CNN architectures generally
explore the spatial-spectral contexts simultaneously and achieves better
accuracy. Nevertheless, our proposed method has the advantage over the
compared methods by exploring inter-spectral relations and learning a graph
embedding hierarchy representation with each level emphasizing on specific
spectral features exhibiting discriminative power toward the classification.
Fig. 4 shows a visual comparison of our proposed SGR and the second best
method [13], which confirms that our SGR produces more accurate and coherent
predictions.

Figure 3: The university of Pavia dataset: (a) False color image (b)
Groundtruth map (c) Prediction of SSLSTMs [63] (d) Prediction of the proposed
SPGAT

Figure 4: Indian Pines dataset: (a) False color image (b) Groundtruth map (c)
Prediction of LBMSELM [1] (d) Prediction of the proposed SPGAT

.

###  3.3 Ablation Study

We conduct ablation study on the University of Pavia dataset to quantitatively
verify the effectiveness of the proposed architecture. The first baseline is
applying a predicting layer (Conv3D-FC-Softmax) on top of the encoder network,
_i.e_. Baseline-1. We also investigate spectral decoupling module with two
different depth 1 and 3, dubbed as Baseline-2 and Baseline-3 respectively. To
verify the contribution from spectral ensembling, we replace it with a naive
graph upsampling and embedding summation operation, _i.e_. Baseline-4.

As shown in Table 5, a significant accuracy gain of $6.13\%$ is achieved by
our proposed spectral graph reasoning network, by comparing Baseline-1 and
SGR. Adding one level of spectral coupling module improves the accuracy from
$88.24\%$ to $91.87\%$, which confirms its effectiveness. However, adding
three levels slightly decreases the accuracy, _i.e_. a loss of $0.16\%$
compared with SGR which has two levels. The possible reason is using the
limited training samples for learning a more complex architecture.
Furthermore, introducing the spectral ensembling module boosts the performance
by $1.83\%$.

| Baseline-1 | Baseline-2 | Baseline-3 | Baseline-4 | SGR
---|---|---|---|---|---
OA | 88.24 | 91.87 | 94.21 | 92.54 | 94.37
Table 5: OA achieved by different baselines on the University of Pavia
dataset.

##  4 Conclusions

In this paper, we proposed a novel spectral graph reasoning network for
hyperspectral image classification which has significantly advanced the state-
of-the-art. At the core of our network architecture lies a new spectral
feature reasoning and ensembling paradigm. We demonstrated that applying
interpretable graph reasoning in the spectral feature domain enables learning
a spectral-specific graph embedding which in turn improves the discriminative
capability. We further proposed a spectral ensembling module that explores the
interactions and interdependencies across the graph embedding hierarchy via a
novel recurrent message propagation mechanism.

## References

  * [1] Cao, F., Yang, Z., Ren, J., Chen, W., Han, G., Shen, Y.: Local block multilayer sparse extreme learning machine for effective feature extraction and classification of hyperspectral images. IEEE Trans. Geoscience and Remote Sensing 57(8), 5580–5594 (2019)
  * [2] Chen, S., Zhao, Y., Jin, Q., Wu, Q.: Fine-grained video-text retrieval with hierarchical graph reasoning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10638–10647 (2020)
  * [3] Chen, Y., Nasrabadi, N.M., Tran, T.D.: Hyperspectral image classification using dictionary-based sparse representation. IEEE transactions on geoscience and remote sensing 49(10), 3973–3985 (2011)
  * [4] Chen, Y., Jiang, H., Li, C., Jia, X., Ghamisi, P.: Deep feature extraction and classification of hyperspectral images based on convolutional neural networks. IEEE Transactions on Geoscience and Remote Sensing 54(10), 6232–6251 (2016)
  * [5] Chen, Y., Lin, Z., Zhao, X., Wang, G., Gu, Y.: Deep learning-based classification of hyperspectral data. IEEE Journal of Selected topics in applied earth observations and remote sensing 7(6), 2094–2107 (2014)
  * [6] Cho, K., van Merrienboer, B., Gülçehre, Ç., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. In: EMNLP. pp. 1724–1734. ACL (2014)
  * [7] Chung, F.R., Graham, F.C.: Spectral graph theory. No. 92, American Mathematical Soc. (1997)
  * [8] Defferrard, M., Bresson, X., Vandergheynst, P.: Convolutional neural networks on graphs with fast localized spectral filtering. In: Advances in neural information processing systems. pp. 3844–3852 (2016)
  * [9] Deng, F., Zhi, Z., Lee, D., Ahn, S.: Generative scene graph networks. In: International Conference on Learning Representations (2021)
  * [10] Hamida, A.B., Benoit, A., Lambert, P., Amar, C.B.: 3-d deep learning approach for remote sensing image classification. IEEE Transactions on geoscience and remote sensing 56(8), 4420–4434 (2018)
  * [11] Han, K., Wang, Y., Guo, J., Tang, Y., Wu, E.: Vision gnn: An image is worth graph of nodes. Advances in neural information processing systems 35, 8291–8303 (2022)
  * [12] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 770–778 (2016)
  * [13] He, M., Li, B., Chen, H.: Multi-scale 3d deep convolutional neural network for hyperspectral image classification. In: 2017 IEEE International Conference on Image Processing (ICIP). pp. 3904–3908. IEEE (2017)
  * [14] Hu, R., James, S., Wang, T., Collomosse, J.: Markov random fields for sketch based video retrieval. In: Proceedings of the 3rd ACM conference on International conference on multimedia retrieval. pp. 279–286 (2013)
  * [15] Hu, X., Wang, X., Zhong, Y., Zhao, J., Luo, C., Wei, L.: Spnet: A spectral patching network for end-to-end hyperspectral image classification. In: IGARSS 2019-2019 IEEE International Geoscience and Remote Sensing Symposium. pp. 963–966. IEEE (2019)
  * [16] Jiang, C., Chen, Y., Chen, C., Jia, J., Sun, H., Wang, T., Hyyppa, J.: Implementation and performance analysis of the pdr/gnss integration on a smartphone. GPS Solutions 26(3), 81 (2022)
  * [17] Jiang, C., Chen, Y., Chen, C., Jia, J., Sun, H., Wang, T., Hyyppä, J.: Smartphone pdr/gnss integration via factor graph optimization for pedestrian navigation. IEEE Transactions on Instrumentation and Measurement 71, 1–12 (2022)
  * [18] Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. In: Proceedings of International Conference on Learning Representations (2017)
  * [19] Kuo, B.C., Huang, C.S., Hung, C.C., Liu, Y.L., Chen, I.L.: Spatial information based support vector machine for hyperspectral image classification. In: 2010 IEEE International Geoscience and Remote Sensing Symposium. pp. 832–835. IEEE (2010)
  * [20] Lee, H., Kwon, H.: Contextual deep cnn based hyperspectral classification. In: 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS). pp. 3322–3325. IEEE (2016)
  * [21] Lee, H., Kwon, H.: Going deeper with contextual cnn for hyperspectral image classification. IEEE Transactions on Image Processing 26(10), 4843–4855 (2017)
  * [22] Li, W., Chen, C., Su, H., Du, Q.: Local binary patterns and extreme learning machine for hyperspectral imagery classification. IEEE Transactions on Geoscience and Remote Sensing 53(7), 3681–3693 (2015)
  * [23] Li, Y., Zhang, H., Shen, Q.: Spectral–spatial classification of hyperspectral imagery with 3d convolutional neural network. Remote Sensing 9(1), 67 (2017)
  * [24] Lu, X., Wang, W., Danelljan, M., Zhou, T., Shen, J., Van Gool, L.: Video object segmentation with episodic graph memory networks. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16. pp. 661–679. Springer (2020)
  * [25] Ma, L., Crawford, M.M., Tian, J.: Local manifold learning-based $k$-nearest-neighbor for hyperspectral image classification. IEEE Transactions on Geoscience and Remote Sensing 48(11), 4099–4109 (2010)
  * [26] Mou, L., Ghamisi, P., Zhu, X.X.: Deep recurrent neural networks for hyperspectral image classification. IEEE Transactions on Geoscience and Remote Sensing 55(7), 3639–3655 (2017)
  * [27] Qi, X., Liao, R., Jia, J., Fidler, S., Urtasun, R.: 3d graph neural networks for rgbd semantic segmentation. In: Proceedings of the IEEE international conference on computer vision. pp. 5199–5208 (2017)
  * [28] Qin, A., Shang, Z., Tian, J., Wang, Y., Zhang, T., Tang, Y.Y.: Spectral-spatial graph convolutional networks for semisupervised hyperspectral image classification. IEEE Geosci. Remote Sensing Lett. 16(2), 241–245 (2019)
  * [29] Shi, L., Zhang, L., Yang, J., Zhang, L., Li, P.: Supervised graph embedding for polarimetric sar image classification. IEEE Geoscience and Remote Sensing Letters 10(2), 216–220 (2012)
  * [30] Song, W., Li, S., Fang, L., Lu, T.: Hyperspectral image classification with deep feature fusion network. IEEE Transactions on Geoscience and Remote Sensing 56(6), 3173–3184 (2018)
  * [31] Wang, H., Raiko, T., Lensu, L., Wang, T., Karhunen, J.: Semi-supervised domain adaptation for weakly labeled semantic video object segmentation. In: Asian conference on computer vision. pp. 163–179. Springer (2016)
  * [32] Wang, H., Wang, T.: Boosting objectness: Semi-supervised learning for object detection and segmentation in multi-view images. In: ICASSP. pp. 1796–1800 (2016)
  * [33] Wang, H., Wang, T.: Primary object discovery and segmentation in videos via graph-based transductive inference. Computer Vision and Image Understanding 143, 159–172 (2016)
  * [34] Wang, H., Wang, T., Chen, K., Kämäräinen, J.K.: Cross-granularity graph inference for semantic video object segmentation. In: IJCAI. pp. 4544–4550 (2017)
  * [35] Wang, T., Xu, N., Chen, K., Lin, W.: End-to-end video instance segmentation via spatial-temporal graph neural networks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10797–10806 (2021)
  * [36] Wang, T.: Apparatus, a method and a computer program for image processing (Nov 15 2016), uS Patent 9,495,755
  * [37] Wang, T.: Method, apparatus and computer program product for segmentation of objects in media content (Apr 25 2017), uS Patent 9,633,446
  * [38] Wang, T.: Submodular video object proposal selection for semantic object segmentation. In: 2017 IEEE International Conference on Image Processing (ICIP). pp. 4522–4526. IEEE (2017)
  * [39] Wang, T.: Method, an apparatus and a computer program product for object detection (Nov 1 2018), uS Patent App. 15/956,878
  * [40] Wang, T.: Method for analysing media content (Apr 26 2018), uS Patent App. 15/785,711
  * [41] Wang, T.: Semantic segmentation based on a hierarchy of neural networks (Dec 22 2020), uS Patent 10,872,275
  * [42] Wang, T., Collomosse, J.: Probabilistic motion diffusion of labeling priors for coherent video segmentation. IEEE Transactions on Multimedia 14(2), 389–400 (2011)
  * [43] Wang, T., Collomosse, J., Hilton, A.: Wide baseline multi-view video matting using a hybrid markov random field. In: 2014 22nd International Conference on Pattern Recognition. pp. 136–141. IEEE (2014)
  * [44] Wang, T., Collomosse, J., Hu, R., Slatter, D., Greig, D., Cheatle, P.: Stylized ambient displays of digital media collections. Computers & Graphics 35(1), 54–66 (2011)
  * [45] Wang, T., Collomosse, J., Slatter, D., Cheatle, P., Greig, D.: Video stylization for digital ambient displays of home movies. In: Proceedings of the 8th International Symposium on Non-Photorealistic Animation and Rendering. pp. 137–146 (2010)
  * [46] Wang, T., Collomosse, J.P., Hunter, A., Greig, D.: Learnable stroke models for example-based portrait painting. In: BMVC (2013)
  * [47] Wang, T., Collomosse, J.P., Slatter, D., Cheatle, P., Greig, D.: Video stylization for digital ambient displays of home movies. In: NPAR. pp. 137–146 (2010)
  * [48] Wang, T., Fan, L.: Watermark embedding techniques for neural networks and their use (Jan 16 2020), uS Patent App. 16/508,434
  * [49] Wang, T., Guillemaut, J.Y., Collomosse, J.: Multi-label propagation for coherent video segmentation and artistic stylization. In: 2010 IEEE International Conference on Image Processing. pp. 3005–3008. IEEE (2010)
  * [50] Wang, T., Wang, G., Tan, K.E., Tan, D.: Spectral pyramid graph attention network for hyperspectral image classification. arXiv preprint arXiv:2001.07108 (2020)
  * [51] Wang, T., Wang, H.: Graph transduction learning of object proposals for video object segmentation. In: ACCV. pp. 553–568 (2014)
  * [52] Wang, T., Wang, H.: Graph transduction learning of object proposals for video object segmentation. In: Asian Conference on Computer Vision. pp. 553–568. Springer (2014)
  * [53] Wang, T., Wang, H.: Method and an apparatus for automatic segmentation of an object (May 5 2016), uS Patent App. 14/930,392
  * [54] Wang, T., Wang, H.: Graph-boosted attentive network for semantic body parsing. In: Artificial Neural Networks and Machine Learning–ICANN 2019: Image Processing: 28th International Conference on Artificial Neural Networks, Munich, Germany, September 17–19, 2019, Proceedings, Part III 28. pp. 267–280. Springer (2019)
  * [55] Wang, T., Wang, H., Fan, L.: Robust interactive image segmentation with weak supervision for mobile touch screen devices. In: Proceedings of International Conference on Multimedia and Expo. pp. 1–6. IEEE (2015)
  * [56] Wang, T., Wang, H., Fan, L.: A weakly supervised geodesic level set framework for interactive image segmentation. Neurocomputing 168, 55–64 (2015)
  * [57] Wang, W., Lu, X., Shen, J., Crandall, D.J., Shao, L.: Zero-shot video object segmentation via attentive graph neural networks. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 9236–9245 (2019)
  * [58] Xing, Y., He, T., Xiao, T., Wang, Y., Xiong, Y., Xia, W., Wipf, D., Zhang, Z., Soatto, S.: Learning hierarchical graph neural networks for image clustering. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3467–3477 (2021)
  * [59] Yang, Y., Ren, Z., Li, H., Zhou, C., Wang, X., Hua, G.: Learning dynamics via graph neural networks for human pose estimation and tracking. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 8074–8084 (2021)
  * [60] Zhang, L., Zhang, Q., Du, B., Huang, X., Tang, Y.Y., Tao, D.: Simultaneous spectral-spatial feature selection and extraction for hyperspectral images. IEEE Transactions on Cybernetics 48(1), 16–28 (2016)
  * [61] Zhao, G., Ge, W., Yu, Y.: Graphfpn: Graph feature pyramid network for object detection. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 2763–2772 (2021)
  * [62] Zhong, Z., Fan, B., Duan, J., Wang, L., Ding, K., Xiang, S., Pan, C.: Discriminant tensor spectral–spatial feature extraction for hyperspectral image classification. IEEE Geoscience and Remote Sensing Letters 12(5), 1028–1032 (2014)
  * [63] Zhou, F., Hang, R., Liu, Q., Yuan, X.: Hyperspectral image classification using spectral-spatial lstms. Neurocomputing 328, 39–47 (2019)
  * [64] Zhu, L., Wang, T., Aksu, E., Kamarainen, J.K.: Cross-granularity attention network for semantic segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops. pp. 0–0 (2019)
  * [65] Zhu, L., Wang, T., Aksu, E., Kamarainen, J.K.: Portrait instance segmentation for mobile devices. In: 2019 IEEE International Conference on Multimedia and Expo (ICME). pp. 1630–1635. IEEE (2019)
  * [66] Zhu, L., Wang, T., Aksu, E., Kamarainen, J.: Portrait instance segmentation for mobile devices. In: ICME. pp. 1630–1635 (2019)